Setup

# installs
! pip install -q duckdb
! pip install -q mongita
! pip install -q kuzu
! pip install -q lancedb
! pip install -q sentence-transformers
! pip install -q requests

# imports
import pandas as pd
import requests
from sentence_transformers import SentenceTransformer

# get an example database for sqlite
! wget -qO music.sqlite https://github.com/lerocha/chinook-database/raw/refs/heads/master/ChinookDatabase/DataSources/Chinook_Sqlite.sqlite

# get an example for the graph database
! wget -q https://files.grouplens.org/datasets/movielens/ml-latest-small.zip

With the libraries installed and some sample data downloaded from the web, let’s define the connections to each database.

# instantiate each database

# oltp - single file
import sqlite3
oltp = sqlite3.connect("music.sqlite")

# olap - single file
import duckdb
olap = duckdb.connect("olap.duckdb")

# doc - creates a folder
from mongita import MongitaClientDisk
doc = MongitaClientDisk("docdb")

# graph - creates a folder
import kuzu
g = kuzu.Database("graphdb")
graph = kuzu.Connection(g)

# vector - creates a folder
import lancedb
vec = lancedb.connect("vectordb")

OLTP via SQLite

I won’t spend much time on SQlite and python, as there are plenty of tutorials on the web. Below will demonstrate how to interface with sqlite via Python and pandas.

# execute a query to show the tables
# list tables in sqlite database
oltp.execute("SELECT name FROM sqlite_master WHERE type='table';").fetchall()

[('Album',),
 ('Artist',),
 ('Customer',),
 ('Employee',),
 ('Genre',),
 ('Invoice',),
 ('InvoiceLine',),
 ('MediaType',),
 ('Playlist',),
 ('PlaylistTrack',),
 ('Track',)]

# of course, we can query data directly into pandas
# join the artists and albumns
sql = """
select a.*,
       b.albumid,
       b.title
from artist a
join album b
on a.artistid = b.artistid
"""
df = pd.read_sql(sql, oltp)
df.head(3)

# create a silly table
create_table_query = """
CREATE TABLE IF NOT EXISTS applicants (
    id INTEGER PRIMARY KEY,
    name TEXT,
    position TEXT,
    salary REAL
)
"""
# Execute the query
oltp.execute(create_table_query)
oltp.commit()

Ensure the table exists.

oltp.execute("SELECT name FROM sqlite_master WHERE type='table';").fetchall()

[('Album',),
 ('Artist',),
 ('Customer',),
 ('Employee',),
 ('Genre',),
 ('Invoice',),
 ('InvoiceLine',),
 ('MediaType',),
 ('Playlist',),
 ('PlaylistTrack',),
 ('Track',),
 ('applicants',)]

# a simple dataset
applicants = [
    ("Alice", "Manager", 70000),
    ("Bob", "Analyst", 50000),
    ("Charlie", "Clerk", 30000)
]

# insert into the table,
insert_query = """
INSERT INTO applicants (name, position, salary) VALUES (?, ?, ?)
"""

oltp.executemany(insert_query, applicants)
oltp.commit()

# lets grab the data with pandas
pd.read_sql("SELECT * FROM applicants", oltp)

# of course we can update records
update_query = """
UPDATE applicants SET salary = ? WHERE name = ?
"""
oltp.execute(update_query, (80000, "Alice"))
oltp.commit()

pd.read_sql("SELECT * FROM applicants", oltp)

# we can even use pandas to insert/append to the table
new_data = pd.DataFrame({
    "name": ["David", "Eve"],
    "position": ["Engineer", "HR"],
    "salary": [60000, 55000]
})
new_data.to_sql("applicants", oltp, if_exists="append", index=False)

2

# review the data
applicants_table = pd.read_sql("SELECT * FROM applicants", oltp)
applicants_table

OLAP via DuckDB

I am a huge fan of DuckDB.

• Like SQLite, it’s an in-process database. You can also use duckdb in-memory as well.
• Incredibly performant
• Can be used as an analytics engine or a backend to power data applications.
• Motherduck is a cloud hosted data warehouse based on Duckdb.

There are a variety of applications. If you are interested in learning more about how DuckDB is empowering data teams, I refer you to the free resource here.

# we can create a table directly from pandas
olap.execute("CREATE TABLE applicants as select * from applicants_table")

<duckdb.duckdb.DuckDBPyConnection at 0x1453148f0>

# verify that the data exist
olap.execute("SELECT * FROM applicants").df()

# create two csv files
applicants_table.to_csv("e1.csv", index=False)
applicants_table.to_csv("e2.csv", index=False)

# duckdb is amazing - scan a directory of files and return as a single dataframe
combined_csvs = olap.execute("SELECT * FROM '*.csv'").df()
combined_csvs

Document Database via Mongita

From the repo, Mongita is a lightweight embedded document database that implements a commonly-used subset of the MongoDB/PyMongo interface.

To demonstrate some basic data operations, I am going to use the Fruityvice API.

# get the data from the API

resp = requests.get("https://www.fruityvice.com/api/fruit/all")
fruits_resp = resp.json()
fruits_resp[0]

{'name': 'Persimmon',
 'id': 52,
 'family': 'Ebenaceae',
 'order': 'Rosales',
 'genus': 'Diospyros',
 'nutritions': {'calories': 81,
  'fat': 0.0,
  'sugar': 18.0,
  'carbohydrates': 18.0,
  'protein': 0.0}}

# from the documentation
# create the database within the document store
fruits_db = doc.fruits_db

# the collection
fruits = fruits_db.fruits

# add documents to the fruits collection
fruits.insert_many(fruits_resp)

<mongita.results.InsertManyResult at 0x112bea0f0>

# how many documents are there - no query filter
fruits.count_documents({})

49

# get one document
fruits.find_one()

{'name': 'Persimmon',
 'id': 52,
 'family': 'Ebenaceae',
 'order': 'Rosales',
 'genus': 'Diospyros',
 'nutritions': {'calories': 81,
  'fat': 0.0,
  'sugar': 18.0,
  'carbohydrates': 18.0,
  'protein': 0.0},
 '_id': ObjectId('677187a7635da2c06e2f038a')}

# we can query documents where the calories key is greater than 100
list( fruits.find({'nutritions.calories': {'$gt': 100}}) )

[{'name': 'Durian',
  'id': 60,
  'family': 'Malvaceae',
  'order': 'Malvales',
  'genus': 'Durio',
  'nutritions': {'calories': 147,
   'fat': 5.3,
   'sugar': 6.75,
   'carbohydrates': 27.1,
   'protein': 1.5},
  '_id': ObjectId('677187a7635da2c06e2f038f')},
 {'name': 'Avocado',
  'id': 84,
  'family': 'Lauraceae',
  'order': 'Laurales',
  'genus': 'Persea',
  'nutritions': {'calories': 160,
   'fat': 14.66,
   'sugar': 0.66,
   'carbohydrates': 8.53,
   'protein': 2.0},
  '_id': ObjectId('677187a7635da2c06e2f03ac')},
 {'name': 'Hazelnut',
  'id': 96,
  'family': 'Betulaceae',
  'order': 'Fagales',
  'genus': 'Corylus',
  'nutritions': {'calories': 628,
   'fat': 61.0,
   'sugar': 4.3,
   'carbohydrates': 17.0,
   'protein': 15.0},
  '_id': ObjectId('677187a7635da2c06e2f03b3')}]

Graph Database via Kuzu

Kuzu is an embedded graph database. A graph databases allow us to store our data as nodes and edges, where nodes are the entities in our application, and the edges, or relationships, are how the nodes are related.

While I am a huge fan of Neo4j, Kuzu provides an attractive solution that also aims to focus on graph analytics tasks.

For this demo, I am going to use a small subset Movielens dataset, which was downloaded via wget above.

# unzip the dataset
! unzip ml-latest-small.zip

Archive:  ml-latest-small.zip
   creating: ml-latest-small/
  inflating: ml-latest-small/links.csv  
  inflating: ml-latest-small/tags.csv  
  inflating: ml-latest-small/ratings.csv  
  inflating: ml-latest-small/README.txt  
  inflating: ml-latest-small/movies.csv  

# read in two of the datasets
movies = pd.read_csv("ml-latest-small/movies.csv")
ratings = pd.read_csv("ml-latest-small/ratings.csv")

movies.sample(3)

ratings.sample(3)

Kuzu can import from a variety of sources, including Pandas dataframes. For the dataset above, we need to parse out the users into their own dataframe.

# kuzu can read from pandas dataframes
user = ratings[['userId']].drop_duplicates()
user.to_csv("user.csv", index=False)

user.head(3)

Unlike Neo4j, it appears that we have define Node and Edge/Relationship tables prior to import. We will do that below.

# create the schema - kuzu specific

# nodes
graph.execute("CREATE NODE TABLE Movie(movieId INT64, title STRING, genres STRING, PRIMARY KEY (movieId))")
graph.execute("CREATE NODE TABLE User(userId INT64, PRIMARY KEY (userId))")

# relationships/edges
graph.execute("CREATE REL TABLE Rating(FROM User TO Movie, rating FLOAT, timestamp INT64)")

<kuzu.query_result.QueryResult at 0x1459c6b70>

# import the data directly from the DataFrames
graph.execute('COPY Movie FROM movies')
graph.execute('COPY User FROM user')
graph.execute('COPY Rating FROM ratings')

<kuzu.query_result.QueryResult at 0x1471a2ed0>

Below will be a cypher query that queries the graph database via a pattern that looks at user’s rating movies, and then summarizes the data to sort the movies based on the number of ratings while also displaying the average rating, which is a property on the rating relationship.

# query
cql = """
MATCH (u:User)-[r]->(m:Movie)
WITH  m.title AS title,
      AVG(r.rating) AS avg_rating,
      COUNT(r) AS num_ratings
ORDER BY num_ratings DESC
LIMIT 10
RETURN title, avg_rating, num_ratings
"""

graph.execute(cql).get_as_df()

Vector Database via lancedb

LanceDB is an embeddable vector database, a format that has become incredibly popular with the rise of LLMs and Generative AI.

Below is a search of Vector Database over the last 5 years, which spikes just after the release of ChatGPT in November 2022.

# a synthetic dataset of text abstracts

abstracts = [
  {
    "id": "1",
    "title": "Optimizing Transformers for Real-Time Applications",
    "abstract": "This paper explores optimization techniques for transformer architectures to reduce latency in real-time applications. We evaluate pruning, quantization, and hardware-specific tuning, achieving up to 40% reduction in inference time without compromising accuracy.",
    "topic": "Transformer Optimization"
  },
  {
    "id": "2",
    "title": "Adversarial Robustness in Deep Neural Networks",
    "abstract": "We present a novel adversarial training algorithm that enhances the robustness of deep neural networks against various attack strategies. Our method outperforms baseline approaches by 15% on benchmark datasets like CIFAR-10 and ImageNet.",
    "topic": "Adversarial Robustness"
  },
  {
    "id": "3",
    "title": "Generative Adversarial Networks for Data Augmentation",
    "abstract": "This study proposes a GAN-based data augmentation technique to improve model generalization. Our experiments demonstrate significant accuracy improvements in medical imaging and natural language processing tasks.",
    "topic": "GANs and Data Augmentation"
  },
  {
    "id": "4",
    "title": "Explainability in Large Language Models",
    "abstract": "We introduce a framework for probing the decision-making processes of large language models, offering insights into token importance and context understanding. Case studies reveal increased transparency in summarization and classification tasks.",
    "topic": "Explainability in AI"
  },
  {
    "id": "5",
    "title": "Efficient Training of Sparse Neural Networks",
    "abstract": "Our research focuses on sparse neural networks and their potential for reducing computational overhead. Using structured pruning techniques, we achieve up to 60% parameter reduction while maintaining competitive performance.",
    "topic": "Sparse Neural Networks"
  }
]

# initialize an embedding model
model = SentenceTransformer("all-MiniLM-L6-v2")

# apply the model the abstracts
for a in abstracts:
  a["vector"] = model.encode(a["abstract"])

# create a table index
tbl = vec.create_table("abstracts", data=abstracts)

# database inspection
vec.table_names()

['abstracts']

# get the embedding for a new abstract
new_abstract = [
  {
    "id": "20",
    "title": "Automating Feature Engineering with Generative AI",
    "abstract": "We propose a generative AI framework for automating feature engineering in machine learning pipelines. Case studies demonstrate reduced development time and improved model performance.",
    "topic": "Automated Feature Engineering"
  }
]

## nn search
embedding = model.encode(new_abstract[0]["abstract"])

# search
results = tbl.search(embedding).limit(3).to_pandas()
results

Summary

Above is a brief review of how to get started with a variety of databases that can power application and analytical workloads, especially those that are targeted at embedded applications that do not require a client/server architecture.

It’s worth noting that there are online hosted versions of the technologies above. The list below highlights offerings that have generous free tiers, which have proven to be helpful for in-class projects.

• OLAP: MotherDuck
• Document: MongoDB Atlas Cloud
• Graph: Neo4j AuraDB
• Vector: Pinecone Serverless

The Dataset

For the sake of illustration, I will going to use play-by-play data files from the NHL to highlight how a data team might store and analyze these datasets outside of a full load into a “data warehouse”. Common query patterns might summarize statistics by season, or more granularly, by month or game. With this in mind, let’s jump into how we might store the data on S3 to enable the analysis of the data.

Let’ assume I have a bucket called nhl-pbp. I am using “folders” to partition the data via a number of core fields. For example, here is the S3 URI for one file.

s3://nhl-pbp/pbp/season=2024/year=2024/month=12/day=22/gameid=2024020539/pbp.parquet

• Think of the first folder pbp as the table in our database. Other tables might be players, or teams, or venue which would reside at /players, /teams and /venue accordingly.
• Within the pbp table, we then partition the data by season
• Within a given season, we can partition the data by calendar items like year month and day to enable granular queries such as “How many goals did Brad Marchand have in the month of December 2024?”
• Finally, because there can be multiple games per calendar day, we add a partition for the game id itself.

Let’s consider a nightly pipeline that collects the data files from yesterday’s games. Our pipeline might do the following:

• Run overnight via a schedule (e.g. EventBridge Scheduler)
• Determine if there were any new games for today() - 1 day
• If there are games, grab the JSON, perform a light transform to flatten into a tabular format, and write a parquet file to a path on S3 that conforms with the partitions.

For the sake of illustration, I already have some data on S3; it’s a handful of games to keep the surface area small.

Querying the Data via DuckDB

Without hyperbole, I tell everyone I can in the data space about DuckDB. I bring it up in nearly every conversation. To me, DuckDB will be considered the swiss army knife for data teams if it isn’t considered that already. The benefits are many, but for this post, I will use DuckDB as a query engine, which will use S3 as a data store. If you are new to DuckDb, let that sink in for a second.

To get started, you will need to ensure you have the HTTPFS duckdb extension installed.

# import the library
import duckdb

# create an in memory connection
db = duckdb.connect()

# ensure the extension is ready to go
db.execute("INSTALL httpfs; LOAD httpfs;")

<duckdb.duckdb.DuckDBPyConnection at 0x16a2e50b0>

Newer versions of duckdb have made it really simple to authenticate with Cloud storage services. In my case, I have already installed and configured my AWS profile, and the snippet below will tell duckdb to use that configuration when making calls to S3.

sql = """
CREATE SECRET aws_secret (
    TYPE S3,
    PROVIDER CREDENTIAL_CHAIN
);
"""

db.execute(sql)

<duckdb.duckdb.DuckDBPyConnection at 0x16a2e50b0>

For a basic example, below I will print out the number of plays for one specific file on S3.

sql = """ 
select count(*)
from 's3://nhl-pbp/pbp/season=2023/year=2023/month=12/day=14/gameid=2023020448/pbp.parquet'
"""

db.execute(sql).df()

While this hopefully highlights the power, simplicity, and speed of duckdb, this is a simple example. The true power comes when we want to use S3 as our data store.

Below we are counting the number of rows from all of the games in the month of December across all years.

sql = """
SELECT COUNT(*)
FROM read_parquet(
  's3://nhl-pbp/pbp/season=*/year=*/month=12/day=*/gameid=*/pbp.parquet'
);
"""

db.execute(sql).df()

You may have noticed above that we are using the read_parquet function as way to scan for files that match our path definition. DuckDB is smart enough to review the files on S3 and calculate the results which span multiple parquet files on S3. How cool is that?

If you want to calculate how many unique games are being evaluated, it’s as simple as:

sql = """
SELECT COUNT(distinct gameid)
FROM read_parquet(
  's3://nhl-pbp/pbp/season=*/year=*/month=12/day=*/gameid=*/pbp.parquet'
);
"""

db.execute(sql).df()

It’s not too hard to imagine the type of data apps you could build by simplying writing parquet files to S3 and using DuckDB as the query engine that sits in between the the frontend and the S3 backend!

AWS Glue Data Catalog and Athena

AWS Glue includes a number of execellent services to empower data teams. One central service is AWS Glue Data Catalog, which allows us to formalize a “Database” and tables over the data stored in S3. There are some tradeoffs as it relates to DuckDB above. While it’s entirely possible to programmatically configure the steps below, I am going to use the console for sake of demonstration.

select year, month, day, count(*) as total from "AwsDataCatalog"."nhl-blog-example"."pbp" group by 1,2,3;

Adding new Data

Let’s step back, and imagine that our nightly service has added data to S3. To simulate this, I am going to add a new play-by-play parquet file for a single game. To add context, this will be for the 21st of December 2024. The S3 URI is below.

s3://nhl-pbp/pbp/season=2024/year=2024/month=12/day=21/gameid=2024020539/pbp.parquet

To demonstrate the final point of this post, let’s hop back to Athena and re-run the query.

We get the same results. This happens because Athena is using the Glue Data Catalog to support the queries against our table, and because above was a new partition for the data lake, our Data Catalog, and as a result Athena, are unaware to look for that file.

To emphasize our core different relative to the DuckDB example, we will run the same query.

sql = """

select year, month, day, count(*) as total 
FROM read_parquet(
  's3://nhl-pbp/pbp/season=*/year=*/month=*/day=*/gameid=*/pbp.parquet'
)
group by 1,2,3;
"""

db.execute(sql).df()

DuckDB correctly returns the results as expected.

What’s important to note above is that if our data pipeline adds new files to S3, DuckDB will automatically factor these files in as long as it meets the path in the read_parquet definition. You could consider this schema-on-read.

You likely are asking what we need to do to have Athena properly identify the new data added as part of our pipeline. There are ways to programmatically update the table in the Data Catalog, or, we could have our pipeline fire a new run of the crawler. For the latter, it’s important to note that you are billed for the processing time, so keep that in mind on which approach makes the most sense.

Summary

We saw above, via a simple example, that data teams can leverage services like S3 as the data store for their pipelines. Tools like DuckDB or services from AWS such as Glue’s Data Catalog and Athena remove the need to have a “warehouse-first” data strategy.

There isn’t always one right answer, but in future posts, we will explore the concepts of data lakes and lakehouses via Apache Iceberg in order provide alternatives to the traditional cloud data warehouse pattern.

Step 1: EventBridge Connection

The HTTP Endpoint State requires an Eventbridge Connection, which in this case, stores our API token.

In the web console, navigate to Eventbridge Connections. Click Create Connection.

On the first portion of the screen, give your connection a valid name and select Public. Optionally provide a description.

At the bottom of the page, select API Key.

This part is important!.

• For the API Key Name, enter Authorization
• for the Value, enter Bearer sk-XXXXXX where your sk-XXXXXX is your OpenAI API token.

Click Create.

In my case, this is what I see after creating the connection.

Step 2: SFN Workflow

The first iteration will be as simple as it can get. Create a new Step Function on the AWS Console. NOTE: When prompted, select JSONata as the Query language.

After selecting the State, we will need to add a few bits of information to configure our HTTP call out.

1. API Endpoint: https://api.openai.com/v1/chat/completions
2. Method: POST
3. Connection: Select the Connection Name you created in the Step 1 above

Below is a basic request body in the format OpenAI expects

{
  "model": "gpt-4o",
  "messages": [
    {
      "role": "developer",
      "content": "You are a helpful assistant."
    },
    {
      "role": "user",
      "content": "Hello!"
    }
  ]
}

As I noted in my previous article, beyond the low-code Workflow builder on the Web Console, we also can fully define our workflows via ASL, enabling Infrastructure-as-Code deployments.

Below is the JSON ASL for the workflow

{
  "QueryLanguage": "JSONata",
  "Comment": "Dec 2024 flow updated as POC for alternatives, and also to show we don't always need to hop into a Lambda.",
  "StartAt": "Call HTTPS APIs",
  "States": {
    "Call HTTPS APIs": {
      "Type": "Task",
      "Resource": "arn:aws:states:::http:invoke",
      "Arguments": {
        "ApiEndpoint": "https://api.openai.com/v1/chat/completions",
        "Method": "POST",
        "InvocationConfig": {
          "ConnectionArn": "arn:aws:events:us-east-1:00000000000000:connection/OpenaiDec24/6a2df29c-c945-4e9f-a446-9957ee9de7e1"
        },
        "RequestBody": {
          "model": "gpt-4o",
          "messages": [
            {
              "role": "developer",
              "content": "You are a helpful assistant."
            },
            {
              "role": "user",
              "content": "Hello!"
            }
          ]
        }
      },
      "Retry": [
        {
          "ErrorEquals": [
            "States.ALL"
          ],
          "BackoffRate": 2,
          "IntervalSeconds": 1,
          "MaxAttempts": 3,
          "JitterStrategy": "FULL"
        }
      ],
      "End": true,
      "Comment": "- Authorization is the key, Bearer <TOKEN> is the value via event Bridge connection"
    }
  }
}

Step 3: Execute the Workflow

If you haven’t done so, we need to rename and save the Workflow. Click the Config tab, rename the workflow, select your Execution Role which allows HTTP invocations, and click save.

With our worfklow defined and saved, we can now test the Execution. I am not going to define a Payload. Leave this screen blank and click Start Execution.

In the graph view, if everything is configured correctly (i.e. the Eventbridge Connection is setup as outlined above), you should see a successful execution.

For the Task State Output, you should see a RequestBody key, with the data sent back from the API call out to ChatGPT.

That’s it. In my case, ChatGPT responded with Hello! How can I help you today?.

Optional: Variables and JSONata

Above was a simple example to ensure everything is wire up correctly in order to call out to OpenAI APIs from within our Step Function workflow. Below will demonstrate how we can dynamically pass in the prompt to OpenAI. As I did with the previous post, I am going to use a Pass state to enter the prompt, and then refer to this variable within the defition of the request body.

The updated workflow will make use of Variables and JSONata to refer to the prompt defined the Pass state.

Below is the ASL definition of the updated workflow

{
  "QueryLanguage": "JSONata",
  "Comment": "Dec 2024 flow updated as POC for alternatives, and also to show we don't always need to hop into a Lambda.",
  "StartAt": "Pass",
  "States": {
    "Pass": {
      "Type": "Pass",
      "Next": "Call HTTPS APIs",
      "Assign": {
        "prompt": "How many wins do the Boston Bruisn have this season?"
      }
    },
    "Call HTTPS APIs": {
      "Type": "Task",
      "Resource": "arn:aws:states:::http:invoke",
      "Arguments": {
        "ApiEndpoint": "https://api.openai.com/v1/chat/completions",
        "Method": "POST",
        "InvocationConfig": {
          "ConnectionArn": "arn:aws:events:us-east-1:000000000000:connection/OpenaiDec24/6a2df29c-c945-4e9f-a446-9957ee9de7e1"
        },
        "RequestBody": {
          "model": "gpt-4o",
          "messages": [
            {
              "role": "developer",
              "content": "You are a helpful assistant."
            },
            {
              "role": "user",
              "content": "{% $prompt %}"
            }
          ]
        }
      },
      "Retry": [
        {
          "ErrorEquals": [
            "States.ALL"
          ],
          "BackoffRate": 2,
          "IntervalSeconds": 1,
          "MaxAttempts": 3,
          "JitterStrategy": "FULL"
        }
      ],
      "End": true,
      "Comment": "- Authorization is the key, Bearer <TOKEN> is the value via event Bridge connection"
    }
  }
}

A few notes on above:

• We define a variable called prompt in the Pass state.
• In the defintion of the Request body, you can see that we are using JSONata to dynamically pull in the prompt via the variable defined above via "content": "{% $prompt %}"

After a successfull execution, we see that ChatGPT acknowledges that the model does not have the ability to recall current information.

Optional: Input Prompt

Finally, a more realistic workflow would expect that the prompt would be included in the payload that triggers the execution of the Step Function.

The changes:

• Remove the Pass state that was added above to highlight Variables
• Include the prompt in the payload at the start of the execution.

We will make use of the reserved variables, namely the input key, to isolate the prompt passed into the Workflow Execution.

The ASL definition is below:

{
  "QueryLanguage": "JSONata",
  "Comment": "Dec 2024 flow updated as POC for alternatives, and also to show we don't always need to hop into a Lambda.",
  "StartAt": "Call HTTPS APIs",
  "States": {
    "Call HTTPS APIs": {
      "Type": "Task",
      "Resource": "arn:aws:states:::http:invoke",
      "Arguments": {
        "ApiEndpoint": "https://api.openai.com/v1/chat/completions",
        "Method": "POST",
        "InvocationConfig": {
          "ConnectionArn": "arn:aws:events:us-east-1:0000000000000:connection/OpenaiDec24/6a2df29c-c945-4e9f-a446-9957ee9de7e1"
        },
        "RequestBody": {
          "model": "gpt-4o",
          "messages": [
            {
              "role": "developer",
              "content": "You are a helpful assistant."
            },
            {
              "role": "user",
              "content": "{% $states.input.prompt %}"
            }
          ]
        }
      },
      "Retry": [
        {
          "ErrorEquals": [
            "States.ALL"
          ],
          "BackoffRate": 2,
          "IntervalSeconds": 1,
          "MaxAttempts": 3,
          "JitterStrategy": "FULL"
        }
      ],
      "End": true,
      "Comment": "- Authorization is the key, Bearer <TOKEN> is the value via event Bridge connection"
    }
  }
}

• Notice that the body of our request to OpenAI was modified to refer to the prompt key from states.input via "content": "{% $states.input.prompt %}" which passes the input directly to OpenAI.

Save and execute the workflow with the input shown below.

{
 "prompt":"Tell me a joke"
}

That’s it! We were able to pull in the input that triggered the execution into the call to OpenAI.

Summary

The post above walks you through the basics of getting started with making callouts to external APIs from inside an AWS Step Function workflow. Above, I used OpenAI’s API to make simple calls to Chat GPT, highlighting that you have options beyond AWS’s Bedrock inside your Agentic Solutions built on top of Step Functions.

Finally, you can call public apis that do not require user/pass by creating a EventBridge Connection with placeholder username/password. Even though the API will you want to leverage may not require a user/pass combintation, this will allow you to get around the required connection input.

Our First Flow

Go ahead and create a new Step Function from the AWS Console. For my purposes, I am going to use a Standard workflow, but note that in practice we very likely would be leverage Express workflows. Discussing the tradeoffs between the two is outside the scope of this article.

I am going to step through the logic in order to hopefully help with the intuition of Variables. If you haven’t used Step Functions in the past, below is the visual Workflow editor.

The Pass state is incredibly powerful, and in this case, we can set variables and apply other logic as needed at the start of our execution.

If you select the Pass State in the canvas, and then the Variables tab, you can manually specify the variables.

From the pills at the top of the screen, you can select Config to rename the Workflow and define the execution role for the Step Function. I already have an Execution Role, but for this purpose of this post, you can let AWS create one for you.

As I noted earlier, not only can we define our workflow logic via drag-and-drop, but this solution is backed by a templating language that we can later use to package our projects via IaC. Below is the first workflow via JSON.

{
  "QueryLanguage": "JSONPath",
  "Comment": "I can set a comment about my workflow, what is required at invocation, the task this solves, etc.",
  "StartAt": "Pass",
  "States": {
    "Pass": {
      "Type": "Pass",
      "End": true,
      "Comment": "This will set the variables that will pass through the execution",
      "Assign": {
        "max_iter": 3,
        "counter": 0
      }
    }
  }
}

If you are following along with this post, it’s important to note that you can copy the code above and paste the contents into the code editor in the console. Select the Code pill, and paste the workflow definition on the left.

With our workflow created, we can now review the execution logic. Click Execute, this will run the workflow. No need to enter a payload yet, just click Start Execution in the lower right.

We can see that the execution successfully ran, and that the variables were assigned. These will be accessible throughout the execution of this pipeline.

Second Flow

Let’s update the workflow to add some additional logic. As with above, you can paste the code definition directly into the console to see the changes.

{
  "QueryLanguage": "JSONPath",
  "StartAt": "Pass",
  "States": {
    "Pass": {
      "Type": "Pass",
      "Assign": {
        "variableName": "$.states.input",
        "max_iter": 3,
        "counter": 0
      },
      "QueryLanguage": "JSONata",
      "Comment": "Manually creating an array for now, but ideally just say if you get to this state, we can specify an iterator as a for loop.  Might be missing something, but this expects \"data\" to iterate over, versus a for loop wih an exit (for max 3 iteratons, do this, else exit) -> like training, but Agent loops is the obvious use case.",
      "Next": "Choice"
    },
    "Choice": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$counter",
          "NumericLessThanPath": "$max_iter",
          "Next": "update counter"
        }
      ],
      "Default": "Success"
    },
    "update counter": {
      "Type": "Pass",
      "Next": "Success",
      "Assign": {
        "counter": "{% $counter + 1 %}"
      }
    },
    "Success": {
        "Type": "Succeed"
      }
    },
    "Comment": "Play around with the new JSONata things.  Really expressive syntax (is there a plugin/vs AI help) and the variables is huge.  Seems like just reference states.input.<object> and then assign variables as needed.  ONly keep what you need.  The concept of jsut passing JSON around is kinda gross, but also, JSONPath is very simple"
  }

Above we have added a few additional nodes to our workflow, namely the Choice which allows us to conditionally route the execution path within our workflow.

Let’s step back for a moment.

• We have defined two variables, max_iter and counter. The former defines the upper bound for how many times we would allow the self-loop, as I may refer to it, self-reflection, to occur. The counter variable allows us to keep track of how many times we evaluted this path.
  
• For each evaluation, we increment the counter by 1.
  
• When the expression counter < max_iter is no longer True, the choice state will follow a different path and complete the exeuction.

Save the workflow and execute the workflow. In reviewing the output, you will notice the the counter doesn’t properly update.

JSONata

Let’s update the workflow to explicilty define the use of JSONata in the update counter state/task.

{
  "QueryLanguage": "JSONPath",
  "StartAt": "Pass",
  "States": {
    "Pass": {
      "Type": "Pass",
      "Assign": {
        "max_iter": 3,
        "counter": 0
      },
      "QueryLanguage": "JSONata",
      "Comment": "Manually creating an array for now, but ideally just say if you get to this state, we can specify an iterator as a for loop.  Might be missing something, but this expects \"data\" to iterate over, versus a for loop wih an exit (for max 3 iteratons, do this, else exit) -> like training, but Agent loops is the obvious use case.",
      "Next": "Choice"
    },
    "Choice": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$counter",
          "NumericLessThanPath": "$max_iter",
          "Next": "update counter"
        }
      ],
      "Default": "Success"
    },
    "update counter": {
      "Type": "Pass",
      "Next": "Success",
      "Assign": {
        "counter": "{% $counter + 1 %}"
      },
      "QueryLanguage": "JSONata"
    },
    "Success": {
      "Type": "Succeed"
    }
  },
  "Comment": "Play around with the new JSONata things.  Really expressive syntax (is there a plugin/vs AI help) and the variables is huge.  Seems like just reference states.input.<object> and then assign variables as needed.  ONly keep what you need.  The concept of jsut passing JSON around is kinda gross, but also, JSONPath is very simple"
}

After saving and executing the workflow, review the output. You can see above that the interface highlights that our counter is changing values, a subtle but nice touch!

However, you can see from the logging that the update counter task didn’t fire three times.

The Final Edit

Above, everything is working correctly, but we are immediately ending our workflow’s execution without going back for another evaluation pass.

Below we will update the workflow to go back to the Choice state in order to enable to allow our workflow to perform another pass.

{
  "QueryLanguage": "JSONPath",
  "StartAt": "Pass",
  "States": {
    "Pass": {
      "Type": "Pass",
      "Assign": {
        "max_iter": 3,
        "counter": 0
      },
      "QueryLanguage": "JSONata",
      "Comment": "Manually creating an array for now, but ideally just say if you get to this state, we can specify an iterator as a for loop.  Might be missing something, but this expects \"data\" to iterate over, versus a for loop wih an exit (for max 3 iteratons, do this, else exit) -> like training, but Agent loops is the obvious use case.",
      "Next": "Choice"
    },
    "Choice": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$counter",
          "NumericLessThanPath": "$max_iter",
          "Next": "update counter"
        }
      ],
      "Default": "Success"
    },
    "update counter": {
      "Type": "Pass",
      "Next": "Choice",
      "Assign": {
        "counter": "{% $counter + 1 %}"
      },
      "QueryLanguage": "JSONata"
    },
    "Success": {
      "Type": "Succeed"
    }
  },
  "Comment": "Play around with the new JSONata things.  Really expressive syntax (is there a plugin/vs AI help) and the variables is huge.  Seems like just reference states.input.<object> and then assign variables as needed.  ONly keep what you need.  The concept of jsut passing JSON around is kinda gross, but also, JSONPath is very simple"
}

Save and execute.

By looking at the events, we can now see that the update counter State/Task was invoked three times, as expected.

Summary

• Variables allow us to keep track of core pieces of information throughout the full execution of our workflow. There is a limit to how much data we can pass around, but the limit is quite reasonable.
  
• JSONata allows us to reference these variables, and shown above, update the values conditionally as needed
• It’s worth pointing out that you can specify if you want to use JSONata at the creation of your workflow. I created my workflow with the previous JSONPath evaluation to highlight that it is possible to mix-and-match as needed, and it’s just a matter of defining the engine within the task state.

The above example is silly and a basic proof-of-concept, but let’s zoom out to why I find this new capability incredibly interesting. Agentic workflows allow us to combine multiple generative AI calls together in a system that aims to perform a given task.

As one example, consider an LLM-as-a-judge workflow which will use one LLM to generate output, and another to evalute the output. Above demonstrated that we could allow up to three calls to the LLM judge. When thinking about structured output, we could have the judge provide feedback for the initial agent to improve, or, if the judge “approved”, move onto new tasks in our workflow.

Setup

First, you will need to ensure strealmit is installed:

pip install streamlit 

Now, let’s setup the project structure:

mkdir streamlit
cd streamlit

Finally, make sure you have enabled APIs on GCP

gcloud services enable run.googleapis.com
gcloud services enable cloudbuild.googleapis.com
gcloud services enable artifactregistry.googleapis.com
gcloud services enable secretmanager.googleapis.com
gcloud services enable iam.googleapis.com

Files

Below we are going to define the files for our application.

streamlit/requirements.txt

streamlit 
matplotlib
networkx
google-cloud-secret-manager
numpy
pandas
duckdb

streamlit/app.py

import streamlit as st
import duckdb
import pandas as pd
from google.cloud import secretmanager
import numpy as np

from itertools import combinations
import networkx as nx
from collections import Counter
import matplotlib.pyplot as plt

project_id = '_your project id_'
secret_id = 'mother_duck'   #<---------- this is the name of the Google Cloud Secret
version_id = 'latest'

db = 'awsblogs'
schema = "stage"
db_schema = f"{db}.{schema}"

sm = secretmanager.SecretManagerServiceClient()

name = f"projects/{project_id}/secrets/{secret_id}/versions/{version_id}"

response = sm.access_secret_version(request={"name": name})
md_token = response.payload.data.decode("UTF-8")

md = duckdb.connect(f'md:?motherduck_token={md_token}') 

############################################ Streamlit App

st.set_page_config(page_title="My Fancy Streamlit App", layout="wide")

sql = """
select 
    min(published) as min,
    max(published) as max,
from
    awsblogs.stage.posts
"""
date_range = md.sql(sql).df()
start_date = date_range['min'].to_list()[0]
end_date = date_range['max'].to_list()[0]


st.title("Streamlit - Example")
st.subheader("To demonstrate Streamlit concepts - It's just a python script!")


st.sidebar.header("Filters")
st.sidebar.markdown("One option is to use sidebars for inputs")
author_filter = st.sidebar.text_input("Search by Author")
date_filter = st.sidebar.date_input("Post Start Date", (start_date, end_date))

st.sidebar.button("A button to control inputs")

st.sidebar.file_uploader("Users can upload files that your app analyzes!")

st.sidebar.markdown("These controls are not wired up to control data, just highlighting you have a lot of control!")


############ A simple line plot

num_days = 365  # Number of days for the time series
start_date = '2023-01-01'  # Start date

date_range = pd.date_range(start=start_date, periods=num_days, freq='D')

np.random.seed(42)  # For reproducibility
values = np.random.randint(50, 150, size=num_days)  # Example: random sales values between 50 and 150

time_series_data = pd.DataFrame({
    'date': date_range,
    'value': values
})

st.line_chart(time_series_data, x="date", y="value")

############ Graph of co-association of tags, a touch forward looking
st.markdown("---")

pt_sql = """
select post_id, term from awsblogs.stage.tags
"""
pt_df = md.sql(pt_sql).df()

st.markdown("### You _can_ show data tables")
st.dataframe(pt_df)

st.markdown("### A static network graph")
st.markdown("We can think of relationships as a graph")

cotag_pairs = []

for _, group in pt_df.groupby('post_id'):
    # Get the unique list of authors for each post
    terms = group['term'].unique()
    # Generate all possible pairs of co-authors for this post
    pairs = combinations(terms, 2)
    cotag_pairs.extend(pairs)

cotag_counter = Counter(cotag_pairs)

G = nx.Graph()

for (term1, term2), weight in cotag_counter.items():
    G.add_edge(term1, term2, weight=weight)

degree_centrality = nx.degree_centrality(G)

node_sizes = [100 * degree_centrality[node] for node in G.nodes()]

edge_weights = [G[u][v]['weight'] for u, v in G.edges()]

pos = nx.spring_layout(G, k=0.3, seed=42)  

fig = plt.figure(figsize=(12, 12))

nx.draw_networkx_nodes(G, pos, node_size=node_sizes, node_color='skyblue', alpha=0.7)

nx.draw_networkx_edges(G, pos, width=edge_weights, alpha=0.5, edge_color='gray')

plt.title("Tag Graph")
st.pyplot(fig)


############ There are some chat support features, more coming

st.markdown("---")

st.markdown("### There are even some chat features - more coming on the roadmap.")

prompt = st.chat_input("Say something")
if prompt:
    st.write(f"User has sent the following prompt: {prompt}")

streamlit/Dockerfile

FROM python:3.11-slim

WORKDIR /app

COPY . /app

RUN pip install --no-cache-dir -r requirements.txt

EXPOSE 8501

ENV STREAMLIT_SERVER_HEADLESS=true
ENV STREAMLIT_SERVER_ADDRESS=0.0.0.0

CMD ["streamlit", "run", "app.py", "--server.port", "8080", "--server.address", "0.0.0.0"]

Deployment

The bash script below will build the image and deploy it to Google Cloud run as a service.

streamlit/deploy.sh

gcloud config set project {your_project_here}

echo "======================================================"
echo "build (no cache)"
echo "======================================================"

docker build --no-cache -t gcr.io/{your_project_here}/streamlit-poc .

echo "======================================================"
echo "push"
echo "======================================================"

docker push gcr.io/{your_project_here}/streamlit-poc

echo "======================================================"
echo "deploy run"
echo "======================================================"


gcloud run deploy streamlit-poc \
    --image gcr.io/{your_project_here}/streamlit-poc \
    --platform managed \
    --region us-central1 \
    --allow-unauthenticated \
    --service-account {service_account_name}@{your_project_here}.iam.gserviceaccount.com_ \
    --memory 1Gi

A few notes on the deploy script

• You will need to replace {your_project_here} with your actual project id
• I am using the us-central1 region
• Above specifies a service account that Cloud Run will use to access various services on GCP. You will need to ensure that when you define the Service Account, you provide access to Secret Manager

From your shell, deploy the application.

bash streamlit/deploy.sh

When the process completes, you should see something similar to below:

======================================================
deploy run
======================================================
Deploying container to Cloud Run service [streamlit-poc] in project [{your-project-here}] region [us-central1]
OK Deploying... Done.                                                                                                          
  OK Creating Revision...                                                                                                      
  OK Routing traffic...                                                                                                        
  OK Setting IAM Policy...                                                                                                     
Done.                                                                                                                          
Service [streamlit-poc] revision [streamlit-poc-00003-588] has been deployed and is serving 100 percent of traffic.
Service URL: https://streamlit-poc-111222333444.us-central1.run.app

That’s it! Navigate to the URL and you have deployed a publicly available, serverless, Streamlit app on GCP via Cloud Run.

Save a dataset to a RapidMiner Repository

Before diving into the code, let’s get some terminology out of the way. RapidMiner has the notion of repositories. These are simply folders on your machine that will act as the container for your work. Once you have a repository setup, you can reference these repositories via //repository. You will see that in the code below.

from ast import operator
from concurrent.futures import process
import os
import rapidminer
import pandas as pd

URL = "https://vincentarelbundock.github.io/Rdatasets/csv/ggplot2/diamonds.csv"
dia = pd.read_csv(URL)
connector.write_resource(dia, "//BU/data/diamonds")

Let’s step through above:

1. We import a number of libraries, including pandas. I would recommend using a package manager like conda. In my case, I create an environment called rapidminer and pip-install all of the tools that I need there. I also use this environment within RM when calling python.
2. URL is simply a pointer to a csv file on the web.
3. I use pandas to read the csv file into a DataFrame, a python object that RapidMiner can easily work with.
4. Using the connector object, I am writing the dataset (called diamonds) into the data sub-folder in my repository called BU. Note the //BU shortcut I referenced above.
5. You should now see the data object within the repository of your choice. TIP: You may need to refresh the view within RM Studio to see the change.

While RapidMiner also has the concept of a project (i.e. a git-backed folder), I tend to simply create repositories in RM that point to folders that are already under version control on my machine. I prefer to use other tools (e.g. CLI, Github Desktop) to commit/push my diffs instead of doing these tasks within RM.

Save a dataset to a RapidMiner Repository

Now this is where functionality gets exciting. The code below will use the same connector, but this time, it is pointed at a RapidMiner process stored in the repository.

hw = connector.run_process("//BU/hw1")

A few notes:

• The process is stored in my repository and is called hw1.
• Based on the configuration above, the entire RapidMiner process will be executed. We will see another example below where we can execute a specific operator within the process.
• Python will invoke RapidMiner and start the engine in order to execute the process. This will take a few moments depending on your machine and installed extensions. NOTE: You can tell python whether or not you want to see the messages while RM is getting started and executing the process flow.
• Above you can notice that the I am assigning the output of the process to a variable called hw. The results stored in this python object will be dependent on the process setup. Above, my process simply Retrieves the diamonds dataset from the earlier example, performs an aggregation, and connects the resulting ExampleSet to the results port. As such the hw variable is a pandas DataFrame with the aggregated results.

The key takeaway from above is that we don’t have any knowledge of the steps in the process, or the work that was done along the way, but the python library allows us to extract the result ports and bring those objects back into our python session.

You might be wondering why I used the ExampleSet reference above. An ExampleSet is simply a dataset in RapidMiner, and in most cases from my experience, will be a pandas DataFrame, but it doesn’t have to be.

Call an Operator within the Process

Finally, the last task that appears to be supported is the ability to call a specific operator (e.g. a function or unit of work in RM) and get the output from that run. This is a touch tricky because I suspect that when calling this operator, we must properly pass the inputs the operator expects.

# using the dia object from earlier, only keep rows where the cut value is Good
dia2 = dia.loc[dia.cut=='Good', :]

# the Aggregate operator has two outputs, as such, specifying two output objects
agg, ori = connector.run_process("//BU/hw1", inputs=dia2, operator="Aggregate")

A few notes on above:

• I am keeping a subset of the original diamonds dataset that we grabbed from the web. This is to test the output of the Aggregate operator, which is essentially a group-by utility. In this case, there should only be one row since the process will be grouping by the variable, or Attribute cut.
• We are still using the run_process method, but now are specifying the inputs to use, and the name of the Operator within the process.
• We are also storing multiple output, as the Aggregate operator exports the results as an ExampleSet (that is coverted to a DataFrame), as well as the original ExampleSet passed to the operator.

How neat is that?

Looking Ahead

This is a great start, and considering that RapidMiner processes are simply XML files, I do wonder if I will be able to think about ways to write tests against my students’ assignments by parsing the XML files and then running various operators along the way.

This leads me to my wishlist:

1. Instead of parsing the XML document, it would be nice if we could inspect both the process itself, as well as get feedback on the entire run of the process. Ideally we would be able to see the data flow through, understand the changes that occur along the way, etc. Having this information in python, after the call to run_process would enable us to write tests to ensure that a process is configured as expected, the output matches expectations, etc.
2. In addition to running a single operator, it could be helpful to run a portion, or sequence of the process. The same rationale still applies; write tests against the process.

Why

• Remind my future self on the basic stack needed to deploy ML APIs.
• Act as a Quick Start guide that data product owners can use to have discussions with downstream engineers
• Highlight how easy it is to get started serving models with scikit-learn, a robust ML framework that allows for reproducible pipelines. Current SOTA models tend to be built in DL frameworks like Tensorflow, but for a large set of business needs, the marginal gain is not worth the marginal effort.

The last point is especially true for teams that are starting to figure out how to deploy models, even if they are batch-oriented and not via API responses.

Clusters/Groups of Services

from diagrams import Cluster, Diagram
from diagrams.aws.compute import ECS
from diagrams.aws.database import ElastiCache, RDS
from diagrams.aws.network import ELB
from diagrams.aws.network import Route53

with Diagram("Clustered Web Services", show=False):
    dns = Route53("dns")
    lb = ELB("lb")

    with Cluster("Services"):
        svc_group = [ECS("web1"),
                     ECS("web2"),
                     ECS("web3")]

    with Cluster("DB Cluster"):
        db_main = RDS("userdb")
        db_main - [RDS("userdb ro")]

    memcached = ElastiCache("memcached")

    dns >> lb >> svc_group
    svc_group >> db_main
    svc_group >> memcached

How cool is that?!?

Step 1 - Enable the REST API

By default, airflow does not accept requests made to the API. However, it’s easy enough to turn on:

# auth_backend = airflow.api.auth.backend.deny_all
auth_backend = airflow.api.auth.backend.basic_auth

Above I am commenting out the original line, and including the basic auth scheme.

To be validated by the API, we simply need to pass an Authorization header and the base64 encoded form of username:password where username and password are for the user created in Airflow.

For example:

Above, I have blurred out a series of text, but that is the username:password Base64 encoded. There are plenty of tools on the web that can encode this for you.

NOTE: You see the encoded information is prefaced by Basic.

Step 2: Test the API by Listing Dags

With above in place, we can list the dags in Airflow easily via /dags.

Because I am running locally, it’s as simple as a GET request to http://localhost:8080/api/v1/dags. Just remember to include the Authorization bits. The call to list the DAGs is also shown in the screenshot above.

Step 3: The DAG setup and configuration

Of course, if we are going to pass information to the DAG, we would expect the tasks to be able to consume and use that information. Below provides snippets of my DAG to help refer to the core pieces.

# airflow bits
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.utils.dates import days_ago

# a function to read the parameters passed
def main_task(ti, **context):
    # the sql
    SQL = """
    select count(a.dim_user_id) as total
    from table1 a,
        table2 b
    where a.dim_account_id = b.id
    and   b.global_id = {}
    and   a.dim_site_id = {}
    and   a.message NOT IN ('JOINED', 'LEFT')
    and   a.activity_date between '{}' and '{}'
    """
    
    # connect to redshift
    rs = connect_redshift()  
    
    # extract the parameters passed from the REST API Trigger 
    gid = context['dag_run'].conf['gid']
    sid = context['dag_run'].conf['sid']
    date_start = context['dag_run'].conf['date_start']
    date_end = context['dag_run'].conf['date_end']
    
    # build the SQL and get the data
    SQL = SQL.format(gid, sid, date_start, date_end)
    df = rs.redshift_to_pandas(SQL)
    
    # do more things and 


args = {
    'owner': 'brock',
    'depends_on_past': False,
    'start_date': days_ago(2),
    'email': ['brocktibert@gmail.com'],
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(seconds=7),
    'schedule_interval': None,
    'provide_context': True
}

dag = DAG(
    dag_id='rest-trigger',
    default_args=args,
    tags=["brock"]
)

t1 = PythonOperator(task_id = "main_etl", 
                    python_callable = main_task,
                    dag = dag)

Let’s review:

1. We are importing the airflow bits that we need.
2. I am not showing all of my code, or even the real code I created for the client, but I am simply creating a function that reads from **context and extracts the parameters from the DAG’s conf object, where those params are stored. This will make more sense in a moment.
3. I am adding provide_context=True to the args.
4. I use the function main_task in a PythonOperator and associate it to the DAG.

Note: The dag_id is really important. This is what is available airflow uses as the id for a DAG. Please be mindful of the values you use here.

Step 4. Trigger the dag

Earlier in the screenshot we saw that we can use a GET request to /dags to get a simple list. We can use a POST request to trigger the dag by name. Above, the DAG I want to trigger is called rest-trigger.

Let’s review above:

1. We are making a call to POSTing data to http://localhost:8080/api/v1/dags/rest-trigger/dagRuns. Note that we are including the dag name / dagRuns.
2. The Authorization bits still need to be included and are not shown above, but are the same as the earlier screenshot.
3. In the body, we are passing json and including the information that we want as inside the conf key.
4. I am also specifiying my own ID for the dag_run_id. This will be auto generated for us, but is helpful if we have systems/logic that sits above the Airflow API.

Review the Triggered Dag

With the DAG triggered, we can use the UI to review the process, but we can also use the REST API to poll the job for it’s status via a GET call to http://localhost:8080/api/v1/dags/rest-trigger/dagRuns where again, we are passing in the dag name. In this case, we are passing in rest-trigger but you would use the name of your own dag.

Closing Remarks

This was a harder to get up and running than I would like to admit. In the end,

• If you hit issues, the logs can be helpful. print statements will be included in the logs, so if you need to, you can leverage that flow to identify hang ups.
• The name of the dag drives a lot of the functionality, which is why I stressed this earlier in the post.
• From the web UI, you can access the REST API docs.

While not shown above, you can pass the parameters around your DAG. Below is a simple example of taking in a single parameter and passing it around via XCOM and the ti parameter that was included but not used above.

from datetime import timedelta

from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.utils.dates import days_ago


dag = DAG(
    dag_id="trigger-me",
    default_args={"start_date": days_ago(2), "owner": "brock", "provide_context": True},
    schedule_interval=None
)


def push(ti, **context):
    # gets the parameter gid which was passed as a key in the json of conf
    gid = context['dag_run'].conf['gid']
    # use the task
    # https://marclamberti.com/blog/airflow-xcom/
    ti.xcom_push(key='global_id', value=gid)
    return gid

def pull(ti):
    gid2 = ti.xcom_pull(key="global_id", task_ids=['pusher'])
    print(gid2)

t1 = PythonOperator(task_id = "pusher", python_callable=push, provide_context=True, dag=dag)
t2 = PythonOperator(task_id = "puller", python_callable=pull, provide_context=True, dag=dag)

t1 >> t2

That’s it. I had to piece together above from a range of sources, so I hope this helps you (and my future self) if you need to explore this functionality.

TL;DR

• Combine Tensorflow/Keras with R
• NHL Data to estimate Shooter Player Embeddings
• Export to Tableau for exploration (yes we could use ggplot et. al, but highlights we have other options, especially for those new to the language)

R Setup

# packages
library(keras)

suppressPackageStartupMessages(library(tidyverse))
library(reticulate)
suppressPackageStartupMessages(library(caret))

# options
options(stringsAsFactors = FALSE)
use_condaenv("tensorflow")

Python setup

# imports
import pandas as pd
import numpy as np
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split
from tensorflow.keras.layers import Activation, concatenate, Dense, Dropout, Embedding, Input, Reshape, Flatten
from tensorflow.keras.models import Model
from tensorflow.keras.utils import plot_model
from tensorflow.keras.preprocessing.text import Tokenizer

Get the data

URL = "http://peter-tanner.com/moneypuck/downloads/shots_2019.zip"
download.file(URL, destfile="shots.zip")
shots_raw = read_csv("shots.zip")

dim(shots_raw)

[1] 88592   124

URL = "http://peter-tanner.com/moneypuck/downloads/shots_2019.zip"
shots_raw = pd.read_csv(URL)

shots_raw.shape

(88592, 124)

Filter rows

We want to keep shots on net, and not on an empty net, as well as remove records where the shooter id is 0.

# keep shots that were on goal
shots_raw = shots_raw %>% filter(shotWasOnGoal == 1 )
shots_raw = shots_raw %>% filter(shotOnEmptyNet == 0)
shots_raw = shots_raw %>% filter(shooterPlayerId != 0)
shots_raw = shots_raw %>% filter(!is.na(shooterPlayerId))

dim(shots_raw)

[1] 64318   124

shots_raw = shots_raw.loc[shots_raw.shotOnEmptyNet == 0, :]
shots_raw = shots_raw.loc[shots_raw.shotWasOnGoal == 1, :]
shots_raw = shots_raw.loc[shots_raw.shooterPlayerId != 0, :]
shots_raw = shots_raw.loc[~shots_raw.shooterPlayerId.isna(), :]

shots_raw.shape

(64318, 124)

Select Columns

With the rows select, let’s keep the columns that we want to include in this analysis.

# keep just the columns that we need
shots_raw = shots_raw %>% select(shooterPlayerId, shotType, goal, arenaAdjustedShotDistance, 
                                   arenaAdjustedXCord, arenaAdjustedYCord,  shotAngle, offWing)

dim(shots_raw)

[1] 64318     8

COLS = ['shooterPlayerId', 'shotType', 'goal', 'offWing', 
        'arenaAdjustedShotDistance', 'arenaAdjustedXCord', 'arenaAdjustedYCord', 
        'shotAngle']
shots_raw = shots_raw[COLS]

shots_raw.shape

(64318, 8)

Encode the shot types

I am going to one-hot the shot types, though in the future I will explore the use of keras.preprocessing.text.one_hot. The result will be new columns added to our shots_raw dataset, with each shot type flagged as 0/1.

x <- dummyVars(" ~ .", data = shots_raw)
shots_raw <- data.frame(predict(x, newdata = shots_raw))
rm(x)

glimpse(shots_raw)

Observations: 64,318
Variables: 14
$ shooterPlayerId           <dbl> 8480801, 8476853, 8476331, 8476853, 8475197…
$ shotTypeBACK              <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0…
$ shotTypeDEFL              <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0…
$ shotTypeSLAP              <dbl> 0, 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0…
$ shotTypeSNAP              <dbl> 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0…
$ shotTypeTIP               <dbl> 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0…
$ shotTypeWRAP              <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0…
$ shotTypeWRIST             <dbl> 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1…
$ goal                      <dbl> 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0…
$ arenaAdjustedShotDistance <dbl> 4.123106, 59.000000, 30.000000, 40.000000, …
$ arenaAdjustedXCord        <dbl> 85, -30, 60, -56, -40, -48, -34, -77, -61, …
$ arenaAdjustedYCord        <dbl> -1, -2, -7, -22, -30, -8, 41, -13, -34, 11,…
$ shotAngle                 <dbl> -14.036243, 2.009554, -12.994617, 33.690068…
$ offWing                   <dbl> 1, 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1…

shots_raw = pd.get_dummies(shots_raw, columns=['shotType'])
print(shots_raw.shape)

(64318, 14)

print(shots_raw.head(3).T)

                                      0             2             5
shooterPlayerId            8.480801e+06  8.476853e+06  8.476331e+06
goal                       1.000000e+00  0.000000e+00  0.000000e+00
offWing                    1.000000e+00  0.000000e+00  0.000000e+00
arenaAdjustedShotDistance  4.123106e+00  5.900000e+01  3.000000e+01
arenaAdjustedXCord         8.500000e+01 -3.000000e+01  6.000000e+01
arenaAdjustedYCord        -1.000000e+00 -2.000000e+00 -7.000000e+00
shotAngle                 -1.403624e+01  2.009554e+00 -1.299462e+01
shotType_BACK              0.000000e+00  0.000000e+00  0.000000e+00
shotType_DEFL              0.000000e+00  0.000000e+00  0.000000e+00
shotType_SLAP              0.000000e+00  0.000000e+00  0.000000e+00
shotType_SNAP              0.000000e+00  1.000000e+00  1.000000e+00
shotType_TIP               1.000000e+00  0.000000e+00  0.000000e+00
shotType_WRAP              0.000000e+00  0.000000e+00  0.000000e+00
shotType_WRIST             0.000000e+00  0.000000e+00  0.000000e+00

Scale the numeric data to 0/1

# clunky, but break out columns to standardize
tmp = shots_raw %>% select(arenaAdjustedShotDistance:shotAngle)
tmp2 = preProcess(tmp, method = "range")
pp = predict(tmp2, tmp)
rm(tmp, tmp2)

# drop the original and append these
shots_raw = select(shots_raw, -arenaAdjustedShotDistance:-shotAngle)
shots_raw = cbind(shots_raw, pp)
dim(shots_raw)

[1] 64318    14

scaler = MinMaxScaler()
COLS = ['arenaAdjustedShotDistance', 'arenaAdjustedXCord', 'arenaAdjustedYCord', 'shotAngle']
shots_raw[COLS] = scaler.fit_transform(shots_raw[COLS])
shots_raw.shape

(64318, 14)

Setup the tokenizer and fit to the Player IDs

For this exercise, instead of converting the player ids to be 0-based, I am going to treat the player ids as if they are unique words, with the unique number of players representing our complete vocabulary. As such, document represents a shot of the puck on net, and each document only includes one “word”, or shooter.

The trick here is that we have to treat our player ids as character strings.

# ensure that the shooter ID is a string
shots_raw$shooterPlayerId = as.character(shots_raw$shooterPlayerId)

# setup the tokenizer
shooter_tokenizer = text_tokenizer()

# fit the shooters 
fit_text_tokenizer(shooter_tokenizer, shots_raw$shooterPlayerId)

shooter_tokenizer$index_word[1:3]

$`1`
[1] "8477492"

$`2`
[1] "8471214"

$`3`
[1] "8474157"

shooter_tokenizer$word_index[1:3]

$`8477492`
[1] 1

$`8471214`
[1] 2

$`8474157`
[1] 3

length(shooter_tokenizer$index_word)

[1] 869

# make an integer so zero is not parsed
shots_raw.shooterPlayerId = shots_raw.shooterPlayerId.astype('int')

# ensure that the player ID is a string
shots_raw.shooterPlayerId = shots_raw.shooterPlayerId.astype('str')

# setup the tokenizer
shooter_tokenizer = Tokenizer()

# fit the tokenizer to shooters
shooter_tokenizer.fit_on_texts(shots_raw.shooterPlayerId)

list(shooter_tokenizer.index_word.items())[:3]

[(1, '8477492'), (2, '8471214'), (3, '8474157')]

list(shooter_tokenizer.word_index.items())[:3]

[('8477492', 1), ('8471214', 2), ('8474157', 3)]

len(shooter_tokenizer.index_word.items())

869

Create the Shooter sequences

These are size 1 sequences that do not require padding, as we only allow 1 word (or player) per shot. The key here is that we are using keras to help us easily map our data to the new id system.

# make sequences with the new index
shooters = texts_to_sequences(shooter_tokenizer, shots_raw$shooterPlayerId)
shooters = unlist(shooters)

class(shooters)

[1] "integer"

length(shooters)

[1] 64318

shooters = shooter_tokenizer.texts_to_sequences(shots_raw.shooterPlayerId)
shooters = [x[0] for x in shooters]
shooters = np.array(shooters)

type(shooters)

<class 'numpy.ndarray'>

len(shooters)

64318

Isolate the other features/targets

# Was the shot a goal?   This is our target.
goal = shots_raw$goal

# the shot info
shot_info = shots_raw %>% select(-shooterPlayerId, -goal)
shot_info = as.matrix(shot_info)

length(goal); mean(goal);

[1] 64318

[1] 0.09103206

dim(shot_info)

[1] 64318    12

colnames(shot_info)

 [1] "shotTypeBACK"              "shotTypeDEFL"             
 [3] "shotTypeSLAP"              "shotTypeSNAP"             
 [5] "shotTypeTIP"               "shotTypeWRAP"             
 [7] "shotTypeWRIST"             "offWing"                  
 [9] "arenaAdjustedShotDistance" "arenaAdjustedXCord"       
[11] "arenaAdjustedYCord"        "shotAngle"                

# Was the shot a goal?   This is our target.
goal = np.array(shots_raw.goal)

# the shot info
shot_info = shots_raw.drop(columns=['shooterPlayerId', 'goal'], axis=1, inplace=False)

len(goal)

64318

goal.mean()

0.09103205945458503

shot_info.shape

(64318, 12)

shot_info.columns

Index(['offWing', 'arenaAdjustedShotDistance', 'arenaAdjustedXCord',
       'arenaAdjustedYCord', 'shotAngle', 'shotType_BACK', 'shotType_DEFL',
       'shotType_SLAP', 'shotType_SNAP', 'shotType_TIP', 'shotType_WRAP',
       'shotType_WRIST'],
      dtype='object')

Define the model architecture

# the setup
NUM_SHOOTERS = length(unique(unlist(shooter_tokenizer$index_word))) +1
SHOT_COLS = ncol(shot_info)
VEC_SIZE = 50

# the input layers
shooter_input = layer_input(shape=c(1), name = "shooter_input")
shot_input = layer_input(shape=c(SHOT_COLS), name = "shot_input")

# shooter layers
s1 = layer_embedding(input_dim = NUM_SHOOTERS, 
                     output_dim = VEC_SIZE, 
                     input_length = 1, 
                     name="shooter_embedding")(shooter_input)
s2 = layer_flatten(name = "shooter_flat")(s1)
s3 = layer_dense(units = 1, activation = "sigmoid")(s2)

# put the model together
model = keras_model(inputs = shooter_input, outputs = s3)

summary(model)

Model: "model"
________________________________________________________________________________
Layer (type)                        Output Shape                    Param #
================================================================================
shooter_input (InputLayer)          [(None, 1)]                     0
________________________________________________________________________________
shooter_embedding (Embedding)       (None, 1, 50)                   43500
________________________________________________________________________________
shooter_flat (Flatten)              (None, 50)                      0
________________________________________________________________________________
dense (Dense)                       (None, 1)                       51
================================================================================
Total params: 43,551
Trainable params: 43,551
Non-trainable params: 0
________________________________________________________________________________

# setup
NUM_SHOOTERS = len(np.unique(shooters)) + 1
SHOT_COLS = shot_info.shape[1]
VEC_SIZE = 50

# the input layers
shooter_input = Input(shape=(1, ), name="shooter_input")
shot_input = Input(shape=(SHOT_COLS, ), name="shot_input")

# shooter layers
s1 = Embedding(NUM_SHOOTERS, VEC_SIZE, input_length=1)(shooter_input)
s2 = Flatten()(s1)
s3 = Dense(1, activation="sigmoid")(s2)

# put the model together
model = Model(inputs = shooter_input, outputs = s3)

model.summary()

Model: "model_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
shooter_input (InputLayer)   [(None, 1)]               0
_________________________________________________________________
embedding (Embedding)        (None, 1, 50)             43500
_________________________________________________________________
flatten (Flatten)            (None, 50)                0
_________________________________________________________________
dense_1 (Dense)              (None, 1)                 51
=================================================================
Total params: 43,551
Trainable params: 43,551
Non-trainable params: 0
_________________________________________________________________

# below might choke RMD
plot_model(model, to_file='model.png')

Train and Evaluate the Model

model %>% 
  compile(optimizer = "adam", 
          loss="binary_crossentropy", 
          metrics =c("accuracy"))

history = 
  model %>% 
  fit(x=list(shooters), 
      y=goal, 
      epochs = 5, 
      verbose = 2)

model.compile(optimizer="adam", loss = "binary_crossentropy", metrics = ['accuracy'])

X = [shooters, shot_info]
history = model.fit(X, goal, epochs=5)

Get the Embeddings

With our simple model, we have estimated embeddings for each shooter. Let’s grab those.

shooter_embeddings = get_weights(model)[[1]]

shooter_embeddings[1:3, 1:3]

            [,1]       [,2]        [,3]
[1,] -0.03280475 0.02733728 -0.01639561
[2,]  0.08887081 0.12082704  0.12113131
[3,]  0.09737433 0.06078419  0.05767342

dim(shooter_embeddings)

[1] 870  50

shooter_embeddings = model.layers[1].get_weights()[0]

shooter_embeddings[1:4, 1:4]

array([[-0.07189947, -0.08770541,  0.04801337],
       [-0.03720884, -0.04936351,  0.04588475],
       [-0.1379198 , -0.04528455,  0.08142862]], dtype=float32)

shooter_embeddings.shape

(870, 50)

Map the embeddings to the players

The embeddings are related to a player, so we are intereseted extracting these vectors and looking at player similarity, etc.

# build our vocabulary (player) dataframe
# https://www.r-bloggers.com/word-embeddings-with-keras/
players = data.frame(
  playerid = names(shooter_tokenizer$word_index), 
  id = as.integer(unlist(shooter_tokenizer$word_index)), stringsAsFactors=FALSE)

players = dplyr::arrange(players, id)

# keep only those rows where the indexes align - R is 1-based
shooter_embeddings = shooter_embeddings[players$id, ]
rownames(shooter_embeddings) = players$playerid
colnames(shooter_embeddings) = paste0("e", 1:ncol(shooter_embeddings))
shooter_embeddings[1:3, 1:3]

                 e1         e2          e3
8477492 -0.03280475 0.02733728 -0.01639561
8471214  0.08887081 0.12082704  0.12113131
8474157  0.09737433 0.06078419  0.05767342

# make the embed vectors a pandas dataframe
shooter_embeddings = pd.DataFrame(shooter_embeddings)

# a list of true shooter ids
#shooter_id = [v for k, v in shooter_tokenizer.index_word.items()]
shooter_id = {k:v for k, v in shooter_tokenizer.index_word.items()}
shooter_df = pd.DataFrame.from_dict(shooter_id, orient='index', columns=["playerid"])

# name the columns
shooter_embeddings.columns = ["e" + str(i + 1) for i in range(shooter_embeddings.shape[1])]

# align the data by index
shooter_embeddings = pd.merge(shooter_embeddings, shooter_df, how='inner', left_index=True, right_index=True)

# clean up the index so its the player
shooter_embeddings.index = shooter_embeddings.playerid

# the first few
shooter_embeddings.iloc[:3, :3]

                e1        e2        e3
playerid                              
8477492   0.093861 -0.071899 -0.087705
8471214   0.072550 -0.037209 -0.049364
8474157   0.059455 -0.137920 -0.045285

Export the data to Tableau

Whether it is R or python, you might be asking why I am exporting the data to Tableau. That is a fair question, but the point is to show how the ecosystem of data science programming libraries can also leverage best-of-breed data visualization suites such as Tableau. The tool plays a key role in my exploratory analysis pipeline, and the goal below is show how in 1-line of code, we can export our data for rapid exploration, which can aid in our data cleaning and modeling tasks within R/python.

Rstudio and Reticulate

There has been plenty written about reticulate, so I will let you dive into the tutorials and background. It is not without its quirks, but by and large, the combo works really well, especially for a younger solution. Historically there have been other attempts to bridge the gap, but from a feature and usability perspective, this is by far the most robust offering if you ask me.

Over the last year, I have been (slowly) working on a python package to help facilitate the collection and analysis of datasets that are openly available within higher education. I mention the tools above because my development has been really slow outside of RStudio. Last week, I got fed up and came back to RStudio. If I must say, the experience has been really pleasant, but more importantly, I am writing code at a much faster rate. Is it because I am more comfortable with the Rstudio interface? Perhaps, but I really do believe RStudio could be THE data science IDE of the future.

With that said, here are a few things that tripped me up along the way. This is not meant to call out the quirks of developing python packages using RStudio and reticulate, but it is a note to my future self as to the tricks necessary to work around some issues that are pretty annoying.

Generate a dataset

This is probably more easily explained via a toy dataset.

I am using the wakefield package, which I believe to be fantastic. Check it out here.

The README really starts to highlight what is possible in regards to synthentic data generation.

# load the libraries
suppressPackageStartupMessages(library(tidyverse))
library(wakefield)
library(jsonlite)

If you have any issues with above, it should be as easy as install.packages('<package_name_here>') to install the required package(s).

Now that we are good to go, let’s build out a fake dataset to demonstrate my use-case.

## synthetic dataset using wakefield -- super expressive, right?
dat = r_data_frame(
  n = 4,
  id,
  race,
  age,
  died
) %>%
  r_na(prob = .5)

## take a quick peak at the data
glimpse(dat)
Rows: 4
Columns: 4
$ ID   <chr> "1", "2", "3", "4"
$ Race <fct> NA, White, Bi-Racial, NA
$ Age  <int> NA, 62, NA, 52
$ Died <lgl> NA, TRUE, TRUE, NA

What we have is 4 rows of data, and 4 columns of data. However, the bigger point is that we can see that each of the final 3 variables have some NA injected into their values. This is done via the r_na function available in wakefield.

You should really check out the docs with ?r_na

Ok, moving on.

Parsing

The goal is to generate JSON datasets where each entry is a record from our dataset, with the appropriate key:value pairs representing the features for each observation.

The jsonlite package is feature-rich, and while its totally RTFM, I only just noticed the dataframe parameter for toJSON, which takes one of 3 options:

• rows
• columns
• values

Not really knowing what each did, the code below applies each transformation, and then generates an R list for each.

What I expect is a list of length that is equivalent to 4, with each entry being another list (the actual row of data) with a length of 1 to 4, but not exactly 4 for each entry as the NA values should be omitted from the JSON entry.

The code below creates three objects, one testing each value.

## write to a json file - note how to handle dataframes
dat_r = toJSON(dat, dataframe = "rows")
dat_c = toJSON(dat, dataframe = "columns")
dat_v = toJSON(dat, dataframe = "values")

## bring the character vectors back but as lists in R
## might be a way to do this in jsonlite, but my old habits stay here
dat_rl = rjson::fromJSON(dat_r)
dat_cl = rjson::fromJSON(dat_c)
dat_vl = rjson::fromJSON(dat_v)

In recent versions of Rstudio, it’s possible to inspect lists within the Environment pane. Given these are small objects, I would encourage you to poke around with the dat_[r|c|v]l objects from above.

rows to the answer

Jumping ahead, let’s look at the printout of both dat and the list which was built using the columns as the value to the dataframe parameter.

dat
# A tibble: 4 x 4
  ID    Race        Age Died
  <chr> <fct>     <int> <lgl>
1 1     <NA>         NA NA
2 2     White        62 TRUE
3 3     Bi-Racial    NA TRUE
4 4     <NA>         52 NA

And now the list form, which represents exactly the JSON format I am looking for.

lapply(dat_rl, names)
[[1]]
[1] "ID"

[[2]]
[1] "ID"   "Race" "Age"  "Died"

[[3]]
[1] "ID"   "Race" "Died"

[[4]]
[1] "ID"  "Age"

We can see from above that the length is what we expected, and the columns of data for each observation are not the same, as the NA values were omitted from the data structure.

So what

The next logical question is why should I care, right?

• When converting between various data formats, if we store the fields with missing data, it’s just taking up space on disk
• I am a huge fan of neo4j, and a quick way to create nodes is to pass a JSON entry, but the key is that you start to get into trouble when attempting to write properties (keys) that have missing values.
• Don’t attempt to reinvent the wheel

Why this post

Previously, installing rasa was a breeze. Admittedly I never really had issues until recently. I am uncertain if the issue occurred on my machine with the new Mac OS (I am a Mac user), or if new features were added and downstream dependencies were missing from my machine. Regardless, I want to remind myself of how I was able to fix the issue with the help of this Github Issue. I will do the setup on a Mac, as well as a simple cloud droplet on Digital Ocean.

Install on a Mac

The first thing that we need to do is create an environment in conda. An environment encapsulates our tooling and helps avoid issues where an upgrade to a python package could break your ability to run your previous code. Basically, use environments whenever possible.

conda create -n rasabot python=3.6

With above, the conda tool will manage the dependencies for you and get a base install of python 3.6 contained with an environment called rasabot. Let’s activate this environment:

source activate rasabot

On my machine, my terminal now tells me that I am in the rasabot environment

(rasabot) QST-ML-0208:brocktibert btibert$

Ok, one thing that I want to do is enforce my version of numpy. Let’s install that below.

pip install numpy==1.14.5

Ok, with that covered, let’s get Rasa Core setup.

pip install rasa_core

As noted from the solution on the Github ticket, your machine may need to have the Mac Command Line tools installed (which is found in the Stackoverflow Answer). In short, you will create an Apple Developer account and install the tool as described here.

Ok, with rasa seemingly good to go, let’s validate:

python -c "import rasa_core; print(rasa_core.__version__)"

This returned 0.12.3.

Ok, we also will want to leverage Rasa NLU. We will install that below.

pip install rasa_nlu[tensorflow]

And we will do the same for validation:

python -c "import rasa_nlu; print(rasa_nlu.__version__)"

yields 0.13.8

We are off to the races.

Setup on Digital Ocean

In addition to local dev, I wanted to make sure that I could spin up rasa in the cloud. I really enjoy using Digital Ocean, but this is applicable anywhere.

Once you have ssh’d into your remote box:

apt-get update
apt-get install
apt-get install libxml2-dev libxslt-dev
sudo apt-get install gcc
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
exit
conda create -n rasabot python=3.6
source activate rasabot
pip install numpy==1.14.5
pip install rasa_core
pip install rasa_nlu[tensorflow]

Above, when you install Miniconda, you will be prompted to hit enter and confirm the locations. I accepted all of the defaults.

That’s it!

Snippet 1: Load CSV from within R

First off, I use RNeo4j to connect R to Neo4j. It is totally possible to load data within dataframes into Neo4j using the ?cypher command; Nicole shows you how to do this in the Import section of the README on the project page. But be warned, once you get to larger datasets, this might not be your best option with respect to speed performance.

If you haven’t already, play around with the neo4j-shell and the LOAD CSV functionality. It’s pretty fast and can handle files of a few million records.

The snippet below is a quick way to simply call that procedure from within your R script.

NEO_SHELL = "~/neo4j-community-2.3.1/bin/neo4j-shell"
build_import = function(neo_shell, cypher_file) {
  cmd = sprintf("%s -file %s", neo_shell, cypher_file)
  system(cmd)
}

Just point to your shell and store it in the variable NEO_SHELL. From calling it is simple.

build_import(NEO_SHELL, "../cypher/add-contstraints.cql")

I simply reference the shell, and pass a text file that contains my cypher statements.

From my experiences, there are huge performance gains when using the shell to execute commands, so the helper function above let’s me stay within R but benefit from the performance gains.

Snippet 2: Read a Cypher query file into R

When developing your code, inevitably you will be playing around with the incredibly helpful browser tool. I use it to prototype my queries, especially before running LOAD CSV on a larger file.

When using R, I have two options.

1. Type in the cypher query into a string, and then pass the query to the cypher function within the R package. Or,
2. Development my queries in separate text files. I use a .cql extension. To me, it’s easier to maintain, but needs to be brought into R.

The process below will bring in a cypher query as a statement into R that can be further passed to the cypher function.

## read the cypher query into a string variable  
## http://stackoverflow.com/questions/9068397/import-text-file-as-single-character-string  
FILE = "../cypher/get-edges.cql"
cypher_edges = readChar(FILE,file.info(FILE)$size)

## get the edges
edges = cypher(graph, cypher_edges)

Above, I have a query file that I read into the variable cypher_edges and pass that to cypher. I use this query to return data back to R. To me, I would rather manage my queries in separate files, not within R, and this allows me to do that. Moreover, I believe it makes organizing your project’s structure much easier.

Aside: My current workflow

I recently implemented the two helper snippets above when working through a dataset for work. After trying to load data, and calculate similarities within the database (a future post), I arrived at this workflow. Given my tool of choice is R, the workflow below is both manageable and pretty fast.

1. Query my data warehouse for new data
2. Use R to clean and tidy the data for import into Neo4j
3. Write the data to CSV files that can be consumed by LOAD CSV. I use Snippet 1 to do that.
4. Once the data are in the database, bring back a subgraph of interest into R using Snippet 2.
5. Manipulate the data as needed. For example, calculate jaccard similarities across the nodes of interest.
6. Write the similarities to another csv file.
7. Use Snippet 1 again to write the similarities back into the database for use in other queries.

For one project, in less than 5 minutes, I was able to import two years of applicants, along with various demographics on that pool, calculate the similarity based on interactions, and write those similarities (nearly 2.8 million rows) back to the database. In terms of speed, this workflow will enable me to opertationalize all sorts of models that can be further implemented into marketing and recruitment efforts. I will post on this shortly …

Summary

This post was meant to capture two helpful code snippets that improve my workflow, and speed, of analyzing datasets in R and Neo4j.

A Basic Model

The code and data model below are intended to highlight a simple proof-of-concept about how you might leverage graphViz to make the data-modeling tasks in Neo4j easier.

grViz("
      digraph neo4j {

      # a 'graph' statement
      graph [overlap = false, fontsize = 10]

      # several 'node' statements
      node [shape = circle, fontname = Helvetica]
      a [label = 'Student'];
      b [label = '@@1-1'];
      c [label = '@@1-2'];
      d [label = '@@1-3'];
      e [label = '@@1-4'];
      f [label = '@@1-5'];
      g [label = 'Marketing Persona'];
      h [label = 'Gender'];
      i [label = 'State'];
      j [label = 'Region'];
      k [label = '@@2-1'];
      l [label = '@@2-2'];
      m [label = '@@2-3'];
      n [label = '@@2-4'];

      # several 'edge' statements
      a -> b [label = 'WAS_SENT' fontsize = 9.5];
      b -> c [label = 'NEXT'];
      c -> d [label = 'NEXT'];
      d -> e [label = 'NEXT'];
      e -> f [label = 'NEXT'];
      a -> g [label = 'FROM_PERSONA'];
      a -> h [label = 'HAS_GENDER'];
      a -> i [label = 'LIVES_IN'];
      i -> j [label = 'IS_IN'];
      a -> k [label = 'HAS_INTERACTION' fontsize = 9.5];
      k -> l [label = 'NEXT'];
      l -> m [label = 'NEXT'];
      m -> n [label = 'NEXT'];
      }

      [1]: rep('Email', 5)
      [2]: rep('Interaction', 4)
      ",
engine = "circo")

From above, a few things that I wanted to call out:

• My example graph is very specific to Enrollment Management. In this case, the data are very student-centric, in that a Student is sent marketing emails, has various demographics associated with them, and interacts with you in a variety of ways (i.e., visit campus, request’s information, etc.). Your domains might yield “prettier” graphs.
• I am leveraging the @@ option to dynamically build the labels, which you reference at the end of the script through a footnote.
• You can control a large number of elements. In a couple of cases, I manually specify the font size for the edge label.

Summary

By no means is this a robust demonstration, but simply a quick post to demonstrate an option that you might want to leverage when documenting and building out your database. As mentioned above, the fact that you can reproduce your graph is why I will probably use DiagrammeR to work through my data modeling tasks.

Intro

The use of graphs to solve business problems is not new, as companies like Amazon, Netflix, and nearly all major social media sites have been doing this for some time. I have been obsessed with graphs for just as long, and after learning as much as I can about analysis of graphs and graph databases, I am finally getting the time to take what I have learned and apply it to real world data problems I face within Enrollment Management.

In this post, I am going to explore the network structure of competition at the undergraduate level. To get the data, I crawled a popular “College Search Site” to extract the information that I needed. I won’t go into specifics, but the web is one huge dataset, and sometimes you need to be able to write a bot (computer program) to crawl and build your datasets.

In short, this post attempts to predict if one college is “similar” to another by leveraging the other institutions they are similar to, and then looking at the connections those schools have as well. For the time being, I am only going to look 1 hop away from the school of interest, but there is so much more that we can do with this type of database.

I am going to leverage Neo4j, a graph database, for this analysis. Neo4j fits perfectly with this use case, as it not only belongs to the NoSQL family of databases, but with only a few lines of code, you can quickly generate recommendations, or predictions, from your graph. Pretty powerful!

Lastly, it’s worth noting that I am only looking at similarity between institutions. A more robust analysis would be the connections students make with these schools. The data exist, but outside of one company, I don’t know if other college search sites are taking advantage of this data structure.

If you work at one of these companies and aren’t sure, reach out. I would love to chat about what we might be able to do! I am always looking for real-world datasets for the academic research I do outside of my role in Enrollment Management, so maybe we can work together!

The Dataset

As always, I am using R for my analysis. Nicole White has created an awesome R package RNeo4j to connect to Neo4j. I am using that library below to connect to the database and wipe out all my data for a clean environment.

## load the library
library(RNeo4j)

## connect to the graph
graph = startGraph("http://localhost:7474/db/data/")
graph$version

# clear it out entirely
## CAVEAT: deletes the data without user confirmation.  Do not copy and paste!
# cypher(graph, "MATCH (n) RETURN COUNT(n)")
clear(graph, input = F)

Now that we have a clean database, let’s talk about the data.

Lets suppose you have a dataset that looks like this:

Here is a description of what these fields represent.

1. from_unitid: The IPEDS unitid for the school of interest.
2. to_unitid“: The IPEDS unitid for the school that is similar to from_unitid
3. from_rank: A rank-ordered estimate of how similar the from institution is to the to institution
4. from_rating: The rating - out of 5 stars - for the to_unitid school scraped from the source website
5. to_rating: The rating - out of 5 stars - for the similar school

There already has been a ton of research on the prediction of edges (links) in graphs, so my approach to this problem is only scratching the surface, and quite frankly, basic.

Since we are trying to build a model that predicts if one school is defined as smililar to another, we need to remove a small portion of our dataset in order to validate the accuracy of that estimate.

The code below is how I randomly deleted a few edges from our graph in order to retain a test dataset

## ensure that there are 25 rows for each school
tmp = tbl_df(edges) %>% group_by(from_unitid) %>% summarise(num = length(to_unitid))
tmp = filter(tmp, num == 25)
edges = subset(edges, from_unitid %in% tmp$from_unitid)
rm(tmp)

## split into the traing and test dataset
ROWS = sample(1:nrow(edges), 150, replace=F)

## training graph
graph_test= edges[ROWS, ]
graph_train = edges[-ROWS,]
rm(ROWS)

## how many unique schools in the test dataset
length(unique(graph_test$from_unitid))
save(graph_test, graph_train, edges, file="data/graph-data.rdata")

Database Creation

The image below is how we are translating the data into a network graph.

Our dataset is more comparable to Twitter than Facebook. School 1 is similar to School 2, but School 2 isn’t similar to School 1. In graph terms, this is a directed network. Unlike Facebook, where users both agree to to a friendship, our dataset is comprised of connections between institutions that may not be reciprocated.

Because of this, we can start to map out the competitive landscape within higher education. Even though the dataset comes from only one college search site, it’s a start, and allows us to look at who are competitors are competing with. That’s some powerful stuff!

Before we do any analysis, we need to put the data into Neo4j.

## set the constraints first
addConstraint(graph, "School", "unitid")

## the query
query = "
MERGE (s1:School { unitid:{from_unitid} } )
ON CREATE SET s1.rating = {from_rating}
MERGE (s2:School { unitid:{to_unitid} } )
ON CREATE SET s2.rating = {to_rating}
CREATE (s1) -[r:SIMILAR_TO {rank:{from_rank}}]-> (s2)
"

## start the initial transaction
tx = newTransaction(graph)

## just using the small dataset for exploring and valid queries
start = Sys.time()
for (i in 1:nrow(graph_train)) {
 ## for every 500 rows, commit the transaction
 if (i %% 500 == 0) {
  # commit the transaction
  commit(tx)
  cat("Batch ", i/500, " committed \n")
  # open a new transaction
  tx = newTransaction(graph)
 }

 ## create the parameter values
 from_unitid = graph_train[i, ]$from_unitid
 from_rating = graph_train[i, ]$from_rating
 to_unitid = graph_train[i, ]$to_unitid
 to_rating = graph_train[i, ]$to_rating
 from_rank = graph_train[i, ]$from_rank

 ## appead the query
 appendCypher(tx,
              query,
              from_unitid = from_unitid,
              from_rating = from_rating,
              to_unitid = to_unitid,
              to_rating = to_rating,
              from_rank = from_rank)
} #endfor

## committ the last transaction and record the time it takes
commit(tx)
end = Sys.time()
end - start

Using cypher transactions, as I did above, isn’t the fastest way of getting data into Neo4j, but it’s clean and easy to read. For reference, it took a tad over 12 minutes to put our training dataset into the database.

If you need to throw larger volumes of data into the database, you should check out the shell tools that are part of the base install. More specifically, neo4j-shell and neo4j-import.

Explore the data

Now let’s confirm that we have some data.

## make sure we are connected
graph = startGraph("http://localhost:7474/db/data/")

## the number of nodes
cypher(graph,
       "
       MATCH (n)
       RETURN COUNT(n) as `Total Nodes`
       ")

  Total Nodes
1        3477

## summary of the data model
summary(graph)

    This         To   That
1 School SIMILAR_TO School

## how many relationships
cypher(graph,
       "
       MATCH () -[r]- ()
       WITH r
       RETURN type(r), count(*) as total
       ")

     type(r)  total
1 SIMILAR_TO 161860

Finally, let’s look at a quick plot from a small subset of the data

## the query
query = "
MATCH (n)-[r]->(b)
RETURN n.unitid AS from, b.unitid AS to, r.rank AS rank
LIMIT 50
"

## the dataframe
dat = cypher(graph, query)
dat$rand = NULL

## graph the dataframe
g = graph.data.frame(dat, directed = T)

## simple plot
ggnet(g, size=10)

The plot above just pulls out 50 nodes. I won’t go into describing the graph, but I find it interesting that there are two schools that are common to the larger clusters, but these two institutions are not similar to each other.

Predicting Edges

As mentioned above, I wanted to create a very basic predictive model to test if leveraging the structure of the graph is better than just guessing.

I don’t know if it’s my machine, a sub-optimal CYPHER query, or simply the expected run-time, but generating the predictions for 150 schools took nearly two hours. I have had much better performance on larger data, so something doesn’t feel right here.

Let’s take a quick at the predictions to ensure the output matches our expectations.

Note that column freq is how many schools were included in the prediction. If we wanted to extend this, a school can have up to 25 SIMILAR schools. With this in hand, you could estimate a confidence of the prediction by freq / 25, or more accuractley, freq / # Schools Similar to. This would yield a percentage, with numbers closer to 1 indicating that nearly all other schools have this recommendation as a competitor.

We finally have what we need!

Results

Let’s see how the recommendation did.

First, in order to measure how well our simple model performs, we need to understand the compared to what question. To create this baseline, we go back to our original dataset, identify the most popular schools, and then select the most popular insitution not already listed by the school of interest. This is just a Top-N prediction.

pop_accurate
FALSE
  150

From above, we can see that the Top-N prediction did not predict any connections correctly, as evidenced by the 150 underneath the FALSE. In reality, if my test set was larger than 150 edges, undoubtedly we would have got a handful of correct guesses.

Now, let’s look at the model.

pred_accurate
FALSE  TRUE
  105    45

From above, we see that the model accurately predicted 45 of the 150 connections. That is 30 % correct, not bad.

Summary

Overall, there is some evidence to suggest that the network structure of the competition higher ed can be leveraged to predict which schools compete with each other. The holdout sample of 150 cases wasn’t that large, but predicting 30% of the cases with a very simple model isn’t a bad start.

Neo4j is a pretty powerful database, but I still have a lot to learn. Generating predictions (recommendations) for 150 cases shouldn’t take nearly two hours. This would never fly in a production environment, but as mentioend above, I suspect that there are some things that I can do on my end to greatly enhance performance. In another study (not published), I was able to generate predictions for 1000 entities in less than ten minutes. Neo4j admits that it’s not an analytical engine, which is why there are some great tools being devolped on top, like Mazerunner which aims to bring graph ETL capabilities to the database.

Stepping back, I am excited to extend the use of Neo4j into other projects within Enrollment Management, as I strongly believe that analyzing our data in graph format can yield strategic insights above and beyond what we can already do with more traditional methods like regression, clustering, and machine learning.

Overview

Generally, you can set environment variables by:

export YOUR_VAR=abc123

within a terminal. In a new session,

echo $YOUR_VAR

should yield what you need.

Within python, you can get at it by:

import os
os.getenv("YOUR_VAR")

but if you are using R and Rstudio, Sys.getenv("YOUR_VAR") will return "".

No bueno.

A solution

Navigate to ~, and create the .Renviron file if it doesn’t already exist

cd ~
touch .Renviron
open .Renviron

And in the file, type

YOUR_VAR="abc123"

Save the file and restart/reopen Rstudio.

From there, Sys.getenv("YOUR_VAR") should be good to go.

Deeper Dive

For a more granular look at this functionality, feel free to reference the links below

• Startup
• The reference that I used

About this post

For this post, I will be using using the R language to download the data from WICHE, an amazing resource for projections of High School graduates by state. Using these data, we can do all sorts of fun analyses.

In a future post, I will show you how to link WICHE to IPEDS data in order to forecast college participation rates by state.

While I will provide a few code snippets below, you should feel free to clone my Github Repo which everything you need to replicate this post.

Also included is a Tableau Workbook. If you have Tableau Desktop, this super basic workbook highlights how you can leverage parameters to create your own forecasts.

Below is a screenshot of the workbook, which is a basic “Create-Your-Own College Enrollment Forecast” of sorts.

Why this post

The changing demographics and volume of students that would be considering a college education should not be news to anyone in Enrollment Management. I hope to highlight how with just a few lines of code, we can:

1. Grab data that forecasts the volume of high school graduates
2. Use R to parse, clean, and reshape the data (originally stored in Excel)
3. Save out the data and leverage Tableau to do some basic forecasting

For those of you that might be new to R, reading code can be extremely helpful when attempting to learn a new language. When possible, I always try to comment the heck of out my code. Hopefully these comments can help you in your journey.

Get the data

With R, it’s super simple to grab data from the web. The command below will download the WICHE Excel Workbook.

## download the dataset into your working directory
## use mode option below so the file can open in R, error w/o it
WICHE_DATA = "http://wiche.edu/info/knocking-8th/tables/allProjections.xlsx"
download.file(url=WICHE_DATA, destfile="raw/wiche.xlsx", mode="wb")

It should be noted that the code above assumes that your current directory (where you are running the code) has a folder called raw. To assure that this is the case, just do this:

## ensure that we have a directory to store the raw data
if (!file.exists("raw")) dir.create("raw")
if (!file.exists("figs")) dir.create("figs")

Now we can use the RODBC package (on Windows) to connect to the workbook and query it as if the sheets were database tables.

xl = odbcConnectExcel2007("raw/wiche.xlsx")

Because each state is a tab in the workbook, let’s use R to define an object that holds the state abbreivations, which we will use while looping through the workbook.

## how cool is it that R has the State names and Abbreviations preloaded?
?state.name
(states = state.name)
length(states)
states = c(states, "District of Columbia")

Finally, let’s loop and build a dataset in the format we want:

## use a for loop -- not ideal but easy to read and debug
wiche = data.frame(stringsAsFactors=FALSE)
for (state in states) {
 raw = sqlFetch(xl, state, stringsAsFactors=FALSE)
 ## bc there is a structure to each sheet, we can reference each column by index
 ## no way is this ideal, but quick when data doesnt change
 ROWS = 9:40
 COLS = c(1, 3:10)
 ## create a flag for actual/projected -- hard coded from looking at Excel file
 status = c(rep("actual", 13), rep("projected", 19))
 ## keep the data
 df = raw[ROWS, COLS]
 colnames(df) = c('year',
                  'pub_amind',
                  'pub_asian',
                  'pub_black',
                  'pub_hisp',
                  'pub_white',
                  'pub_total',
                  'np_total',
                  'total')
 ## remove the commas -- using a for loop not ideal, but intuitive
 for (i in 2:ncol(df)) {
  df[,i] = as.numeric(gsub(",", "", df[,i]))
 }
 df$state = state
 df$status = status
 ## bind onto the master data frame
 wiche = rbind.fill(wiche, df)
 ## status
 cat("finished ", state, "\n")
}

A quick plot

When playing around with data, it’s usually a good practice to visualize what you have. Below is a quick plot which represents both the actual and forecasted volume of high school graduates going until the 2027/28 Academic year.

Summary

I would encourage the reader to browse the code, and if possible, fire up the Tableau workbook. As an Enrollment Scientist, R and Tableau are my two tools that I use on a daily basis.