Photo by nine koepfer on Unsplash | Branded content disclosure

Data lakes provide a myriad of benefits. They are data agnostic and don’t require you to define a schema up-front. However, without a proper structure, it may be challenging to find the data that you need. In this article, we’ll address this problem by creating a FastAPI abstraction allowing us to query the AWS Glue metadata catalog.

If we would implement it as a Python package instead, we would assume that everyone knows enough Python to use it. However, your company may have other developers (Java, C#, JavaScript, Go, …) that need access to data from a data lake. By building a REST-ful service for data discovery, you are providing a programming-language agnostic interface for everyone. This API-first approach has further benefits:

We can start by listing all the methods that can be potentially useful for data discovery:

We have three choices: either using lower-level AWS SDKs, doing queries on Athena’s information_schema, or leveraging awswrangler. In the end, we may combine all of them to satisfy the requirements from the previous section.

#1. To list all databases (i.e., schemas) and tables, we can use the awswrangler package, especially the module wr.catalog.

Image by the author

#2. To filter for table names with a specific prefix or suffix, we can also use wr.catalog.tables. The same is true for retrieving table definition and doing a full-text search on those definitions:

Image by the author

#3. List all partitions of a specific table:

Image by the author

#4. Show table’s DDL:

Image by the author

#5. List S3 directory path or show specific objects according to file type (ex. parquet files) and last modified date:

Image by the author

#6. Query column comments to find the dataset that you need:

Image by the author

#7. Describe Athena table definition:

Image by the author

#8. Search for specific column names:

Image by the author

The above methods constitute a basic MVP for a data discovery service. It allows querying schemas, tables, columns, column comments (aka data dictionary), showing a preview of the data, as well as exploring the underlying files, directories, and partitions.

By leveraging services such as AWS X-Ray and Lake Formation, we could add methods to query usage access patterns and identify the most and the least used datasets.

The full code for this MVP can be found in this Github repository. Let’s briefly discuss some of the details.

In the following demo, we can see all the endpoints applied on a Brazilian E-commerce dataset from Kaggle.

The screencast above demonstrates how we can use this API to discover and query e-commerce data stored in an S3 data lake. The end users can identify tables or schemas that they need. By drilling down into the underlying files, data scientists and analysts can explore this data in an interactive notebook or other tools of their choice. Reading a specific parquet, CSV, Excel, or JSON file from S3 with awswrangler is as simple as:

Image by the author

There are several options to deploy a REST API. If you already use AWS, you may find it useful to leverage serverless services built specifically for building resilient and scalable APIs: AWS Lambda and API Gateway.

With the excellent package called mangum, converting our FastAPI to a Lambda handler is as simple as importing this package (from mangum import Mangum) and adding a single line of code: handler = Mangum(app=app).

The Dockerfile to deploy the API to AWS Lambda looks as follows:

Image by the author

We use the official Python Lambda image with Python 3.8 and install all required packages. Then, we copy our remaining API code and specify the Lambda handler that will be used as an entry point to our container.

Finally, we have to push the container to ECR — AWS container registry. To replicate this demo, replace 123456789 with your AWS account ID, and adjust your AWS region name. In case you wonder: dda is my abbreviation for data-discovery-api.

Image by the author

The container image is deployed to ECR. Now, we need to create a Lambda function. We choose the container image option and select our ECR image.

Image by the author

Since querying some tables can take longer than Lambda’s default timeout of just three seconds, we can extend it to three minutes.

Image by the author

Finally, we need to attach IAM policies to grant our Lambda function permissions to retrieve data from AWS Glue, Athena, and S3 (of course, we should use more granular permissions for production).

Image by the author

After configuring the Lambda function with our container image, we can create a new API in the API Gateway console. We need to choose REST API and specify a name.

Image by the author

Then, we can add a method and resource to configure Lambda Proxy integration.

Image by the author

Once all that is set up, we can deploy the API and start serving it.

To see it step by step, here is a screencast that demonstrates API Gateway configuration, deployment, and testing the API:

To see how you can additionally configure API keys to make your API secure, check my previous article. It also shows how to configure distributed tracing with AWS X-Ray for even more control over your FastAPI service on AWS:

FastAPI + AWS = secure API (Part 2)Let’s build a secure and observable Python REST API with FastAPI, AWS Lambda, API Gateway, CloudWatch, and AWS X-Raytowardsdatascience.com

In this article, we investigated how we can build a REST-ful service for data discovery. By leveraging Python libraries such as FastAPI, awswrangler, boto3, and Mangum, we can build useful APIs in just a few hours rather than weeks. Additionally, by deploying this service to API Gateway and Lambda, we can serve this API at scale with no operational headaches. Lastly, by leveraging Dashbird, you can add observability to your serverless resources without having to install any CloudWatch agents, pulling logs, or building any dashboards.

Thank you for reading! If this article was useful, follow me to see my next posts. Also, have a look at related articles: