Today, we release the Tinybird DynamoDB Connector in public beta. The DynamoDB Connector is a fully managed data source integration for importing data from Amazon DynamoDB to Tinybird. With the DynamoDB Connector, developers can quickly sync data from the popular AWS database into Tinybird, which is better suited for aggregations and real-time, OLAP workloads.
The DynamoDB Connector is now available in public beta on all plans across all supported AWS regions. For detailed instructions on how to use the DynamoDB Connector, please refer to the documentation.
For more information on how the DynamoDB Connector works, keep reading or watch the screencast below.
Connecting DynamoDB to Tinybird
The Tinybird DynamoDB Connector gives developers using Amazon's DynamoDB a fully managed solution to move real-time analytics use cases off of DynamoDB and onto Tinybird without writing any additional code or maintaining any external ETLs.
While DynamoDB can be a powerful real-time database for the right type of workload, it can struggle under more intense analytical workloads. Customers like Canva have transitioned analytics previously run on DynamoDB to Tinybird for better performance, and the DynamoDB Connector now makes this transition much simpler.
Canva captured telemetry into DynamoDB in an attempt to build user-facing analytics, but the approach failed because DynamoDB struggled to handle Canva's aggregations, and Canva found it complex to design the right schemas. They'd spend several months in the design phase trying to find the optimal schema and couldn't afford to redesign these schemas anytime they wanted to evolve or explore new use cases.
Read the full story
Backfills and streaming upserts
The DynamoDB Connector syncs DynamoDB to Tinybird in two parts: 1) an initial backfill, and 2) continuous streaming updates. Tinybird transparently manages both processes.
For initial backfills, Tinybird uses DynamoDB's Point-in-time Recovery (PITR) feature to export a snapshot of the DynamoDB instance to files in Amazon S3, which are ingested into the Tinybird Data Source when the connection is first created.
To stay in sync with changes to DynamoDB tables, Tinybird automatically sets up and subscribes to DynamoDB Streams, using a Change Data Capture (CDC) approach to handle upserts and maintain a stateful copy of your DynamoDB tables in Tinybird.
Latency for this continuous sync is no more than 5 seconds under normal loads.
Security through AWS IAM Roles and Policies
As with Tinybird's Amazon S3 Connector, the DynamoDB Connector uses the AWS Identity and Access Management (IAM) service to securely manage permissions between Tinybird and DynamoDB.
When you create DynamoDB connection, Tinybird will automatically generate the JSON snippets needed to create the IAM Role and Access Policy that allow Tinybird to read from your DynamoDB tables. All you need to do is add your DynamoDB table name and S3 bucket, then copy and paste.
Out-of-the-box ingestion observability
As with every Tinybird Data Source Connector, observability for the DynamoDB Connector is available out of the box using the datasource_ops_log Service Data Source. All write operations from DynamoDB to Tinybird are logged in this automatically generated Data Source, and you can query and analyze the logs free of charge to build observability APIs and dashboards.
Pricing for the DynamoDB Connector
Pricing for the DynamoDB Connector is based on data processing and data storage.
The initial write operation to backfill data and the continuous CDC-based syncs will incur write processing costs at the standard $0.07/GB.
The data stored in your DynamoDB Data Source will incur a monthly charge of $0.34/GB compressed. The precise amount of compression will depend on your schema, but the typical compression ratio for Tinybird Data Sources varies between 3.5x and 10x.
In addition, you'll need to consider the costs incurred on AWS, specifically for any storage and egress costs to handle snapshotting and continuous syncs. These will depend on your data size, region, etc.
A plug-and-play transition for Rockset customers
Real-time database Rockset recently announced its acquisition by OpenAI and its intention to sunset the service by the end of the month. Many Rockset customers used Rockset's DynamoDB connector to quickly ingest tables from DynamoDB to Rockset.
The Tinybird DynamoDB Connector provides an experience familiar to former Rockset customers, enabling a seamless transition to Tinybird's real-time data platform.
For more information about migrating from Rockset to Tinybird, check out the migration guide.
Using the DynamoDB Connector
Connecting Tinybird to DynamoDB is straightforward. First, create a new Data Source and select the DynamoDB Connector.
Then, follow the guide to create the Access Policy and Role in the AWS IAM Service. Tinybird will automatically generate the JSON snippets for each. All you need to do is copy and paste into the AWS IAM console and supply the DynamoDB table name and S3 bucket you want to use for backfill.
Once you've established a secure connection between Tinybird and your DynamoDB instance, choose the DynamoDB table and an S3 bucket to support the snapshot and initial load. Tinybird will analyze the DynamoDB table and generate a preview with inferred schema. You can modify the schema as necessary before creating the Data Source.
Once the Data Source has been created, Tinybird will perform an initial load, backfilling existing data from DynamoDB into the Tinybird Data Source. Once the initial load is complete, Tinybird will automatically sync changes from the DynamoDB table into Tinybird, transparently handling upserts to maintain an accurate state.
You can also use the Tinybird CLI to set up a connection to DynamoDB. Simply use the command below and follow the guided prompts to create the connection:
Get started with the DynamoDB Connector
The DynamoDB Connector is now available in public beta for Workspaces in all supported AWS regions. To get started, create a new Data Source and follow the guided steps to sync your DynamoDB tables with Tinybird.
For more details about the DynamoDB Connector, refer to the documentation.
As always, if you have any questions, please join our Slack Community.
New to Tinybird?
If you're currently working with DynamoDB and looking for a real-time analytics database to scale your performance, try Tinybird. The Build plan is free, with no credit card requirement and no time limit. Sign up today to unify data from DynamoDB and other sources, shape it with SQL, and publish your queries as scalable REST API Endpoints that you can integrate into your application.
And if you're coming from Rockset, check out our Rockset Migration Guide for more information on making the transition from your existing Rockset environment to Tinybird's purpose-built real-time data platform.
