Aurora DSQL: reinventing the traditional SQL database for the cloud

Originally published on https://builder.aws.com/content/2zgTHreh1tBYirkDs6SuGUktx45/aurora-dsql-reinventing-the-traditional-sql-database-for-the-cloud

The cloud was never someone else’s computer - it was always meant to be reinvention of traditional on-premise hardware and software. If you are using the cloud just as someone else’s computer to do the exact same things you do on your own servers, then you are completely missing the point . Using cloud means you don’t need to do many of the things you usually did on-premise, like installing and managing Operating Systems, tuning JVMs , upgrading and patching software, installing and managing database clusters - all the stuff that is not your core business. And for the purposes of this post, we talking about why using the cloud means not having to install and manage databases, clusters, backups and restore, and all the things required to getting databases to scale, which are all pretty complex . All driven by one of the strengths of AWS : to build services that allowed our customers to focus on their unique application logic, not on the undifferentiated heavy lifting of provisioning servers, autoscaling, patching software, and logging.

Scaling in the cloud

Of the many benefits of cloud computing , the one that customers often find fascinating about AWS, is the ability to scale. Because in an on-premise data center, you can only scale based on the amount of hardware that you currently have. While with AWS, Amazon Prime Day 2024 was Amazon's biggest shopping event ever, powered by massive scaling on AWS, including 250K Graviton chips, processing 146 million DynamoDB requests per second, handling over 1.3 trillion Lambda invocations, Aurora processed more than 376 billion transactions, and serving a quadrillion ElastiCache requests in a single day.
Lets looks at the multiple options you have for scaling the different layers of your architecture:

• For your application layer, you can vertically scale (scaling up) an instance by resizing it . And you can horizontally scale (scaling out) using AWS Auto Scaling , usually with Elastic Load Balancing . Scaling your compute with Serverless is even more effortless, using Lambda or Fargate .
• When you need to scale your file systems, EFS automatically scales its storage capacity and throughput
• Scaling of your caching layer is achieved with ElastiCache Serverless , instantly scaling vertically and horizontally to meet demand.
• And scaling of the middleware layer, using SQS and  EventBridge , is native and automatic.
• With traditional SQL databases, scaling becomes tricky, and is usually the bottleneck you will eventually hit, after all other layers of your architecture has been scaled. You can vertically scale up the database, until you hit the largest instance size. But you cant just vertically auto-scale by adding more writer instances (with RDS you can add read replicas ). At least until now....onwards!

Introducing Amazon Aurora

Basically, you really benefit from cloud when you are using fully managed and serverless services, which takes away all the undifferentiated heavy lifting , and thats why AWS launched Amazon RDS in 2009 . RDS allows you to simply just get a working database, without having to worry about all the stuff - installing, patching, backups  - that goes with it. It offered the common DB engines that customers wanted: MySQL, PostgreSQL, SQL Server, Oracle, MariaDB, and even Db2. So while RDS is a fully managed service, you still needed to think about the underlying server. Even though RDS made it easy to scale with Read replicas and Multi-AZ deployments , you still needed to think about the underlying instances of the primary writer when thinking about scaling. Specifically, for these above DB engines, that existed for many years before the cloud, the compute and storage were tightly coupled to each other. So in order to improve scaling and performance, AWS released Amazon Aurora in 2014 (Aurora wasn’t the only big launch at re:Invent 2014. Another big one was AWS Lambda ), a custom built MySQL and PostgreSQL compatible database born and built in the cloud., with a completely different storage architecture. Aurora uses a distributed, fault-tolerant, self-healing storage system that auto-scales. This is fundamentally different from the traditional database architecture where compute and storage are tightly coupled. This improved performance, allowing Aurora to deliver up to five times the throughput of standard MySQL and up to three times the throughput of standard PostgreSQL. It improved High availability, as Aurora replicates data across multiple Availability Zones and can provide up to 15 low-latency read replicas. The storage layer maintains six copies of your data across three Availability Zones.

Evolution of Aurora

AWS improved Aurora and made it serverless. Aurora Serverless v1 (announced in 2017 and GA in 2018) is on-demand, auto-scaling, serverless Aurora. You don’t have to think about instances or scaling and you pay only for what you use. AWS continued improving and released Amazon Aurora Global Database in 2018. Aurora Global Database is designed for globally distributed applications, allowing a single Amazon Aurora database to span multiple AWS Regions. It replicates your data with no impact on database performance, enables fast local reads with low latency in each Region, and provides disaster recovery from Region-wide outages. Aurora Serverless v2 (announced in 2020 and GA in 2022) improved the serverless offering including more granular scaling, faster response to workloads, and higher concurrency. Customers asked for still better ways to scale horizontally, so in 2024 AWS released Aurora Limitless Database , a managed sharding option built on top of Aurora Serverless. It provides automated horizontal scaling to process millions of write transactions per second and manages petabytes of data while maintaining the simplicity of operating inside a single database. With Aurora PostgreSQL Limitless Database, you can focus on building high-scale applications without having to build and maintain complex solutions for scaling your data across multiple DB instances to support your workloads.

This is how I summarise the evolution of Aurora as discussed thus far:

Evolution of Aurora

• While RDS started offered a fully managed service with multiple DB engines, they were limited in their ability to scale because compute was tightly coupled to storage
• Aurora disaggregated DB storage from compute, improving performance, while still offering MySQL and PosgreSQL compatibility
• Aurora improved scaling, performance and DR with Read replicas and Multi-AZ deployments,
• All the above still meant the writer instance was limited by its ability to scale vertically, to Aurora Servlerless v1 and v2 allowed for vertical auto-scaling of the writer instance
• Aurora Global Database improved DR by allowing a single Amazon Aurora database to span multiple AWS Regions
• Aurora Limitless Database managed sharding further improved scaling by allowing for horizontal scaling of the writer

Serverless is the game changer

It’s important at this time to talk about serverless computing. In 2014, AWS heralded the era of serverless with AWS Lambda , that allowed you to run code without thinking about servers or clusters. Talking specifically about scaling, Lambda scales crazily fast , and if its connecting to RDS, can scale too fast , potentially bringing down the database. So in 2019, AWS released RDS Proxy , which helps applications and Lambda functions to pool and share connections to RDS, to protect the database.
However, as Marc Brooker notes:

I was hearing two consistent things from serverless and container customers:
Existing relational database offerings weren’t a great fit for the fast-moving scalable world of serverless and containers. These customers loved relational databases and SQL, for all the reasons folks have loved relational for forty years, but felt a lot of friction between the needs of serverless compute and existing relational products. Amazon RDS Proxy helped with some of this friction, but it wasn’t going away.
Large, highly-regulated, AWS customers with global businesses were building applications across multiple AWS regions, but running into a tricky architectural trade-off. They could pick multi-region active-active (with DynamoDB Global Tables, for example), but lose out on SQL, ACID, and strong cross-region consistency. Or they could choose active-standby (with Aurora Global Database, for example), but lose the peace of mind of having their application actively running in multiple places, and the ability to serve strongly consistent data to customers from their closest region. These customers wanted both things.So, in 2021 I started spending a lot more time with the databases teams at AWS, including the incredibly talented teams behind Aurora and QLDB. We built a team to go do something audacious: build a new distributed database system, with SQL and ACID, global active-active, scalability both up and down (with independent scaling of compute, reads, writes, and storage), PostgreSQL compatibility, and a serverless operational model.

Introducing Aurora DSQL

Amazon Aurora DSQL (announced in 2024, GA in 2025) is the fastest serverless, distributed SQL database with active-active high availability and multi-Region strong consistency. Marc Bowes describes it better :

I’ve been working on DSQL for a number years. Before working on DSQL, I used to work for Amazon EC2, where we managed a very large MySQL installation. The performance and availability of these databases has a direct impact on the performance and availability of customers’ ability to launch and discover their instances.
As it turns out, running (many) 1000s of databases is quite the challenge. Over the years, we’ve hardened our understanding of how to operate at scale, how to use the database effectively, and how to build applications that interact with the database responsibly.
Ultimately, this journey led us to develop new database technology. It led us to develop a database that doesn’t have a single point of failure, that doesn’t rely on vertical scaling, or application-level sharding.
What is DSQL? Imagine an AWS service like DynamoDB or S3. A service where you don’t have to pick an instance type (do I want large or 4xlarge?). A service where there is no infrastructure to manage, or patch. A service that is pay-as-you-go, that scales to meet your needs. Then, add in the capabilities of a relational database. The ability to run complex transactions, do joins, and evolve your schema over time.

DSQL is the best of both worlds: performance and scaling of DynamoDB, but with traditional SQL and ACID of Aurora PostgreSQL. Its truly serverless, as you open more connections, it scales like crazy. Read this for more on how DSQL works , comparing DSQL to Aurora Serverless v2 , and this for pricing.

To get started, use this pattern  to deploy a API Gateway and python Lambda function connecting to DSQL. And just see how simple it is:

In CloudFormation with AWS SAM, its a single line to declare a DSQL single-region cluster:

DSQL:
    Type: AWS::DSQL::Cluster

Compare that to Aurora Serverless v2 , where it takes 28 lines in CloudFormation to declare a cluster.Its still early days, but DSQL is going to be massive. DSQL should (one day) join the category of the core services that power AWS customers and AWS’ own systems: S3, DynamoDB, Kinesis, EBS, EC2, and Lambda. The fundamentals behind DSQL’s dynamic multi-tenancy and scale-to-zero look just like those other foundational services. The Firecracker-driven compute, the horizontally scalable journaling, and the storage layer from the rest of the Aurora family combine into a strikingly simple architecture on top of AWS’ global infrastructure.