Sep 13, 2024

Too Long; Didn't Read This blog post is inspired by the [2023 State of Databases for Serverless & Edge](https://leerob.io/blog/backend) and focuses on 2024, incorporating my personal experience working with various developer tools startups. As we're into the last quarter of 2024, we have seen an explosion of new dev tools, companies, and programming models transforming how developers approach data storage and retrieval. Many of these databases offer high availability, geographic scaling, and seamless integrations with other cloud services, but they come with challenges. Costs can also scale quickly, especially with large datasets, and control over data can be limited, which might be a concern for developers, founders, and companies. 💡 this blog was originally posted on my new open-source blog: In this post, I've shared my overview of the databases that pair exceptionally well with modern app architectures, focusing on serverless. 💡 This blog post is inspired by the 2023 State of Databases for Serverless & Edge and focuses on 2024, incorporating my personal experience working with various developer tools startups. Criteria Services that pair well with serverless. Solutions compatible with JavaScript and TypeScript. Who should use and who should avoid Serverless? Disclaimer: I recently did consulting work for Neon , and it blew my mind to learn how easy it is to go serverless these days. I have also used many of these tools for my projects and do consulting work for other developer tools. What is Serverless? Before we get started, let's talk about the term "serverless" which should never have become a thing because there's always a server, and it is a marketing term that doesn't refer to any specific computer term. I find the term "serverless" misleading to many people in tech. Essentially, all the code in the world has to run on a machine, so there is nothing really "server-less" about it. Cloud means running software on other people's computers in a data center. Serverless is a subset of the cloud. It's a way of building and deploying cloud apps. Serverless implies a very high level of abstraction when interacting with cloud infrastructure. Traditionally, app developers have consumed cloud infrastructure on the level of individual boxes running operating systems. These don't usually correspond to physical machines, but the OS box is the unit of abstraction presented to people running, building, and deploying cloud apps. Serverless apps are typically based on the "function" (one invocation of some tiny bit of logic) as the unit of consumption for cloud infrastructure, e.g: AWS Lambda. They also typically embrace high-level abstractions for managing data. The term "serverless" is sometimes annoying as heck to many developers in tech and yay maybe we should have called It "servermore". Serverless is a term that refers specifically to servers and says that you don't have them. It's similar to wireless, fanless, and other terms. So maybe the term serverless became a thing because configuring your server is a huge hassle and a headache and not a great idea for many beginners. So, today, the term serverless makes a lot less sense because the configuration level is nearly the same. For a time, such easy-to-configure servers almost felt like there weren't any servers at all. Hence, "serverless" And yeah, the naming can be confusing, so it's mostly "I don't want to think about managing any server," so the point of serverless isn't that there isn't a server anywhere. As a user, you and I do not need to worry about a server, which means there is no provisioning, scaling, upgrading, or updating the server. Serverless just means not hosting your own infrastructure + boxes. Amazon Lambda is a good example where you run arbitrary scripts on their infrastructure on demand without having always-available servers. So, being serverless means you don't have to think about servers. Of course there are servers hard at work handling your application's requests, but that's their problem, not yours. They do all the work behind the scenes of allocating, configuring, and maintaining the servers. Instead of paying for servers, you and I pay for the requests that your application makes to the database and the storage that your data consumes. so "Serverless" means: Never being paged when a rack goes down. Never needing to rollout software on a hundred nodes. Never paying for idle infrastructure. Never losing business because you can't scale. A New Programming Model Relational databases have been the backbone of application development for over 25 years, but serverless computing demands a shift in how we interact with these databases. Connectionless Protocols: Traditional databases use stateful protocols, which can be inefficient for serverless environments where functions scale to zero. New solutions offer connectionless access via HTTP APIs or WebSockets, abstracting connection management away from developers. Web-Native Interfaces: Modern databases are increasingly designed to work seamlessly with browser-based APIs and lighter runtime environments, making them ideal for edge computing. This shift makes databases more compatible with web-centric development, where data is fetched using standard web protocols. Lightweight Client Libraries: The burden of managing complexity is shifted to the database vendor to simplify development. This allows client libraries to be thin and easy to integrate, with the database provider handling tasks like connection pooling and caching. Type Safety: TypeScript's popularity has driven demand for databases and libraries that support type-safe interactions, ensuring better developer experience and reducing runtime errors. Example: Consider solutions like Supabase and Neon . Supabase offers a Postgres-based solution with an HTTP API, making it serverless-friendly. Neon, on the other hand, uses a WebSocket-based approach to manage Postgres connections efficiently in serverless and edge environments. For type-safe development in serverless environments, several tools and databases stand out. Here are some examples: Supabase : An open-source alternative to Firebase, Supabase provides a complete backend as a service, including a Postgres database, authentication, and storage. It's designed with TypeScript support, ensuring type safety across the board, from database interactions to API requests. Neon : Neon is a serverless PostgreSQL database focusing on providing a type-safe experience for front-end developers. It offers features like branching and scaling, making it a powerful tool for projects that require a scalable and resilient database. Turso : Built on SQLite, Turso is an edge-hosted, distributed database that integrates well with TypeScript. It allows developers to define APIs in TypeScript, providing a type-safe interface handy for serverless and edge computing applications. Xata : Xata is a serverless database that integrates relational and search functionalities. It offers type-safe APIs and has built-in AI features, making it a powerful tool for modern applications requiring strong type safety and advanced data handling capabilities. Serverless Database Firebase Firestore Firestore has been a reliable choice for developers building real-time applications, especially in mobile contexts. It offers seamless integration with Firebase Authentication, making it an excellent choice for cross-platform applications. Example: Retrieving Documents from Firestore import { getFirestore, doc, getDoc, DocumentData } from "firebase/firestore";const db = getFirestore();const docRef = doc(db, "users", "userId123");const docSnap = await getDoc(docRef);if (docSnap.exists()) { const data: DocumentData = docSnap.data(); console.log("Document data:", data);} else { console.log("No such document! ");} PlanetScale PlanetScale, built on Vitess, offers MySQL with a serverless twist. One of its standout features is the ability to branch databases, similar to how you might branch code in Git, making it easier to manage schema changes and collaborate on database development. Example: Querying PlanetScale with Kysely import { Kysely, sql } from 'kysely';import { PlanetScaleDialect } from 'kysely-planetscale';interface Database { users: { id: number; name: string; email: string; }}const db = new Kysely<Database>({ dialect: new PlanetScaleDialect({ host: process.env.DATABASE_HOST, username: process.env.DATABASE_USERNAME, password: process.env.DATABASE_PASSWORD, }),});async function getUsers() { const users = await db.selectFrom('users').selectAll().execute(); console.log(users);}getUsers(); PlanetScale's serverless capabilities and powerful features like branching make it a top choice for modern web applications. AWS Aurora Serverless AWS Aurora Serverless offers a robust, fully managed PostgreSQL and MySQL-compatible database with the ability to automatically scale based on application demand. Example: Connecting to Aurora Serverless with Node.js import { Client } from 'pg';const client = new Client({ host: process.env.AURORA_HOST, user: process.env.AURORA_USER, password: process.env.AURORA_PASSWORD, database: process.env.AURORA_DB,});async function queryDatabase() { try { await client.connect(); const res = await client.query('SELECT NOW()'); console.log(res.rows[0]); } catch (err) { console.error(err); } finally { await client.end(); }}queryDatabase(); Aurora Serverless is ideal for applications that need to scale in and out automatically without manual intervention, offering the performance and reliability of traditional RDBMS systems. Rising Serverless Databases In addition to the established players, several rising stars are pushing the boundaries of what's possible with serverless databases. Neon: Serverless PostgreSQL with a Twist Neon is a serverless PostgreSQL offering that uniquely separates storage and compute and enables instant scaling and cost-effective operation, especially in edge environments. Neon also supports database branching, making it a powerful tool for modern development workflows. Example: Using Neon's Serverless Driver import { Client } from '@neondatabase/serverless';const client = new Client(process.env.DATABASE_URL);async function getTime() { await client.connect(); const { rows: [{ now }] } = await client.query('SELECT NOW();'); await client.end(); return now;}getTime().then(console.log); For more details on how Neon's serverless driver works and its impact on the developer experience, check out Neon's blog post . There are plenty of serverless database options already: Firestore, DynamoDB, CosmosDB, FaunaDB, and even MongoDB, and there are "newsql" distributed relational systems like CockroachDB and Planetscale with serverless plans, but very few that separate storage and compute and fewer that are open source end-to-end. Neon is also 100% compatible with Postgres (unlike CockroachDB) because the compute is Postgres. There is a significant architectural difference between Neon and PlanetScale, CockroachDb, and Yugabyte. Neon is shared storage (storage is distributed but shared), and the others are shared nothing. Shared-nothing systems are hard to build with supporting all the features of the base system. Neon is 100% compatible with Postgres b/c they didn't (or almost didn't) change the Postgres engine. Supabase: The Open-Source Firebase Alternative Supabase is an open-source alternative to Firebase, built on PostgreSQL. It offers a full suite of backend services, including database, authentication, and storage. Example: Querying Supabase with TypeScript import { createClient } from '@supabase/supabase-js';const supabaseUrl = 'https://xyzcompany.supabase.co';const supabaseKey = process.env.SUPABASE_KEY as string;const supabase = createClient(supabaseUrl, supabaseKey);interface Country { id: number; name: string; code: string;}async function getCountries() { const { data: countries, error } = await supabase .from<Country>('countries') .select('*'); if (error) { console.error('Error fetching countries:', error); return; } console.log(countries);}getCountries(); Other Solutions In addition to the above, several other solutions deserve mention for their unique approaches to data storage and management in serverless and edge environments: Xata: Combining database, search, and analytics, Xata offers an intuitive, spreadsheet-like UI that appeals to both developers and non-developers alike. EdgeDB: Challenging the traditional ORM/database separation, EdgeDB offers an optimized approach to querying and managing data. Hasura: With its ability to connect multiple data sources and expose them via a GraphQL API, Hasura simplifies the process of building scalable, serverless applications. Who should go serverless, and who should avoid going serverless? Choosing to go serverless depends on your project's specific needs and scale. Let's discuss. Who Should go serverless: Many people might say, "We can optimize it later" for a scale-up of orders of magnitude, but this is nearly always wrong. Reddit is a perfect example of running into scaling issues recently but is locked into architecture with few escape hatches built in; when a bridge fails, it'll just fail. When software runs into a scaling limit, it'll degrade and fall on its arse constantly and be terrible as long as it takes for the software team to completely rework its entire architecture, often having to learn entirely new technologies. I don't know the relevant details about Reddit, but the assumption is that the early Reddit people could have quickly built something more scalable, yet there are tech reasons why the later people, with far more resources, can't. If you are designing a blog, then yeah, you probably don't need to worry about it. If you are starting a social network or SAAS business app, then you probably do. A lot of successful businesses start with things that are not scalable, and it is a strength, not a weakness. For instance, you can't beat Facebook at its own game if you start a social network. You have to do something that Facebook can't do because it is too big. Scalability problems will be tackled as you grow. Among the many things Facebook can't do is running its service on a single database. It makes things much more challenging for them. Thankfully, you are much smaller than Facebook, and you can. Take that advantage. Who Shouldn't go serverless: If you have less than 10k req/s, you shouldn't start thinking about multiple DB servers or migration. No need to be obsessed with "scaling." focus on growing your business. Tradeoffs are tradeoffs. For example - "by running a k8s cluster when you don't need one, a cost you pay is the overhead." but 👇 but make sure your know all the rules of the game otherwise...you'll see the huge bills once your app goes viral. Nowadays, there's a ton of idling compute capacity running on the client side that could be used to spin up a personal cloud environment with all core services if one has the proper knowledge. For anyone that has a core to spare and a few GBs of memory, which should be easy for most modern midrange hardware, I would recommend they save that $5 paying for a third party VPS, go deploy an open source hypervisor such as KVM, and run the virtual server on your hardware. Comparison of free tiers: https://paul.totterman.name/posts/free-clouds/ Final Thoughts There are many other databases; I just named several, and there are hundreds more for every possible niche, but DBaaS seems to be rapidly evolving. It looks like there will be a lot of new developments in the next few years to help reduce the overall dev effort. 🔥 Serverless is my secret weapon for hobby projects. When my lambdas are snoozing and S3's collecting dust, it's like having a VIP pass to free cloud computing. This blog is? Powered by Neon and Vercel, and my wallet's pretty happy too. Only splurged on the domain name 🚀💸 Some people would say just use "$5/month VPS." But on that $5/month VPS there's management overhead - you need to have at least basic Linux knowledge, and ideally more than that to know not to do stupid things like chmod 777 and database exposed on the public internet. You also need to do your updates, etc. I have realized recently that, even when someone has a good product, you have to be very strategic within Devtools and enterprise space. Doing it well is an art. With the rise of serverless, it's super easy for an average developer or a new developer to build a backend. It's also super easy to deploy a front-end with Vercel, Netlify, or GitHub Pages but I believe there's no easy solution for the database part yet. In the future, it should be the norm to include a database in an application with just a single env variable without worrying about anything else. And last, maybe I'm in my own bubble in "database/backend,” but there have been many new startups in this space, which is fun to see 👊 It's also interesting to see more database services support HTTP APIs and as a person living in the EU, an unpopular saying is that the current situation in the EU makes these SaaS offerings almost irrelevant. Companies and governments are very reluctant to pick any cloud services, and It's a NIGHTMARE to sell SaaS to European Govs if it's hosted with a US company (except, of course, you are one of said US companies. Then you can sell whatever you want. Yes, it's bizarre). I'm not against or in favor of cloud services, but I wish more people understood their costs upfront and picked them based on merits. Some teams benefit greatly from cloud services, but usually if they're not cost sensitive. In my last job, we had everything on AWS because we'd never scaled to somewhere where it'd become expensive and convenient. The moment the bill starts to bite, people ought to at least price out alternatives and consider hybrid setups.