Rendered at 09:11:17 GMT+0000 (Coordinated Universal Time) with Netlify.
olucasandrade 16 hours ago [-]
You've heard the advice: "Use the right tool for the right job." Sounds reasonable. Even wise. So you followed it. You picked Redis for caching, Elasticsearch for search, Kafka for messaging, MongoDB for documents, Pinecone for vectors, InfluxDB for time-series, and Postgres for... well, the relational stuff.
Congratulations. You now have 7 databases to maintain, 7 backup strategies to manage, 7 monitoring dashboards to watch, 7 security audits to run, and 7 monsters that can break at 3 AM.
The thing nobody talks about, cause it doesn't sell: that advice "the right tool for the right job" is the battle cry of every vendor's marketing department.
The uncomfortable truth is that PostgreSQL is not "just a relational database." It hasn't been for over a decade. It's a data platform that does what most of these specialized tools do, using the same algorithms, with a single connection string, a single backup strategy, and a single place to debug when everything breaks at 3 AM.
Not "close enough." Not "good enough at small scale." The same algorithms.
All these buzzwords, unless we're talking about truly unbelievable scale, are achievable with just Postgres.
Let me show you. Actually, no... simulate it and see for yourself.
dirtbag__dad 3 hours ago [-]
OLAP conveniently not mentioned. Anyone have any suggestions for this?
We tried to get as much mileage as we could out of Postgres before we switched to clickhouse and it was night and day.
Const-me 8 hours ago [-]
I like the article, but why PostgreSQL specifically? I have recently needed some of these features in a server, MariaDB did the job reasonably well.
howardYouGood 8 hours ago [-]
My two cents from beers and tech talks around the industry; a lot of DBAs were put off even trying MariaDB altogether due to the sale to Sun. They felt abandoned after putting the MySQL team on the map to begin with. It was felt the same could happen with MariaDB, so why bother?
I heard from a lot of teams then who planned MySQL migrations to Postgres before the Sun sale to Oracle due to the MySQL sale to Sun.
allthetime 7 hours ago [-]
Why did you choose Maria over PG?
Const-me 54 minutes ago [-]
I like engine=memory tables. Compared to data structures found in programming languages, memory tables are more powerful: arbitrary columns, indices. The DB server solves concurrency with transactions and row-level locks; need B-tree primary key which is not the default for memory engine but easy enough to do at table creation.
I think they save quite an amount of software complexity, delegating these problems to the DB server.
KronisLV 10 hours ago [-]
I haven't seen people often refer to the stack in the site as a "microservices architecture": databases, key-value stores, message queues, object stores and other specialized solutions are typically viewed in a category of their own, orthogonal to what the application architecture is, unless you'd have for example 3 different PostgreSQL instances for the same project.
From what I've seen, even if you'd have PostgreSQL, Redis, Garage (S3) and RabbitMQ, that might still be considered a monolith if you just have a singular deployment unit for your app (even if scaled to multiple instances). It'd still be a distributed system, sure, much the same way how an app that's connected to a singular DB is (just fewer points of failure), you'd just be separating the concerns across specialized solutions.
Aside from that, you can get pretty far with just PostgreSQL alone, but sometimes it's nice to have the more specialized solutions because the libraries and APIs that they have are optimized for working within a particular domain - for example, object storage easily letting you handle file uploads, downloads, Range requests, presigned URLs and so on, so you don't have to build anything yourself or think that much about what would be the best way to store a bunch of binary data in a RDBMS.
On the flip side, if you can somehow avoid the pain of trying to put everything in PostgreSQL, the idea of having a single source of truth for all of the concerns in your app is pretty darn nice! Might as well take a step further and get rid of the front-end/back-end altogether and make the DB serve web requests (or use an embedded DB within your app process), but that gets dangerously close to the mess that was Oracle Forms or that is Oracle ADF.
banashark 10 hours ago [-]
I'm a big fan of Postgres
I've used it for a while and prefer it (partly due to experience) over other options.
There is a lot that it _can_ do, especially when you take FDW, extensions, PLs, etc into the equation.
However, it's not an "always better" situation. These other specialized services excel in ways that Postgres cannot at the moment. _If_ you have a small system, _and_ the featureset you need overlaps with postgres' abilities, _and_ you don't expect to outgrow those two properties, then it could definitely make sense to use Postgres.
Let's imagine that adding all the pressure to the same system _didn't_ impact other parts of the same system (you wouldn't want a surge in kafka write traffic causing latency on your basic crud routes, right?).
Redis vs Postgres unlogged tables:
* Redis has a bunch of algorithms which are battle-tested and ease implementation of common patterns. Need a bloom filter, expiration with ttl that you don't need to write extra code for?
Kafka/SQS vs postgres queueing:
* There are pros and cons here, but you definitely don't want your other Postgres work being impeded by high spikes in traffic. Distributed logs and dedicated message queues are built to handle elastic scalability in ways that are difficult to achieve with Postgres. What if you have certain, super busy tenants whose queueing traffic comes in giant batches at unexpected times? With SQS and their recent fair queues, you not only don't need to worry about the spikes (as long as your writers can write fast enough), but you also don't need to worry about the distribution of in-flight consumer work being imbalanced due to the spikiness of a single tenant
ElasticSearch:
* Postgres FTS can work for a number of simple scenarios, but there's a number of edge-cases where performance deteriorates as well. What happens when large documents are normal? https://old.reddit.com/r/PostgreSQL/comments/1q5ts8u/postgre... shows some metrics on what happens when you get into TOAST territory. FTS also puts a decent load on the db both in indexing as well as searching.
MongoDB:
* Mongo, for all of the deficiencies that it's had over the years (which I hear are mostly resolved these days), can scale writes in a _muuuuch_ more simple way than Postgres.
The claim is that "unless you're at unbelievable scale" you won't hit it, however I don't think that's true.
I've worked on multiple Postgres databases that tapped out due to the amount of work it would take to scale things up further (full tenant sharding by db, multiple shards for some larger tenants, keeping that all going and working at a larger scale).
Every additional piece of logic you add into Postgres complicates the story of how you fix things once it becomes too much. Once you've got a single write that triggers 10 different table writes, with 80 index updates, and you want to scale those writes up, you might hit scenarios where you need to choose what gets migrated out. Or you get smart with materialized views, but those require full refreshes. So you create your own version of incrementally-maintained-views, which is more writes and work.
That all being said, I do think Postgres can work for more concerns than it currently is used for, even if I think the recent glazing is a bit much.
Postgres is great, and having everything in a single transactional workload is extremely convenient and can remove a lot of race conditions and buggy behavior.
The thing nobody talks about, cause it doesn't sell: that advice "the right tool for the right job" is the battle cry of every vendor's marketing department.
The uncomfortable truth is that PostgreSQL is not "just a relational database." It hasn't been for over a decade. It's a data platform that does what most of these specialized tools do, using the same algorithms, with a single connection string, a single backup strategy, and a single place to debug when everything breaks at 3 AM.
Not "close enough." Not "good enough at small scale." The same algorithms.
All these buzzwords, unless we're talking about truly unbelievable scale, are achievable with just Postgres.Let me show you. Actually, no... simulate it and see for yourself.
We tried to get as much mileage as we could out of Postgres before we switched to clickhouse and it was night and day.
I heard from a lot of teams then who planned MySQL migrations to Postgres before the Sun sale to Oracle due to the MySQL sale to Sun.
I think they save quite an amount of software complexity, delegating these problems to the DB server.
From what I've seen, even if you'd have PostgreSQL, Redis, Garage (S3) and RabbitMQ, that might still be considered a monolith if you just have a singular deployment unit for your app (even if scaled to multiple instances). It'd still be a distributed system, sure, much the same way how an app that's connected to a singular DB is (just fewer points of failure), you'd just be separating the concerns across specialized solutions.
Aside from that, you can get pretty far with just PostgreSQL alone, but sometimes it's nice to have the more specialized solutions because the libraries and APIs that they have are optimized for working within a particular domain - for example, object storage easily letting you handle file uploads, downloads, Range requests, presigned URLs and so on, so you don't have to build anything yourself or think that much about what would be the best way to store a bunch of binary data in a RDBMS.
On the flip side, if you can somehow avoid the pain of trying to put everything in PostgreSQL, the idea of having a single source of truth for all of the concerns in your app is pretty darn nice! Might as well take a step further and get rid of the front-end/back-end altogether and make the DB serve web requests (or use an embedded DB within your app process), but that gets dangerously close to the mess that was Oracle Forms or that is Oracle ADF.
However, it's not an "always better" situation. These other specialized services excel in ways that Postgres cannot at the moment. _If_ you have a small system, _and_ the featureset you need overlaps with postgres' abilities, _and_ you don't expect to outgrow those two properties, then it could definitely make sense to use Postgres.
Let's imagine that adding all the pressure to the same system _didn't_ impact other parts of the same system (you wouldn't want a surge in kafka write traffic causing latency on your basic crud routes, right?).
Redis vs Postgres unlogged tables:
* Redis has a bunch of algorithms which are battle-tested and ease implementation of common patterns. Need a bloom filter, expiration with ttl that you don't need to write extra code for?
Kafka/SQS vs postgres queueing: * There are pros and cons here, but you definitely don't want your other Postgres work being impeded by high spikes in traffic. Distributed logs and dedicated message queues are built to handle elastic scalability in ways that are difficult to achieve with Postgres. What if you have certain, super busy tenants whose queueing traffic comes in giant batches at unexpected times? With SQS and their recent fair queues, you not only don't need to worry about the spikes (as long as your writers can write fast enough), but you also don't need to worry about the distribution of in-flight consumer work being imbalanced due to the spikiness of a single tenant
ElasticSearch: * Postgres FTS can work for a number of simple scenarios, but there's a number of edge-cases where performance deteriorates as well. What happens when large documents are normal? https://old.reddit.com/r/PostgreSQL/comments/1q5ts8u/postgre... shows some metrics on what happens when you get into TOAST territory. FTS also puts a decent load on the db both in indexing as well as searching.
MongoDB: * Mongo, for all of the deficiencies that it's had over the years (which I hear are mostly resolved these days), can scale writes in a _muuuuch_ more simple way than Postgres.
The claim is that "unless you're at unbelievable scale" you won't hit it, however I don't think that's true. I've worked on multiple Postgres databases that tapped out due to the amount of work it would take to scale things up further (full tenant sharding by db, multiple shards for some larger tenants, keeping that all going and working at a larger scale).
Every additional piece of logic you add into Postgres complicates the story of how you fix things once it becomes too much. Once you've got a single write that triggers 10 different table writes, with 80 index updates, and you want to scale those writes up, you might hit scenarios where you need to choose what gets migrated out. Or you get smart with materialized views, but those require full refreshes. So you create your own version of incrementally-maintained-views, which is more writes and work.
That all being said, I do think Postgres can work for more concerns than it currently is used for, even if I think the recent glazing is a bit much.
Postgres is great, and having everything in a single transactional workload is extremely convenient and can remove a lot of race conditions and buggy behavior.
But, what about...