What I emphasized in this article (author here) is scaling postgres backups. This works because there's already rock-solid systems built into postgres + surrounding tooling to build from.
The broader takeaway is the principle of, "how do I take something that doesn't scale on its own and make it so?" This applies to backups, compute, storage layers, proxies. It's why Neki and Vitess are so powerful for everything from small 1GB databases to petabytes.
Hello! Very nice article. I have a couple of questions if you don't mind.
When doing the last streaming of the wal from the primary the article mentions that the nodes will catch up to replication time `T`.
How do the nodes coordinate this time `T`? Is it simply just choosing a time in the future (after the backup has started) and waiting till they all catch up or is there more realtime coordination happening?
Also, in another part it's mentioned that "Time T is saved to ensure we know the precise time, down to the second, included in this backup." My question is that if 1 second is granular enough? I'm assuming that this is a simplification for the sake of explanation and Time T is a timestamp with at least millisecond granularity. I regularly play with otel data that can have nano-second granularity so I'm assuming is millisecond or more
What I emphasized in this article (author here) is scaling postgres backups. This works because there's already rock-solid systems built into postgres + surrounding tooling to build from.
The broader takeaway is the principle of, "how do I take something that doesn't scale on its own and make it so?" This applies to backups, compute, storage layers, proxies. It's why Neki and Vitess are so powerful for everything from small 1GB databases to petabytes.
Hanging around to answer questions, too :)
Hello! Very nice article. I have a couple of questions if you don't mind.
When doing the last streaming of the wal from the primary the article mentions that the nodes will catch up to replication time `T`.
How do the nodes coordinate this time `T`? Is it simply just choosing a time in the future (after the backup has started) and waiting till they all catch up or is there more realtime coordination happening?
Also, in another part it's mentioned that "Time T is saved to ensure we know the precise time, down to the second, included in this backup." My question is that if 1 second is granular enough? I'm assuming that this is a simplification for the sake of explanation and Time T is a timestamp with at least millisecond granularity. I regularly play with otel data that can have nano-second granularity so I'm assuming is millisecond or more
Interesting, last I checked PlanetScale still doesn't have in place Postgres version updates.