Consensus offloaded from raft to journal

[maxin93]

Hi, I recently came across a surprising article: Amazon EKS can scale up to 100K nodes. In the article, they mentioned that they offloaded etcd's raft to their own journal. Is there more detailed information on this? Or does anyone know if Amazon will open-source the journal?
https://aws.amazon.com/cn/blogs/containers/under-the-hood-amazon-eks-ultra-scale-clusters/

[serathius]

AWS announcement follows GKE's 65k node scale (https://cloud.google.com/blog/products/containers-kubernetes/gke-65k-nodes-and-counting) where etcd was replaced by Spanner. While both articles quote replacing etcd/raft as reason for reaching higher scale, I personally think it played smaller role than it might seem. Both achievements primarily show the results of recent work in Kubernetes control plane performance, which has unlocked a much higher scale (e.g., https://kubernetes.io/blog/2025/09/09/kubernetes-v1-34-snapshottable-api-server-cache/).

The Kubernetes scale target (5k nodes) was set to represent the needs of the large majority of users, leaving it to advanced users like cloud providers to push it further. I've personally seen 30k node clusters running on etcd, so it shouldn't be immediately abandoned if you need higher scale.

The magic piece behind both improvements isn't just replacing etcd; it's that both Amazon's journal and Google Spanner use atomic clocks to resolve conflicts in their consensus algorithms. While Raft needs two network hops to confirm the order of two operations, these systems can just look up the real time from an atomic clock to achieve the same result. This makes things that much easier to scale horizontally compared to purely software-based consensus.

I don't expect such advanced systems, which require dedicated hardware, to be open-sourced anytime soon.

(Full disclosure: I work on GKE)

[maxin93]

Thank you very much for your patient response to my questions. I have one more question: the maximum storage space recommended by the current community is still 8GiB. Will this be increased in the latest version? Or is there already ongoing work related to this matter?

[serathius]

We haven't increased the default etcd database quota because we were still not finished developing a benchmark that allow quantity what does it really mean to support X gigabytes of storage or to prevent regressions in the future. Still there is nothing holding you to increase the quota further. Effort is tracked in #16467, but the progress is slow.

I have seen clusters running with 20GB quota and more in production, however there are some kinks to it, for example defrag locking database for 10s+, which can be disruptful for some workloads.

[maxin93]

Thanks again. I will try using the latest version to test whether it meets our requirements.

[hwdef]

The magic piece behind both improvements isn't just replacing etcd; it's that both Amazon's journal and Google Spanner use atomic clocks to resolve conflicts in their consensus algorithms.

This infromation is awesome. Never thought of it.
Scale determines the upper limit of technology. This technology can only be seen in cloud vendors of such a large scale.