Cluster

A cluster is a set of nestor endpoints that together hold each block once. Nodes do not talk to each other and share no state. Routing is entirely client side: every client computes the same owner for a block from the node list and the block's identity, so the cluster has no coordinator, no membership protocol and no rebalancing traffic.

Rendezvous hashing

For every block the client scores each node with a hash of the node address, the object and the block index, and the highest score owns the block. All clients use the same seeded hash, so they agree without exchanging anything. Adding a node moves to it exactly the blocks it now wins, about 1/n of the total, and nothing else. Removing a node reassigns only that node's blocks, each to whatever was second in its ranking. There is no ring to rebalance and no virtual node table to distribute.

Because ownership is per block rather than per object, one large object is spread across the whole cluster. A 10 GiB scan is served by every node in parallel, and no single node has to hold a hot object entirely.

Routing unit

The cluster has its own block_size, 1 MiB by default, which must be a multiple of the block size the nodes cache with. A read is split into cluster blocks, the first is requested to learn the object's size and ETag, and the rest are requested in order with read_window blocks in flight and their bodies chained into one stream. Every request is a ranged S3 GET to a node's endpoint, so from a node's point of view the client is just another S3 client and a routed block lands exactly on whole node blocks.

Subsequent blocks carry the ETag from the first as If-Match, so a read across nodes is one version or a 412, the same guarantee as a single node, see Consistency.

Bounded load

Each node has a load_limit of in-flight requests from a given client, 256 by default. A block whose owner is at the limit is sent to the next node in its ranking instead, which either has the block from an earlier spill or fetches it from the origin. Hot spots cost some duplication rather than queueing behind one node. When every node is saturated the owner is used anyway.

Failures and tails

A node whose connection fails is marked down for down_for, 5 s by default, and skipped in every ranking until then. The request that observed the failure is retried once on the next ranked node. Application-level errors such as 404 or 412 are answers and are not failed over.

Hedging works across nodes. Each node keeps a latency estimate, and a request that has not answered after factor times that estimate is duplicated to the second ranked node, the first answer wins. The defaults are the same as the engine's, see Origin fetches.

Membership

The node list is either static or a DNS name.

[cluster]
nodes = ["10.0.1.5:9000", "10.0.1.6:9000", "10.0.1.7:9000"]

[cluster]
dns = "nestor.cache.svc:9000"
refresh = "10s"

A DNS name is re-resolved every refresh and the node set updated in place, so scale-out and node loss are picked up without a restart. A resolution that returns no addresses is ignored and the current set kept. A headless Kubernetes service or an ECS service discovery name is the intended source.

Nodes authenticate clients like any endpoint. cluster.credentials is what the nodes' [auth] expects and cluster.tls selects https.

Two tiers

A nestor binary with a [cluster] section is a gateway. Its own RAM and disk are a first tier in front of the cluster, and misses go to node owners rather than to the origin. Writes still go to [origin] directly, the cluster is read only. With warm_on_write = true a PUT that passes through the gateway is followed by a fetch of each block on its owning node, so the object is hot cluster wide before anyone reads it.

The same layout is available in process. A service embeds the nestor crate with a ClusterOrigin from nestor-client as the namespace's origin and gets a local tier over the shared one without running a gateway, see nestor-client.