Overview
Nestor is one engine with three front doors. The engine caches blocks and talks to origins. Everything else is an adapter on one of its two sides: what asks for bytes, and where bytes come from.
Crates
| Crate | Role |
|---|---|
nestor | The engine. Namespaces, blocks, tiers, fetch scheduling, consistency. Depends on the Origin trait only. |
nestor-store | Both directions of object_store. ObjectStoreOrigin makes any ObjectStore an origin, NestorStore makes a cached namespace look like an ObjectStore. |
nestor-s3 | The S3 endpoint as an axum router. SigV4 verification, addressing, GET and HEAD from cache, forwarding and re-signing of everything else. |
nestor-client | Client-side cluster routing. Rendezvous hashing, bounded load, membership, hedging across nodes. Exposes the cluster as an Origin. |
nestor-cli | The nestor binary. TOML configuration, TLS, Prometheus export, and the wiring that turns the crates above into a node or a gateway. |
nestor-e2e | End-to-end scenarios against RustFS in Docker Compose. |
Dependencies point one way. nestor knows nothing about S3 wire formats, HTTP or clusters. nestor-s3 and nestor-client depend on nestor and nestor-store. nestor-cli depends on all of them and holds no logic of its own beyond configuration.
Core model
A namespace is an origin plus caching policy: block size, fetch and read windows, consistency mode, readahead, hedging. A Nestor instance holds many namespaces over one shared cache. In the S3 endpoint every bucket becomes a namespace with the [buckets] policy. In a library, a namespace is whatever the application decides, one per object class is typical.
A block is a fixed-size slice of an object identified by namespace, object name, a content tag and its index. The tag is the reason a changed object cannot be served from old blocks, see Consistency.
A read is a byte range against one object. It is resolved to a block range, served block by block as a stream with a bounded number of blocks in flight, and finished without ever buffering the whole object. Blocks & reads covers the path in detail.
An origin is anything implementing head and get with ranges and preconditions. object_store gives Nestor every major backend. A cluster of other Nestor nodes is also an origin, which is how a local tier stacks on a shared one.
Deployment shapes
The shapes compose. A nestor binary with a [cluster] section is a gateway: its own RAM and disk are a local tier, and misses go to the cluster instead of the origin. A service can embed the nestor crate with a ClusterOrigin and get the same two-tier layout in process. Writes always go to the real origin, the cluster only ever serves reads.