Consistency
A cache in front of mutable objects has two failure modes: serving bytes from an old version, and mixing blocks from two versions in one response. Nestor rules out the second by construction and bounds the first with a per-namespace policy.
Content tags
A block key is (namespace, object, tag, index). The tag is a 64 bit hash of the object's ETag, or of its size when the origin returns none. Two versions of an object have different tags, so their blocks are different keys that never collide in the cache. There is no invalidation race because old blocks are never overwritten, they stop being referenced and age out.
A read pins its tag when it starts and sends the matching If-Match with every origin GET. If the object changes mid-read the origin answers 412 instead of new bytes, the fetch fails with a stale error and the reader restarts once from the current metadata. A reader therefore observes one version or fails, never a splice.
Modes
Each namespace declares how its objects behave.
| Mode | Tag | Revalidation |
|---|---|---|
etag, ttl | Hash of ETag | Metadata older than ttl is revalidated with a conditional GET on the next read |
immutable | Constant | Never. An object name is assumed to map to one content forever |
ETag mode is the default, with a 60 s TTL. A fresh metadata entry is trusted as is. An expired entry is not discarded, its ETag goes out as If-None-Match on the read's first GET. A 304 costs one round trip with no body and confirms every cached block is still current. A 200 with a new ETag brings a new tag and the read proceeds against it while the old blocks fall out of the cache on their own. The TTL is the staleness bound: a reader may see the previous version for at most that long after an overwrite, unless the write went through Nestor.
Immutable mode skips the metadata round trip entirely for bounded reads. It is the right setting for content-addressed or append-only layouts, segment files with a sequence number in the name, build artifacts keyed by digest, or any store where overwriting a key is a bug. The saving is real: a bounded read on a cold object is one GET with no probe, and on a warm object is zero origin requests forever.
Writes through Nestor
When a write passes through the S3 endpoint or a NestorStore, the staleness bound disappears for that object. PUT invalidates the old metadata and, when the body is small enough, inserts the new blocks under the new tag before the response reaches the client. DELETE and CompleteMultipartUpload invalidate. The next read sees the new version with no TTL wait.
Writes that bypass Nestor, another client uploading straight to the origin, are covered by the TTL in ETag mode and not at all in immutable mode. A deployment that mixes the two paths should keep the TTL short or route all writers through the cache.
Metadata cache
Size and ETag per object live in a separate LRU, 100000 entries by default and shared across namespaces. Entries record when they were fetched. Each read consults it once, and every origin response refreshes it, so a hot object's metadata never expires in practice: the reads themselves keep it current.
Metadata is in RAM only. After a restart the block tiers may still hold data from disk but every object's first read pays one probe to learn its tag again, after which the on-disk blocks are hits.