Origin fetches

Every byte Nestor serves was fetched from an origin exactly once, by one fetch group, on behalf of every reader waiting for it. This page covers how those fetches are issued: how a cold read finds out about the object, how many requests can be in flight, and what happens when the origin is slow or fails.

Cold reads

A read needs the object's size and ETag before it can resolve open ranges or tag blocks. Fetching them with a HEAD would put a full round trip in front of every cold read. Instead the first GET doubles as the probe.

The probe requests the blocks the read would fetch first anyway. The Content-Range total gives the size, the ETag header gives the tag, and the body is consumed as a normal fetch group so nothing is thrown away. If the metadata cache had an expired entry, the probe carries If-None-Match and a 304 refreshes the entry for free. Bounded reads in immutable namespaces skip the probe entirely because they need neither the size nor a tag.

The only read that still pays a HEAD is a suffix range on an object of unknown size, since its first block cannot be computed without the total.

Concurrency

Three semaphores bound outstanding origin work for the whole instance.

Permit	Default	Held by
origin_concurrency	64	Foreground fetch groups, one per origin GET
readahead_concurrency	16	Background fetch groups issued by readahead
hedge_concurrency	16	Secondary requests issued by hedging

Foreground and background pools are separate so a burst of readahead cannot starve a cache miss a caller is waiting on. Hedges take from their own pool and when it is exhausted the primary request is awaited, so hedging degrades to nothing rather than amplifying load on an origin that is already struggling.

Fetch policy

One FetchPolicy per namespace decides how a miss goes to the origin. A read overrides it with ReadOptions in the library or X-Nestor-Fetch on the endpoint. A fetch group runs under the policy of the read that opened it.

Setting	Default	Purpose
attempts	3	Tries per fetch, including the first
backoff, backoff_max	50ms, 2s	Delay before a retry, doubling per attempt
first_byte	5s	Time for one attempt to return headers
attempt	30s	Time for one attempt to deliver its body
deadline	60s	The fetch as a whole, across attempts and backoff
hedge	{ factor = 3.0, min = "50ms", max = "2s" }	Secondary request timing, false disables

The object_store client under nestor-store runs with its own retries and request timeout off (Transport), so this is the only policy in effect and nestor_origin_retries_total counts every retry.

Timeouts

Timeout	Catches	Scope
first_byte	Hung connection, origin that accepted and stalled	Headers of one attempt, hedge included
attempt	Body that trickles while holding a concurrency permit	One attempt, headers and body
deadline	Worst case a reader waits	All attempts and backoff, a retry runs only if its backoff still fits

A timeout is OriginError::Timeout and retryable. Blocks are inserted only once complete, so an attempt cut off mid-body leaves nothing partial and the retry resumes at the first unfinished block.

Hedging

An attempt has two stages and each is hedged against its own distribution. Each namespace keeps two histograms over the last one to two minutes, time to first byte and time per block, the latter measured from headers or from the previous block to the next complete block on whichever stream delivered it. They are log2 histograms with eight sub-buckets per octave, updated with atomic increments and rotated in three generations, so a quantile read is a walk over 192 counters and never below the true value by construction, at most 12.5% above it.

The hedge delay is either factor times the mean, 3.0 by default so a 20 ms origin hedges at 60 ms, or the quantile of the window, { quantile = 0.99 } hedges at the recent p99. Exactly one of the two is set. Either way the delay is clamped to [min, max], and with no sample in the window it is max. The delay in force per stage is exported as nestor_hedge_delay_seconds.

Headers unanswered after the delay get a second identical request and the first answer wins. A block that has not completed after the delay gets a second request for the range still owed, from the first unfinished block to the end of the fetch, and the first stream to complete the next block carries on while the other is dropped. Nothing already delivered is fetched twice. In both stages a retryable failure on one side hands over to the other, so a hedge already in flight is used rather than backed off and retried. A backup whose ETag no longer matches is dropped, the primary is still the version the read started on. The cluster client hedges across nodes with the same rule, see Cluster.

Retries

Error	Retried
I/O, short read, timeout	Yes, with backoff
404, precondition failed	No, they are answers

A GET that fails while the object's size is unknown may have asked past the end, and origins report that like any other error. One HEAD settles it: a range starting at or beyond the size resolves to empty blocks, anything else goes to the retry policy with the original error.

Out of attempts fails the waiting readers with the last error, out of deadline fails them with a timeout. A short read is a body that ended before the requested range did.

Readahead

A read that starts where the previous read of the same object ended is sequential. When a namespace has readahead set, such a read schedules the next readahead blocks past its own range as background fetch groups, bounded by the object size. Blocks already cached or in flight are skipped, so readahead costs nothing on a warm object.

The default is 8 blocks. Consumers with their own prefetching, such as a stream store that already reads whole segments in order, should set it to 0 and keep the cache purely demand driven.

Origin trait

All of this sits on one small interface:

#[async_trait]
pub trait Origin: Send + Sync + 'static {
    async fn get(&self, object: &str, options: GetOptions) -> Result<GetResponse, OriginError>;
    async fn head(&self, object: &str) -> Result<ObjectMeta, OriginError>;
}

GetOptions carries an optional byte range plus If-Match and If-None-Match. GetResponse carries the object metadata, the range actually returned and a body stream. OriginError distinguishes not found, precondition failed, not modified, short read, timeout and I/O so the fetcher can decide what to retry. nestor-store implements it over object_store, nestor-client over a cluster of nodes, and MemoryOrigin in the nestor crate is an in-process origin with injectable latency and failures for tests.