TENT Deadline Scheduling

TENT Deadline Scheduling#

Overview#

This document describes how TENT schedules transfers that carry a deadline: the admission queue that orders, promotes and drops requests before they reach a transport, and the RDMA bandwidth arbitration that orders slices on a NIC before they are posted. Both are opt-in and both predict deadline feasibility with one shared formula; they differ in what they can see ahead of a request, and in what they do about it.

It also sets the two next to device selection, which runs on the same NICs and keeps its own bandwidth estimate, because the three are easy to confuse and are deliberately not the same thing.

The document is ordered from the shared parts to the specific ones: the path a deadline takes, the prediction both layers make and the bandwidth series that feeds it, then the admission queue and the arbitration each with a worked example, then configuration, metrics and known limits.

Background#

A KV-cache transfer in disaggregated serving is only useful if it lands before the consumer needs it. TENT’s QoS priorities (see Quality of Service) separate business classes, but within one class every flow is served alike: a NIC splits its bandwidth equally among contenders, and the engine dispatches in arrival order. Neither knows that one flow has ten milliseconds left and another has a second.

RFC #2519 introduced an absolute deadline on Request and built on it in three steps, each opt-in and each leaving the previous behavior byte-identical when off:

Observe: on completion, record how much of its window a transfer actually took, as measured latency over window. This ratio is the MLU (tent_deadline_mlu_permille).
Order: dispatch the request with the earliest deadline first.
Degrade: predict which requests cannot make their window and cancel them up front, so the caller can fall back (recompute locally, for example) instead of waiting for a late transfer.

RFC #2792 added the same prediction at the NIC: among slices that are about to be posted on one queue pair, post the most urgent first.

The Path of a Deadline#

Request{deadline_ns, intent_type, priority}
        |
        v
  submitTransfer
        |
        v
  Admission queue (enable_runtime_queue)          <- runtime_queue/*
     tryAdmit      : capacity limits, EDF insert
     pickForDispatch: promotion, drop, dispatch window
        |
        v
  Transport (RDMA)
     device selection : which NIC                 <- transports/rdma/* (slice-spraying.md)
     arbitration      : post order on that NIC    <- deadline_bw_arbitration
        |
        v
  Completion -> tent_deadline_mlu_permille

A request with deadline_ns == 0 takes the same path and is never reordered, promoted or dropped: every predicate below treats “no deadline” as “nothing to predict”.

Predicting Feasibility: MLU#

The scheduling layers use the same ratio as the metric, with a predicted transfer time in place of the measured latency: the metric reports what a transfer took, the predictor estimates what it will take before it runs. Both layers use one definition, DeadlineMlu() in tent/runtime/deadline_mlu.h:

predicted_mlu = ((bytes_ahead + length) / bandwidth) / (deadline_ns - now_ns)

length is the request’s own bytes.
bytes_ahead is what must move before them. The deadline is absolute, so this wait counts against the window as an additive delay over the wire rate. It is not folded into a slower bandwidth, which would multiply the wait by the request’s size.
bandwidth is the transmit estimate, one of the two per-NIC series described in Bandwidth Series below, never the selection EWMA that device selection reads.

MLU below 1 means the request is predicted to make its window; 1 is the boundary; above 1 it is predicted to miss. Two edge rules: no deadline or no usable bandwidth (<= 0) yields 0, so such a request is never urgent and never dropped; a deadline already in the past is infinitely urgent, which in the queue means it is dropped whenever the drop is on, and in the arbitration means it posts first.

The two layers deliberately read different bytes_ahead, each counting the bytes it can actually see at its decision point:

Layer	Decides	`bytes_ahead`
Admission queue	before the request reaches a worker	every drop-eligible owner already dispatched and not yet completed, including slices still waiting in worker queues (`dispatching_bytes_`)
RDMA arbitration	at posting time, on one NIC	bytes already posted to that NIC’s hardware (`getPostedBytes`), plus the slots the arbitration has already placed ahead in this pass

The admission side reads the wider set on purpose: its decision (a drop) is irreversible, so it is the conservative one. The arbitration only reorders, and the worker-queue slices it would otherwise count are its own contenders.

How Each Layer Evaluates It#

The formula is shared; every input to it is chosen per layer.

	Admission drop	RDMA arbitration
Evaluated	once per queued owner, in `pickForDispatch`, before the request reaches any worker	once per slice per slot, in `orderByDeadline`, just before `submitSlices`
`length`	the whole request	one slice
`bandwidth`	sum of the transmit estimates of all available RNICs (`getEstimatedBandwidth`)	the transmit estimate of the NIC these slices will post on
`bytes_ahead` on entry	`dispatching_bytes_`: every eligible owner dispatched and not yet completed, on any NIC, including slices still in worker queues	`getPostedBytes(dev)`: bytes on this NIC’s hardware, not yet completed; nothing from worker queues
`bytes_ahead` during the pass	grows by each owner dispatched in this call; a dropped owner adds nothing	grows by each slice placed in an earlier slot; every slice is placed eventually
`now`	read once per `pickForDispatch`	read once per `orderByDeadline`
Who is scored	eligible owners only (RDMA route, not staged); others are dispatched without a prediction	every slice in the group; a slice without a deadline scores 0
Threshold	`mlu >= mlu_local_threshold` → cancel	none; scores are only compared with each other
Effect of a high value	the request never runs	the slice posts earlier
Past deadline	infinite → cancelled	infinite → first slot
No usable bandwidth	drop disabled for this pass, everything dispatched	order left as it arrived

Two consequences follow. The admission prediction treats all in-flight bytes as if they were served serially ahead of the request, although they are spread over several NICs and move in parallel, so it is systematically pessimistic; that is why θ_local sits above 1.0. The arbitration prediction is per NIC and per slice, so it is close to what that queue pair will actually do, but it only ever sees the slices that reached this worker in this tick: a more urgent slice arriving one tick later takes its turn in the next batch.

Bandwidth Series#

Each RDMA device keeps two bandwidth estimates, learned from the same completion events but along two different loops. Both use the update new = α × old + (1 − α) × sample and both are clamped to ewma_min_bandwidth_multiplier .. ewma_max_bandwidth_multiplier of the link speed; everything else differs.

One Slice, Two Meters#

 worker posts slice            NIC completes it          poller handles it
 ---------+------------------------+-------------------------+----------------
          | submit_ts recorded     |                         | poll_ts
          | posted_bytes += len    |                         | posted_bytes -= len
          | (busy stretch opens    |                         | completed_bytes += len
          |  if it was 0)          |                         |
          |                        |                         |
          |<------- post -> completion latency ------------->|  selection sample:
          |                                                  |  len / latency
          |                                                  |
          |  busy time: only while posted_bytes > 0          |  transmit sample, at most
          |  (this and every other slice on the NIC)         |  every 10 ms, last completion
          |                                                  |  of a poll pass:
          |                                                  |  Δcompleted_bytes / Δbusy_time

Selection EWMA (device selection)#

Sample source: one slice, on its own successful completion. The worker takes poll_ts − submit_ts, the time from this post to this completion, and the sample is slice length / that time.
What it contains: the slice’s wire time plus its wait behind every work request the NIC already held when it was posted. A backed-up NIC therefore produces low samples.
When: every successful completion, unconditionally. Failed, flushed, timed-out, cancelled or re-routed slices contribute nothing.
Update: α = bandwidth_learning_rate = 0.01, so the estimate is almost entirely the latest sample; it reacts within one or two completions.
Reader: device selection, as predicted_time = (inflight + slice) / ewma when scoring NICs. Queueing being inside the sample is what the selector wants: it is choosing among NICs, and a NIC that is slow because it is busy should lose the comparison.

Transmit Estimate (admission drop and arbitration)#

Sample source: the NIC as a whole, over a time interval. Two counters run per device: completed_bytes, added to by every completion, and busy_ns, which accumulates only while the NIC has at least one posted, uncompleted work request (a stretch opens when posted_bytes goes from 0 to non-zero and closes when it returns to 0).
What it contains: bytes moved per unit of time the NIC was actually working. Idle gaps between bursts are not charged, and a slice’s own wait behind earlier work is not visible, because the ratio does not look at individual slices at all.
When: at most once per transmit_meter_interval_ns (10 ms), taken at the last completion of a poll pass (every completion in a pass carries the same timestamp, so a sample mid-pass would split one burst across two intervals). One sampler per interval is chosen by CAS when several lanes poll at once. The first sample only sets the baseline. An interval that spans more than transmit_meter_max_interval_ns (50 ms) of wall clock is discarded: it describes a link too far in the past. A slice that ends without moving its bytes (failed, flushed, timed out) invalidates the current stretch and the meter starts fresh.
Update: α = transmit_bandwidth_learning_rate = 0.9, so a single sample moves the estimate by a tenth of the difference; about ten intervals, roughly 100 ms under load, to follow a real change. With no usable interval the estimate keeps its last value, or the link-speed seed from openDevice(): the optimistic direction, which cannot cause a false drop.
Readers: the admission drop, as the sum over available devices (getEstimatedBandwidth()), and the arbitration, as this NIC’s value. Both add the queueing term themselves through bytes_ahead, so the rate must not contain it, or the wait would be counted twice.

Side by Side#

The two series, input by input:

	Selection EWMA	Transmit estimate
Unit of measurement	one slice	one NIC over one interval
Numerator	slice length	bytes completed in the interval
Denominator	post → completion of that slice	NIC busy time in the interval
Includes queueing behind earlier work	yes, on purpose	no, on purpose
Sampled	every successful completion	at most every 10 ms, last completion of a pass
α	0.01 (follows the latest sample)	0.9 (~100 ms to converge)
Fed by	`release()`	`maybeSampleTransmit()`
Read by	`DeviceSelector::allocate()`	`getEstimatedBandwidth()` (sum), `orderByDeadline()` (per NIC)
Question answered	which NIC is the better choice right now	how fast does this NIC move bytes once posted

Why per-completion timing cannot serve the predictors: up to max_qp_wr work requests are posted in one call with effectively one timestamp, and a poll pass timestamps every completion it collects alike, so a slice’s own post-to-completion time grows with the depth of the batch it travelled in. Deep enough, the selection sample would sit on its lower clamp on a perfectly healthy link. Bytes over busy time does not depend on how the work was batched.

Admission Queue#

The queue lives in LocalTransferAdmissionQueue (tent/runtime/admission_queue.*) and is owned by TransferEngineImpl. It is off unless enable_runtime_queue is true; with it off, submitTransfer hands requests straight to the transport, as before.

What Enters the Queue#

A submit is queued as a whole or not at all. A submit that contains any staged owner (one that must be copied through a staging buffer, see the proxy path) bypasses the queue entirely; staging-internal submits always queue. Each owner records whether it is degradation eligible: routed to RDMA and not staged. Only eligible owners can be dropped and only their bytes count toward bytes_ahead, because the bandwidth provider is the RDMA transport’s and says nothing about a TCP, NVLink or staging transfer.

tryAdmit enforces the capacity limits: max_outstanding_owners and max_outstanding_bytes bound everything admitted and not yet terminal, with staging_owner_reserve and staging_byte_reserve held back so staging-internal work can always make progress. A request longer than max_dispatch_bytes is rejected at submit, since it could never fit a dispatch window.

Ordering#

With deadline_aware off the queue is FIFO. With it on, owners carrying a deadline are inserted in earliest-deadline-first order at admission time, and owners without a deadline keep FIFO order behind them. Dispatch then consumes from the front, so the hot path stays O(picked) rather than re-sorting on every call.

Promotion#

promotion_slack_ns, when positive, moves any queued owner whose remaining slack (deadline_ns - now) has fallen below that value to the front of the queue, ahead of owners with more slack or none. A stable partition keeps EDF order inside each group. This is how a request that was comfortable when it was admitted but has since become urgent overtakes the ones admitted before it. It requires deadline_aware.

Deadline-Infeasible Drop#

mlu_local_threshold (θ_local), when positive and combined with deadline_aware and an installed bandwidth provider, turns on the drop. pickForDispatch walks the queue from the front; for each eligible owner it computes the predicted MLU against the current dispatching_bytes_ and, if the value reaches θ_local, marks the owner CANCELED instead of dispatching it, releases its capacity, and raises the on_local_decode_suggested hook. The scan continues, since later owners have looser deadlines. A dropped owner does not consume dispatch budget.

The caller sees the drop as a CANCELED task status. The hook is the intended signal for “recompute locally”; today the engine installs an empty hook set, so nothing beyond the status is delivered (see Known Limits).

Bandwidth Provider#

The provider installed by TransferEngineImpl is RdmaTransport::getEstimatedBandwidth(): the sum over the local RNICs of each device’s transmit estimate. It is installed only when the RDMA transport is present; without it the drop is inactive even if θ_local is set, and a warning is logged at startup.

Choosing θ_local#

MLU 1.0 is the boundary between met and missed, so a threshold below 1.0 drops requests the predictor itself expects to succeed. Measured on hardware with θ = 0.5, dispatch fell to a fraction of the offered load while every request that was dropped would have made its window. Dropping does not depress the bandwidth estimate in return: the transmit meter charges bytes to the NIC’s busy time, so a NIC carrying half the load reads the same rate over half the time. The loss is throughput, in proportion to how much feasible work the threshold refuses.

A value slightly above 1.0 is the useful setting, because the predictor is conservative by construction. bytes_ahead is every eligible byte dispatched and not yet completed, and the formula charges all of it ahead of the new request as if it were served serially; in practice those bytes are spread over several NICs and queue pairs and move in parallel, and the new request’s own slices are spread the same way. So a predicted MLU of 1.1 usually still lands on time. θ in the range 1.2 to 1.5 absorbs that bias and drops only requests the wait alone rules out. The code does not currently validate the lower bound; treat 1.0 as the floor.

Dispatch Window and Progress#

max_dispatch_owners and max_dispatch_bytes bound how much the queue hands to transports at once. The window refills on every submit and on every poll, and the ProgressWorker refills it again whenever a transport reports task completion. RDMA completions do not yet wake the worker, so with the queue active the worker also refills on a timer, progress_fallback_interval_us (50 ms by default). That timer is the upper bound on how long a dispatch slot can sit free after an RDMA transfer finishes.

Worked Example: Three Requests#

Two RNICs whose transmit estimates sum to 800 Gb/s (100 GB/s), deadline_aware on, θ_local = 1.2, max_dispatch_bytes raised to 1 GiB so the sizes below fit one window, and an empty queue. Three submits arrive in this order:

Request	Length	Deadline window at submit	Eligible
R1	64 MiB	20 ms	yes (RDMA)
R2	512 MiB	5 ms	yes (RDMA)
R3	16 MiB	3 ms	no (routed to TCP)

Admission. Each tryAdmit inserts by deadline, so the queue reads R3 (3 ms), R2 (5 ms), R1 (20 ms) regardless of arrival order. Suppose the first pickForDispatch runs after all three are queued.

Dispatch pass 1. bytes_ahead starts at 0 (dispatching_bytes_).

R3 is first. It is not eligible, so the drop check is skipped; it is dispatched to TCP. Its bytes do not enter dispatching_bytes_.
R2: predicted time 512 MiB / 100 GB/s ≈ 5.4 ms against a 5 ms window, MLU ≈ 1.07. Below 1.2, so it is dispatched; dispatching_bytes_ becomes 512 MiB.
R1: predicted time (512 + 64) MiB / 100 GB/s ≈ 6.0 ms against 20 ms, MLU ≈ 0.30. Dispatched; dispatching_bytes_ is now 576 MiB.

A fourth request. R4, 256 MiB with a 6 ms window, arrives while R2 and R1 are still in flight. Its predicted time is (576 + 256) MiB / 100 GB/s ≈ 8.7 ms, MLU ≈ 1.45. That reaches 1.2: R4 is marked CANCELED at pick time and never reaches a transport. The caller sees CANCELED on its next poll and can recompute locally instead of waiting almost 9 ms for a transfer that would land close to 3 ms late.

Had θ_local been 0.9, R2 would have been dropped in pass 1 with MLU 1.07: a request the predictor expected to miss by only 0.4 ms, and one that on a slightly better link would have made it. That is the sense in which values below 1.0 are too aggressive and values just above 1.0 are the useful range.

Completion. When R2 completes, complete() subtracts its 512 MiB from dispatching_bytes_, and the transmit meter has meanwhile learned from the bytes R2 moved: if the NICs sustained 380 Gb/s rather than 400, the provider’s sum drifts down over the next ~100 ms and later predictions become slightly more conservative.

RDMA Bandwidth Arbitration#

Arbitration is in Workers::orderByDeadline with the pure ordering policy in tent/transport/rdma/bw_arbitration.h. It is off unless transports/rdma/deadline_bw_arbitration is true; off, the post order is exactly the arrival order.

It runs within one priority tier: QoS priorities still decide which tier posts first, arbitration decides the order inside the tier. When a worker has gathered the slices it is about to post on one NIC, it:

Reads that NIC’s transmit estimate. Zero or below means nothing to predict from and the order is left alone.
Checks whether any slice carries a deadline; if none does, every MLU would be 0 and the pass is skipped.
Takes bytes_ahead as the NIC’s posted bytes: work that has reached the hardware and has not completed. None of the candidates is in it, and neither is work still sitting in a worker queue.
Builds the order one slot at a time: the slice with the highest predicted MLU takes the next slot and its bytes join bytes_ahead for the ones still waiting, since the queue pair will post them in that order. Ties keep arrival order. The first 64 slots are resolved exactly; the remainder is ranked once against the bytes those slots accumulated. submitSlices posts a prefix of the order, as long as the queue pair’s remaining budget (max_qp_wr, 256 by default, minus what is already outstanding), and the rest wait for the next tick.

Arbitration never drops, admits or moves a slice to another NIC; it only decides the order in which already-selected slices are posted.

Worked Example: Three Slices#

A NIC has 32 MiB posted and not yet completed, and its transmit estimate is 400 Gb/s (50 GB/s). One worker is about to post three 16 MiB slices on it:

Slice	Deadline window remaining
A	10 ms
B	2 ms
C	none

First slot, bytes_ahead = 32 MiB. A and B each predict (32 + 16) MiB / 50 GB/s ≈ 1.0 ms of transfer: A’s MLU is 1.0 / 10 = 0.10, B’s is 1.0 / 2 = 0.50, C’s is 0. B takes the slot and its 16 MiB join bytes_ahead, now 48 MiB.

Second slot: A now predicts (48 + 16) MiB / 50 GB/s ≈ 1.34 ms, MLU 0.134; C is still 0. A takes it. C posts last.

Post order: B, A, C. In arrival order (A, B, C) B would have waited behind A’s 16 MiB, about 0.34 ms of its 2 ms window. The example also shows the two rules a reader should expect: a slice with no deadline never overtakes one with a deadline, and a slice’s urgency rises as the slots ahead of it fill.

Comparison with Device Selection#

The earlier tables compared the two predictors’ inputs and the two bandwidth series. This one compares the three mechanisms themselves: what each is for, where it runs, and what it can and cannot do.

	Device selection	Admission queue	RDMA arbitration
Question	which NIC	whether and when to dispatch	in what order to post on a NIC
Runs	per slice, in the worker	per submit and per poll, in the engine	per post batch, in the worker
Reads	selection EWMA, NUMA tier, inflight charge	transmit estimate (sum), `dispatching_bytes_`	transmit estimate (this NIC), posted bytes
Can	choose, split across NICs	reorder, promote, cancel	reorder within a tier
Cannot	see deadlines	choose a NIC	drop or move a slice
Switch	`enable_smart_scheduling`	`enable_runtime_queue` + `deadline_aware` (+ `mlu_local_threshold`)	`deadline_bw_arbitration`

Device selection is documented in Slice Spraying.

Configuration#

Enabling the Queue#

{
  "enable_runtime_queue": true
}

Parameter	Type	Default	Description
`enable_runtime_queue`	bool	`false`	Route non-staged submits through the admission queue. Everything under `runtime_queue/` is read only when this is on

Admission Queue#

{
  "runtime_queue": {
    "max_outstanding_owners": 1024,
    "max_outstanding_bytes": 1073741824,
    "staging_owner_reserve": 0,
    "staging_byte_reserve": 0,
    "max_dispatch_owners": 64,
    "max_dispatch_bytes": 67108864,
    "deadline_aware": true,
    "promotion_slack_ns": 0,
    "mlu_local_threshold": 1.2,
    "progress_fallback_interval_us": 50000
  }
}

Parameter	Type	Default	Description
`max_outstanding_owners`	int	`1024`	Owners admitted and not yet terminal, queued or dispatched
`max_outstanding_bytes`	int	`1073741824` (1 GiB)	Same bound in bytes
`staging_owner_reserve`	int	`0`	Owners held back from user submits so staging-internal work can always be admitted
`staging_byte_reserve`	int	`0`	Same reserve in bytes
`max_dispatch_owners`	int	`64`	Owners in flight at the transports at once
`max_dispatch_bytes`	int	`67108864` (64 MiB)	Bytes in flight at the transports at once; a single request longer than this is rejected at submit
`deadline_aware`	bool	`false`	Earliest-deadline-first dispatch; owners without a deadline keep FIFO order behind those with one. Required by promotion and drop
`promotion_slack_ns`	int	`0`	Owners with less remaining slack than this move to the front of the queue. `0` disables promotion
`mlu_local_threshold`	float	`0.0`	θ_local. Eligible owners whose predicted MLU reaches this value are canceled instead of dispatched. `0` disables the drop. Values below `1.0` cancel requests predicted to succeed
`progress_fallback_interval_us`	int	`50000`	Timer on which the progress worker refills the dispatch window when no completion wake arrives. `0` disables the timer

RDMA Arbitration and the Transmit Estimate#

{
  "transports": {
    "rdma": {
      "deadline_bw_arbitration": true,
      "transmit_bandwidth_learning_rate": 0.9,
      "transmit_meter_interval_ns": 10000000,
      "transmit_meter_max_interval_ns": 50000000
    }
  }
}

Parameter	Type	Default	Description
`deadline_bw_arbitration`	bool	`false`	Order slices most-urgent-first before posting on a NIC
`transmit_bandwidth_learning_rate`	float	`0.9`	α for the transmit estimate (`1.0` = never learn, `0.0` = always take the latest sample)
`transmit_meter_interval_ns`	int	`10000000` (10 ms)	Minimum wall-clock span of one meter interval
`transmit_meter_max_interval_ns`	int	`50000000` (50 ms)	An interval longer than this is discarded instead of learned from

The transmit estimate also inherits the clamp and seed shared with device selection: ewma_min_bandwidth_multiplier, ewma_max_bandwidth_multiplier, default_bandwidth_gbps, min_bandwidth_gbps, max_bandwidth_gbps. See the Slice Spraying configuration.

Request Fields#

Field	Type	Default	Description
`deadline_ns`	uint64	`0`	Absolute steady-clock time in nanoseconds by which the transfer should complete. `0` means no deadline
`intent_type`	enum	`INTENT_UNSPEC`	What the transfer is for (foreground get, background prefetch, migration). Used by the transport selector’s intent policies; not read by the scheduling described here
`priority`	int	`PRIO_HIGH`	QoS tier. Arbitration reorders only within a tier

C++:

Request req;
req.opcode = Request::WRITE;
req.source = local_buffer;
req.target_id = remote_segment;
req.target_offset = 0;
req.length = 1 << 20;
req.deadline_ns = now_ns() + 20'000'000;  // 20 ms from now, steady clock
engine.submitTransfer(batch, {req});

Python:

req = tent.Request(tent.OpCode.WRITE, src, target_id, 0, length,
                   deadline_ns=now_ns + 20_000_000)

now_ns must come from the same steady clock the engine uses (std::chrono::steady_clock on the C++ side). A wall-clock timestamp is not comparable and will be read as a deadline far in the past or future.

Metrics#

Metric	Type	Description
`tent_deadline_mlu_permille`	Histogram	Measured MLU at completion, times 1000: submit-to-completion latency over the window the transfer had at submit. `1000` is the met/missed boundary. Recorded only for transfers that reached `COMPLETED` with a deadline still ahead at submit; this is the measured ratio, not the predictor’s estimate
`tent_deadline_infeasible_total`	Counter	Transfers whose deadline was already in the past at submit time. Kept separate from the histogram so they are not read as high-MLU samples

A dropped owner surfaces as a CANCELED task; it produces neither metric, because it never ran. See Metrics for labels and export.

Where Each Effect Is Observable#

Ordering and promotion are engine-level and apply on any transport. The drop and the arbitration only ever act on RDMA: the drop requires an RDMA-routed, non-staged owner and an installed RDMA bandwidth provider, and the arbitration runs in the RDMA workers. On a host with TCP only, deadline_aware and promotion_slack_ns change dispatch order, while mlu_local_threshold and deadline_bw_arbitration do nothing. That is by design, not a misconfiguration.

Known Limits#

Drop notification. The engine installs an empty DegradationHooks, and there is no public API to register one, so a caller learns of a drop only by observing CANCELED. The hook exists for an upper layer to trigger local recompute.
Threshold validation. mlu_local_threshold below 1.0 is accepted and cancels requests the predictor expects to succeed. Treat 1.0 as the floor.
Mixed submits. A submit that mixes staged and non-staged owners bypasses the queue as a whole, so its RDMA owners do not enter dispatching_bytes_ and the drop predictor underestimates the queue ahead of later requests.
Quarantined owners. An owner that is dispatched and then never reaches a terminal status (a batch abandoned by the lazy free path) keeps its bytes in dispatching_bytes_, so the drop predictor grows more pessimistic over time in that failure mode.
RDMA completion wake. RDMA does not yet signal the progress worker on completion, so dispatch-window refill after an RDMA completion waits for the fallback timer.
Failover keeps the RDMA charge. Eligibility is fixed at admission from the planned route. A request that enters as RDMA, fails there and is recovered over TCP keeps its bytes in dispatching_bytes_ until it completes, so the drop predictor is pessimistic by one transfer’s length for the duration of that recovery.
One provider, one transport. The only bandwidth provider is the RDMA transport’s, and bytes_ahead is a single sum, so only RDMA-routed, non-staged requests can be predicted and dropped. Extending the drop to another transport would need a provider per transport and a queue-ahead term per transport, with each owner recording which one predicts it. The structure allows this; nothing implements it.

References#

RFC #2519: deadline-aware transfers (observe, order, degrade)
RFC #2792: deadline-aware NIC bandwidth arbitration
TENT Slice Spraying: device selection and the two bandwidth series
TENT Quality of Service: priority tiers the arbitration works within
TENT Metrics

TENT Deadline Scheduling

Contents

TENT Deadline Scheduling#

Overview#

Background#

The Path of a Deadline#

Predicting Feasibility: MLU#

How Each Layer Evaluates It#

Bandwidth Series#

One Slice, Two Meters#

Selection EWMA (device selection)#

Transmit Estimate (admission drop and arbitration)#

Side by Side#

Admission Queue#

What Enters the Queue#

Ordering#

Promotion#

Deadline-Infeasible Drop#

Bandwidth Provider#

Choosing θ_local#

Dispatch Window and Progress#

Worked Example: Three Requests#

RDMA Bandwidth Arbitration#

Worked Example: Three Slices#

Comparison with Device Selection#

Configuration#

Enabling the Queue#

Admission Queue#

RDMA Arbitration and the Transmit Estimate#

Request Fields#

Metrics#

Where Each Effect Is Observable#

Known Limits#

References#