Completion notification#

Classic Transfer Engine can publish one terminal record per opted-in batch on an engine-owned eventfd (RFC #5164 v1). The facility is a wake for asynchronous batch completion. It does not replace getBatchTransferStatus, and it does not change batches that never opt in.

The callable contract is on TransferEngine in transfer_engine.h. This page records the design choices behind that contract.

Scope#

TENT does not implement the facility: completionFd() is -1, and drainCompletions returns CompletionNotificationUnsupported. On classic TE the fd is created once per engine. A non-negative fd means the engine can notify. It does not mean every batch can finish without polling.

A batch opts in with enableBatchCompletionNotification after allocateBatchID and before its only submitTransfer. The opt-in is rejected when the fd is unavailable, when slice_timeout is enabled, or when the batch already has tasks. A second submit on an opted-in batch is rejected. Transports that complete only by foreground polling are not eligible; those batches stay on today’s submit and status-query path.

Poll-driven transports, a disabled facility, and an application deadline all fall back to getBatchTransferStatus. An application deadline ends the wait. It does not by itself settle a batch that still has a running task.

One fd for every opted-in batch#

Each engine has one counter-mode eventfd. Every opted-in batch on that engine shares it. A readable fd means at least one terminal record is queued. It does not name the batch.

drainCompletions is the only way to learn which batches reached a terminal snapshot. It removes the records it returns and does not free batch handles. One call at a time owns that consumer. The queue tail and the eventfd counter are not shared by concurrent drains. A dispatcher therefore matches batch_id itself. Reading the fd and ignoring the records drops completions that belonged to other batches.

The counter is level-triggered. The engine writes it when the queue goes from empty to non-empty. Further records that arrive before the next drain stay in the queue and do not each produce another write. A waiter that is already awake observes them on the drain it is about to run, or on the next one.

close on the original fd does not reliably wake a thread blocked on a dup. Engine destruction writes the counter before closing it. That write is the shutdown wake.

One publication per batch#

Publication is a latch, separate from finished_task_count and from the status-query walk.

After the submit path has accepted the tasks, the latch is armed to one more than the number of submitted tasks. Each task that reaches finishIfComplete releases one token. The submit call releases the extra token only when it returns success. The call that drops the counter to zero publishes a single snapshot. A failed submit never releases the extra token, so the latch does not reach zero and no record is queued for a batch the caller did not accept.

The snapshot is COMPLETED or FAILED, plus a byte count. A failed task contributes no bytes, matching a settled getBatchTransferStatus walk. The failure flag is stored before that snapshot is taken, so a failed event batch is not published as COMPLETED. Batches that did not opt in still record the flag after the completion count, and their status sequence is unchanged.

Tasks in the same batch are not signaled one by one. The record appears when the last submitted task finishes. Notification latency for an earlier task therefore waits on the slowest task in that batch. The RDMA or TCP completion of that earlier task is not delayed.

TIMEOUT and CANCELED are not published in v1. A path that finishes a task without going through finishIfComplete would also skip the latch. Foreground timeout handling stays on the status query.

Notify stays on the status query#

The eventfd reports local completion only. Peer notify is still sent from getBatchTransferStatus through sendNotifyByID. The completion callback does not send it and must not call getBatchTransferStatus.

submitTransferWithNotify on an event-enabled batch returns InvalidArgument. Storing the notify after submit would race a drain and freeBatchID that reuse the batch id. The caller sends the notify after it has observed the terminal record.

Drain and free are different lifetimes#

A terminal record names a batch. It does not own the batch. drainCompletions copies the record out. freeBatchID invalidates the handle and releases the descriptor once no completion callback is still inside it.

If freeBatchID runs before the record is published, the publication is suppressed and nothing is queued. If it runs after publication but before drain, the queued record is marked skipped and a later drain does not return that id. freeBatchID may return before the transport has finished touching buffers. A deferred free means in-flight work may still own those buffers. The caller keeps them until freeBatchID returns success.

Buffers also stay allocated until the batch itself is fully settled. Draining some other batch, or observing one failed task while a sibling task is still running, is not that point.