A voice assistant has to answer in a few hundred milliseconds. An agent doing real work takes minutes. Bolt those together naively and you get the two failure modes everyone has met: the assistant goes silent while it thinks, or you interrupt it and something unrelated dies.
The architecture
Two clocks, deliberately uncoupled
Speech and interruption live in the foreground. Sessions, tools and long-running work live in the background, and results are reinserted when they matter rather than when they finish. Speech, foreground turns, background tasks and disconnects each get their own cancellation scope, so cutting off a sentence cancels a sentence, not the task it was describing. That sounds obvious and is what most implementations get wrong, because hanging everything off one context is easier.
Turn state is a deterministic machine rather than something inferred from timers: naming the states is what makes the behaviour testable.
What counts as heard
This is the detail I would defend hardest, and it is the same instinct as any eval harness. Synthesizing audio is not delivering it. Queuing is not playing. Playback starting is not playback finishing.
So delivery is accounted across distinct stages through to a confirmed receipt, and interrupted or stale playback is never counted as heard. Get this wrong and the assistant believes it told you something it did not, a bug invisible in testing and infuriating in use. It is the voice-domain version of a green test on a feature that does nothing.
Fail closed at the boundary
Dispatch runs over a versioned protocol with retry deduplication, rejection of conflicting task IDs and monotonic acknowledgments. Acceptance is emitted only after the agent returns a real delegation handle; missing context or a response without a genuine handle fails closed rather than optimistically reporting success. Reconnects preserve session identity and ordered task state, because a dropped connection is normal in a voice product, not exceptional.
What is not claimed
Experimental, and not affiliated with Nous Research or LiveKit. Automated gates cover typed input through to remote audio, but microphone quality still needs a constrained human test, so natural-conversation quality is not claimed yet. Being precise about that boundary is the only thing that makes the verified part mean anything.
Ken Faiman · Applied AI, agent & evaluation systems · faiman.com