Building full-duplex intelligence

We are building a device-centered architecture for voice. The live conversation stays on the device while longer reasoning and tool execution proceed in the background. We train compact control models alongside our language and speech models, and coordinate them through the Inference Framework, the on-device runtime that keeps track of every task in flight and every sample that has actually been played. New speech can revise a task, interrupt its explanation, or start independent work while an earlier request continues. We call this system-level full duplex. The aim is simple to state and hard to build. A person should be able to talk the way they talk to a friend, and everything behind the voice should keep up.

Consider what such a system has to handle. A person asks an assistant to compare two suppliers for delivery before Friday. While the comparison runs, they ask what a term in the quotation means. Then they move the deadline to Wednesday. A result arrives while they are still speaking. Later, they interrupt the answer to correct a quantity. The system should answer the separate question without losing the comparison, apply the new deadline to the right task, and return a conclusion that reflects the correction. Every one of those utterances changes work already underway, and the system has to know which work.

The Personal Language Model, or PLM, interprets requests and draws on relevant memory. KRORAINA 2 generates contextual speech. Five control models, totaling 16.76M additional parameters beyond the shared codec encoder, predict turn transitions, classify interruptions, and decide when a quiet “mm-hm” belongs. Smart Router selects remote capability when the work requires it. The control models gate playback, and they also feed PLM. Every backchannel, laugh, cough, and silence they classify reaches the language model as a signal, so PLM perceives how a person is responding as well as what they said. All five are trained on decisions that no transcript shows, at the instants they have to be made. We design their inference schedules, streaming interfaces, and training supervision together so this concurrency fits within a phone’s compute and memory budget.

Our first article, The Second History of Intelligence, introduced the model families and research direction. This article examines how those components work together. We first follow the conversation itself, from hearing the person through speaking and being interrupted, and then the work underneath it, from a request into execution, through changing requirements, and back into the answer.

A conversation with several things in progress

Full duplex, in the usual sense, means incoming and outgoing audio stay active together. An assistant that also carries out work faces a second kind of concurrency. The conversation has to stay available while a task waits on a tool, a database, or a remote model. The two kinds meet whenever new speech arrives while that work is in flight.

We organize the system into acoustic, language, and task processes. Acoustic processes run streaming ASR and the control models on incoming audio. Language processes run PLM over the transcript, the events the control models label, relevant memory, and active task state, and stream committed response clauses to KRORAINA 2. Task processes retain execution state across conversational turns. A deterministic Voice Controller grants or revokes playback independently of language generation and task execution, and the Inference Framework schedules all three kinds of process on the device.

This separation lets the live conversation carry complete answers, clarifications, and new requests while background work continues. PLM can explain a term or handle another question during a pending comparison. Only conclusions that depend on the missing evidence have to wait. Two things travel from the conversation into the work in Figure 1. The words reach the task they change. Everything around the words, a pause, a “yeah,” an interruption, reaches PLM as well, so it knows how the person is responding and can respond to it.

The system tracks four independent states. Floor ownership records who is leading the exchange. Generation records what the system is preparing. Playback records what can become audible. Task execution records the objective and progress of the underlying work. The person can hold the floor while PLM integrates one result and another task continues. A single “listening” or “speaking” state cannot represent that moment.

Why we separate these components

GPT-Live handles the live exchange with a native model that listens and speaks, and delegates reasoning asynchronously.1 Open pipelines take the modular route. They chain streaming ASR, a text LLM, and TTS, add a learned end-of-turn model and sometimes an acoustic interruption classifier, and several run locally.

We take the modular route further. We train all five control models, Acoustic Guard, KR-VAP, KR-EOT, KR-Interrupt, and KR-BC, for different decisions in the same exchange. Their outputs feed a Voice Controller that knows which audio is still revocable, while PLM and the Inference Framework track what a correction changes. That gives us specific interfaces to train and test, from an acoustic onset through a task revision to the next audible clause. Keeping the parts separate also follows from how our models are used.

SADAIR shapes one system for many products. Developers and device makers build on these models, and enterprise customers customize them for their own domain, usually one part at a time. One customer fine-tunes the language model on its own data. Another keeps that model and changes the voice. A third adjusts the interruption policy, since different products want different rules about when the assistant should yield. In a native full-duplex model, language, voice, and timing share one set of weights, so a change to any of them is a change to the model that carries all three. Keeping the parts separate lets a customer change one and leave the rest as they were, and lets a task move to a different remote model on the same terms. Every such change still runs through integrated regression tests, because a new delivery style or inference profile can shift the timing of the whole exchange.

The separation also serves the device budget. On a phone, listening, speaking, and reasoning share memory bandwidth, thermal limits, and one accelerator. The five control models range from a 0.21M-parameter interruption classifier to an 8.19M-parameter end-of-turn model. They share the codec encoder’s features instead of duplicating it, and they run on the CPU or DSP so that time-critical control never waits on an occupied, non-preemptible accelerator. Language and speech generation are then scheduled around the live audio deadlines. Our 1-bit (ternary) PLM work, described in the first article, aims to run 20B-class language models on the same hardware. It changes the language model alone and leaves the acoustic path untouched.

The components also run at different cadences. Acoustic Guard advances every 32 ms, shared codec features and KR-VAP every 80 ms, and KR-EOT runs on candidate turn endings. PLM prefill and decoding have a different workload profile, and remote inference completes asynchronously. Those differences determine when to run a model, reuse its state, or discard a superseded result.

Modularity also creates problems that coordination must solve. Passing text between components can lose prosody. Separate components can act on different versions of the conversation, and their caches compete for device memory. A continuous acoustic path and context-aware speech address the first problem, explicit task revisions the second, and deadline-aware scheduling the third. The rest of this article works through them in that order.

Hearing the person while the device speaks

During an overlapping exchange, the microphone receives the person’s voice together with sound from the device’s speaker. The system first needs a reliable account of which sound it produced itself. Without one, the assistant hears its own voice as an interruption, or lets it mask the person’s.

Our acoustic path uses WebRTC AEC3 for acoustic echo cancellation (AEC).2 Its render reference is tapped after mixing, gain changes, ducking, and resampling, immediately before device submission. A pre-duck reference would describe a different signal from the one entering the room. A platform-owned AEC backend requires separate qualification; without a qualified backend and render path, the Inference Framework falls back to half duplex.

Capture and render timestamps align the two streams, with drift tracking between their clocks. An output-route change invalidates the AEC alignment and affected streaming state. Headset and speaker routes therefore have their own buffering and recovery behavior.

Acoustic Guard detects speech onset, residual echo, and double-talk in the post-AEC signal. Streaming ASR publishes stable transcript prefixes and a revisable suffix for PLM. In parallel, KR-VAP forecasts speaker activity, KR-EOT estimates turn completion, and KR-Interrupt classifies overlap while assistant audio is being presented. Their fast timing decisions do not wait for ASR stabilization. The Inference Framework supplies playback and acoustic-health state alongside those predictions, so a detected onset can be interpreted in relation to the sound the device is actually producing. The next question is when that evidence justifies speaking.

Preparing an answer without taking the floor

People often pause before finishing a thought. “Compare the two suppliers, but only if…” leaves the request unfinished even if the room goes quiet. A complete instruction can yield the floor with very little silence. A fixed silence threshold cannot reliably distinguish an unfinished thought from a completed request.

KR-VAP is a 3.74M-parameter causal model built on Voice Activity Projection research.3 It consumes two synchronized streams of 512-dimensional features from the shared, frozen Mimi encoder, one for the person and one for the assistant’s presented audio. The streams share encoder weights but keep separate streaming state. Across a 20-second context, KR-VAP predicts a joint distribution over both speakers’ activity in four future intervals spanning two seconds. The joint prediction can represent a brief listener response while the person continues, as well as a complete turn transition.

KR-EOT is an 8.19M-parameter, audio-native end-of-turn model that puts our endpointing heads on a compact pretrained speech encoder. It combines an eight-second audio window with Acoustic Guard history to estimate whether the person is yielding the floor or continuing. It retains an uncertain state when the evidence is ambiguous. Candidate-triggered inference concentrates computation around possible turn endings. KR-VAP predicts how speaker activity will evolve; KR-EOT estimates whether the current contribution is complete, independently of final ASR punctuation.

The Voice Controller combines these signals with the current interaction state and checks that they are still relevant. A prediction made before the person’s latest continuation should not authorize speech after it. When the evidence conflicts, the controller leaves the floor with the person. Its operating point balances the risk of cutting off an unfinished thought against the delay after a completed one.

In use, the difference is felt as two absences. There is no rush to fill a pause. The person can say “Compare the two suppliers, but only if…” and think, and nothing happens. And there is no dead air after a finished request. The reply is under way before a silence timer would have expired.

Speculative response generation can overlap with listening. PLM works from the available context, and KRORAINA 2 begins streaming synthesis from stable, committed clauses. A playback grant from the Voice Controller is still required before any samples become audible. Speculative compute and the turn-taking decision stay separate, so useful work starts before endpointing completes.

A continuation has different consequences depending on its meaning. “And include shipping” changes the comparison and can invalidate an affected draft. Speech about an unrelated subject can leave the draft valid while postponing delivery. Canceling every prepared answer whenever speech begins would waste useful work and undermine the concurrency the system is designed to support.

Responding as a listener

Three exchanges show what the listener side produces. A person reads out a long list of constraints. They hear a quiet “mm-hm” at the end of a phrase, never in the middle of a word, and keep going, and when they finish, the reply already has every constraint in it. The same person says “make it two hundred, no, wait,” and stops. The assistant neither confirms two hundred nor asks what they meant. It waits, and when “two fifty” arrives, that is the version it works with.

Later they mention that a supplier’s last shipment slipped a week, and the “oh” they hear is shaped by that sentence; when a name is lost under a passing truck, they hear “sorry, which one?” and get the floor straight back. Most of the time the assistant says nothing at all. These contributions, called backchannels, fit into the person’s turn without taking it over, and their timing and delivery have to match the exchange.

Context determines how the same sound should be delivered. KR-BC, our 4.23M-parameter listener-response policy, runs a causal Transformer over eight seconds of typed interaction features. It combines VAP and EOT posteriors, prosody, partial-transcript cues, playback state, and recent response history. Affective cues influence the response without establishing the person’s internal emotional state.

KR-BC predicts whether to contribute, then selects the response class, its intensity, and whether it is lexical or non-lexical. Its training distinguishes continuers, neutral acknowledgments, affective reactions, and repair bids. Affective supervision includes interest, surprise, and concern. Agreement is not among the accepted responses. “Exactly” endorses a claim, whereas a receptive sound only acknowledges it, so agreement examples train the policy as high-commitment negatives. A repair bid may ask the person to pause and is evaluated separately from a backchannel that leaves them the floor.

A listener response has to land on a phrase boundary, and it has to sound like it belongs to the sentence it answers. The two demands pull against each other. The boundary leaves a window of a few hundred milliseconds, and rendering a sound for that sentence takes time on an accelerator the main reply may also be using. Most systems give up the second demand and play a stored sound. We keep both, using the lead time KR-VAP provides.

KR-VAP forecasts a listener-response opportunity up to two seconds before it arrives, from the shape of the person’s speech. KR-BC proposes on that forecast, with class, intensity, and form chosen for what the person is saying. KRORAINA 2 starts rendering the candidate at once, conditioned on the exchange so far, including the person’s own prosody in the phrase still under way. The candidate carries a validity window. When the boundary arrives as forecast, the Voice Controller rechecks eligibility and plays a sound made for that sentence. When the person continues instead, the candidate is revoked and no one hears it.

This is the discipline the main reply already follows, speculative generation before the grant and playback only on the grant, applied to a sound that lasts a fraction of a second. The voice profile’s contextual bank of prerendered short responses remains as the fallback, used when the forecast arrives too late to render or the accelerator is committed elsewhere, so a response that belongs at a boundary is never lost to a busy device. Response selection and acoustic realization therefore have separate deadlines, and the bank protects the first without giving up the second.

The initial production profile enables silence, continuers, and conservative neutral acknowledgments by default. Affective reactions and repair bids, the “oh” and the “sorry, which one?” above, are trained and evaluated with the rest, and each is enabled once it clears a separate eligibility path. The learning problem is to choose a contribution that helps the exchange continue, including when to remain silent.

Making room for an interruption

The meaning of an interruption takes time to become clear. When speech begins, the system may detect that the person is talking before it knows whether they are correcting a detail, acknowledging the answer, or addressing someone else.

Our interruption path begins with reversible ducking when an onset passes Acoustic Guard’s checks. KR-Interrupt then classifies the event using acoustic posteriors and renderer state. Its eight classes distinguish floor-taking barge-in, cooperative overlap, user backchannels, laughter, breath, or a cough, third-party speech, assistant echo, noise, and uncertainty. The classifier itself is a 0.21M-parameter recurrent network that fuses what Acoustic Guard, KR-VAP, KR-EOT, and the renderer already know, and it updates at causal cutpoints 80, 150, 250, and 400 ms after onset. The Voice Controller uses that evidence to suspend, cancel, or resume playback, with explicit-stop handling taking priority.

A suspended response keeps its audio, decoder state, and alignment for a short window, so a false alarm resumes with a crossfade from the sample where it stopped instead of restarting the sentence.

This design gives the person room to enter while preserving the option to continue an answer. The acoustic decision can stop speech before the semantic layer has resolved which task the correction affects. Once the words become clear, the Inference Framework applies the corresponding update.

If the person takes the floor before the assistant becomes audible, pending playback can be revoked directly. The system records that the answer was prepared but never presented. Recovery then starts from a different point than it would after an audible interruption.

Heard from the person’s side, the stages run together, and what differs is what happens next. They say “wait, Wednesday” over the answer. Their first syllable is enough to soften the voice. By the end of the word, KR-Interrupt has decided this is a takeover, the queued audio never plays, and the reply opens with the change, “Wednesday, then,” before resuming at the clause that was cut.

They say “hang on” and turn to someone in the room. The voice stops while the comparison keeps running. When they come back with “okay, go on,” the reply marks the return, “right, back to the two suppliers,” and picks up at the clause that was cut, without the part they already heard. They ask “what’s lead time?” in the middle of the answer. They get a one-sentence explanation, then “so, the cheaper one,” and the comparison was never touched.

Other insertions never take the floor at all. They say “yeah,” and the voice dips and comes back within the same sentence. They cough, or someone else speaks, and they hear the same dip and the same return. If they start talking before the assistant has become audible, nothing plays at all, and there is no half-word to apologize for. None of this asks the person to learn how to talk to the system. They interrupt, correct, ask, and acknowledge the way they would with another person, and the system’s job is to keep both the conversation and the work intact. Every one of those classifications also goes to PLM. That is what lets the reply after “wait, Wednesday” open with the change, and what lets the assistant notice, a minute later, that the person has agreed with everything it said.

The exchanges above, and the listener responses in the previous section, can be heard in our first article and in the demo.

From a codec frame to the loudspeaker

KRORAINA 2 uses a Temporal Transformer that predicts the first codebook of each frame and carries the sequence forward, and a Depth Transformer that fills in the remaining codebooks within the frame. Our streaming waveform decoders reconstruct PCM from those codes. The first article describes the generator, the waveform decoders, their training, and the mobile-inference choices.

The generator advances in 80-millisecond codec frames. The renderer divides each decoded frame into eight 10-millisecond playback blocks. It can therefore change gain or stop submitting samples at a finer boundary than the generation frame.

The 10-millisecond block sets the granularity of playback control. The complete response to an interruption also includes detecting speech onset, interpreting the event, and accounting for audio already committed to the output device. Each stage needs its own measurement.

We distinguish five stages of audio delivery. Audio can be generated, eligible for playback, queued by the application, committed to the device, or estimated to have been presented. Only part of a generated response may reach each stage. Keeping these ranges separate distinguishes work the system can still discard from sound already on its way to the person.

Before every 10-millisecond block submission, the audio callback checks the current playback grant, generation epoch, task revision, and expiry. The Voice Controller is the sole writer of playback authority. A renderer plan prepared before cancellation cannot override a newer revocation observed at the submission boundary. Obsolete application-held samples are discarded, while the device-committed tail remains subject to the output route’s actual buffering. The wired profile caps that tail at 20 ms. A Bluetooth route can hold far more, and when its buffer exceeds the committed-horizon limit the system refuses the duplex grant and falls back to half duplex.

Continuing from what was actually presented

Suppose the assistant prepares an answer explaining a supplier’s lower price, followed by a qualification about delivery. The person interrupts around “cheaper,” and the qualification may never become audible. Treating the whole generated sentence as shared history would make the next answer assume information the person may never have received.

The Inference Framework maintains a presented-range estimate, using text-to-audio alignment to map committed words and codec frames onto PCM sample ranges. Hardware timestamps, route latency, and gain envelopes determine which ranges were presented, which remain uncertain, and which were never submitted. The estimate includes device-committed audio that may continue after application cancellation. A text-generation cursor alone cannot establish that boundary, and the presented range does not claim to measure what the person understood.

PLM uses this record to repair the explanation. If the interruption changes the quantity, PLM updates the comparison and explains the effect. If the person only asks for a term to be repeated, the comparison can remain valid. When the playback boundary is uncertain, resuming from a complete clause can preserve meaning without repeating the entire answer. People who are cut off restart at the beginning of a clause, never mid-word, and they do not repeat what was already heard. The repair target follows the same rule.

In the example above, the reply after “wait, Wednesday” names the change, returns to the price, and this time includes the delivery qualification, because with Wednesday as the deadline it decides the comparison.

The interruption itself is information, and how the assistant re-enters is a decision of its own. PLM receives the interruption as an event, with its type and how long the person was away, and sizes the return to it. After a two-second aside it simply continues from the clause. After a longer one it marks the return, “so, back to the two suppliers,” and may restate the one fact the rest depends on. After a correction it names the change first. The words come from PLM; KRORAINA 2 gives them the delivery of someone picking a thread back up, lighter and quicker than a fresh start.

This requires different views of context. The task view contains the current objective, relevant memory, and accepted evidence. The speech view contains committed text, voice identity, and conversational cues for delivery. The presentation view records what the Inference Framework estimates it has played. Generated but unplayed content can remain useful internally without becoming part of the shared spoken history.

The Memory System also needs to distinguish a discarded draft from a spoken proposal or an outcome confirmed by the person. A provisional conclusion should not become a durable preference simply because a model generated it. Retaining these distinctions helps later exchanges begin from what was actually established.

What the language model hears

Everything the control models classify also goes to the language model. This is a principle of the design, and we know of no other speech system that follows it. Speech pipelines discard what a person does while the assistant speaks. The words reach the language model; a backchannel is passed or suppressed, a cough is filtered out, and a silence is nothing at all.

We keep those events. Acoustic Guard and KR-Interrupt label each one, a backchannel, a laugh, a cough, third-party speech, or a takeover, with its prosody and its position in the assistant’s sentence, and a stretch with no acknowledgment is recorded as an event of its own.

Those labels enter the dialogue state that PLM reads alongside the transcript, so a run of agreement across a long explanation, a laugh at an unexpected place, or a minute of silence can shape the next reply. Someone who has said “yeah” to every point may hear the assistant check whether it has missed an objection; someone who has gone quiet may be asked whether this is still a good time.

Over a longer stretch the pattern itself becomes the thing to answer. A person who has cut in three times in a minute, each time a little sooner, may hear “Do you want the short version?” before the fourth. A person who has gone quiet through the whole comparison and then asks about lead time for the second time may hear “Is it the term, or the two-week gap between them?” Each of these is a question about what the person is doing, drawn from what the control models classified and where in the answer it happened, and none of them claims to know what the person feels. That is the behavior people recognize as attention. Anyone who sees the nodding stop asks what they have missed. A system that only hears words cannot.

This works because the control models, the Inference Framework, and PLM are designed and scheduled together. The labels exist at the same cadence as the audio, and PLM has learned to read them. A pipeline assembled from separate services has no such channel, since only text crosses its boundaries. A native model hears the same sounds, but what it makes of them stays inside its weights. A product built on it cannot see what the model concluded, and cannot change how it reacts. The control models are therefore two things at once, gates on playback and a channel of perception for the language model. They turn how a person sounds and responds into semantic signals PLM can use, and PLM treats them as something to notice and ask about, never as a fact about the person.

Turning an utterance into work

So far the conversation has survived pauses, backchannels, and interruptions. The work underneath it has to survive them too, and the next three sections turn to that work. In most voice systems the conversation triggers work and then waits for it. We built the conversation as the way a person steers work. A request starts it, a correction changes it, a question inspects it, a word cancels it, and the person never has to stop talking to do any of these. For that to hold, work needs an identity of its own that outlives the sentence that started it, and the Inference Framework has to own that identity, because a language model that could invent a task could also lose one.

At a likely handover, the system has to determine what the completed turn calls for. It may contain a new instruction, a correction, an acknowledgment, or a question about work already running.

A task becomes actionable when a committed endpoint and stable transcript support a semantically valid request. PLM interprets the turn against active work and relevant memory, then emits a typed proposal. Memory supplies what the person would otherwise have to repeat, and a remembered preference never overrides a correction made in the present exchange. The Inference Framework resolves references, validates the action, and binds it to a task record containing an objective, constraints, and execution context. Task IDs and revision IDs belong to the Inference Framework, so a language-model guess can never invent or overwrite the identity of work already running. They persist when a conversational turn ends or a speech generation is canceled.

Consider three versions of the supplier request. “Can you compare…” leaves the objects unspecified. “Can you compare these two suppliers?” becomes actionable when the references can be resolved. “Compare these two suppliers for delivery before Friday” changes what counts as a successful comparison. The system needs those distinctions before choosing how to proceed.

Some preparation can begin before the request is committed. Tokenization, tool and contract registry checks, and session warmup within existing permissions can reduce the delay once it is ready. Sending a request to an external service or starting paid speculative inference requires a committed request and the corresponding authorization. This separation keeps preparation useful while the person is still forming the task, and it means nothing leaves the device before they have finished.

Once the request is defined, PLM may answer locally, ask for a missing detail, invoke an available tool, or propose delegation. Before delegating “Compare those,” the system still needs to resolve which suppliers the person means, so it asks.

Subsequent speech is interpreted against this active task context. “Include shipping” updates the comparison. “What does lead time mean?” can open an independent question. “Any progress?” inspects the comparison’s execution state. “Cancel the comparison” ends it. “Okay, go ahead” confirms the plan and creates nothing new. If “that one” could refer to several tasks, the system asks which.

Detecting the handover gives the system a moment to respond. Interpreting the utterance decides what that moment is for, and the next two sections follow the work it starts, first to where it runs, then through everything the person does to it.

Delegating without handing over the conversation

Delegation is where a cascaded system hands the conversation to a bigger model and waits. We built it the other way around. PLM keeps the conversation and decides what work exists. A remote model is a contractor that receives a defined request under a contract the device holds, and it can be swapped between one task and the next without the person noticing. The frontier gets stronger every few months; the part that belongs to the person, their context, their permissions, and the record of what was asked and answered, stays on the device, and it is what the system learns from.

PLM identifies when work requires information or capability beyond the live path. The request names the capability it needs, never a provider. The Context Compiler then builds the view that request is allowed to see, from the task, the tools it may use, and the entries in the Memory System that bear on it. Each entry carries its status, confirmed, provisional, conflicted, or expired, so an expired constraint or an unrelated fact stays home. Smart Router selects among eligible remote contracts using predicted quality, cost, and response time. A remote model sees that view, and nothing else.

Delegation may become appropriate after local reasoning has begun. During a supplier comparison, PLM may discover that it needs current delivery availability not held on the device. PLM keeps the work it has already done and requests only what is missing. Within existing application permissions, routine requests proceed automatically while the conversation continues; anything outside them waits for the person.

While the comparison runs, the person can ask what lead time means and receive a complete local explanation. They can discuss information already available, start another task, or revise the comparison. PLM bases its replies on the available evidence and actual execution state. A “running” status does not tell it when a result will arrive, so it never promises one. When there is nothing useful to contribute, the system stays silent while it keeps listening and working.

The comparison keeps its own identity through all of this. Answering the lead-time question, starting a second task, or a remote provider finishing late touches nothing the comparison depends on. What can change it is the question itself, which the next section takes up.

Changing, canceling, and returning work

Keeping the conversation going while a backend thinks is the easier half of background work. The harder half is keeping the work right. The “wait, Wednesday” that earlier sections followed through the interruption and the re-entry now has to be followed through the work. When the person says Wednesday, the Friday answer must never reach them. When they say stop, only what they meant should stop. When a result comes back mid-sentence, it waits its turn. This section is about the machinery that makes those three sentences true.

Changing a requirement

A new requirement becomes a new task revision, and the revision is what every result has to match. Below it, each dispatch carries its own attempt identity, so a retry is never mistaken for the original. Speech has its own version, separate from the task. That is why an interrupted sentence can end while its task keeps running, and why a changed requirement can supersede one attempt without touching unrelated work.

When the deadline changes, the Inference Framework marks the affected attempt as superseded, requests cancellation where supported, and dispatches revised work as needed. It does not depend on inserting a new instruction into a remote model’s hidden state midway through inference. Unaffected tasks remain active.

Result callbacks race with task updates and cancellation. Policy & commit, the acceptance boundary introduced in our first article, applies a revision fence before a result can affect current work. A result has to match the current revision and the attempt still wanted, and it has to come from a request that was never revoked. A duplicate callback passes the same checks and has no second effect. Canceling a remote request and rejecting its eventual result are separate operations, so correctness does not depend on the provider stopping immediately.

Stopping the right thing

Cancellation also needs a scope. “Let me finish” can stop audible speech while leaving the comparison alive. “Cancel the comparison” ends the underlying work. “Use Wednesday instead” changes the objective and supersedes affected execution. All three can begin with the same acoustic interruption, so stopping playback must not silently choose among them.

Cancellation stops work that is still running. An external action that has already completed needs a separate reversal, where one exists. A sent message, for example, may no longer be retractable. Recording the outcome lets the assistant explain what stopped and what had already happened, so “did that go out?” gets a true answer.

Returning a result at the right moment

Work also comes back, and often at an awkward moment. A result can arrive while the person is speaking. The system continues processing it and waits for an appropriate moment to deliver the answer. In the running example, the Wednesday result lands while the person is asking what lead time means. PLM folds it into the comparison while that question is answered, and the comparison follows as soon as the person has finished, opening with the fact that changed. Had the Friday result arrived instead, the fence would have stopped it, and the person would never hear it.

Once Policy & commit has accepted a result, PLM integrates it with the task context while the conversation continues. No result enters the conversation any other way. A result is evidence for the next thing the system says, and what it says depends on what was asked, what has changed since, what has already been said, and how the person is responding. A number read out from a template knows none of that.

Integration and delivery finish at different times. A valid result can become stale while an answer is being generated, so the commit step checks the task state again. A prepared answer stays tied to its task and revision until it is spoken, and the Voice Controller decides when its audio can enter the exchange. A result can therefore be valid and still wait, because the person is mid-sentence.

Together these rules make the conversation a control surface the person can trust. They can change their mind mid-task and know the old answer is gone. They can say stop and have only the right thing stop. They can ask what happened and get the truth.

Training decisions that happen mid-conversation

Everything above depends on decisions that no transcript records. The system must know when a request is ready to begin and whether a correction changes work already underway. It must choose whether a short reaction belongs at this moment and where an interrupted explanation should resume. A finished exchange contains the words on both sides and none of these decisions, so none of them can be learned from transcripts alone. We had to build the data that contains them.

We train at three linked levels. PLM task trajectories teach how context and evidence change the next action. Causal acoustic supervision trains the control models on information available at a particular instant. Contextual speech and backchannel preference training shape when and how a contribution fits the exchange. The Inference Framework’s records connect these levels through task revisions, transcript versions, and presented audio ranges.

Learning the consequence of a changed context

For PLM, the capability-boundary curriculum constructs controlled variations of the same task. A supplier comparison may have complete evidence in one branch, an unresolved reference in another, and missing live availability in a third. A fourth branch returns an answer computed before the deadline changed. Holding the underlying task fixed makes the changed condition determine the appropriate action.

The curriculum expands 100,000 seed episodes into 10 million candidate forks. Acceptance filtering and per-model capability checks produce 1.6M accepted episodes for PLM 1.8B, 1.8M for PLM 3B, and 2.2M for PLM 8B-A3B. The models share task lineages but receive model-conditioned rollouts, labels, and hard negatives.

A valid action depends partly on the local model. A task that PLM 8B-A3B can complete may require delegation from PLM 1.8B. The supervision must be conditioned on that capability boundary. Otherwise, the same dataset could teach a smaller model to overreach or a larger one to delegate work it could already perform.

Where several paths are reasonable, the label is a set of acceptable actions. Clarification and an authorized lookup may both resolve an uncertainty, so the action loss rewards their combined probability instead of penalizing an arbitrary alternative. Personal-context examples distinguish confirmed, provisional, conflicted, and expired information. They teach the model to let a correction supersede a memory, to keep a change inside its scope, to use the memory that matters, and to leave the rest alone. Hard negatives include an update applied to the wrong task and a fluent answer based on a superseded result.

Boundary SFT combines trajectory loss, valid-set action loss, and schema loss. User text, task context, and tool observations condition the model and are masked out of the response loss. General SFT replay accounts for 30% of this stage’s token-passes, preserving broad capability during specialization. Subsequent preference optimization contrasts corrected and stale trajectories, grounded answers and unsupported fluency, and justified delegation and unnecessary escalation.

Interrupted responses create another kind of task example. The continuation is conditioned on the revised objective, the estimated presented range, and the interruption event itself, its type and how long the person was away, so the model learns to size its re-entry to the gap. A suitable target restores an unheard qualification when it matters, omits an obsolete conclusion, and avoids restarting an explanation that the person has already received. Trajectories also carry the events the control models labeled, so PLM learns to read a run of agreement or a long silence as part of the context. The perception channel described earlier is trained here.

Preserving the information available at each instant

Acoustic supervision needs a stricter temporal boundary. Later audio can tell an annotator that a person eventually continued speaking, but that future must remain outside the input used to decide at an earlier pause.

For KR-Interrupt, we construct 200,000 onset events covering floor-taking, cooperative overlap, echo, third-party speech, transient sounds, and ambiguous cases. We split source events before deriving views at 80, 150, 250, and 400 milliseconds after onset. The 150,000 training onsets produce 600,000 causal windows. At each cutpoint, the target reflects what is decidable from the evidence so far; early views can defer even when the completed event is unambiguous.

Labels need to preserve that progression. Training every early fragment against the certainty available at the end of an event would reward a decision rule that cannot be executed online. Examples from the same event and source conversation also stay together in a data split. Otherwise, later parts of a training exchange could reappear as apparently new evaluation material.

We trained KR-VAP on 38,400 hours of synchronized dyadic audio. Each example exposes 20 seconds of history; the following two seconds supply activity labels and never enter the model input. KR-EOT uses 19.2M training cutpoints selected from a 12,000-hour registry. Its supervision combines activity-derived labels, teacher-accepted examples, and human annotation, including a separately tracked consensus subset. These are different training problems even when both ultimately affect turn-taking.

KR-VAP also faces a deployment mismatch. Recorded human dialogue supplies the activity targets, but the deployed assistant’s speech is controlled by our own playback policy. We therefore calibrate the integrated path with KRORAINA 2 in the loop and condition it on presented assistant audio. Predicting a human interlocutor’s next sound and scheduling the assistant’s own next sound require different treatment.

The generator’s own contextual training, which holds text and voice identity fixed while changing the preceding exchange, is described in our first article.

Learning when a reaction belongs

A listener response is a rare positive among constant negatives. KR-BC training contains 1,198,500 root opportunities, including 16,500 affective-reaction examples, and the positives sit among silence, continued floor ownership, echo, third-party speech, and other no-response cases. The accept head learns whether to contribute; class, intensity, lexical form, and semantic-commitment objectives shape the proposal. Concern and surprise are therefore trained response functions with eligibility conditions, alongside neutral acknowledgments and continuers.

The annotation design covers additional functions, including light support, encouragement, laughter, and breath responses. These go beyond the four classes the accepted-class head can emit. Annotators retain disagreements and can mark several responses as appropriate, including silence. They also flag reactions that imply understanding or agreement unsupported by the available context.

A second stage uses 90,000 training preference pairs, split equally between two comparisons. One holds the response class fixed and changes its placement to a randomly selected eligible moment. The other compares the proposal with silence. The first trains timing; the second trains whether to contribute at all. Evaluation matches response count and duration in the random-timing baseline, preventing a policy from gaining an apparent timing advantage simply by making fewer or shorter sounds.

A recording follows a fixed sequence, but a live listener response can change what the person does next. They may continue through a small acknowledgment or stop because it sounds like a takeover. Calibrating the integrated system therefore requires testing the Voice Controller and rendered sound together, accounting for echo conditions and the person’s subsequent behavior.

The three levels share one discipline. Each model is trained on what it will actually have at the moment it has to decide, and nothing more.

Keeping the device responsive

The rule for a live media path is that audio never waits. A server can give that path its own machines. A phone cannot, so the path has to hold beside the language model, the speech generator, and everything else the person is running. On-device full duplex therefore requires bounded latency under concurrent inference. The five control models add 16.76M stored parameters, but parameter count alone does not determine the real-time workload. Shared feature extraction, state caches, memory traffic, and generator kernels all compete for the same device resources. We separate their execution budgets and schedule them according to the cost of missing a deadline.

The Inference Framework keeps capture and render callbacks bounded. They never wait for PLM, network I/O, memory retrieval, dynamic allocation, or unbounded locks, so inference stays outside the audio callback. Each 10-millisecond block boundary re-checks the render-authority snapshot described above, so no queued render plan outlives its grant.

Thread priority cannot preempt an accelerator kernel already using a shared resource. Our scheduler keeps acoustic control independent of long, non-preemptible generator work, using CPU or DSP paths where required. Committed speech generation takes priority over speculative generation and contextual backchannel rendering. Optional reactions yield first under load, and a listener response falls back to its prerendered form before it is dropped. The order is the order of what the person would miss least. A remote result still consumes local resources when PLM integrates it and KRORAINA 2 prepares the response, so asynchronous network execution does not remove scheduling pressure.

Backpressure is decided stream by stream, and each stream is allowed to lose a different thing. An acoustic posterior can be replaced by a newer one, because only the latest matters. Pending ASR work can be merged as the transcript advances. KR-EOT keeps a short candidate queue, with a separate slot for an explicit stop so it can never be crowded out. A backchannel that misses its window is dropped. Committed speech loses nothing until it finishes or is canceled, because dropping one frame of it could erase a negation or half a qualification.

How much speech to prepare ahead is a resource tradeoff. A deeper queue protects against a temporarily slow generator but holds more work that may become obsolete. A shallow queue holds less prepared audio and demands steadier generation. Audio held by the application and audio already committed to the device have different cancellation limits, so their queue occupancy needs to be measured separately.

Keeping language state warm

Prefix caching reuses the KV cache for an unchanged language-model prefix. Incremental decoding extends that state as tokens arrive. Warm sessions avoid repeated setup and prefill, but their KV allocations remain resident alongside the speech models and acoustic state. Cache reuse therefore requires a valid context identity as well as enough memory to retain it.

Long conversations eventually require compaction. The shortened context must retain active task identities, current constraints, unresolved work, and relevant accepted evidence. A fluent summary that drops an outstanding task has lost information required for execution.

Context compaction creates a new token sequence and therefore requires a matching KV cache. The Inference Framework pre-fills that snapshot before switching at a supported request boundary. A returned tool result likewise enters through a versioned context update; it cannot retroactively alter the prefix used by an in-flight decode.

On a server this is a managed handoff between two warm instances. On a phone there may be memory for only one. With enough headroom, the active session stays available while the replacement context is prepared. A constrained device re-prefills within a single session instead, and language generation pauses for that computation. The design makes that stall explicit and keeps listening and playback out of it, though both still need their share of the device to meet their deadlines.

Deleting a middle span from a KV cache is insufficient because later states were computed from the original prefix. Re-prefill recomputes those dependencies for the compacted context. The scheduler must account for that transient workload as well as steady-state decoding, especially when speech synthesis and result integration are active. The cost the person can notice is a later reply. The costs they must not notice are a dropped frame and a lost task, and the scheduler is built around those two.

Evaluating a complete exchange

Most voice benchmarks score a final answer or a turn-taking decision in isolation. Neither can see what this article is about, which is whether the conversation and the work stay consistent with each other while both change. To evaluate continuity, we follow a request through changes to the work and into the answer delivered back to the person. This lets us test how a decision at one stage affects the rest of the exchange.

The supplier scenario provides a useful test structure. A comparison begins, an independent question is answered, and a changed deadline supersedes the affected work. Both old and current results are then delivered under controlled event orderings. The person interrupts the resulting explanation. Each transition exposes a different way the system could lose continuity.

Testing state under competing events

A task-state test specifies which work must survive each event. An unrelated question leaves the comparison intact. A changed deadline supersedes the affected work. A repeated callback produces no second completion, and a canceled result cannot return as a current answer. Testing only the final wording can overlook these errors when the answer still sounds plausible.

Our evaluation design varies both the timing and order of events. A result can precede a correction, arrive during integration, or arrive after cancellation. An answer can be prepared before new speech starts or partially presented when it is interrupted. The same validity rules must hold across these sequences.

Repair is judged against both the current objective and the delivered history. A continuation that repeats everything may be coherent but burdens the exchange. One that omits an unheard qualification may be concise but misleading. Task correctness and continuity of explanation need distinct judgments.

Measuring responsiveness without rewarding interruption

Conversational timing has competing errors. Aggressive entry shortens response delay while increasing premature speech. A policy that rarely enters can reduce false takeovers while missing useful response opportunities. Latency, recall, false-positive rate, and decision coverage need to be read together at an operating point fixed before the test.

The interaction-timing comparison follows the protocol of our first article. There, the full Voice Controller reached 89.2% end-of-turn recall at a 6.7% false-positive rate with 95.8% decision coverage, and listeners preferred its backchannels to random timing and to silence. A VAD-and-silence baseline, KR-VAP, KR-VAP with KR-EOT, and the full Voice Controller run on the same events, with operating points and class denominators frozen before the test. An uncertain decision stays in its denominator and does not count as a yield. Each step of that ladder isolates the value of projection, endpointing, and playback policy in turn.

Listener responses need comparisons with silence and with matched-count, matched-duration random timing. Contextual delivery tests hold words and voice identity fixed while changing relevant history. Keeping these tests distinct makes it possible to tell whether a benefit comes from timing, conversational function, or the sound of the response.

Measuring the workload that produces the conversation

The voice-stack measurements in our first article include the concurrent control models and exclude concurrent PLM generation. Full-system evaluation adds PLM prefill and decoding, memory retrieval, task updates, and result integration to that workload. Tokens per second and isolated voice latency cannot establish whether the render path meets its deadlines under this combined load.

Sustained runs track frame-time distributions, render underruns, buffer occupancy, memory pressure, and recovery events alongside audible response timing. They include thermal throttling and output-route changes. Comparing isolated and concurrent workloads on the same configuration shows where integration introduces contention.

A synchronized trace from the Inference Framework connects these measurements back to behavior. It records when a request became actionable, which task a correction changed, whether a late result was excluded, and how much of an answer reached the output path. Audio and task events must share a usable time reference for those relationships to be inspected.

The complete exchange tests one thing, whether a person can keep changing a task while the system works on it and hear only answers that are still true. Component tests locate a failure. Only the whole exchange shows whether the speech, task, and scheduling decisions held together.

A system that can learn from the exchange

The training section opened with a problem. The decisions this system makes leave no trace in a transcript, so we had to build the data that contains them. A live exchange supplies that data from the other side. Every conversation the system holds produces those decisions and, a moment later, the person’s response to them. They kept talking through the “mm-hm.” They never asked where the answer had got to. They said “no, the other one.” None of these is a verdict on its own. Each is a signal about a decision made a moment earlier, and the record shows what the system had heard when it made it. Across many exchanges, those signals become the feedback the training section could only construct by hand.

That record is personal, and it never has to leave the device to be learned from. It holds one person’s context, their memory as the Memory System keeps it, and the way they talk, with the pauses, the corrections, and the phrases that mean more to them than to anyone else. Continual learning is a problem the device is placed to solve. The data is already there, the person is already there to correct what it learns, and the loop never has to close in a data center. A model that learns from these exchanges on the person’s own device starts to diverge from every other copy of itself. Two people who bought the same device and asked the same first question end up, months later, with different intelligence. One has learned where its person likes a pause before a decision. The other has learned that “the usual” means the Tuesday order, and that a joke about it lands.

As intelligence enters different lives, its distribution becomes its diversification. A plural ecology of intelligence needs local centers that preserve what is particular, and a device that learns is such a center. A model trained only in the center converges on one account of how a conversation goes and one account of what people mean. A model that keeps learning at the point of use carries a particular person’s way of thinking, and the person keeps the authority over what it learns. That matters for the person, whose system fits them and stays theirs. It matters for everyone, because human history advanced through many centers of judgment, and an intelligence that learned the same lessons everywhere would flatten the differences that make exchange between people generative.

Learning this way has to respect what the record can and cannot say. Asking once for the cheaper supplier is one fact. A standing preference for the cheapest option is a different one, and only a recurring correction, a verified outcome, or explicit approval can establish it. Memory holds the first kind. A weight update has to earn the second, and it has to keep the capability the model already had and pass evaluation before it is adopted. We are integrating Thinking Machines Lab’s Tinker for post-training the shared models,4 and building lightweight on-device adaptation for the personal part, which should never leave the device.

The immediate goal is a system that stays with the person while it works. The longer one is an intelligence that becomes, over months, particular to the person it talks with, and stays answerable to them.

What full duplex is for

The supplier comparison ran through this article because it makes the machinery visible. The machinery exists for something else. Most of what a person has to give is never a task. It is a half-formed idea said out loud to see whether it holds, a laugh in the wrong place, a silence that means the point was missed, a “wait” that means the good part is coming. People give each other this all day, and it is how ideas get made between two people. Software has mostly stood outside it, taking the words and dropping the rest.

This article describes a system that can take part. It hears a person while it speaks, hears how they respond as well as what they say, answers in a voice that carries the feeling of the moment, and picks up from where the person actually was. That is the interaction layer we are building between a person and machine intelligence, and it runs on the person’s own device. The aim is an exchange natural enough that the person forgets the interface and stays inside their own thought.

What that unlocks is the part of a person that only comes out in conversation. An idea gets said badly, heard generously, and said better. A person met with an “oh” at the right moment keeps going and finds the thing they did not know they thought. For most people this is where imagination happens, in exchange with someone who is paying attention. We want it available to anyone, whenever they want it, from something that learns their context over time and stays theirs.

The frontier will keep supplying more capable models, and this layer will keep handing them the work. What it keeps is the person. It keeps their context, learned over months, how they sound when they are sure and when they are not, and what they meant the last time they said “the usual.” That learning stays on the device and under the person’s authority, and everything expressive in the voice is in service of it. The system notices how the person is doing, and it answers in a way the person can feel.

Our first article said that AI’s second history should be measured in the capabilities humanity keeps. This article is about the one the others grow from, the ability to think out loud with something that listens.

A person who is heard keeps thinking. That is what full duplex is for.