Jev Mode on a 3090: 26 ms per Token You Don't Write

📚 More on this topic: Agent Skill Compilation Tested · Skills in the Weights · LoRA Skill Compilation Is a Double-Headed Coin Flip · Best Local LLMs for Structured Output · InsiderLLM Benchmarks

You have a classifier in front of an agent, and it answers by writing JSON one token at a time: open brace, quote, field name, quote, colon, quote, value, and on for twenty tokens before the parser gets anything to parse. At 25 ms a token on a 27B that is half a second per decision, most of it spent spelling out punctuation the schema already fixed. Jev is Typesafe AI’s API that returns structured answers in milliseconds by choosing values rather than writing them — every field of a finite schema scored in one forward pass, each with a probability. Harsha Gondala’s open reconstruction put the mechanism on Apple Silicon under Apache 2.0, and Codacus’s parallel-decision branch of llama.cpp, demonstrated in his video, is the first time I have seen it run on CUDA. His figure was 300 ms against 3.5 s on Gemma 4 12B.

I ran his fork on my RTX 3090 with Qwen3.6-27B fully resident, against the 47-item test split this site’s skill-compilation series has been scoring since August, and put the write path on the same server in the same session. Two questions: how much faster, and does it change what the model gets right.

Should you use it

Routing and triage with short answers and many fields: yes. Every answer is schema-valid by construction — 47 of 47 here, where the written path produced one unparseable answer in every run it has ever made — and the cost is one batched forward pass regardless of how many fields you add. If your written answer today runs to fifty tokens, this is a 5.5x cut in latency with no change to the model.

Anything where the written answer is under about ten tokens: barely worth it. Choosing costs about what prompt processing already costs — so at four written tokens the gain is 1.3x. The parser guarantee still holds, and that may be reason enough, but do not expect the clock to move.

Any hope of an accuracy gain: no. Choosing scored 21 of 47 and writing scored 23. The two paths gave the same three-field prediction on 36 of the 47 items and made the same confusion on the field they both get wrong most. It is the same model reading the same message; only the output path changed.

The line: 275 ms plus 26 ms per token you write

The write path was run four ways on the same 47 items, same system prompt, temperature zero, three passes each, changing only the requested output shape: bare tool/mode/scope values, the compact JSON the prompt already asks for, the same JSON pretty-printed, and JSON with a one-sentence reason field the scorer ignores. Choosing was run single-pass in the same session.

A line through the four write formats is wall = 0.275 s + 25.8 ms × tokens, on Qwen3.6-27B Q4_K_M on a 3090, and every point sits on it. The fixed 275 ms is prompt processing for the roughly fifty uncached tokens of each request — the user message and chat template; the system prompt is cached — and it is the same on all four shapes. Generation runs 22 to 25 ms a token and is linear in the count. Client overhead is 5 ms on every format; sampling is not where the time goes.

Now look at the choose row. Single-pass choosing took 0.298 s: 124 ms to prefill the same context, 169 ms to score 21 rows in one batched decode. That is the whole finding in one number — on a 27B, scoring the schema costs about what prompt processing costs, so choosing is roughly what writing would cost at zero tokens. The speedup is exactly the tokens you no longer write, at 26 ms each, and nothing else. The 5x line is crossed at about fifty written tokens.

Codacus’s 300 ms against 3.5 s on Gemma 4 12B is an 11.7x ratio. On this line that is a long JSON answer, well over a hundred tokens on a model that writes faster than a 27B, and it is consistent with what I measured. I did not reproduce it — different model, different card, different schema.

Accuracy: the same model, the same mistakes

The task is the intent classifier from my own agent, three fields with four labels each, scored exact-match on all three with the deployed sanitise pipeline, against the frozen 47-item split the ten-seeds article established. Writing is the exact request that article’s llama.cpp path used. Choosing sends the same system prompt as the decision endpoint’s instructions and the three label sets as the schema.

Both paths are deterministic within a session: three full passes each, predictions identical, and choose’s probabilities identical to nine decimals. Item by item, both are right on 20, write alone on three, choose alone on one, and neither on 23. Those 23 are the same block the ten adapters could not move — mostly messages whose gold is rag / recall / session that both paths call answer / chat / all.

The two-item gap is entirely in scope, and I had a hypothesis about it. The decision endpoint scores every field from the same prefix, so no field can see what the others chose. Scope is the field whose right answer depends on the other two — the prompt says so in as many words — so I expected that independence to be the cost. Pre-registered: a two-pass version, tool and mode first, then scope with “tool: X, mode: Y” appended to the context, recovers scope to 30 and lands overall at 23 plus or minus one.

It landed at 21 — scope stayed at 28. The two-pass run changed scope on four items and every one of them moved from one wrong value to another. Then I went back to the single-pass records: choose and write disagree on scope on exactly two of 47 items, and on both the tool and mode already matched. Of choose’s 19 scope errors, write shares 12 outright. The confusion is the same shape on both paths, gold session called all (11 times for choose, 10 for write). My hypothesis was wrong. The field did not need its neighbours; the model misreads which messages refer back to a past conversation, and it misreads them the same way whether it writes the answer or picks it.

Confidence: informative, not a guarantee

Each chosen value comes back with a probability, and the pitch is that this tells you when to trust the answer. On the 26 items choose got wrong, the minimum field probability averaged 0.62 (median 0.56); on the 21 it got right, 0.80 (median 0.83). The signal is real. So is the overlap.

A 0.5 gate on the weakest field catches a third of the errors for one lost correct answer, which is a reasonable trade for a router that can fall back to a slower path. Past 0.7 it starts refusing right answers faster than it catches wrong ones. And the tail does not gate at all — six wrong items sit above 0.77, and item 82 is wrong at tool 1.00, mode 0.995, scope 0.98 — gold explore, chose recall, and the written path made the identical call. Type-safe is not correct. That is the line under his video, and the number bears it out: the guarantee is that the value is in the schema, not that it is the right one.

Per field, scope is both the weakest (28 of 47) and the least confident, 0.80 when right against 0.68 when wrong, which fits everything above.

Update, 2026-09-21: the overlap has a cause. In the Latent Space episode with Diogo Almeida, TypeSafe’s co-founder and CEO, the founder’s own account is that a chat-tuned base is trained to be overconfident on strings, and that Jev’s calibration comes from RLCD, a training step on top of the base. The open fork runs on a chat-tuned base and has no RLCD, so the probabilities in this section are the uncorrected ones the product was built to fix. Item 82 at tool 1.00 is what uncorrected looks like; treat the table above as a floor for the open fork, not a reading on Jev.

Two traps

Alternating schemas evicts the cached prefix. The endpoint caches instructions plus the generated field catalogue as one prefix, and the server holds one. My first two-pass harness alternated the tool/mode schema and the scope schema per item, so every request re-prefilled about 700 tokens: cached_tokens read zero on every second call and the run came in at 1.34 s per item, 4.5x single-pass. Batched by pass — all 47 first calls, then all 47 second calls — both prefixes stay warm and the same two passes cost 0.328 s, 1.10x. If you build anything multi-stage on this endpoint, group the calls by schema.

Marginal items flip across server sessions. Within one server process every run was byte-identical. Across two sessions of the same model, same flags, same file, single-pass choose flipped two items, with field probabilities moving up to 0.108, and the same first-pass request flipped two items depending on whether its prefix came from cache or a fresh prefill. The items that flip are the ones sitting at p ≈ 0.5 — item 69’s mode at 0.50 against 0.48, item 77’s tool at 0.506 against 0.512. This is the prompt-cache-reuse mechanism from llama.cpp #26207, which the ten-seeds re-run already pinned on the chat path, showing up on the decision path too. Deterministic means deterministic within a session and a cache state; if a threshold is doing work in your pipeline, a coin at 0.50 will land differently after a restart. The same batch-shape dependence is llama.cpp #25618, where a token scored inside a multi-token batch gets slightly different logits than the same token scored alone, and the greedy divergence measured on 2026-09-22 with the DFlash drafter on the 35B-A3B is the third sighting: six of nine prompts diverged at temperature 0, each at a near-tie the target resolved differently in a 16-token verify batch than in single-token decode.

One more, smaller: the catalogue is part of the prompt. Dropping scope from the schema for the first pass changed the tool and mode answers on five items. The fields cannot see each other’s answers, but every field sees the schema text in the prefix.

The series: four substrates, none beat the base

This is the fourth piece in the skill-compilation series. The first compiled a prompt skill ten times from the same input and got a spread of 4.65 points on this split. The second priced the prompt tax and set up the weights alternative. The third trained the adapter ten times and got a spread of 3.62, every seed at or below the base. This one asks what happens with no compilation at all: the base model, the deployed prompt, and a different way of reading the answer out.

Prompt: σ 4.65, best draw above base, mean not. Weights: σ 3.62, nothing above base. No compilation: 21 against the base’s 23, same model, same errors — the only thing gained is time and a parser that cannot fail. On this task, on this split, nothing has beaten the base yet, and the fastest of the four is the one that changed the model least.

Limits

One model, Qwen3.6-27B at Q4_K_M, one card. One quant path: through llama.cpp the base writes 23 of 47, or 48.94 percent, reproducing the ten-seeds record for this GGUF on 46 of 47 predictions; the ten-seeds headline of 44.68 percent was the same base scored through PEFT in NF4, and both stand. Choose landing on exactly 21 of 47 is a coincidence with that second figure, not a reproduction of it. One task, 47 items — so one item is 2.1 points and the two-item gap is inside the noise the series already carries. The server ran with three decision slots, the fork’s minimum, because the desktop was up and its README’s sixteen did not fit beside it; this schema still scored in a single round. His two caveats, that hybrid models with recurrent layers lose the fused batch and that an offloaded MoE copies experts over PCIe on any batch over about thirty tokens, are untested here — everything in this article is a dense model, fully resident.

Method and full results

Fork. thecodacus/llama.cpp, branch parallel-decision, commit 14d04e755fa28653e87b9a07072892265bdc0fad (2026-09-20, b9891-1161-g14d04e755), cloned to its own tree beside the v0.4.0 and b10088 builds. cmake -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_NATIVE=OFF -DLLAMA_CURL=OFF, CUDA 12.8.

Model. Qwen_Qwen3.6-27B-Q4_K_M.gguf, sha256 8739a0cbb80036e5dbdced2085f142b8ba86e3235db8b8039b3769fe5fc70843, verified on the day and equal to the ten-seeds record. 22.0 GB on the card beside the desktop’s 4.2 GB.

Rig. Miu: RTX 3090 24 GB, i7-8086K, 62 GB DDR4-2667, Ubuntu 24.04, driver 580.178.04. Desktop up throughout.

Server. llama-server -m <gguf> -ngl 99 -fa on -c 4096 --jinja --load-mode mmap --port 8081 -kvu --decision-seqs 3. The first four flags are the ten-seeds section-7 line. --no-mmap is gone from the fork’s parser; its equivalent --load-mode none failed with a CUDA out-of-memory two seconds into load, before any weights reached the card (swap full, 17 GB of RAM free), so the default mmap path was used. --decision-seqs 16 failed a 202 MiB cudaMalloc at 22 GB used. One server process per phase; every run in a phase shares it.

Choose. POST /v1/decision, instructions = the deployed intent system prompt unchanged (solo.py sha256 32456044…), schema = three enum fields with the deployed label sets in the order the prompt lists them, mode: tree, one context per request. The endpoint refuses a field without a description, so each carries the prompt’s own header phrase verbatim, asserted present at import. The decision object is JSON-dumped through the same parse_and_sanitise() as the written answer, then exact-match score().

Write. POST /v1/chat/completions, the ten-seeds score_lora.py request byte for byte: temperature 0, thinking off, stock prompt; max_tokens 100 in phase 1 and 160 in phase 2. The three extra formats append an output-format override to the system prompt; the scorer parses JSON, or the slash string, and ignores reason.

Phase 1 (write ×3, choose ×3, 141 samples per path), server-reported means beside the client wall clock:

That is 3.38x, and rounds was 1 on every request. Cold first call with the prompt cache empty: choose 2.05 s, write 1.24 s; every tabled request is warm.

Phase 2 (same server line, new session: choose ×1, two-pass ×3 alternating and ×3 batched, two-pass with the line in instructions ×1, four write formats ×3). Single-pass choose in this session 0.298 s mean, 0.345 s p95, and 21 of 47 with two items different from phase 1. Two-pass batched: pass one 128 ms scoring, pass two 38 ms scoring plus 79 ms prefill of the 54-token context. Write formats: prompt 288 to 292 ms on all four; generation 98 / 489 / 704 / 1,340 ms; 21.8 to 25.4 ms per token.

Confidence figures are from phase 1, choose rep 0; probabilities were identical across the three reps.

Raw per-item records for all 29 runs, the harnesses, the analysis scripts and the server logs are in the repo at docs/bench-results/miu-jev-decision-2026-09-21/ (phase 1 commit 519a68f3, phase 2 2f088af5). No rows go into the benchmarks dataset: this is a latency and accuracy result on a task split, not a tok/s measurement.