Introducing TypeLLM: Type-Safe Decoding for Autoregressive LLMs

TypeSafe AI's Jev highlighted a useful idea: software needs decisions, not more strings to parse. Jev answers it with a purpose-built model. TypeLLM asks the complementary question — whether the same typed-decision interface can come from any open autoregressive model you already run.

So that is what it is: you hand TypeLLM context and an ordered JSON Schema, and it hands back values your software can act on — enums, booleans, integers and floating-point numbers — with no proprietary model API, no retraining, and no parser for you to maintain.

The guarantee is narrow and absolute: a categorical value is always inside the schema's declared domain, and a numeric value always has the requested type. Not usually. Not after a retry. Invalid continuations are never eligible to be selected.

How a type-safe answer is generated

Start with one field of the schema:

"expense_type": {
    "type": "string",
    "enum": ["meal", "travel", "equipment"],
    "question": "What type of expense is this?",
}

TypeLLM presents that field as a user turn:

USER

Receipt from Hilton London
Total: £324
Employee travelled to London for a client meeting.

Choice(
  name="expense_type",
  question="What type of expense is this?",
  choices={"A":"meal","B":"travel","C":"equipment"}
)

ASSISTANT

B

The model answers with one token, and two things make that enough.

Map every value to one token. meal, travel and equipment tokenize to different lengths, so they cannot be compared at one position. Each gets a single-token control label — A, B, C — written on the choices line. Labels are checked against the server's tokenizer: one token in, the same text back out, or the label is skipped.

Only those labels are selected. TypeLLM reads the log probability of each candidate token and nothing else, so an out-of-schema token can never be chosen: it was never a candidate. Renormalizing over the candidates gives a distribution on your declared values — meal 0.04, travel 0.93, equipment 0.03 — which sampling draws from and you can threshold on. Argmax does not need it: restricting the candidates already fixes the ranking.

Only the labelled candidates are read; their scores are renormalized to sum to one.

All of this needs the model's per-token probabilities, which closed APIs do not expose. TypeLLM gets them from an open-source model served with SGLang.

One forward pass covers a decision however many values it allows, so a boolean and an eleven-value numeric enum cost the same. Temperature acts on the candidate scores, never on free generation: turning it up makes meal likelier than travel, never something the schema never declared.

Beyond finite domains

Enums and booleans have a finite domain, so one label can stand for each possible value. Integers and floating-point numbers do not. TypeLLM supports them directly with JSON Schema integer and number fields, without requiring you to enumerate the answers in advance:

result = client.generate(
    context="Calculate the requested value accurately.",
    schema={
        "type": "object",
        "properties": {
            "answer": {
                "type": "number",
                "question": "What is 17.5 multiplied by 4?",
            },
        },
        "required": ["answer"],
    },
)

print(result)
# {"answer": 70.0}

A numeric decision may use several tokens rather than one. At every step, TypeLLM permits only continuations that can form a valid signed integer or decimal number, and it permits the answer to end only after the number is complete. The returned value is a Python int for integer and an int or float for number, ready for the rest of the program to use.

Two execution modes

Everything above describes one decision. A schema is several of them, and there are two ways to run the set. They differ in exactly one thing: what each decision is conditioned on.

Sequential: each decision sees the last

Generated one after another: every decision is conditioned on the context and on each value chosen before it.

The completed turn stays in the conversation before the next question is appended. The earlier Choice says that B means travel, and the assistant's answer records that B was selected. Field order is therefore decision order: the second request extends the first conversation with one answer and one new question.

Each decision adds one short turn. The next decision sees the original context, the earlier choice mapping, and the selected label.

The completed first turn is byte-identical at the start of the second request, so the server's prefix cache matches it and reuses that KV state. The client holds one growing conversation and never touches a KV tensor; only the new tail is computed.

The long context is read once; each later decision adds only its own record.

The server reports how many tokens it reused. With D decisions, a context of C tokens, and roughly S new tokens per decision:

without prefix reuse: O(D*C + D^2*S)
with prefix reuse:    O(C + D*S)
output generation:    O(D), plus the digits of any numeric field

Batch: independent fields in one request

Generated together: every decision is conditioned on the context alone, so none has to wait for another.

Not every schema needs that. When every field is answerable from the context alone, running them in sequence makes each one wait for a round trip it never needed.

So TypeLLM forks instead of extending. The shared context is prefilled once and each field becomes a branch: the same context plus its own question. All branches go out in one call and come back as a single batched decode step, one token per field.

Because every branch starts with identical text, the prefix cache serves all of them. In one local run — a ~1,100-token shared context, sixteen boolean fields, one GPU, sixteen concurrent requests allowed — every branch reused 1,088 cached tokens:

Execution	End-to-end	Per decision	Relative
Sequential	9.35 s	0.584 s	1.0x
Batch	1.61 s	0.101 s	5.8x

One measurement on one machine, not a portable benchmark; it moves with K, the model, the context length and the server. The shape is the durable part: prefill of about C + sum(Q_k) for K fields, then one batched decode.

The cost is the dependency — each branch sees only the context and its own question — so use batch when the fields genuinely are independent. Everything else holds: each branch still scores only single-token labels and renormalizes over its own declared domain.

Sequential or batch?

The two modes are not a speed knob. They compute different conditionals, so the question is never "which is faster" but "does this field's answer depend on another field's answer". A severity that should follow from the system already identified, a rollback that should follow from both — those belong in sequence, and the cost is D round trips. Sixteen flags that each read the same document independently belong in one batch, and the cost is that none of them can see the others.

Getting it wrong in one direction is slow; in the other it is wrong. A field batched away from a dependency it actually has will answer as though that dependency did not exist, and nothing in the output will say so. Nothing stops you from splitting a workflow either — run the dependent fields in sequence, then batch the independent ones over a context that already carries the first results.

Conclusion

TypeLLM shows that a typed-decision interface does not require a purpose-built model. Restricting the candidates at one position, renormalizing over them, and appending the result to the prompt is enough to get categorical values that are typed by construction. Constraining every continuation extends the same idea to integers and floating-point numbers. No proprietary API and no retraining: TypeLLM reads a distribution the model already produces and reuses one cached context across the workflow.

More benchmarks and worked examples are coming.