The workflow model

We need a model that runs the workflow

Many useful AI features are workflows. A support ticket is classified, checked for urgency, assessed for a refund amount if it is a billing issue, and answered using the category and urgency. Some decisions can run together; others need earlier answers.

TypeLLM makes the workflow the unit of inference. You declare the decisions, their types and how they depend on each other; TypeLLM executes the graph and returns typed results in one API call.

Chat APIs are not optimal for executing workflows

Most of a workflow's cost is reading. Take a support ticket with a 10,000-token history, read by the four steps. Before the model can answer anything about it, the server runs the ticket through the model once (prefill) and keeps the result: the KV cache, the keys and values of every token at every layer. Every answer is generated on top of it. A workflow should compute that cache once and let every step use it.

Through a chat API, each step is a separate request with its own prompt, and the server decides on its own what it can reuse:

Prefix caching only matches a shared beginning. A cached prefix is reused only when the new prompt starts with exactly the same tokens. Put each step's instructions before the ticket, as chat prompts usually do, and the prompts diverge at the first token: the ticket is read again for every step.

Earlier answers come back as text. A dependent step gets the category pasted into its prompt, so the server encodes it again, though the previous request had just computed it.

Parallel steps race for the same cache. Send category and urgency together, and both requests can prefill the ticket at the same moment: neither can reuse a cache the other has not finished.

Nothing guarantees the cache is still there. Between requests, it can be evicted under load, or the next request can be routed to another replica that never saw the ticket.

Asking for all four answers as one JSON keeps the cache, but turns the workflow into one sequence. The fields are written in a fixed order, so category and urgency cannot be decoded side by side. The refund is skipped only if the model follows the prompt. A confidence the model writes is text it generated, not the probability it gave each category.

A server cannot plan for a graph it is never shown.

A workflow model plans its execution from the graph

Workflows should be declared, not orchestrated

With SQL, you declare the result you want. The database chooses how to execute the query. You do not write the join loops or schedule each read yourself.

A workflow model is the same split, for AI workflows. Here is the ticket workflow, declared:

{
  "category": {"type": "string", "enum": ["billing", "technical", "account"]},
  "urgent":   {"type": "boolean"},
  "refund":   {"type": "number", "when": {"category": "billing"},
               "instructions": "Refund in dollars."},
  "reply":    {"type": "string", "depends_on": ["category", "urgent"],
               "instructions": "One sentence to the customer."}
}

Category and urgency can run together. The refund is generated only for billing tickets. The reply reads category and urgency; it does not wait for the refund.

For the ticket, "I upgraded to the Pro plan on the 3rd and my card was charged twice, $49 each time. I can't find a way to get the second charge back in the billing page. Please help, this is my company card," the call returned:

{
  "category": "billing",
  "urgent": true,
  "refund": 49,
  "reply": "I am sorry to hear about the duplicate charge on your company card, and I have flagged this as urgent for our billing team to process your refund immediately."
}

Four steps, one API call, every answer in its type.

Time and compute are planned from the dependencies

Because the engine sees the graph, it plans two things at once: when each step runs, and what each step has to compute.

The call takes as long as its longest chain. Category and urgency need nothing from each other, so they run side by side, and the reply starts as soon as both are done: the call takes as long as its longest chain of dependencies, not the sum of its steps.

Independent fields fork from one cache. The ticket is prefilled once, before category and urgency run, and both fork from its KV cache. Every request of the call runs on the same replica, so the cache is there.

A dependent field continues its parent. A field that reads one earlier answer continues that answer's prompt. The ticket, the parent's question and its answer are already in the cache; only the new question is computed.

With several parents, it continues the longest. The reply reads category and urgency. It continues the longer of the two prompts, so it reuses the most cache, and receives the other answer as a value in its question.

A skipped branch costs nothing. For a technical ticket, the refund's when fails: its prompt is never sent, and nothing waits for it.

On the ticket above, a 15-run measurement compared one TypeLLM call with four serial calls, earlier answers pasted into later prompts:

The serial version read 2.1× the input tokens and took 1.9× as long.

Tools and people are nodes in the graph

Some steps need what only your side knows: an order's status in your database, a result from your own API, a person's approval. They are declared in the graph too, with "source": "user":

{
  "order_id":     {"type": ["string", "null"]},
  "order_status": {"type": "string", "enum": ["shipped", "delivered", "lost"],
                   "source": "user", "when": {"order_id": {"ne": null}}},
  "refund_ok":    {"type": "boolean", "source": "user",
                   "when": {"order_status": "lost"}},
  "reply":        {"type": "string", "depends_on": ["order_status"]}
}

The call runs every step that does not wait on your side, then stops with status "needs_input". Your code looks up the order, and the call resumes where it stopped, its context still cached; the steps after see your answer like any of the model's. For a lost order it stops once more, for a person to approve the refund. With when, it stops only when your side is needed: an email without an order number runs straight through. See Tools and human input.

From deterministic to probabilistic graphical models

A deterministic graph needs typed nodes

In a deterministic graph, each node passes one answer to the steps that depend on it.

A type, held during generation. Every answer, intermediate ones included, is decoded within its type: a string, a number or an integer, a boolean, one of an enum's values, an object, an array as long as the input says, or a box around something in an image.

Control flow. Types are what make control flow possible. A when tests a value: category equal to billing, an amount greater than 100. A depends_on hands a value to the next step. Both work only if the answer is exactly one of the declared values, or a number: a "Billing" never matches billing, and a "$49" cannot be compared with 100.

Thinking, when the step is hard. A node can reason before it answers, like adding up the lines before saying whether an invoice adds up. With "thinking": "auto", each node thinks only as much as it needs, on each call.

A probabilistic graph needs probabilities

Uncertainty and confidence. A decision is not one answer. For an enum or a set of choices, TypeLLM reads the probability of each option from the model as it decides, and a confidence for the choice. The graph can already branch on them: refund can run only when category is billing with a confidence of at least 0.8, and stop for a person otherwise.

Uncertainty propagation. A reply built on a category that was 60% billing should carry that uncertainty forward. The next step is to pass distributions over categorical choices along the edges of the graph, so every node conditions on its parents' uncertainty. With distributions in the graph, a workflow can also choose among declared next steps, weighing the cost of a check against the cost of being wrong. In fault diagnosis, it ranks the possible causes, runs the check that narrows them most, and stops when the evidence meets a rule you declared.

The workflow model is a path to safe intelligence

An agent is useful when the next action has to be discovered at run time. A workflow model is for processes you already know: it executes the steps you declare, and nothing else. It answers typed fields; it does not choose tools, call APIs or act on the world. Your application acts, after whatever checks you require.

Every step, and when it runs, is written down before the call. As models become more capable, the answers get better while what the model can do stays the same: the workflow, not the model, decides what can happen.

Safety also means knowing how sure the model is. With a probability on every decision, a workflow stops where the model is unsure, for a person to decide, instead of passing a guess along as a fact; and as the graph becomes probabilistic, that uncertainty reaches every step that depends on it.

The goal is a model that runs your workflow, and tells you where it might be wrong.

Chat models are optimised to generate sequences. Workflow models are designed to execute computation graphs. Declare the process once. Get its typed results in one call. Try a ticket workflow in the playground, or start with the quick start.