Can Jev roll a die?

TypeSafe's newly launched Jev has been getting people talking about a different way to interact with AI: give it a decision to make, define the possible answers, and get back probabilities your software can use.

In its launch announcement, TypeSafe introduces Jev as its first System One Model, built for fast, structured decisions. For a classification task, you supply the categories, and Jev returns a choice, a probability for each option, and a confidence score. TypeSafe describes these probabilities as calibrated: the numbers should meaningfully reflect uncertainty.

That is an appealing interface. But how well do the probabilities behave when the uncertainty is something we already know?

Can Jev roll a die?

A fair-die experiment experiment was raised on X by @sanyamsatia, prompting this experiment. Here is the version we tested:

What number will come up on a single roll of a fair six-sided die?

A fair die gives each outcome a one-in-six chance (16.67%), making it a simple check of whether Jev's probabilities reflect the known uncertainty. We tried it with Jev using the official Python SDK.

from typesafe_sdk import Choice, TypeSafeClient

with TypeSafeClient() as client:
    response = client.system_one(
        state="A single roll of a fair die.",
        questions={"roll": Choice(
            instructions="what number will come up on a single roll of a fair six-sided die?",
            criteria=dict.fromkeys(["one", "two", "three", "four", "five", "six"]),
        )},
    )
print(response.choices["roll"].probabilities)

The standalone Jev call returned:

Why does Jev favor “one”?

Jev assigned an 83% probability to one, which was also the first option in our request. We have two hypotheses about what might be causing this bias:

• A preference for the answer itself. The label one, or its meaning, attracts more probability wherever it appears.
• A preference for position. The model favors the first option, which happens to be one here.

These can coexist. To tell them apart, we can move the options around and see whether the highest probability follows the answer or its position.

Does changing the order change the answer?

We tested all 720 possible orderings of the six options. Every label appeared exactly 120 times in every position.

We also wanted to know whether another model would behave differently. Using TypeLLM, we ran the same experiment with Qwen, with thinking disabled. For the original ordering, Qwen gave one 87.20%—so this first prediction was no closer to the expected 16.67%.

Across all orderings, however, a different pattern emerged:

Jev kept favoring one, wherever it appeared. Its probability for that label ranged from 77% to 91%. Qwen almost always favored whichever option came first.

This supports different explanations for the two systems. For Jev, the preference follows the label, although this experiment alone cannot tell us whether its cause is semantic or something else. For Qwen, the strong position preference suggests a simple intervention.

Reducing Position Bias with Permutation Averaging

A decision model should be permutation invariant when the options have no meaningful order: rearranging the same choices should not change their probabilities. The chance of rolling one should be the same whether it appears first or last.

We can achieve this by averaging over all possible orderings. For each ordering, we match the probabilities back to their labels and take their arithmetic mean. Because every ordering is included, the average is independent of the initial option order, assuming each ordering produces a fixed prediction. Sampling a smaller set of permutations approximates this approach. See pijev for details.

Here is what changed after averaging all 720 orderings:

Qwen on its own strongly favored the first answer. Qwen with permutation averaging came much closer to the fair-die probabilities. Its averaged probabilities range from 13.47% to 25.41%, against the expected 16.67% per face.

How many permutations do we need?

With more choices, enumerating every ordering quickly becomes expensive: six choices have 720 permutations, while ten have over 3.6 million. Instead, we can sample a small set of distinct permutations and average their predictions.

Each permutation is an independent prediction conditioned on the same context, so TypeLLM can batch them for parallel scoring. SGLang can reuse the shared context's KV cache, avoiding repeated computation of that prefix and keeping the extra overhead low when the shared context dominates the input. Each reordered question still requires its own scoring.

Even a small sample helps in this example. The chart measures error against a fair die using KL divergence: lower is better, and zero is a perfect match.

For TypeLLM + Qwen, averaging 8 permutations reduced error by 79%, rising to 97% with all 720, relative to its average single-order error. Jev improved only slightly.

Putting it together: TypeLLM + Qwen + permutation averaging

TypeLLM now handles the averaging for you. Add "permutations": 8 to an enum question to sample eight distinct orderings, or use "all" for every ordering. The returned probabilities are already aligned and averaged.

With Qwen served through SGLang, call:

from typellm import TypeLLMClient

client = TypeLLMClient(
    "http://127.0.0.1:30000",
    model="qwen3.8-27b",
    seed=42,
)
result = client.generate(
    context="A single roll of a fair die.",
    questions={"roll": {
        "type": "string",
        "enum": ["one", "two", "three", "four", "five", "six"],
        "instructions": "what number will come up on a single roll of a fair six-sided die?",
        "permutations": "all",
        "return_probabilities": True,
    }},
)
print(result["roll"])

To compare 1, 8, and all 720 orderings, run the included examples/fair_die/fair_die.py script. From the updated TypeLLM checkout containing this feature:

pip install -e .
python examples/fair_die/fair_die.py --url http://127.0.0.1:30000

The script expects RadixArk/Qwen3.8-27B-NVFP4-BF16-LMHead running on a GPU through SGLang with the served name qwen3.8-27b. Change --url to your server address. Results, including the probabilities and KL error for each budget, are saved to examples/fair_die/result.json.

For Jev users, pijev offers permutation averaging through a TypeSafe-compatible client. In this example, the strongest improvement came from combining Qwen with averaging.