Probabilities & sampling

Add return_probabilities=True to an enum or boolean field. That field returns a value and its candidate distribution; other fields still return plain values.

result = client.generate(
    context="The expense was a train ticket.",
    questions={
        "category": {
            "type": "string",
            "enum": ["meal", "travel", "equipment"],
            "instructions": "Classify the expense.",
            "return_probabilities": True,
        },
    },
)

Example return:

{
    "category": {
        "value": "travel",
        "probabilities": {"meal": 0.04, "travel": 0.93, "equipment": 0.03},
    },
}

Argmax is the default. Use mode="sample" to sample instead. Probabilities describe the supplied candidates, not the likelihood that an answer is factually correct.

Return format and sampling rules

• return_probabilities defaults to false and must be a boolean. It is only accepted on enum and boolean fields, not open numeric or text fields.
• Probability-map keys retain enum candidate values. Boolean probability keys are Python True and False; JSON serialization turns them into string keys. The selected value remains a Python bool.
• Probabilities sum to one over the allowed candidates; sampling temperature changes this distribution.
• mode="sample" requires a finite temperature greater than zero. The default temperature is 1.0, and the default seed is None.
• Sampling applies to candidate selection and open numeric/text generation. A seed controls client sampling but does not guarantee deterministic model reasoning or server execution.

client = TypeLLMClient(
    "http://127.0.0.1:30000",
    mode="sample",
    temperature=0.8,
    seed=42,
)

Per-question permutation averaging

Add permutations to an explicit enum field to average predictions over different option orders. TypeLLM aligns probabilities by candidate value before averaging.

client = TypeLLMClient("http://127.0.0.1:30000", seed=42)
result = client.generate(
    context="A single roll of a fair die.",
    questions={"roll": {
        "type": "string",
        "enum": ["one", "two", "three", "four", "five", "six"],
        "instructions": "What number will come up on this roll?",
        "permutations": 8,
        "return_probabilities": True,
    }},
)
print(result["roll"])

• Omit permutations or set it to 1 to use the original ordering once.
• A positive integer requests that many distinct orderings, capped at the number of possible permutations. Smaller budgets sample uniformly without replacement.
• "all" enumerates every ordering. The limit is 720 evaluated orderings per field; larger requests raise SchemaError. Use an integer budget for larger enums.
• Only explicit enums accept this option, including numeric and boolean enums. Setting it on a boolean without enum, open numeric field, or text field raises SchemaError.
• The return structure is unchanged. With return_probabilities=True, probabilities are the arithmetic mean; otherwise the field returns only the selected value.
• Argmax selects from the mean; sample mode applies temperature to each ordering’s distribution, averages, then samples once. Ties follow the original enum order.
• The client seed makes permutation sampling reproducible for the same sequence of calls. It does not guarantee deterministic backend predictions.

Variants are batch-scored with shared context. Each ordering still requires model computation; KV caching can reuse the common prefix. Batch, sequential, and DAG execution are supported. Later dependent fields see the final aggregated choice, not individual permutation results.

Full enumeration makes the averaged probabilities independent of the initial option order for fixed per-order predictions. Sampling approximates this; neither guarantees perfect calibration.

See the GPU-tested fair-die example for the script and recorded results, or read Can Jev roll a die?.