--- name: typellm description: Build Python applications with the hosted TypeLLM API. Use for typed LLM outputs, enum and boolean decisions with probabilities and confidence, scores on a rubric, numeric or text extraction, objects and arrays, boxes that find things in images, image input, per-field thinking, and dependency-driven workflows. metadata: author: TypeLLM --- # TypeLLM TypeLLM returns typed Python values from an LLM: each field you declare comes back as a `str`, `int`, `float`, `bool`, one of the values you listed, a `dict` or a `list`, never as text to parse. It calls the hosted TypeLLM API with an API key. ## Connect and generate Requires Python 3.10+: ```bash pip install -U typellm ``` Get an API key from https://typellm.ai/dashboard/keys (sign in by email; keys need an access code during early access) and keep it in the environment, never in source: ```bash export TYPELLM_API_KEY="tl-sk-..." ``` ```python import os from typellm import TypeLLMClient client = TypeLLMClient(api_key=os.environ["TYPELLM_API_KEY"]) response = client.generate( context="Receipt: Hilton London. Total: £324.50. Two nights for a client meeting.", questions={ "merchant": {"type": "string", "instructions": "Return the merchant name."}, "nights": {"type": "integer", "instructions": "How many nights?"}, "total": {"type": "number", "instructions": "Extract the total in GBP."}, "expense_type": { "type": "string", "enum": ["meal", "travel", "equipment"], "instructions": "Classify this expense.", }, "business_related": {"type": "boolean", "instructions": "Was this for business travel?"}, }, ) print(response.result) # {"merchant": "Hilton London", "nights": 2, "total": 324.5, # "expense_type": "travel", "business_related": True} ``` `generate` is synchronous and returns what the HTTP API does: `.result` (a dictionary, not a JSON string; use its values directly), `.thinking` and `.usage`. One client can be shared by many threads. ## Define fields - Pass `context` as a string and `questions` as a nonempty mapping of field names to definitions. Put field-specific instructions in `instructions`. - Types: `string`, `integer`, `number`, `boolean`, `object`, `array` and `box` (see below). Every declared field is evaluated. - Use `enum` for a finite choice: 2–512 distinct values matching the type (up to 26 inside an array's items), e.g. `{"type": "number", "enum": [0.0, 0.5, 1.0]}` returns the chosen candidate. - When the right choice depends on what each value means, write the candidates as `choices` instead of `enum`, each with a `description`: `"choices": [{"value": "billing", "description": "Payments and refunds."}, {"value": "other"}]`. - Allow a missing value with `"type": ["number", "null"]`; the field returns `None`. - Numeric fields do not accept `minimum` or `maximum`; use an enum for a fixed set of values. - String answers stop at 128 tokens. Ask for the length you want in `instructions` ("in one sentence"); `maxLength` is not supported. - `images=[...]` takes file paths, http(s) URLs, `data:` URIs, bytes or PIL images; the client reads and encodes them. At most 8 per call, 10 MB each. ## Objects and arrays An `object` groups typed `properties`; an `array` returns as many `items` as the input holds, each matching the item schema: ```python response = client.generate( context="Our team: Ada is the CTO, Lin is an engineer, and Sam runs sales.", questions={ "employees": { "type": "array", "instructions": "Return all employees.", "items": {"type": "object", "properties": { "name": {"type": "string"}, "role": {"type": "string"}, }}, }, }, ) # {"employees": [{"name": "Ada", "role": "CTO"}, {"name": "Lin", "role": "engineer"}, # {"name": "Sam", "role": "sales"}]} ``` - Properties take what a field takes. Inside array items, properties take `instructions`, `enum`, `choices` and null only: no thinking, `depends_on`, `when`, `return_probabilities` or scores. - `minItems` and `maxItems` bound an array: an array of objects adds at most 200 items per call, other arrays at most 50. Arrays of arrays and arrays inside objects are not supported. - For an input too long for one call, split it and pass the items so far as `"continue_from": items` on the array; each call returns the whole array. ## Boxes A `box` finds something in the image and returns where it is: `top`, `bottom`, `left` and `right`, each from 0 to 1 across the image, from its top-left corner. An array of boxes finds every match, from none to `maxItems`: ```python response = client.generate( context="Look at the attached photo.", images=["dogs.jpg"], questions={ "dog_head": {"type": "box", "instructions": "The dog's head, ears included."}, "dog_heads": {"type": "array", "items": {"type": "box"}, "instructions": "Every dog's head, ears included."}, }, ) # {"dog_head": {"top": 0.35, "bottom": 0.65, "left": 0.34, "right": 0.63}, # "dog_heads": [{"top": 0.31, ...}, ...]} ``` - A call with a box needs an image. Say what to find, and where it ends, in `instructions`. - There is always a box. To check that the target is there, ask a boolean first and give the box a `when` on it. - A box cannot be nullable, take `enum`, `choices` or `return_probabilities`, sit inside an object's or an array's items, or be tested by a `when`. - An array of boxes takes only `{"type": "box"}` as its items, up to 50 boxes; `continue_from` is not supported. ## Dependencies Fields run in parallel by default. When one needs earlier answers, list them in `depends_on`; it then sees those answers, and the call follows the graph: ```python response = client.generate( context="The payments service started failing after a deployment.", questions={ "severity": {"type": "string", "enum": ["low", "medium", "high"], "instructions": "Assess the incident severity."}, "rollback": {"type": "boolean", "depends_on": ["severity"], "instructions": "Given the incident and its severity, should we roll back?"}, }, ) ``` Dependencies may reference fields declared later; unknown names, duplicates, self-dependencies and cycles are rejected. Returned keys keep declaration order. Dependencies do not interpolate instructions or change enums. `"when"` runs a field only if other fields' answers pass tests: a value (equals), a list or `{"in": [...]}`, `{"not_in": [...]}`, `{"ne": x}`, and on number fields `{"gt" | "gte" | "lt" | "lte": x}`; every test must pass. Use these words, not `>=` or `$gte`. The fields `when` names become dependencies, so `depends_on` need not list them. A skipped field has no key in `response.result` (check `response.skipped` or use `.get()`), and fields depending on it are skipped too: ```python "approval": {"type": "boolean", "when": {"amount": {"gte": 1000}}}, ``` `"source": "user"` makes your code answer a top-level string, number, integer or boolean field, never the model (hosted API only): a database lookup, your own API or tool, or a person. The call answers everything that does not wait on it, then returns `response.status == "needs_input"`, with the fields to ask in `response.needs_input`; `client.resume(response, answers)` goes on and returns every answer so far. A `when` on the field stops only when it passes: ```python "order_status": {"type": "string", "enum": ["shipped", "delivered", "lost"], "source": "user", "instructions": "The order's status in our system.", "when": {"order_id": {"ne": None}}}, ``` ## Thinking, probabilities, scores and sampling Thinking is set per field; the fields that think reason side by side: ```python context = "Travel policy: hotels up to £350 a night. Expense: Hilton London, one night for a client meeting, £324.50." response = client.generate(context=context, questions={ "total": {"type": "number"}, "policy_ok": {"type": "boolean", "instructions": "Within the travel policy?", "thinking": True, "thinking_budget": 1024}, }) response.result # {"total": 324.5, "policy_ok": True}: the typed answers response.thinking # {"policy_ok": "..."}: reasoning per field response.usage # input_tokens and thinking_tokens: what the call is billed for ``` A field that thinks without a `thinking_budget` gets 1,024, and a budget may be at most 4,096. Thinking and its budget are set per field only; do not pass `thinking=` or `thinking_budget=` to `TypeLLMClient`. `"thinking_effort"` sets the budget by level: `"none"` (no thinking), `"low"` (512), `"medium"` (2,048) or `"high"` (4,096). `"thinking": "auto"` lets TypeLLM choose the level on each call, and `response.thinking_effort` reports it, e.g. `{"policy_ok": "low"}`. Do not combine `thinking_effort` with `thinking_budget`, or either of them with `"auto"`. Add `"return_probabilities": True` to an enum, choices or boolean field to get `{"value": ..., "probabilities": {...}, "confidence": ...}`. The probabilities are averaged over option orders by default; `"permutations": 1` reads the options once, as listed. `confidence` runs from 0 (no preference) to 1 (all probability on one answer): route low-confidence answers to review. A score rates the input on an ordered scale: a `number` field with `levels`, lowest first. It returns the probability-weighted average of the level indices (0 to one less than the number of levels): ```python "severity": {"type": "number", "instructions": "How severe is this issue?", "levels": [{"label": "Cosmetic"}, {"label": "Workaround available"}, {"label": "Fully blocked", "description": "No workaround."}]}, ``` A score takes no `permutations`, cannot be nullable or inside an array, and `when` compares it with `gt`, `gte`, `lt` or `lte`. `temperature` (0 by default, the most likely answer; above 0 samples) and `seed` can be set on the client or per `generate` call, as can `timeout` (seconds; 60 by default and at most 90; raises `GenerationTimeout`). ## Handle failures Import `SchemaError`, `SGLangError` and `GenerationTimeout` from `typellm`. `SchemaError` means a field definition is invalid. `ValueError` means invalid settings. HTTP errors raise `SGLangError` with the status in `.status`: 401 is a bad key, 400 `limit_exceeded` a request over a limit, 402 no credits left, and 429 too many requests a minute (200 per account) or too many calls at once (retry after the `Retry-After` header; for a higher limit, contact https://typellm.ai/contact). Preserve errors rather than inventing answers; typed output does not make an answer true. Validate the integration with one small request. If no API key is available, say that inference was not tested. ## Further documentation - [Quick start](https://typellm.ai/docs/quick-start) - [Output types](https://typellm.ai/docs/output-types) - [Thinking](https://typellm.ai/docs/thinking) - [Image input](https://typellm.ai/docs/images) - [Workflows](https://typellm.ai/docs/workflows) - [Probabilities](https://typellm.ai/docs/probabilities) - [Client reference](https://typellm.ai/docs/reference) - [API reference](https://typellm.ai/docs/api) - Code written for OpenAI's Decisions API or Jev's System One runs on TypeLLM with the base URL changed: [OpenAI Decisions](https://typellm.ai/docs/openai-decisions), [Jev System One](https://typellm.ai/docs/jev-system-one)