TypeLLM

Tools and human input in a workflow

Open in Colab →View on GitHub

A support email comes in. To answer it, the model needs facts it cannot read from the email: the order's status from your database, and for a lost order, whether an agent approves a refund. Fields with "source": "user" are answered by your code, not the model. The call answers everything it can, stops with status "needs_input", and client.resume() goes on with your answers.

Set up

Install the library:

pip install "typellm>=0.7.0"

Get an API key from your dashboard:

from typellm import TypeLLMClient

client = TypeLLMClient(api_key="YOUR_API_KEY")

EMAILS holds three support emails: one asks where an order is, one reports an order that never arrived, and one has nothing to do with orders. See the notebook for them.

Write the workflow

The model reads the order number. Your database gives the order's status, only when there is an order number. An agent decides a refund, only for a lost order. The model then writes the reply that fits: about the order, about the refund, or a general one when there is no order.

QUESTIONS = {
    "order_id": {"type": ["string", "null"], "instructions": "The order number in the email, digits only."},
    "order_status": {
        "type": "string",
        "enum": ["processing", "shipped", "delivered", "lost", "not_found"],
        "source": "user",
        "instructions": "The order's status in our system.",
        "when": {"order_id": {"ne": None}},
    },
    "refund_approved": {
        "type": "boolean",
        "source": "user",
        "instructions": "Has a support agent approved a refund?",
        "when": {"order_status": "lost"},
    },
    "reply": {
        "type": "string",
        "instructions": "Reply to the customer about their order's status.",
        "when": {"order_status": {"not_in": ["lost"]}},
    },
    "refund_reply": {
        "type": "string",
        "instructions": "Reply to the customer about their lost order and the refund decision.",
        "depends_on": ["refund_approved"],
    },
    "general_reply": {
        "type": "string",
        "instructions": "Reply to the customer.",
        "when": {"order_id": None},
    },
}

when decides both what your side is asked and which reply is written. A field that waits on a skipped one is skipped too, so each email gets exactly one of the three replies.

Answer from your side

Each field with "source": "user" gets a function that answers it from what the call has so far. Here a dictionary stands in for the order database, and the agent approves every refund; in your code, query the database, call your API, or put the question in front of a person.

ORDERS = {"8812": "shipped", "8840": "lost"}  # your order database


def lookup_order(result):
    return ORDERS.get(result["order_id"], "not_found")


def ask_agent(result):
    return True  # in your code: ask a person, and wait for the answer


ANSWER = {"order_status": lookup_order, "refund_approved": ask_agent}


def run(email):
    """Answer an email: stop for what our side answers, answer it, and go on until done."""
    response = client.generate(context=email, questions=QUESTIONS)
    while response.status == "needs_input":
        answers = {name: ANSWER[name](response.result) for name in response.needs_input}
        response = client.resume(response, answers)
    return response

response.needs_input names the fields the call is waiting for, with their type and instructions, and response.result holds every answer so far. A call can stop more than once: when one of your answers decides that another field applies, the call stops for that one next.

Final result

for email in EMAILS:
    response = run(email)

The first email stops once, for the order's status:

Hi, where is my order #8812? I ordered it last week and haven't heard anything. Thanks, Dana
  stopped for ['order_status'], with {'order_id': '8812'}
  order_id: 8812
  order_status: shipped
  reply: Hi Dana, your order #8812 has been shipped. You should receive it soon. Thanks for your patience!

The second stops twice: for the status, then, since the order is lost, for the agent:

Order 8840 still hasn't arrived and it's been three weeks. [...] I'd like my money back. - Sam
  stopped for ['order_status'], with {'order_id': '8840'}
  stopped for ['refund_approved'], with {'order_id': '8840', 'order_status': 'lost'}
  order_id: 8840
  order_status: lost
  refund_approved: True
  refund_reply: Hi Sam, I am very sorry to hear that your order (8840) has not arrived. I have checked
    our system and confirmed that the package is marked as lost. Since the item is no longer in transit,
    I have approved a full refund for you. [...]

The third has no order number, so it asks nothing of your side and finishes in one call:

How do I change the email address on my account? - Lee
  order_id: None
  general_reply: Hi Lee, to change the email address on your account, please log in and navigate to the
    'Account Settings' or 'Profile' section. [...]

Each email is read once: a resumed call goes on from where the call stopped and is billed only for your answers. A paused call can wait for up to 24 hours, so a person has time to answer. See Tools and human input for the details. The notebook runs all of it with your key: open it in Colab.

← all recipes