How many "r"s are in "strawberry"?

Open in playground →

The letter count models famously get wrong. With thinking on, the count comes back as an integer, with a probability for every candidate.

Integer enum
Thinking
Probabilities

Challenges

A language model reads tokens, not letters, so the 'r's inside strawberry are not directly visible to it. Counting them is a well-known place for models to slip, and many answer 2.
With thinking on, the model spells the word out and counts before it answers, and its reasoning comes back with the result.
return_probabilities works on enum and boolean fields, so the count is an integer enum of 0 to 9. The answer is always one of those numbers, and each comes back with a probability.

import os
from typellm import TypeLLMClient

client = TypeLLMClient(api_key=os.environ["TYPELLM_API_KEY"])
response = client.generate(
    context="Counting \"r\"",
    questions={
      "strawberry": {
        "type": "integer",
        "enum": list(range(10)),
        "instructions": "How many times does the letter \"r\" appear in \"strawberry\"?",
        "thinking": True,
        "return_probabilities": True
      }
    },
)
print(response)

Time 2.10 s · Cost $0.000068

Generation(
    result={
        'strawberry': {
            'value': 3,
            'probabilities': {
                '0': 0,
                '1': 0,
                '2': 0,
                '3': 1,
                '4': 0,
                '5': 0,
                '6': 0,
                '7': 0,
                '8': 0,
                '9': 0,
            },
        },
    },
    thinking={
        'strawberry': 'We need answer user\'s simple counting question. Need produce final JSON {"strawberry": "<label>"}. Count \'r\' in "strawberry". Let\'s spell: s t r a w b e r r y. Letters: r at position 3, then r at position 8? Let\'s index: s(1) t(2) r(3) a(4) w(5) b(6) e(7) r(8) r(9) y(10). Total 3. Choice D. Need final only JSON likely. Ensure label D.',
    },
    usage=Usage(input_tokens=92, thinking_tokens=126),
)