Examples

How many "r"s are in "hello" and "strawberry"?

Open in playground →
  • Integer enum
  • Thinking: auto
  • Probabilities

Task

Count the letter "r" in "hello" and in "strawberry".

Challenges

  • A language model reads tokens, not letters, so the 'r's inside strawberry are not directly visible to it. Counting them is a well-known place for models to slip, and many answer 2. hello, with none, is easy.
  • With "thinking": "auto", each field first gets a level of thinking for this call. hello gets none and answers at once; strawberry gets some, spells the word out and counts. thinking_effort reports the level each one got.
  • return_probabilities works on enum and boolean fields, so each count is an integer enum of 0 to 9. The answer is always one of those numbers, and each comes back with a probability.

Laying out…

Time 0.67 s · Cost $0.000065

Generation(
    result={
        'hello': {
            'value': 0,
            'probabilities': {
                '0': 1,
                '1': 0,
                '2': 0,
                '3': 0,
                '4': 0,
                '5': 0,
                '6': 0,
                '7': 0,
                '8': 0,
                '9': 0,
            },
            'confidence': 1,
        },
        'strawberry': {
            'value': 3,
            'probabilities': {
                '0': 0,
                '1': 0,
                '2': 0,
                '3': 1,
                '4': 0,
                '5': 0,
                '6': 0,
                '7': 0,
                '8': 0,
                '9': 0,
            },
            'confidence': 1,
        },
    },
    thinking={
        'strawberry': 'We need answer user\'s request. Need count \'r\' in "strawberry". Let\'s count: s t r a w b e r r y. Letters: s(1), t(2), r(3), a(4), w(5), b(6), e(7), r(8), r(9), y(10). There are three r\'s? positions 3,8,9 = 3. Choice D: 3. Need output {"label": "D"}. Ensure final only JSON.',
    },
    usage=Usage(input_tokens=157, thinking_tokens=115),
    thinking_effort={
        'hello': 'none',
        'strawberry': 'low',
    },
)