Send low-confidence decisions to a person

Comments come in, and each has to stay up or come down under your community rules. Most are clear; a few are not. Let the model decide the clear ones, and stop for a moderator only when it is unsure: its confidence decides which comments reach a person.

Set up

Install the library:

pip install "typellm>=0.7.0" pandas

Get an API key from your dashboard:

from typellm import TypeLLMClient

client = TypeLLMClient(api_key="YOUR_API_KEY")

RULES holds the community rules, and COMMENTS eight comments, from plain praise and spam to sarcasm and a link to the commenter's own blog. See the notebook for them.

Write the workflow

decision returns its probabilities, so it has a confidence. moderator is answered by your side, with "source": "user", and only when that confidence is below 0.8. Each path writes its own notice to the author: notice after a sure decision, reviewed_notice after the moderator's.

QUESTIONS = {
    "decision": {
        "type": "string",
        "enum": ["allow", "remove"],
        "return_probabilities": True,
        "instructions": "Should this comment stay up under the community rules?",
    },
    "moderator": {
        "type": "string",
        "enum": ["allow", "remove"],
        "source": "user",
        "instructions": "A moderator's decision on this comment.",
        "when": {"decision": {"confidence": {"lt": 0.8}}},
    },
    "notice": {
        "type": "string",
        "depends_on": ["decision"],
        "instructions": "One sentence to the comment's author about the decision.",
        "when": {"decision": {"confidence": {"gte": 0.8}}},
    },
    "reviewed_notice": {
        "type": "string",
        "depends_on": ["moderator"],
        "instructions": "One sentence to the comment's author about the moderator's decision.",
    },
}

Moderate a batch

Every comment is one call, and the calls run side by side. A sure decision finishes its call; an unsure one stops it with status "needs_input":

from concurrent.futures import ThreadPoolExecutor


def moderate(comment):
    return client.generate(context=f"{RULES}\n\nComment: {comment}", questions=QUESTIONS)


with ThreadPoolExecutor(8) as pool:
    responses = list(pool.map(moderate, COMMENTS))

The model is sure of spam, insults and praise, and of harsh criticism of a library, which the rules allow. The three it is unsure of are the ones a person would hesitate over too: sarcasm, a jab at the author rather than the post, and a link that could be spam or a helpful addition.

Let the moderator decide the unsure ones

Here the moderator's decisions are a dictionary; in your code, put the comment in a review queue and resume when a person has decided. A stopped call can wait up to 24 hours, and resuming does not start it over.

MODERATOR = {
    "Wow, what a genius take. Truly, nobody has ever thought of this before.": "allow",
    "People like the author shouldn't be allowed to write about security.": "remove",
    "Check out my blog post on the same topic, I think it adds some context: example.com/caching": "allow",
}

for i, (comment, response) in enumerate(zip(COMMENTS, responses)):
    if response.status == "needs_input":
        responses[i] = client.resume(response, {"moderator": MODERATOR[comment]})

Final result

Five comments were decided at once; a person saw three. The threshold sets that balance: raise it and more comments reach a moderator, lower it and fewer do. Set it on a labelled sample of your own comments, at the lowest value that still gives the accuracy you need. See Tools and human input for how a call stops and resumes. The notebook runs all of it with your key: open it in Colab.