TypeLLM

Discover a schema from your documents

Open in Colab →View on GitHub

You have a pile of rental listings and want them as a table, but no list of columns yet. Let TypeLLM write the schema: show it a few sample listings and ask for the fields, each with a name, a type and a description. Then turn those fields into questions and extract every listing with them.

Set up

Install the library:

pip install "typellm>=0.6.9" pandas

Get an API key from your dashboard:

from typellm import TypeLLMClient

client = TypeLLMClient(api_key="YOUR_API_KEY")

LISTINGS is a list of 6 rental listings, a few lines each; see the notebook for details.

Describe a field: the meta schema

The meta schema is an array of fields, up to 30. Each field has a name, a type and a description:

META_SCHEMA = {
    "type": "array",
    "maxItems": 30,
    "items": {
        "type": "object",
        "properties": {
            "name": {"type": "string"},
            "type": {"type": "string", "enum": ["string", "integer", "number", "boolean"]},
            "description": {"type": "string"},
        },
    },
    "instructions": "The fields of a table that holds one row per document like these. "
                    "One field per fact the documents state, named in snake_case.",
}

Discover the fields from three samples

Pick samples that differ, so the fields cover what the others say too:

samples = "\n\n".join(LISTINGS[i] for i in (0, 1, 3))
fields = client.generate(context=f"Sample documents:\n\n{samples}",
                         questions={"fields": META_SCHEMA}).result["fields"]
price_per_month      number   The monthly rental price in USD.
bedrooms             integer  The number of bedrooms in the unit.
bathrooms            integer  The number of bathrooms in the unit.
square_feet          integer  The total area of the unit in square feet.
location             string   The neighborhood or specific area where the unit is located.
available_date       string   The date when the unit becomes available for move-in.
pets_allowed         boolean  Whether pets are permitted in the unit.
furnished            boolean  Whether the unit is furnished.
lease_term_months    integer  The minimum lease duration in months.
parking_included     boolean  Whether a parking spot is included in the rental price.
deposit_months       number   The security deposit required, expressed as a multiple of the monthly rent.
floor_number         integer  The floor number of the unit within the building.
in_unit_laundry      boolean  Whether the unit has a washer and dryer inside the apartment.
utilities_included   boolean  Whether utilities are included in the monthly rent.
street_parking_only  boolean  Whether the unit offers only street parking and no dedicated or included parking spots.

Turn the fields into questions

Each field becomes a question. Any listing may leave a field out, so every field takes null. Review the fields first if you like: rename, drop or add some.

QUESTIONS = {field["name"]: {"type": [field["type"], "null"], "instructions": field["description"]}
             for field in fields}

Final result

Extract every listing with the discovered questions:

import pandas as pd

rows = [client.generate(context=listing, questions=QUESTIONS).result for listing in LISTINGS]
table = pd.DataFrame(rows).convert_dtypes()
price_per_month bedrooms bathrooms square_feet              location available_date
           3450        2         1         850      Mission District          Nov 1
           1795                            420 Lake Merritt, Oakland         Oct 20
           5200        3         2        1600        Berkeley hills   December 1st
           3900        1                                        SoMa            now
           1350        4                                Inner Sunset         Nov 15
           4600        2         2        1050             Palo Alto      January 5

The first six of 15 columns. A listing that does not say its size or bathrooms leaves them empty. The notebook runs all of it with your key: open it in Colab.

← all recipes