Extracting document data using decision models

Our previous blog explained how we built an invoice data extraction pipeline with a decision model sitting at its core. Now, we have generalised that approach to support any document type, such as CVs, contracts, forms, etc. You can try it here.

This started with a user who uses Strom to extract very specific fields from the asset labels stuck on electronic gadgets, such as:

Two fields were of interest: model name and serial number. Their workflow looks like this:

1. The user takes a picture of the asset label with their mobile app.
2. The app performs on-device OCR and sends the extracted text to the decision model, with a request like the one below. The model names are known in advance, so they are the options of the first question. The serial number isn't, so the options of the second question are the lines OCR'd from the label:

POST https://api.uprelic.com/v1/systemone
{
    "model": "strom-1.0.7",
    "state": "HEWLETT PACKARD\nCOLOR DISPLAY/彩色顯示器/彩色显示器/모니터\nPRODUCT NUMBER: P5303D\nModel No / 型號 / 型号/모델명: PE1165U\nASSEMBLY P/N : 262243-421\n...\nSerial No. 序號/序号:\nCNG34124S0\n...",
    "questions": {
        "q1": {
            "type": "choice",
            "criteria": {
                "PE1165U": null,
                "PE1240": null,
                ...
            },
            "instructions": "What is the model name of the device?"
        },
        "q2": {
            "type": "choice",
            "criteria": {
                "PRODUCT NUMBER: P5303D": null,
                "ASSEMBLY P/N : 262243-421": null,
                "Serial No. 序號/序号:": null,
                "CNG34124S0": null,
                ...
            },
            "instructions": "What is the serial number of the device?"
        }
    }
}

3. The app uses the answers from the model to show the user the details of the device.

This works well with a decision model because:

1. It is fast: ~700 ms including the network round trip. This keeps the UX smooth.
2. It is cheap: ~€0.0001.

But this only works when:

1. The on-device OCR is accurate.
2. A flat list of lines is enough to tell which value belongs to which label. OCR turns "Serial No." and "CNG34124S0" into two separate lines, and their position on the label is lost. On a simple label, the model can still pair them. On a busy one, with several numbers and labels in several languages, it can't.

These limitations are not specific to product labels. Any document that contains text in an irregular layout can trip up this flow. So we built a new endpoint, POST /v1/extract, to address both.

Some interesting details about the endpoint:

1. Strom sees the layout, not just the text. Each line is sent with its page and position, together with an image of the page where every line is outlined and tagged with its ID. Strom can then pair a label with the value next to it or below it, the way a person reading the page would.
2. It runs OCR itself if you send an image or a scanned PDF. We use PP-OCRv6, one of the strongest open OCR models available, on our own GPUs, so the file never leaves our internal network.
3. It chains together multiple decision model calls to improve the data extraction. For example, after the first Strom request returns the top candidates for a field, we run Strom again on a smaller candidate set to pick between candidates that are about equally likely.

You describe the fields you want as a JSON Schema, for example "serial_number": {"type": "string"}, and get back for each field the line it was found on, a typed value, a confidence and its position on the page. To keep picking from a known list, as with the model names above, use an enum: "model_name": {"enum": ["PE1165U", "PE1240"]}.

You can try the endpoint here, or use the API in your existing setup.

Result from the API, visualised: