Self-improving document extraction

Our document extraction API takes a PDF or an image and a list of fields, and returns each field's value with a confidence. It works well on most documents out of the box. But every customer's documents are a little different. One supplier prints "Amount paid" where another prints "Received with thanks", and a receipt from one restaurant chain puts the service charge where another puts the tax. A general setup gets those details wrong over and over, on the same fields, for the same customer.

So we built a way for the extraction to learn those details from each customer's own documents.

Harnesses

Strom is a decision model: it answers questions with calibrated probabilities. On its own, it doesn't read PDFs. A harness is the code around the model that turns a task into questions Strom can answer, and turns the answers back into a result. For document extraction, the harness reads the pages, finds the lines that could hold each field, asks Strom which line is the right one, and cleans the value up to the field's type.

The model is shared by everyone. The harness is where knowledge about one customer's documents can live, without touching the model. That's where self-improvement happens.

How self-improvement works

You turn it on under Billing, and your documents keep going through POST /v1/extract as before. Once enough new documents have come in, the harness looks at a sample of your recent documents, works out which of your fields it gets wrong, and adapts how it reads those fields for your account. A change goes live only if it does better on your own documents than the current setup. Otherwise it's thrown away.

A few things this is not:

• It's not training. Strom itself doesn't change, and your documents are never used to train or fine-tune a model.
• It's not shared. What the harness learns from your documents applies only to your account.
• It's not permanent. If you turn it off, your extraction goes back to the standard setup. Turn it back on, and the improvements apply again.

How we tested it

We used two public datasets with known correct answers:

• CORD: photos of shop and restaurant receipts from Indonesia. We extracted 7 fields: subtotal, tax, service charge, discount, total, cash paid and change.
• DocILE: scanned business documents, mostly invoices. We extracted 12 fields, including invoice number, order number, customer number, dates, totals and amounts paid and due.

For each dataset, we sent documents through the API with self-improvement on, and let the harness run one round of improvements. Then we measured accuracy before and after that round against the dataset's correct answers, on documents that weren't used to build the improvements. That shows how the changes carry over to documents the harness hasn't learned from.

Results

Dataset	Before	After	Gain
CORD receipts	78.1%	85.3%	+7.2 points
DocILE invoices	62.0%	71.8%	+9.8 points

"Before" and "after" are the share of all field answers that were correct, after one round.

The gains come from a few fields that were wrong in the same way on many documents:

Field	Dataset	Before	After
Service charge	CORD	65%	97%
Cash paid	CORD	80%	98%
Amount paid	DocILE	20%	93%
Order number	DocILE	28%	64%
Customer number	DocILE	67%	82%

Fields that were already right on most documents stayed about the same.

Not every change helped. On DocILE, invoice number went from 80% to 74% correct: the harness became stricter about what counts as an invoice number than the dataset is. We're working on catching changes like this before they go live.

What it costs

While self-improvement is on, POST /v1/extract costs twice its usual price. We store the documents you send for 90 days, so the harness has something to learn from. Turning it off returns the usual price and stops storing documents. Details are in our Terms, § 9.9.

You can turn it on for your account under Billing. The API docs explain the endpoint itself.

What's next

Self-improvement is in beta. We'll keep adding new ways for the harness to learn from your documents, so each round gets more out of them. And document extraction is only the first harness to get it: next, we're bringing self-improvement to others, starting with Agent guard.