strom-base: a bigger model for the hard decisions
We added a second, larger model to Strom. Here is where it beats strom-fast, where it doesn't, and what it costs.
Strom now has two models:
strom-fast(strom-1.0.7): the model we launched with. Fast, cheap, calibrated probabilities.strom-base(strom-base-1.0): a larger model, in beta. Slower and more expensive, and better at long, multi-step decisions.
You switch by changing one field:
{
"model": "strom-base",
"state": "...",
"questions": { ... }
}Everything else stays the same: the request format, the question types, the response. It also works on /v1/extract.
The rest of this post is the numbers, and four examples where the two models gave different answers.
How we tested it
We used the hard tier of JevBench, a public benchmark for decision models run by Benchmark Heaven. The public hard tier has 111 questions. Most of them are built so that a quick reading gives the wrong answer: the decisive fact is in a definition, a later amendment or a footnote, sometimes thousands of characters into the document.
We sent all 111 to both models through the normal public API, once each.
Results
| Type of question | Questions | strom-fast | strom-base |
|---|---|---|---|
| Multi-step lookups across documents | 18 | 12 | 17 |
| Long policy documents | 19 | 13 | 15 |
| Judging whether a response is correct | 17 | 14 | 15 |
| Ambiguous cases, probabilities, trade-offs | 23 | 20 | 23 |
| Date and number calculations | 15 | 7 | 6 |
| Adversarial inputs, routing, traps | 19 | 19 | 19 |
| Total | 111 | 85 (77%) | 95 (86%) |
strom-base got 14 questions right that strom-fast got wrong. The other way round, it was 4. Most of the gain is in the long, multi-step questions: 9 of the 14 had more than 8,000 characters of input.
It isn't better everywhere. On date and number calculations it came out one behind, and on the adversarial and trap questions both models got everything right.
Four questions where the answers differed
1. "Someone else's data is in our file"
A support ticket: an Enterprise customer's CSV export fails, and the one file that came through has rows from another tenant, tn_9120. They call it a data leak. But the routing policy defines an organization by Org ID, and tn_9120 is the customer's own subsidiary workspace on the same Org ID. No leak, just a bug, so it goes to engineering.
strom-fast: escalate to security (90%).strom-base: escalate to engineering. Correct.
2. The rounding mistake
Is this answer correct? "250 ft² × (0.3048 m/ft)² = 23.22576 m². To three decimal places: 23.225 m²." The method is right, but 23.22576 rounds to 23.226.
strom-fast: yes.strom-base: no. Correct.
3. The leap day
3 credits per active day, first and last day included. Active from February 28 to March 2, 2028, with February 29 suspended. That's 3 days, so 9 credits. Missing the suspension gives 12.
strom-fast: 12.strom-base: 9. Correct.
4. Six documents deep
About 10,000 characters of grant records, with look-alikes marked old or withdrawn. Following six references leads to an adopted amendment that sends anything above 45,000 to programme review. The total is 46,500.
strom-fast: release the money (92%).strom-base: programme review. Correct.
In examples 1 and 4 (cases where the text was built to mislead), strom-fast wasn't just wrong, it was sure of itself.
What it costs
| strom-fast | strom-base | |
|---|---|---|
| Price per 1M input tokens | $0.042 | $0.24 |
| Median time per request in this test | 0.61 s | 0.72 s |
| 90th percentile | 0.68 s | 1.13 s |
| Probabilities calibrated | Yes | No, not yet |
| Status | Stable | Beta |
The times above include the network latency.
For a single request, strom-base is only a little slower. The difference shows up when there are many questions on the same input. strom-fast reads the input once and reuses it for every question. strom-base reads the whole input again for each question.
Its probabilities are not calibrated yet. When it says 85%, don't read that as "right 85% of the time". If you have a threshold on confidence, set it again for strom-base; don't copy the one you use for strom-fast.
Which one to use
Use strom-fast by default. It's cheaper, faster, calibrated, and gets most decisions right.
Try strom-base when:
- the input is long, like a contract, a policy or a set of records,
- the answer depends on following several references or exceptions
strom-base is in beta and runs on a small cluster for now, so expect lower capacity than strom-fast. Tell us what you find at support@uprelic.com.