← All posts
Accuracy
How we calibrate confidence
A confidence of 0.9 should be right nine times in ten. Here is how we check that on every release, and what we change when it drifts.
·
6 min read
Every field Tolva returns carries a confidence between 0 and 1. The number is only useful if it means what it says: of all the fields we score at 0.9, about nine in ten should be correct. When that holds, you can set one threshold and trust it.
What calibration means here
We group a held-out set of fields by their score, in bands of 0.05, and compare each band’s score with its measured accuracy. A calibrated model puts every band on the diagonal. A band above the line is under-confident and sends good fields to review; a band below it is over-confident and lets wrong values through.
The set we measure on
The set is 1.6 million labelled fields from invoices, receipts, bank statements and handwritten forms, sampled to match the mix customers send. It never trains the model, and it changes only when a new document type ships.
What we change when it drifts
When a release moves a band more than two points off the diagonal, we refit the calibration layer on its own and leave the extraction model alone. In r2.6 that recalibration is what lifted handwritten forms from 96.2 % to 97.9 % without a new model.
The changelog lists the measured accuracy of each release, so you can see the drift yourself rather than take our word for it.