Which DeepSeek-R1 are you testing?

An invoice-review trial compares DeepSeek-R1 checkpoints by correction cost, latency and routing errors. A model that costs more per call can cost less per resolved case.

An accounts clerk receives an invoice for twelve cartons, a delivery note for ten and a credit for two. On the face of it, the credit settles the difference. Then the clerk notices the credit belongs to last month’s delivery. A model can multiply the quantities correctly and still recommend paying the wrong amount. This is a useful test of a reasoning model, provided the procurement team knows which model produced the answer.

Model evaluation A two-model route adds a third decision to test: whether each invoice went to the right model.

The models behind the name

On 20 January 2025, DeepSeek released R1 and made smaller models available alongside it. The full R1 was a mixture-of-experts model. DeepSeek listed 671 billion parameters in total and 37 billion activated per token. A selection of its internal expert networks contributes to each step. The operator still needs a way to hold or access the full set of weights. Taking the active count as a memory specification would understate the deployment task.

The six smaller models had a different origin. The team used examples generated by the larger model to train dense Qwen and Llama models, ranging from 1.5 billion to 70 billion parameters. In a dense model the same network participates in each step. Distillation teaches another network from examples, while quantisation changes the precision used to represent numbers within a model. The two operations yield different test candidates. DeepSeek documented the individual base models in its checkpoint inventory and described training in its January technical report.

Before requesting a quote, name the checkpoint in full. DeepSeek-R1-Distill-Qwen-32B and DeepSeek-R1-Distill-Llama-70B differ in both size and base model. If they score differently, the test measures the difference between those two complete systems. It cannot tell a buyer how much of the gap came from parameter count alone. Save the weights revision or hosted model version, runtime, precision, prompt template and generation limit with the result.

Make the invoice test hard in the right places

Build a small casebook from closed exceptions whose resolutions an accounts specialist can verify. Remove identifying details while preserving the relationships between dates, quantities, currencies and document identifiers. Keep all versions of the same dispute together when separating practice cases from the final test. Otherwise a revised invoice can turn up in the test after its near twin was used to tune the prompt.

Include ordinary discrepancies as well as the awkward ones. A row billed twice tests arithmetic and matching. Two invoices with identical amounts but different purchase orders test attribution. A missing credit note tests whether the system asks for the right evidence. Give each candidate the same packet. Ask for the document and line, the disputed amount and the next action. A specialist who cannot see the model name should score all three against the settled case. Record a confident recommendation to pay without support as its own failure class.

Set an output budget the production process can afford. A model that runs out of room before giving its final answer has failed that case, even if its partial working looks promising. Conversely, a longer explanation earns no extra credit when a clerk needs a line reference and an amount. Keep incomplete runs in the denominator and report output tokens and time to the usable answer.

Put a price on correction

Suppose two candidates each handle the same 100 cases. The following figures are invented to show the calculation. A specialist spends one routine minute checking each of the 100 submissions. A failed first pass takes another four minutes of a specialist’s time to resolve manually. Treat every corrected case as subsequently accepted. Assume US$45 an hour for review and a measured inference bill of US$2 for candidate A and US$7 for B.

Scroll horizontally to compare the figures.

Illustrative review costs for 100 invoice cases
Cost componentCandidate ACandidate B
Accepted on first pass88 of 10094 of 100
Routine review at US$0.75/minUS$75US$75
Extra correction time48 min / US$3624 min / US$18
Inference for all casesUS$2US$7
Cost per completed caseUS$1.13US$1.00

In this example, B costs US$5 more to run and avoids six corrections. Four minutes at the assumed wage cost US$3 per correction, so the six avoided corrections save US$18. After paying for the extra inference the net saving is US$13, or thirteen cents per completed case. If correction really took only one minute, six avoided corrections would save US$4.50 and A would be cheaper by fifty cents. Timing the reviewer can settle a purchasing decision that a model leaderboard cannot.

A severe error changes the decision again. If either model silently directs payment to the wrong account, an average cost per corrected case assumes a reviewer catches it. Put that failure in a separate acceptance criterion. Where the account match is an exact comparison of two trusted identifiers, implement the check directly and give the model the ambiguous items that still need interpretation.

Test the route you would actually buy

A team might send clean quantity mismatches to the smaller candidate and cases involving credits to the larger one. Test this rule on held-out input with no access to the eventual answer. A missing credit note is especially revealing because the word “credit” may never appear in the documents. Record difficult cases mistakenly sent down the inexpensive route, the cost of running both models when the first escalates, and the time the clerk spends waiting. Compare the full routing rule with using either model alone.

A 100-case exercise is a screening tool. One misclassification moves the overall pass rate by a percentage point. Five examples in a rare category say little about the next hundred of that kind. Keep the cutover reversible, retain the test packets and rerun them after a checkpoint or serving change. Ask reviewers to capture a newly observed error with its source lines so the next comparison can reproduce it.

For a team investigating hosted model access, bring that casebook and the exact model identifiers to a Cooma Model API discussion and confirm availability. If the experiment requires a dedicated deployment of specific model weights, the same casebook can accompany a dedicated inference enquiry. Give the accounts manager a list of invoice decisions the chosen configuration can support, including the exceptions a person still owns.