Define the operating requirement
Set the minimum acceptable quality and the limits on cost, latency, volume, auditability, and reliability before benchmarking.
Methodology
The SimplestWins standard.
A benchmark only matters if the decision can be reproduced. The backend must preserve the contract below rather than simply returning a preferred model.
Decision rule
The winner is the lowest-complexity method that satisfies every required constraint.
Every method must first be measured. Incomplete evidence produces no recommendation; a complete comparison with no passing method also names none.
Build the cheapest credible baseline first
Start with deterministic rules. Then add traditional ML, smaller models, and frontier models only when the task justifies them.
Evaluate on the same task and dataset
Every method must see the same inputs and be judged against the same labels or assertion-based success criteria.
Separate measured facts from estimates
Accuracy and latency come from benchmark runs. Cost can be computed from provider pricing, but every result stores its provenance and version.
Recommend by constraint, not hype
Once every method is measured, choose the least complex passing method. A cost estimate cannot break a tie; cost is considered only when all passing methods at that complexity have measured costs.
Keep every run reproducible
Dataset version, code commit, method configuration, model name, provider, prompt, and run timestamp must be saved with every result.
Performance
Accuracy, task success, latency, throughput, and error rate.
Economics
Cost per 1K tasks and projected cost at the selected operating volume.
Control
Determinism, auditability, reproducibility, privacy, and maintenance burden.
Complexity
Rules → traditional ML → small model → frontier LLM, unless the benchmark documents a different hierarchy.
The benchmark pages show measured results: tasks scored against public corpora, published beside the floors they had to beat. The assessment page answers a question about your task, which nobody has benchmarked — you answer structured questions yourself, or optionally ask a model to fill them from your description. Explicit rules assess those fields. That result is labelled ASSESSED, and it is reasoning, never a measurement of your system. Section 3.3 of the project standard governs the difference.
Benchmark detail pages expose the published class precision, recall, F1 and sample sizes. These diagnostics can reveal uneven errors between labels. They do not measure demographic fairness, production distribution shift, or the consequences of deployment. Rules and machine learning both need those checks.