Athena · SignalRank — public calibration snapshot
Which model should you trust, for which question? Measured, not assumed.
SignalRank runs a known-answer challenge harness across frontier and open models, scores every response against ground truth, and maintains a per-domain trust calibration. This is a static snapshot of the live corpus.
440
known-answer questions
10,791
challenges executed
The honest number: router accuracy is currently 16.3% (116/713) — the harness measures well; routing on those measurements is the open research problem. Better priors, decay modelling and per-domain sample efficiency are exactly where academic collaboration takes this next. We publish the weak number because the method only works if the numbers are real.
Trust is also an optimisation engine. Look at the market-analysis table below: a model priced at a fraction of the flagships outperforms them in that domain. Evidence-based routing therefore cuts token spend and raises answer quality at the same time — send each question to the model the data says wins it, at the price that model actually costs. At enterprise volume, calibrated routing is a unit-economics instrument, not just a safety one.
Medical
Top of table — clinical reasoning, guidelines, pharmacology
| claude-opus-4-7 | | 0.944 | 37 |
| gpt-5-mini | | 0.930 | 52 |
| claude-opus-4-8 | | 0.924 | 31 |
| grok-4 | | 0.919 | 63 |
Market analysis
Domain inversion: a small model leads — this is why per-domain calibration exists
| gpt-5-mini | | 0.938 | 10 |
| gpt-5.4 | | 0.928 | 10 |
| grok-4-fast | | 0.809 | 14 |
| claude-opus-4-7 | | 0.783 | 12 |
Legal
Statutory interpretation, case analysis, drafting
| mimo-v2.5-pro | | 0.905 | 6 |
| claude-opus-4-7 | | 0.890 | 7 |
| grok-4-fast | | 0.864 | 10 |
| claude-sonnet-4-6 | | 0.848 | 7 |
Adversarial robustness
Trick premises, injection attempts, manipulative framing
| claude-opus-4-7 | | 0.922 | 62 |
| claude-opus-4-8 | | 0.916 | 17 |
| claude-sonnet-4-6 | | 0.910 | 62 |
| gpt-5.5 | | 0.865 | 17 |
Temporal / live knowledge
Every model degrades — the harness measures knowledge decay directly
| gpt-5.5 | | 0.681 | 3 |
| claude-opus-4-7 | | 0.665 | 13 |
| grok-4 | | 0.591 | 14 |
| deepseek-chat | | 0.471 | 14 |
Method, in brief
Every model in the roster answers the same known-answer questions per domain; responses are scored against ground truth by an independent judge model; scores update a per-model, per-domain trust calibration with recency weighting. Human corrections enter the corpus as new ground truth. Low-scoring outliers (visible in the full tables) are retained, not pruned — some reflect integration or refusal-policy artifacts rather than capability, and distinguishing those is itself part of the calibration research.
Roster note: model list reflects the snapshot date; newest-tier additions (including Anthropic's Fable/Mythos class) are onboarded as calibration batches complete.