Latency
45%Lower p95 latency scores higher only when it comes from authenticated streaming inference runs.
Scoring methodology
Live benchmark rows now, deeper provider telemetry next.
The leaderboard reads from `benchmark_runs` in Neon Postgres. An hourly collection route refreshes the latest provider rows, but the UI computes a composite score only from authenticated inference measurements. When provider keys are not configured, public catalog probes keep source freshness visible while latency and throughput remain marked pending.
Collection contract
The live dataset ships through /api/leaderboard, while the scheduled collector writes fresh rows through /api/collect.
Cost
35%Provider price is normalized per 1M output tokens so routing policies compare like-for-like.
Throughput
20%Higher generated-token throughput improves score only after real completion measurements are available.
Current limitations
- Without provider API keys, the collector falls back to public docs/catalog probes and stores latency, throughput, TTFT, and token counts as unavailable.
- The app stores only the latest hourly snapshots, not long-horizon trend views or provider detail pages.
- Reliability, error-rate, and concurrency stress metrics still need their own collection pipeline.