Picking a Model
Do not start with the leaderboard. Start with the job the model needs to do.
Picking a model is not a personality test. You are matching a workload to constraints.
Ask the boring questions first
What is the task? Chat, code, extraction, classification, embeddings, reranking, long-document analysis, image understanding, or agentic tool use all stress different capabilities.
How wrong can it be? A writing assistant can be a little weird. A refund policy extractor needs to be boring and consistent.
How fast does it need to be? Latency changes product feel. A slow model can be fine for background analysis and terrible inside autocomplete.
What is the budget? Cheap models win when the task is narrow and volume is high. Expensive models win when mistakes cost more than tokens.
What data goes in? Privacy, compliance, and customer contracts can decide local, self-hosted, or specific cloud providers before benchmarks matter.
The sane workflow
Pick two or three candidates. Build a tiny eval set from real examples. Measure output quality, latency, cost, and failure modes. Then choose.
Leaderboards help you shortlist. Your own evals decide.