A new AI model tops the leaderboard almost every week, and the tests it tops are things like multiple-choice trivia and math olympiads. None of that tells you whether it can fix your invoices or draft your client emails. A personal AI benchmark answers that with your own work as the test, and you can build one this weekend.
Why the public leaderboards keep lying to you
Nobody hires a vice president based on SAT scores. Yet every model launch gets judged on benchmarks like MMLU-Pro, which (this is real) asks models for the cranial capacity of Homo erectus and the place named in the title of a Cheap Trick live album from 1979. Wharton professor Ethan Mollick has been pointing this out for a while: those scores are directionally useful, but they say nothing about whether a model helps with your job.
There’s a gaming problem, too. Last year Meta submitted a customized build of a Llama model to a popular voting arena without making that clear, and researchers keep finding that voting arenas reward being likable over being capable. Benchmark questions also leak into training data, so scores inflate without the models getting better at anything you care about. The result: a flagship plan and a free model can look identical on paper and behave completely differently in your inbox.
The fix: test models on your own failures
Here’s the mindset shift, borrowed from developer Steve Yegge. Whenever a model fails a task you genuinely needed done, that task goes in your back pocket. “Then every time a new model drops, it’s like Christmas,” as Yegge puts it. Failed tasks are the only tests that actually separate models, because everything easy has been solved.
Your personal AI benchmark is just that pocket list, formalized: ten tasks from your real work, run through every model you’re considering, scored by rules you set.
Step 1: collect your ten tasks
Scroll back through the past month of AI chats and find the tasks that went badly: lots of back-and-forth, wrong output, or you gave up entirely. Aim for ten. If you use a desktop app like Claude or ChatGPT, you can even have the AI help, with a prompt like this:
Go through our session history for the past month and find tasks that needed heavy back-and-forth or were abandoned. Aim for at least ten. Collect the prompts and any files into a folder for me to review.
For each task, save everything an intern would need: the prompt, the input files, the instructions you’d give a human, and the good version of the output if one exists. Your context is the secret sauce here. “Rewrite this email” is a weak test; “rewrite this email using the tone from these three examples, for this client who hates exclamation points” is a real one.
Step 2: run every model through the same tasks
Now the fun part. Fire up the models you’re choosing between and give each one the identical task, identical files, identical instructions. Three comparisons matter most:
- Big vs small: the flagship plan against the cheap tier of the same provider
- Provider vs provider: Claude against ChatGPT and Gemini, on your actual work
- Same model twice: run one task three times to see how wild the variance gets
If you use an agentic tool like Claude Code, you can ask it to run all ten tasks across models in parallel and build a simple HTML page that shows the outputs side by side. It’s a twenty-minute setup that saves an afternoon of tab-switching.
Before the next step, a quick reality check on what you’re replacing:
| Public benchmarks | Your personal AI benchmark | |
|---|---|---|
| What it measures | Trivia, math, canned puzzles | The tasks you actually get paid for |
| Who it fits | Researchers and marketing teams | You |
| Shelf life | Saturates within months | Grows as your work changes |
| Cost to run | Free to read, useless to act on | A few dollars in API credits |
| Decision value | “Which model is smartest?” | “Which model should I pay for?” |
Step 3: turn your taste into pass/fail checks
Reading ten outputs side by side builds instinct, but instinct drifts. On a good day everything looks brilliant; after a bad meeting, everything sounds like a robot. So write your opinions down and make the machine grade itself.
The trick is called LLM-as-a-judge. Open voice mode on your main model, look at each output, and say what you think. Maybe “this headline was a banger,” or “this one sounds like AI writing,” or a flat “it ignored my format.” Then ask the model to turn that feedback into a rubric of yes-or-no checks it can apply to future outputs. One model does the task, and a second one grades it against your rules.
Keep the checks binary. Models are terrible at scoring things out of 100; their ratings bunch up around 70 no matter what, so a 72 versus a 65 means nothing. “Is the title boring, yes or no” beats any number. Three to ten checks per task is the sweet spot.
Step 4: re-run it every time a model launches
Benchmarks expire. Mollick used to ask models to draw “an otter on a plane using wifi” because almost none could. Now every model passes it, so he escalated to weirder prompts. Your tasks will saturate the same way, and when they do, add harder ones.
There’s a deeper reason to keep raising the bar. Yegge calls it the discernment horizon: on easy tasks, every model clears the bar and they all look identical. The differences only show up on ambitious, open-ended work. When Every’s head of evals ran his own consulting benchmark, the company’s cheapest model came within a few points of the flagship on routine tasks, because routine tasks don’t give smart models room to pull ahead.
That’s also your routing playbook, not just a fun fact. Once your personal AI benchmark shows the cheap model passing your routine checks, send it the routine work and save the expensive one for the tasks where it actually separates. We wrote a full guide on cutting AI costs with model routing if you want to go deeper there, plus one on when free open models are genuinely enough.
What this saves you
Money first. People routinely overpay for flagship models out of habit, and a personal AI benchmark is often all it takes to notice the cheaper plan passes everything you actually do. Time second: you stop decoding launch posts to figure out which model is “best” this week, because you have your own answer. Confidence last: when a teammate asks why you’re paying for X over Y, you have outputs to show, not vibes.
One caveat worth naming: don’t benchmark on stale knowledge questions. Models have training cutoffs, and testing trivia mostly measures when the training data ended, which is its own topic we covered in why your chatbot feels outdated.
Build your first personal AI benchmark this weekend
Don’t build the ten-task cathedral on day one. Start with three tasks that failed this month, two models you’re torn between, and three yes-or-no checks each. That’s an hour, and it already beats every leaderboard on the internet for your purposes. Grow it to ten tasks as new failures pile up, and re-run the whole thing whenever a model you care about launches.
The models will keep improving without asking you. The only edge you control is knowing, with evidence, which one deserves your ten bucks a month.
