Editor’s note (October 7): Added five new models. Haiku 5.5 and Sonnet 5.5 are much better than their predecessors, but not competitive with GPT-6 Luna. On the open side, MiMo V2.6 Pro and Flash trail DeepSeek V4.1 Flash. GPT-6.1 Sol doesn’t improve on GPT-6 Sol.
My agentic OS reads every meeting I have. The transcripts come from Anarlog, an open-source, local-first notes app (similar to Granola) that records and transcribes the call, then hands the transcript to whatever model you pick to write the summary. I run my own fork, which adds custom summary templates and a Linux build of the command-line tool, so the server where my agents live can read my meetings. My chief-of-staff agent treats every summary as memory: who I talked to and what we agreed. Picking the model is on me.
The failure I worry about is a summary that’s confident and wrong. I can notice a thin summary easily. An invented detail gets written down as fact, and a week later it turns up in a brief or a draft under my name. I’ve been bitten by bad agent memory before (an old file once put a job I never held into an email draft), so which model writes these summaries matters to me.
So I stopped guessing and built a test.
Why not just read the leaderboard
Public benchmarks test general skill, mostly coding and math. This job is narrow and ugly: audio that drops words, speakers the transcript never names, company names the speech-to-text turns into something else, people talking over each other. A model can rank well on general skill and still invent a detail to smooth over a garbled sentence.
The only way to know which model handles my meetings is to test it on my meetings.
The yardstick
I picked five meetings from my own calendar, each because it breaks summaries in a different way:
- The longest, about 12,000 tokens (roughly 8,600 words). The other side is never named, and the transcription writes my last name over their company’s.
- Three people with only two speaker labels. It also mentions a company none of them works for, which invites a wrong guess.
- Coarse segmentation, where one chunk of text often spans two speakers.
- A short one, under 2,000 tokens (about 730 words), with everyone named and a clear decision at the end.
- A control, with real names on every line.
For each meeting I wrote down the facts of record by hand, split into what a model could know from the transcript and what it couldn’t. A model only gets marked down for the first kind. Then one rubric, one judge model (Opus 5, pinned so it couldn’t drift), and three runs per model per meeting at temperature 0. Over four rounds that came to 16 models and 231 graded summaries.
The rubric counts defects and ignores style. A critical is an error I’d act on wrongly: a wrong fact, a quote pinned on the wrong person, an outcome that’s hedged or buried, or a detail invented from nothing. The field turned up plenty of real ones. One summary said a company had “successfully built its platform” when the person I was talking to had just told me the platform sucks. Another took the transcription’s habit of writing my last name over other people’s and turned it into a person: an engineering executive named “Marquard.”
The best hour of the project came before any of that ran. Before I trusted the judge, I graded seven summaries of the short meeting myself and rejected two. One got a detail of my background wrong (it said I worked East Coast hours once a month, when I worked East Coast hours and traveled to the office once a month). The other explained the obstacle we’d discussed but never said what I decided. Then I ran the judge on the same seven, and it passed all of them.
The judge had seen both problems. Its notes on the second summary even said my decision was missing. But my rubric had no category for a buried outcome, and it filed a wrong detail like the first one as a lesser error that didn’t count. As written, nothing could fail.
I added both as critical errors and reran it. This time the judge caught both of mine, plus two I’d passed. It was right about one of those (another summary made the same East Coast hours mistake, and I’d missed it). Agreement then bounced between five and six of seven across the next few revisions, which told me I was tuning to seven samples. So I stopped.
The result
GPT-6 Sol averaged 0.20 critical errors per summary. GPT-6 Luna and Opus 5.5 followed at 0.47 and 0.67. Then there’s a cliff. The next best was the model I’d been running in production, GPT-5.6 Luna, at 1.20. All three leaders were released this week, so that cliff is what one round of releases from OpenAI and Anthropic bought on this task.
Price didn’t predict quality
In the chart at the top, GPT-6 Luna cost $0.0007 a meeting, the least of any paid model in the field, and placed second. Opus 5.5 was the priciest at about 10 cents and placed third. The top three span about 150× in price. (I’ve plotted Anthropic’s models at list price, even though mine ran on a subscription, so the comparison stays apples to apples.)
Price does track quality inside a family: Sol beats Luna, and Opus beats Sonnet beats Haiku. That’s the only place it told me anything.
At my volume the gap barely registers anyway. Sol against GPT-6 Luna comes to under a dollar a month for every meeting I have. At personal scale, cost is a tiebreaker at most.
Closed, open, local
The best open-weight models scored 1.27, six times the error rate of the best closed one. The best model I can run on my own Mac scored 1.67.
Closed isn’t a guarantee. If I’d picked by brand, Sonnet 5 looked like a safe choice; it placed 13th of 15 at 2.53. Haiku 4.5 came last at 3.47 (worse than Gemma 4 E4B, a free local model from Google small enough to run on a phone). The best closed models sit a tier above everything else right now. The rest of closed is spread across the whole field.
Open weight sits in a tight, middling band, with seven of eight between 1.27 and 1.67. Cheap and consistent, and never good enough to write into memory unreviewed.
Local costs nothing and gets more wrong. Apple’s on-device model on macOS 27 can’t fit three of my five meetings into its 8,000-token context, and it scored 3.17 on the two it could. I’d treat local as a privacy choice, not a quality one.
Do open weights catch up?
I expected the usual story, with open weights a few months behind the frontier. In my field it looks different. From April until GPT-5.6 Luna pulled even in July, an open model (DeepSeek V4 Pro) held the best score. Then GPT-6 and Opus 5.5 jumped well clear in September, while open weight has plateaued near 1.3 for five months.
Take the early lead with a grain of salt. I picked these models to make a production decision, so the field skips several closed releases from the spring (Opus 5, my judge, among them). The only closed models I tested from before July were Haiku and Sonnet. My guess is open weight closes the gap again at some point, but I wouldn’t bet on when.
Averages hide the bad day
Sol’s worst single run had one critical error. Haiku’s had nine. An average won’t show you that, and when the output is headed for memory, the worst run matters as much as the mean.
What kind of wrong
Sol made zero fabrications across 15 runs. Haiku made 18, and Nemotron 11.
The top three models made no critical errors of omission at all. Every critical they did make was a misreading, stated with confidence. Fewer errors doesn’t mean safer errors. I’ll catch an omission when I skim a summary. A confident misreading reads just like a fact.
Some meetings no model fixes
One meeting beat every model. On the long call where the other side is never named (meeting B in the chart above), none of the 45 runs across the field came back clean, and Sol missed it all three times. When the transcript never says who’s talking (and writes my name over their company’s), there’s nothing for a model to recover. That meeting needs a better transcript.
It’s the exception, though. Across the field, the model mattered far more than the meeting: averages ranged 17× across models (0.20 to 3.47) and only 3.4× across meetings.
What I’m switching to
I’m moving my meeting summaries to GPT-6 Luna. Sol scored slightly better, but at 15 runs each, 0.20 and 0.47 sit in the same tier. Luna is “good enough” for a summary I read before it goes into memory anyway, and it’s a bit cheaper. It’s also a big step up from the GPT-5.6 Luna I was running before: better on every measure at less than half the price.
Regardless, summaries stay drafts. No model in the field was clean often enough to write into memory without me reading it first.
What this test can’t tell you
- The judge is Opus 5, and one contestant is Opus 5.5. The grading is blind, though, and Anthropic’s models didn’t come out ahead: Opus 5.5 placed third behind two OpenAI models, and Haiku came last.
- I calibrated the judge on seven summaries of one short meeting. Treat the error counts as relative.
- Anthropic’s models ran through Claude Code’s command-line tool, which adds about 2,000 tokens of scaffolding. They’re consistent with each other and carry an asterisk against the rest.
- Nemotron ran 4-bit quantized, which may explain part of its deficit.
- Fifteen runs per model is enough to separate tiers. It can’t rank neighbors: 1.20 against 1.40 is a tie.
- Five meetings, all mine. The method should carry over. The rankings might not.
Build your own
The whole thing cost a few dollars in API calls, plus roughly $20 of judge time on my Claude subscription at list price. For a model that writes into something I rely on, that’s cheap.
- Pick the meetings that hurt. The ones with garbled audio and unnamed speakers, and the ones where a wrong outcome would cost you. Five is enough.
- Write the facts first. Before you read a single summary, write down what’s true and what a model could know from the input alone.
- Calibrate the rubric. Grade a handful by hand and run the judge against them. If it can’t fail anything, fix the rubric before you spend a dollar.
- Pin the judge. Model aliases move. Mine did mid-project.
- Run reps. Three per meeting at minimum. The bad day is what you’re testing for.
The same five steps work for a team shipping a feature where a model writes into something customers rely on.
My judge prompt is below, trimmed. Give it the transcript, your facts of record and one summary at a time.
My judge prompt (trimmed)
You are grading a machine-written summary of a meeting transcript.
Don't rate the prose. Find specific defects you can quote.
The bar: would a careful colleague file this summary as the permanent
record without reading the transcript first?
You get the transcript, a facts-of-record list, and the summary.
The facts come in three groups:
- DERIVABLE: stated in the transcript. Contradicting or omitting one
is a defect.
- KNOWN-ONLY: true, but not in the transcript. Leaving one out is not
a defect. Asserting something that contradicts one is.
- TRAP: a plausible guess the transcript invites but doesn't support.
Asserting one is a defect.
Critical defects:
- outcome: the decision (a yes, a no, a firm next step) is missing,
hedged, or buried, even if nothing in the summary is false
- factual_error: a number, date, title, role, or piece of someone's
history is wrong, however small
- attribution: words credited to the wrong speaker
- polarity: a no turned into a yes, or the reverse
- fabrication: a name, number, date, commitment, or quote with no
support in the transcript
Also log major defects (an unsupported inference, a floated idea
promoted to an action item, a topic left out entirely), but only
criticals fail a summary.
First, reading the summary alone, write one sentence on what it says
the meeting's outcome was. Then list each defect with its category,
the quote from the summary, the transcript line it conflicts with
(or null), and one sentence of explanation. Return JSON only.
An empty defect list is a valid answer.
A leaderboard tells you which model is best in general. Five of your own meetings tell you which one to trust with yours.