---
source_url: https://marquard.org/posts/meeting-notes-bakeoff/
title: "I ran a bake-off for my meeting notes. Price didn't help."
date: 2026-09-25
summary: "Public benchmarks measure general skill. A test on five of my own meetings, graded against facts I could check, told me which model to trust with my agent’s memory. Price had little to do with it."
categories:
  - "operator notes"
word_count: 2280
reading_time_minutes: 11
---


*__Editor's note (October 7):__ Added five new models. Haiku 5.5 and Sonnet 5.5 are much better than their predecessors, but not competitive with GPT-6 Luna. On the open side, MiMo V2.6 Pro and Flash trail DeepSeek V4.1 Flash. GPT-6.1 Sol doesn't improve on GPT-6 Sol.*

{{< chart src="price.svg" alt="Scatter of 18 paid models, critical errors per summary against cost per meeting on a log scale. GPT-6 Sol, GPT-6.1 Sol, GPT-6 Luna and Opus 5.5 sit alone at the bottom, spread from $0.0007 to about $0.10 a meeting, with Sonnet 5.5 just above them at 0.93; Sonnet 5 and Haiku 4.5 sit near the top despite mid-to-high prices." caption="Critical errors per summary vs. cost per meeting: 18 paid models, 306 graded summaries. Lower is better." >}}

My agentic OS reads every meeting I have. The transcripts come from [Anarlog](https://github.com/fastrepl/anarlog), an open-source, local-first notes app (similar to [Granola](https://www.granola.ai/)) that records and transcribes the call, then hands the transcript to whatever model you pick to write the summary. I run [my own fork](https://github.com/dave-atx/anarlog), which adds custom summary templates and a Linux build of the command-line tool, so the server where my agents live can read my meetings. My chief-of-staff agent treats every summary as memory: who I talked to and what we agreed. Picking the model is on me.

The failure I worry about is a summary that's confident and wrong. I can notice a thin summary easily. An invented detail gets written down as fact, and a week later it turns up in a brief or a draft under my name. I've been bitten by bad agent memory before (an old file once put a job I never held into an email draft), so which model writes these summaries matters to me.

So I stopped guessing and built a test.

## Why not just read the leaderboard

Public benchmarks test general skill, mostly coding and math. This job is narrow and ugly: audio that drops words, speakers the transcript never names, company names the speech-to-text turns into something else, people talking over each other. A model can rank well on general skill and still invent a detail to smooth over a garbled sentence.

The only way to know which model handles my meetings is to test it on my meetings.

## The yardstick

I picked five meetings from my own calendar, each because it breaks summaries in a different way:

- The longest, about 12,000 tokens (roughly 8,600 words). The other side is never named, and the transcription writes my last name over their company's.
- Three people with only two speaker labels. It also mentions a company none of them works for, which invites a wrong guess.
- Coarse segmentation, where one chunk of text often spans two speakers.
- A short one, under 2,000 tokens (about 730 words), with everyone named and a clear decision at the end.
- A control, with real names on every line.

For each meeting I wrote down the facts of record by hand, split into what a model could know from the transcript and what it couldn't. A model only gets marked down for the first kind. Then one rubric, one judge model (Opus 5, pinned so it couldn't drift), and three runs per model per meeting at temperature 0. Over four rounds that came to 16 models and 231 graded summaries.

The rubric counts defects and ignores style. A critical is an error I'd act on wrongly: a wrong fact, a quote pinned on the wrong person, an outcome that's hedged or buried, or a detail invented from nothing. The field turned up plenty of real ones. One summary said a company had "successfully built its platform" when the person I was talking to had just told me the platform sucks. Another took the transcription's habit of writing my last name over other people's and turned it into a person: an engineering executive named "Marquard."

The best hour of the project came before any of that ran. Before I trusted the judge, I graded seven summaries of the short meeting myself and rejected two. One got a detail of my background wrong (it said I worked East Coast hours once a month, when I worked East Coast hours and traveled to the office once a month). The other explained the obstacle we'd discussed but never said what I decided. Then I ran the judge on the same seven, and it passed all of them.

The judge had seen both problems. Its notes on the second summary even said my decision was missing. But my rubric had no category for a buried outcome, and it filed a wrong detail like the first one as a lesser error that didn't count. As written, nothing could fail.

I added both as critical errors and reran it. This time the judge caught both of mine, plus two I'd passed. It was right about one of those (another summary made the same East Coast hours mistake, and I'd missed it). Agreement then bounced between five and six of seven across the next few revisions, which told me I was tuning to seven samples. So I stopped.

{{< callout >}}
My first rubric couldn't fail anything. If your test can't fail, you don't have a test.
{{< /callout >}}

## The result

{{< chart src="kinds.svg" alt="Horizontal bars of critical errors across 15 runs for each model, split into confident misreadings and omissions. GPT-6 Sol has 3, GPT-6.1 Sol 5, GPT-6 Luna 7, Opus 5.5 10, Sonnet 5.5 14, all misreadings; the next model, GPT-5.6 Luna, has 18, and Haiku 4.5 has 52." caption="Critical errors across 15 runs per model, split by kind. Ranked by errors per summary." >}}

GPT-6 Sol averaged 0.20 critical errors per summary. GPT-6 Luna and Opus 5.5 followed at 0.47 and 0.67. Then there's a cliff. The next best was the model I'd been running in production, GPT-5.6 Luna, at 1.20. All three leaders were released this week, so that cliff is what one round of releases from OpenAI and Anthropic bought on this task.

## Price didn't predict quality

In the chart at the top, GPT-6 Luna cost $0.0007 a meeting, the least of any paid model in the field, and placed second. Opus 5.5 was the priciest at about 10 cents and placed third. The top three span about 150× in price. (I've plotted Anthropic's models at list price, even though mine ran on a subscription, so the comparison stays apples to apples.)

{{< callout >}}
The cheapest model I paid for placed second. The four most expensive placed 3rd, 13th, 5th and 15th. On this task, across vendors, price bought nothing.
{{< /callout >}}

Price does track quality inside a family: Sol beats Luna, and Opus beats Sonnet beats Haiku. That's the only place it told me anything.

At my volume the gap barely registers anyway. Sol against GPT-6 Luna comes to under a dollar a month for every meeting I have. At personal scale, cost is a tiebreaker at most.

## Closed, open, local

{{< chart src="tiers.svg" alt="Dot-and-whisker chart by tier. Closed cloud models (ten) run from 0.20 (GPT-6 Sol) to 3.47 (Haiku 4.5), median 1.07. Open-weight cloud models (eight) sit in a tight band from 1.27 to 2.27, median 1.40. Local models run from 1.67 to 3.13, median 2.40." caption="Critical errors per summary by tier. Dot is the tier median; whisker runs from best to worst model." >}}

The best open-weight models scored 1.27, six times the error rate of the best closed one. The best model I can run on my own Mac scored 1.67.

Closed isn't a guarantee. If I'd picked by brand, Sonnet 5 looked like a safe choice; it placed 13th of 15 at 2.53. Haiku 4.5 came last at 3.47 (worse than Gemma 4 E4B, a free local model from Google small enough to run on a phone). The best closed models sit a tier above everything else right now. The rest of closed is spread across the whole field.

Open weight sits in a tight, middling band, with seven of eight between 1.27 and 1.67. Cheap and consistent, and never good enough to write into memory unreviewed.

Local costs nothing and gets more wrong. Apple's on-device model on macOS 27 can't fit three of my five meetings into its 8,000-token context, and it scored 3.17 on the two it could. I'd treat local as a privacy choice, not a quality one.

## Do open weights catch up?

{{< chart src="catchup.svg" alt="Critical errors per summary by release date, April to October 2026, with a step line for the best score so far in each group. The open-weight line drops to 1.33 in April and stays near 1.3; the closed line falls from 3.47 to 1.20 in July and to 0.20 in late September. GPT-6.1 Sol (0.33), Sonnet 5.5 (0.93), MiMo V2.6 Pro (1.40), Haiku 5.5 (1.53) and MiMo V2.6 Flash (1.60) follow." caption="Critical errors per summary by release date (first seen on OpenRouter). Lines track the best score so far." >}}

I expected the usual story, with open weights a few months behind the frontier. In my field it looks different. From April until GPT-5.6 Luna pulled even in July, an open model (DeepSeek V4 Pro) held the best score. Then GPT-6 and Opus 5.5 jumped well clear in September, while open weight has plateaued near 1.3 for five months.

Take the early lead with a grain of salt. I picked these models to make a production decision, so the field skips several closed releases from the spring (Opus 5, my judge, among them). The only closed models I tested from before July were Haiku and Sonnet. My guess is open weight closes the gap again at some point, but I wouldn't bet on when.

## Averages hide the bad day

{{< chart src="reps.svg" alt="Heatmap of critical errors in every individual run: 20 models by five meetings with three reps each. GPT-6 Sol's only errors are a single critical in each rep of meeting B; Haiku 4.5 ranges up to 9 in one run. Meeting B, outlined, has no clean run from any model." caption="Critical errors in every run: five meetings, three reps each at temperature 0. Rows ranked best to worst." >}}

Sol's worst single run had one critical error. Haiku's had nine. An average won't show you that, and when the output is headed for memory, the worst run matters as much as the mean.

## What kind of wrong

Sol made zero fabrications across 15 runs. Haiku made 18, and Nemotron 11.

The top three models made no critical errors of omission at all. Every critical they did make was a misreading, stated with confidence. Fewer errors doesn't mean safer errors. I'll catch an omission when I skim a summary. A confident misreading reads just like a fact.

## Some meetings no model fixes

One meeting beat every model. On the long call where the other side is never named (meeting B in the chart above), none of the 45 runs across the field came back clean, and Sol missed it all three times. When the transcript never says who's talking (and writes my name over their company's), there's nothing for a model to recover. That meeting needs a better transcript.

It's the exception, though. Across the field, the model mattered far more than the meeting: averages ranged 17× across models (0.20 to 3.47) and only 3.4× across meetings.

## What I'm switching to

I'm moving my meeting summaries to GPT-6 Luna. Sol scored slightly better, but at 15 runs each, 0.20 and 0.47 sit in the same tier. Luna is "good enough" for a summary I read before it goes into memory anyway, and it's a bit cheaper. It's also a big step up from the GPT-5.6 Luna I was running before: better on every measure at less than half the price.

Regardless, summaries stay drafts. No model in the field was clean often enough to write into memory without me reading it first.

## What this test can't tell you

- The judge is Opus 5, and one contestant is Opus 5.5. The grading is blind, though, and Anthropic's models didn't come out ahead: Opus 5.5 placed third behind two OpenAI models, and Haiku came last.
- I calibrated the judge on seven summaries of one short meeting. Treat the error counts as relative.
- Anthropic's models ran through Claude Code's command-line tool, which adds about 2,000 tokens of scaffolding. They're consistent with each other and carry an asterisk against the rest.
- Nemotron ran 4-bit quantized, which may explain part of its deficit.
- Fifteen runs per model is enough to separate tiers. It can't rank neighbors: 1.20 against 1.40 is a tie.
- Five meetings, all mine. The method should carry over. The rankings might not.

## Build your own

The whole thing cost a few dollars in API calls, plus roughly $20 of judge time on my Claude subscription at list price. For a model that writes into something I rely on, that's cheap.

1. **Pick the meetings that hurt.** The ones with garbled audio and unnamed speakers, and the ones where a wrong outcome would cost you. Five is enough.
2. **Write the facts first.** Before you read a single summary, write down what's true and what a model could know from the input alone.
3. **Calibrate the rubric.** Grade a handful by hand and run the judge against them. If it can't fail anything, fix the rubric before you spend a dollar.
4. **Pin the judge.** Model aliases move. Mine did mid-project.
5. **Run reps.** Three per meeting at minimum. The bad day is what you're testing for.

The same five steps work for a team shipping a feature where a model writes into something customers rely on.

My judge prompt is below, trimmed. Give it the transcript, your facts of record and one summary at a time.

{{< collapse title="My judge prompt (trimmed)" copy=true >}}
```text
You are grading a machine-written summary of a meeting transcript.
Don't rate the prose. Find specific defects you can quote.

The bar: would a careful colleague file this summary as the permanent
record without reading the transcript first?

You get the transcript, a facts-of-record list, and the summary.
The facts come in three groups:
- DERIVABLE: stated in the transcript. Contradicting or omitting one
  is a defect.
- KNOWN-ONLY: true, but not in the transcript. Leaving one out is not
  a defect. Asserting something that contradicts one is.
- TRAP: a plausible guess the transcript invites but doesn't support.
  Asserting one is a defect.

Critical defects:
- outcome: the decision (a yes, a no, a firm next step) is missing,
  hedged, or buried, even if nothing in the summary is false
- factual_error: a number, date, title, role, or piece of someone's
  history is wrong, however small
- attribution: words credited to the wrong speaker
- polarity: a no turned into a yes, or the reverse
- fabrication: a name, number, date, commitment, or quote with no
  support in the transcript

Also log major defects (an unsupported inference, a floated idea
promoted to an action item, a topic left out entirely), but only
criticals fail a summary.

First, reading the summary alone, write one sentence on what it says
the meeting's outcome was. Then list each defect with its category,
the quote from the summary, the transcript line it conflicts with
(or null), and one sentence of explanation. Return JSON only.
An empty defect list is a valid answer.
```
{{< /collapse >}}

A leaderboard tells you which model is best in general. Five of your own meetings tell you which one to trust with yours.