Misleading Ads Eval
Can a language model tell when an ad is misleading the way New Zealand's Advertising Standards Authority would? This is a small, open evaluation to find out. Pilot cases are running now; results go here when the full set is done.
The question
Deciding whether an ad misleads is a judgement, not a lookup. The Complaints Board reads the ad as an ordinary consumer would, weighs the overall impression against what the advertiser can back up, and applies the Advertising Standards Code. Models are increasingly asked to make calls like this. How often do they land where the Board did, and do they get there for the right reasons?
Method (planned)
- Cases. Published Complaints Board decisions where the complaint was about misleading claims, upheld and not upheld, across products and media.
- The task. Each model sees the ad, the complaint and the relevant part of the Code, but not the ruling. It gives a decision and its reasons.
- Scoring. Agreement with the Board's decision, and separately, whether the model's reasons turn on the same points the Board's did. A right answer for the wrong reason counts as a miss on the second score.
- Checks. Cases the models may have seen in training are flagged, and a held-back set is scored only once, at the end.
Status
Pilot cases, October 2026. The pilot is for testing the task wording and scoring rules before the full set runs, so its numbers won't be published as results.
Limits
The Board's decisions are the benchmark, not the truth: a model that disagrees isn't always wrong. The case set is small and New Zealand only. And how a model answers depends on how it's asked, so the exact prompts will be published with the results.
Independent work, not affiliated with the Advertising Standards Authority or Anthropic. The shape is borrowed from the kind of questions Anthropic's Conceptual Reasoning Fellowship works on: a judgement-heavy question, real expert decisions as the benchmark, and reasons scored as well as answers.