Asking for the Whole Plan
Prompt design moves language models' scores on a clinical-safety benchmark more than model choice. An exploratory study on the NOHARM v2 public cases.
Public development data scored by a locally run AI evaluator. No organiser-scored held-out result and no clinical validation. The 30 public cases and their rubrics are by the NOHARM authors (ARISE Network, CC BY 4.0).
The finding
NOHARM scores a model's management plan for a real primary-care-to-specialist consultation against specialists' ratings of every action the case calls for. Most severe errors are omissions. Asked for a complete, structured plan, terse models stop omitting, and the ranking between models largely changes with how they are asked.
Every model, asked two ways
Twelve models answered the bare case; five also answered with an instruction. Hover a row for its numbers.
Show as a table
A fuller plan scores higher
Horizontal axis: median number of actions the evaluator extracted per plan, on a log scale. Each arrow joins a model's bare case to its instructed arm.
Which part of the instruction carries the gain
On Sonnet 5.5 and GPT-5.6 terra the instruction was taken apart: the consult framing alone, the plan format alone, and a bare request for 25 numbered recommendations that names no clinical content. Switch the metric to see why length alone falls short.
Show as a table
The instruction
Verbatim, as given to the models. The plan-format paragraph alone scored level with the full instruction. Paste a case where it says {case}.
Plan format alone
Full specialist instruction
Every instruction the study used, including the perspective variants and the kit's own 500-word prompt, is in the paper's Appendix A and in scripts/noharm_system.py.
Did asking for more add harm?
Off-rubric actions by model
What this does not show
- Thirty base cases, not 330 independent consultations; instructions were developed on the same cases they were scored on.
- Every score comes from an AI evaluator applying the benchmark's own scoring, not from the organisers and not from physicians. A second model family re-scored the key contrasts.
- No organiser-scored held-out result: the leaderboard admits models, not prompts.
- The physician review is one reviewer, the author, giving an overall verdict: blinded for the off-rubric actions only, and not an independent panel.
- The design is incomplete: five of twelve models were instructed, with different instructions, and the instruction was taken apart on two models only.
- Nothing here evaluates a plan acting without a clinician; it is not clinical validation.