A critical vulnerability in how AI models are evaluated has emerged from recent research: formatting choices in prompts can swing benchmark scores enough to completely reverse leaderboard rankings. Researchers found that adding line breaks to a multiple-choice question prompt increased one model's accuracy by several percentage points—enough to bump it from a lower tier to a higher one. The same underlying model, when presented with identically-worded questions formatted differently, produced measurably worse results. This means leaderboards may be ranking systems based partly on prompt presentation rather than genuine capability differences.
The study, presented in arXiv:2607.09665v1 titled 'Format Sensitivity Index: Token-Controlled Prompt Wrapper Robustness and Schema Compliance in LLM Benchmarking,' introduces two complementary metrics to quantify this sensitivity. Researchers used a token-controlled protocol to isolate formatting variables from content variables, ensuring that only cosmetic differences changed between test runs. Their work reveals that different model families show wildly varying susceptibility to these format manipulations—some robust across formatting variations, others dramatically affected by seemingly trivial presentation choices.
The implications cut deep into AI evaluation credibility. GPT-4 and Claude demonstrated particular sensitivity to formatting variations, while other models proved more resilient. If a leaderboard position hinges on whether you linebreak your prompts or adjust spacing, how much should that ranking actually influence high-stakes decisions like $100M model acquisitions or deployment choices in critical systems? The research suggests the field urgently needs standardized formatting protocols and Format Sensitivity Indices reported alongside benchmark scores, ensuring comparisons reflect genuine model capabilities rather than prompt engineering artifacts.