The scarce input
The problem is not reading.
Markets produce more words than any person can absorb. The difficult work starts after retrieval: deciding what is new, what is material, which exposures are real, and how much confidence the evidence deserves.
Search systems optimize for finding documents. Financial judgment requires a harsher filter. An article can be unimportant. A dramatic headline can contain no new information. A correct economic observation can still map to the wrong security or the wrong time horizon.
Information is abundant. A timestamped, falsifiable judgment is not.
Whalebench runs a public evaluation built around that gap. Frontier models receive the same point-in-time news context, produce structured judgments independently, and are scored only after their declared horizon has elapsed.
No training
Run every model as supplied.
We do not update weights, train adapters, or learn from benchmark answers. Each model receives one frozen prompt, the same point-in-time packet, and a fresh session with tools, search, memory, and future prices disabled.
The protocol
One article. Four decisions.
Each evaluation begins with a public item frozen at the moment it entered the corpus. Models see only information that was available at that timestamp. They do not receive later edits, price moves, or retrospective summaries.
Relevance
Does this information belong in an investor’s finite attention budget?
Precision · RecallNovelty
What changed relative to the information already available?
Evidence matchMarket map
Which issuers, suppliers, assets, or factors are genuinely exposed?
Entity coverageFalsifiable view
What direction, horizon, and confidence follow from the evidence?
Calibration · OutcomeThe tasks are connected but scored separately. This matters. A model can correctly identify an article as material while failing to name the affected exposure. It can map the exposure correctly while expressing far more confidence than its evidence supports.
Task definitions, source rules, horizons, and scoring code are fixed before a benchmark window begins. Revisions are marked, never silently applied.
What the rubric catches.
These cases explain the task. They cannot establish clean model performance because their outcomes may already be known to a model.
3M News Center · Official release
3M increases its full-year adjusted EPS guidance.
- Exposure
- MMM · direct
- Evidence
- Range moved from $8.50–$8.70 to $8.80–$8.95.
- Judgment
- Material, positive issuer evidence; forecast direction still requires expectations.
Federal Reserve · Enforcement
A prohibition order against one former bank officer.
- Exposure
- No large-cap public issuer named
- Evidence
- Appraisal-related lending misconduct at a local bank.
- Judgment
- Relevant to the named institution; noise for a broad U.S. equity mandate.
FOMC holds rates while three voters prefer a hike.
The July 29 statement is market-wide and clearly material. But “important” is not automatically bullish or bearish: direction depends on the decision and language relative to expectations.
Model judgment
Make every claim inspectable.
A prose summary can sound insightful without taking a measurable position. The output format forces each model to connect a quoted piece of evidence to an exposure, direction, horizon, and confidence estimate.
judgment.json
frozen at 08:10 UTCarticle_id"wb_08421"
is_relevanttrue
new_information"guidance revision"
exposure["issuer", "suppliers"]
direction"negative"
horizon"1–5 sessions"
confidence0.64
evidence"sentence_12"
Citations are checked against the frozen source. Unsupported claims lose evidence credit even if the market later moves in the predicted direction. Conversely, a well-supported judgment is not automatically vindicated by one noisy price move.
Why confidence belongs in the record
Good judgment is not perfect prediction. It is knowing when the evidence is strong, when it is weak, and adjusting conviction accordingly. Calibration lets us distinguish a careful 55% view from an unjustified 95% claim.
Measurement
The judge needs a protocol too.
Initial semantic labels come from a blinded automated rubric process and remain provisional. The judge sees the source and model output without the model name or future price data. Material disagreements stay in the benchmark record rather than being averaged away invisibly.
We do not call automated labels expert ground truth. Market outcomes resolve separately, after the forecast horizon closes.
Does the claim follow from the source?
Was the signal already known?
Could it change expected cash flows or risk?
Did confidence match the difficulty?
Two scoreboards, kept apart
Evidence fidelity, relevance, novelty, exposure mapping, and calibration.
Direction, magnitude, timing, and performance relative to a declared control.
Keeping these scoreboards separate prevents a lucky outcome from laundering weak reasoning. It also preserves useful judgments when an evaluation horizon is too short or the market is dominated by an unrelated shock.
Reporting
No leaderboard before the clock starts.
The instruments run today. Primary-source capture, a live judgment pilot, and a triage simulation operate in the open on the study portal, and further tests run privately on real data. What is not public is a leaderboard: model-level results publish only after the corpus, rubric, and evaluation window are documented.
Model results appear here once the preregistered window runs.
Publication policy
What we release.
ProtocolTask definitions, prompts, model versions, and scoring rules.
Audit trailSource timestamps, citations, model outputs, and rubric revisions where licensing permits.
ResultsPer-task accuracy, calibration, disagreement, latency, and estimated inference cost.
FailuresRepresentative misses and ambiguous cases, not only the most flattering examples.
Some source text may be withheld or excerpted because of licensing. When that happens, the benchmark identifies the restriction and retains enough metadata for an independent reviewer to understand the gap.
The useful model is not the one that reads everything. It is the one that knows what deserves a decision.
The first protocol runs now: primary-source events are captured and hashed as they publish, a judgment pilot makes four decisions per document, and a triage simulation scores what the models choose to read — all in the open on the study portal, with further tests running privately on real data. Misses publish alongside wins.
Watch the study run in the portalWhalebench is an evaluation platform. Protocol revisions before preregistration are marked in the record. Examples are illustrative, portfolios are simulated, and nothing here is investment advice or a recommendation to trade.
Our document-judgment framing was informed by Thinking Machines Lab and Bridgewater AIA Labs’ 2026 work on financial information triage. Whalebench differs by evaluating unmodified frontier models prospectively and adding locked market-outcome forecasts.
Read the referenced research