Judging the news
before the market does.

The scarce input

The problem is not reading.

Markets produce more words than any person can absorb. The difficult work starts after retrieval: deciding what is new, what is material, which exposures are real, and how much confidence the evidence deserves.

Search systems optimize for finding documents. Financial judgment requires a harsher filter. A relevant article can be unimportant. A dramatic headline can contain no new information. A correct economic observation can still map to the wrong security or the wrong time horizon.

Information is abundant. A timestamped, falsifiable judgment is not.

Whalebench is designing a public evaluation around that gap. Frontier models receive the same point-in-time news context, produce structured judgments independently, and are scored only after their declared horizon has elapsed.

No training

Run every model as supplied.

We do not update weights, train adapters, or learn from benchmark answers. Each model receives one frozen prompt, the same point-in-time packet, and a fresh session with tools, search, memory, and future prices disabled.

The protocol

One article. Four decisions.

Each evaluation begins with a public item frozen at the moment it entered the corpus. Models see only information that was available at that timestamp. They do not receive later edits, price moves, or retrospective summaries.

The tasks are connected but scored separately. This matters. A model can correctly identify an article as material while failing to name the affected exposure. It can map the exposure correctly while expressing far more confidence than its evidence supports.

Model judgment

Make every claim inspectable.

A prose summary can sound insightful without taking a measurable position. The proposed output format forces each model to connect a quoted piece of evidence to an exposure, direction, horizon, and confidence estimate.

Citations are checked against the frozen source. Unsupported claims lose evidence credit even if the market later moves in the predicted direction. Conversely, a well-supported judgment is not automatically vindicated by one noisy price move.

Why confidence belongs in the record

Good judgment is not perfect prediction. It is knowing when the evidence is strong, when it is weak, and adjusting conviction accordingly. Calibration lets us distinguish a careful 55% view from an unjustified 95% claim.

Measurement

The judge needs a protocol too.

Initial semantic labels come from a blinded automated rubric process and remain provisional. The judge sees the source and model output without the model name or future price data. Material disagreements stay in the benchmark record rather than being averaged away invisibly.

We do not call automated labels expert ground truth. Market outcomes resolve separately, after the forecast horizon closes.

Two scoreboards, kept apart

Keeping these scoreboards separate prevents a lucky outcome from laundering weak reasoning. It also preserves useful judgments when an evaluation horizon is too short or the market is dominated by an unrelated shock.

Planned reporting

No leaderboard before the clock starts.

This page describes a methodology, not a completed result. Whalebench has not published accuracy, return, or cost claims for this evaluation. We plan to publish model-level results only after the corpus, rubric, and evaluation window are documented.

Publication policy

What we intend to release.

Some source text may be withheld or excerpted because of licensing. When that happens, the benchmark will identify the restriction and retain enough metadata for an independent reviewer to understand the gap.