# DeepSeek-V4-Pro — GAIA `<solution>` format mismatch

DeepSeek-V4-Pro's low GAIA score (**12.7** vs DeepSeek-V3.2-Reasoner's **50.3**) is an
**answer-extraction artifact**, not a capability gap. The model reaches correct answers but reports
them in a way the scorer can't read.

## The mismatch

The GAIA harness (`OpenHands/benchmarks` → `benchmarks/gaia/run_infer.py`) extracts the answer like this:

- The prompt asks for the answer wrapped in `<solution>…</solution>`, sent **in a message, *instead
  of* the `finish` tool** (`run_infer.py:490`).
- `_extract_answer_from_history()` (`run_infer.py:498`) walks events in **reverse** and returns the
  first agent `MessageEvent` **or** `FinishAction` it hits (`run_infer.py:537`) — i.e. the *last*
  agent event.
- `_parse_solution_tag()` (`run_infer.py:590`) runs `re.findall(r"<solution>(.*?)</solution>", …)`;
  with no tag it falls back to raw text, which then fails string matching.

V4-Pro emits a correct `<solution>` tag in a message, then **also calls `finish` with a plain-prose
recap that drops the tag**. Since the `FinishAction` is the last agent event, the harness reads the
prose, finds no tag, and the answer scores wrong.

This only affects GAIA — the other four benchmarks are scored by patch application / test execution,
not `<solution>` extraction.

## Example (instance `b9763138`, answer = `3`)

| order | event | type | content |
|------|-------|------|---------|
| 17:54:22 | `event-00065` | `MessageEvent` | `… 91 mod 11 = **3**  <solution>3</solution>` ✅ tagged |
| 17:54:29 | `event-00072` | `FinishAction` | `"… The check digit is **3**."` ❌ no tag → extracted |

Across 39 sampled GAIA conversations, **28 (72%)** dropped `<solution>` from `finish` despite having
emitted it in an earlier message; only **9 (23%)** kept it — matching the ~12.7% score.