Gemini and the shape of multimodal evaluation
Frontier model releases stopped being single-benchmark comparisons. Gemini's launch made the evaluation surface (text, image, audio, video, code, safety) wider than the interface most users touch.
When Google DeepMind published the Gemini technical report, the headline most outlets ran with was a single number: state-of-the-art on a popular language benchmark (Gemini Team, 2023). That framing undersells what actually shipped. Gemini wasn't released as a better autocomplete. It was released as a family (Ultra, Pro, Nano), each evaluated across text, image, audio, video, and code, with safety policy gates layered on top of all of it.
The interesting part isn't that one score went up. It's that the evaluation surface is now wider than the interface most users actually touch. A chat window hides five different competency profiles behind one text box.
Model families need product-shaped evaluation
A single-task benchmark answers one question: how good is this model at this task. That question made sense when "this model" meant one architecture trained for one modality. It stops making sense once a release is actually three differently-sized models sharing a training recipe, each evaluated across inputs that don't share a representation.
Text was always well covered: it's what benchmarks were built for. Everything else is the gap. MMMU, a benchmark built specifically to stress college-level reasoning across image and text together, found that even strong multimodal models trailed expert human performance by double digits at launch, with the gap widest on questions that required genuine cross-modal reasoning rather than reading text off an image (Yue et al., 2023). The modalities that get the least traditional coverage are exactly the ones where models are least reliable, and least audited.
Evaluation surface vs. interface surface
The practical confusion is that the thing users interact with (a chat box, an app) is a thin slice of what was actually evaluated before ship.
| Layer | Question | Signal |
|---|---|---|
| Capability | Can it solve the task? | Benchmark score |
| Modality | Can it transfer across inputs? | Text, image, audio, video |
| Product | Can it run where needed? | Ultra, Pro, Nano tiers |
| Safety | Can it be deployed responsibly? | Policy and eval gates |
This is also where the holistic-evaluation argument comes from outside the multimodal space specifically: Liang et al.'s HELM framework made the case that a single accuracy number hides tradeoffs across robustness, calibration, fairness, and efficiency that matter just as much for deployment as raw capability (Liang et al., 2022). Multimodal model families inherit that same problem, multiplied by every additional input type.
What this means for evaluation practice
The product layer and the safety layer are not afterthoughts bolted onto a capability score; they're gates a release has to clear independently, and they can fail even when the capability score looks great.
*Asterisk matters: state-of-the-art on which benchmark, evaluated how, against which prior model.
Treat a frontier release the way you'd treat any other system launch: read the modality breakdown, not just the leaderboard line; ask which benchmark a "SOTA" claim is relative to; and check whether the safety evaluation is published with the same rigor as the capability numbers, or just asserted.
Model-family evaluations are really product-system evaluations wearing benchmark clothing.
Here's the announcement itself: read it as a product launch, not a paper abstract, and the modality list reads differently.
We’re excited to announce 𝗚𝗲𝗺𝗶𝗻𝗶: @Google’s largest and most capable AI model. Built to be natively multimodal, it can understand and operate across text, code, audio, image and video - and achieves state-of-the-art performance across many tasks. 🧵 dpmd.ai/announcing-gem…
References
- Gemini Team, Google. (2023). Gemini: A Family of Highly Capable Multimodal Models. arXiv:2312.11805
- Yue, X., et al. (2023). MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. arXiv:2311.16502
- Liang, P., et al. (2022). Holistic Evaluation of Language Models (HELM). Stanford CRFM. arXiv:2211.09110
- OpenAI. (2023). GPT-4V(ision) System Card. openai.com