Gemini 3.5 lands, Co-Scientist gets real, and agent runtimes harden

Flash is the new default for agentic loops where latency dominates cost — Google's 4x throughput claim plus the sub-half-cost framing changes the cost-per-task math more than the headline benchmark scores. If you're still routing everything through Pro tier, rerun the numbers this week.

Trace-driven evals are becoming table stakes — if your agent stack still relies on prompt tweaks plus vibes, you're about to fall behind teams that close the loop on production traces weekly. Pick a trace store and an eval harness now.

Verifiable math is the cleanest signal we have that reasoning models are doing real work — no rubric, no judge model, just a proof that checks out. Worth tracking which classes of problems this generalizes to before assuming it transfers to your domain.

The framing here — "good on in-distribution, weak on out-of-distribution" — is the template more domain benchmarks should adopt. If you're evaluating AI for scientific or industrial forecasting, AIMIP's methodology is worth borrowing.

The pattern here — a generalist agent paired with domain experts who validate leads in wet labs — is the realistic shape of "AI for science." If you're building research-assistant products, the case studies are a better spec than any benchmark.

Instruction-following is what users actually notice when an agent "feels dumb," and it's poorly correlated with reasoning benchmarks. Add IFBench (or a domain-specific variant) to your model selection process — the rankings will shuffle.

Codex and similar agents are now a primary vector for installing compromised npm packages at scale. Lock down agent egress to package registries, pin dependencies, and treat agent-generated PRs the same way you treat external contributions.

The interesting metric is P1 count at launch, not lines of code generated. If you're pitching coding agents internally, deadline-bound projects with measurable defect baselines are the right wedge — not steady-state productivity claims.

On-prem Codex is the gating item for financial-services and defense buyers who've been stuck on internal LLM gateways. Expect procurement cycles that were stalled on data-residency to restart in Q3.

Operational adoption by NHC is a real bar — these forecasts go into emergency-management decisions, not dashboards. Worth citing when buyers ask whether AI models are "trusted in production" outside tech.

OfficeQA Pro is closer to what enterprise agents actually do than MMLU or HumanEval — it's worth adding to your eval suite even if you're not using Databricks. The deltas there predict end-user satisfaction more reliably.