scBench: Evaluating AI Agents on Single-Cell RNA-seq Analysis

scBench comprises 394 evaluations spanning six sequencing platforms and seven task categories (Table 1). Each evaluation pairs a data snapshot (often an AnnData .h5ad file) with a natural-language task prompt and a deterministic grader that scores the agent’s structured JSON output as a pass or fail. The benchmark focuses on the analysis stages with the greatest dataset-specific variation: cell typing (118 evaluations, 30%) and differential expression (71, 18%) together account for nearly half the benchmark. Normalization (44) and QC (36) are smaller because these procedural steps admit fewer distinct problem formulations per dataset. Platform representation ranges from Illumina (85 evaluations) and MissionBio (81) to CSGenetics (42). ParseBio lacks QC evaluations because its vendor workflow omits explicit quality filtering, limiting cross-platform QC comparisons to five platforms. MissionBio Tapestri is a targeted DNA+protein platform rather than RNA-seq; we include it to stress-test whether agents generalize beyond transcriptomic workflows to related single-cell analysis patterns (clustering, cell typing from protein markers, variant interpretation).

The two axes, platform and task, allow stratified analysis of model performance. Task categories reveal a gradient of accuracy: normalization applies standard transformations often with well-understood implementations; an agent need only identify the correct function call. Cell typing and differential expression require multi-step reasoning and contextual scientific judgement: selecting marker genes, interpreting cluster identity, subsetting cells, choosing statistical tests, and identifying tissue-specific signatures. Platform diversity tests generalization beyond training-data familiarity. Chromium and Illumina dominate public repositories and tool documentation; MissionBio and ParseBio appear less frequently, use non-standard data structures, and sport lesser-known technical footguns. Models that overfit on Scanpy tutorials without learning transferable analysis techniques should collapse on underrepresented platforms. Sections 2.3 and 2.4 quantify these effects.

We evaluated eight frontier models from four providers (Table 2). Claude Opus 4.6 achieves the highest accuracy at 52.8% (95% CI: 48.3–57.2%), followed by Claude Opus 4.5 at 49.9% and GPT-5.2 at 45.2%. Claude Sonnet 4.5 reaches 44.2%, placing fourth despite being a smaller model. The bottom tier comprises GPT-5.1 (37.9%), Grok-4.1 (35.6%), Grok-4 (33.9%), and Gemini 2.5 Pro (29.2%).

The 23.6 percentage point spread between best and worst models exceeds SpatialBench’s 18.3 pp spread, indicating that scBench discriminates model capability despite the higher overall accuracy. Anthropic models occupy the top four positions, with both Opus variants and Sonnet outperforming all competitors. Stratified analysis (Sections 2.3–2.4) reveals where models diverge.

Task categories reveal a consistent difficulty gradient (Table 3, Figure 4). Normalization is easiest (cross-model mean 70.4%), followed by QC (55.3%). These procedural tasks vary across biological contexts but often involve applying well-understood transformations. Differential expression is hardest (mean 27.0%), with cell typing (34.9%) and clustering (38.3%) in the middle. Seven of eight models follow the same difficulty ordering.

Differential expression is also most discriminative, with a 27.7 pp spread between best and worst models. Model differences concentrate in judgment-heavy stages—DE and cell typing—rather than procedural ones.

Platform choice affects accuracy as much as model choice (Table 4, Figure 5). Cross-model mean accuracy ranges from 59.1% on CSGenetics to 26.4% on MissionBio—a 32.7 pp gap that exceeds the 23.6 pp spread between best and worst models. CSGenetics is easiest for six of eight models; MissionBio is hardest for all eight.

MissionBio inverts rankings. Grok-4 (sixth overall) beats GPT-5.2 (third overall) on MissionBio (24.7% vs 23.0%), and Sonnet 4.5 surpasses GPT-5.2 by 11 pp. The Anthropic models hold up on MissionBio while most competitors collapse.

Every model shows large platform swings. Gemini drops 42 pp between CSGenetics (52.4%) and MissionBio (10.3%). Even Opus 4.5, the most consistent model, loses 39 pp between its best and worst platforms. These effects likely reflect uneven training data: e.g., MissionBio appears less frequently in public documentation than Chromium pipelines.

scBench and SpatialBench (Workman et al., 2025) together cover the two dominant transcriptional assays (Table 5). The top model reaches 52.8% on scBench versus 38.4% on SpatialBench—scRNA-seq is more tractable. This gap holds across the leaderboard: the bottom model scores 29.2% on scBench versus 20.1% on SpatialBench. Model rankings are preserved at the extremes: Claude Opus leads both benchmarks and Gemini ranks last in both.

The benchmarks share some structure. Normalization is easiest in both (84% vs 76% best-model accuracy). Both show strong platform effects, with 30–40 pp swings depending on sequencing technology. The accuracy gap likely also reflects training data: scRNA-seq has far more public datasets than spatial transcriptomics, and tools like Scanpy dominate the ecosystem with extensive documentation. This is perhaps most obvious in the difference in performance on the QC task category, where knowledge of thresholds and other procedurally simple operations explains variability.

scBench is constructed from real scRNA-seq analysis workflows across six sequencing platforms: Chromium, BD Rhapsody, CSGenetics, Illumina, MissionBio, and ParseBio. Following SpatialBench, we identify analysis steps that satisfy three criteria: (1) the task arises in routine practice—a step that a working bioinformatician would perform as part of a standard pipeline (QC, normalization, HVG selection, clustering, annotation, or differential expression); (2) the answer requires empirical data interaction—it depends on the provided dataset and cannot be produced from textbook knowledge or memorized gene lists alone; and (3) the result is a verifiable quantitative artifact—a structured JSON output that can be graded deterministically by one of five grader families (Section 4.5).

Each candidate problem follows a five-stage construction pipeline. We first reproduce the target analysis step on the provided data using the published workflow. We then define the output artifact as an exact JSON schema with named fields and value types. Next, we select the grader family matching the output shape (e.g., NumericTolerance for cell counts, DistributionComparison for cell type proportions). We calibrate tolerances by running the analysis with multiple valid methods and parameter choices to establish the range of acceptable answers. Finally, we harden against shortcuts by removing precomputed embeddings, cached labels, and any fields that would allow the agent to read the answer without performing the intended computation.

Ground truth values are derived by re-running published pipelines from raw counts using author-specified parameters where available, then verified against domain understanding of expected biological results. When papers do not uniquely specify parameters (e.g., QC threshold not reported), we use standard defaults and widen tolerances to accept the resulting variation. Each problem is assigned an evaluation type—scientific, procedural, or observational (Section 4.3)—governing how aggressively tolerances must accommodate methodological variation. As a final quality-control step, we attempt to solve each problem by (a) reading .obs and .uns fields directly without computation, (b) answering from prior biological knowledge, and (c) running with alternative valid methods to verify tolerance coverage. Problems failing any of these checks are revised or removed.

Each evaluation is a JSON specification with four agent-visible components and one internal component. The data node points to one or more AnnData .h5ad files (Wolf et al., 2018) containing the expression matrix, cell metadata (.obs), and gene annotations (.var); at runtime the harness downloads these files into an isolated workspace. The task prompt describes the analysis goal in natural language and specifies the exact JSON output format, including field names and value types. The deterministic grader defines the grader family and its configuration—ground truth values, tolerance parameters, and pass thresholds—that map the agent’s structured answer to pass/fail (Section 4.5). Metadata tags each problem by task category, evaluation type, sequencing platform, and computational complexity. A fifth component, notes, documents the solution approach, tolerance rationale, and known edge cases; notes are excluded from the agent’s context at the harness level and never appear at runtime.

Eval definitions are validated by a deterministic linter before entering the benchmark. The linter performs static schema validation, checking that required fields are present, grader configurations are well-formed (e.g., tolerance types are valid, thresholds are in range), and that the answer fields specified in the task prompt match what the grader expects. Evals that fail linting are blocked; evals with ambiguous tolerances or shortcut-prone structure are revised or removed during manual review.

Every evaluation is classified into one of three types that govern how aggressively tolerances must accommodate methodological variation.

Following SpatialBench, we apply three design criteria to every evaluation. The overarching rule is specify what, not how: tasks define the scientific goal and the exact output format, but do not prescribe a step-by-step method or parameters (with the exception of procedural evaluations, which name the method). The linter enforces structural compliance; manual review validates each criterion.

Each evaluation is paired with a deterministic grader that maps the agent’s structured JSON answer to pass/fail. We use five grader families; formal specifications are in Appendix C.

We evaluate all models under mini-SWE-agent (Yang et al., 2024), an open-source harness that implements a simple action loop: the LLM generates a free-form response, the harness extracts the first fenced code block (delimited by markdown triple-backtick syntax), executes it in a local bash shell, and returns stdout/stderr to the model as the next observation. Each evaluation is capped at 100 action steps (LLM turn $\to$ code extraction $\to$ execution $\to$ observation); if the agent exhausts the step budget without writing eval_answer.json, the evaluation scores zero.

The runtime environment provides scanpy, anndata, numpy, pandas, scipy, and matplotlib; all models share the same package versions. Network access is enabled, allowing agents to install additional packages if needed. Each evaluation runs in an isolated workspace: data files are symlinked from a local cache and the agent has read/write access only within that workspace. No GUI or interactive tools (Jupyter, plot display) are available.

Two timeout layers bound execution. An operation timeout of 300 seconds caps any individual bash command; an evaluation timeout of 600 seconds (configurable per evaluation) caps total wall-clock time via SIGALRM. On timeout or runtime crash, the harness grades whatever is in eval_answer.json at that point; if no answer file exists, the evaluation scores zero. There is no retry logic—each replicate is a single attempt. The harness records a complete trajectory for every run (conversation history, tool calls, and outputs), enabling post-hoc analysis of agent behavior.

We follow the same two-stage aggregation used in SpatialBench. Each model–evaluation pair is run $K{=}3$ times. Replicates share the same prompt, data, and harness; the only source of variation is the model’s sampling nondeterminism (no explicit seed or temperature control). Each run receives a binary outcome from the grader, $s_{i,r}\in\{0,1\}$.

In the first stage we compute the per-evaluation mean $\bar{s}_{i}=\frac{1}{K}\sum_{r}s_{i,r}$, yielding a value in $\{0,\tfrac{1}{3},\tfrac{2}{3},1\}$. In the second stage we treat the $\{\bar{s}_{i}\}_{i=1}^{n}$ as independent observations and compute the aggregate accuracy $\hat{\mu}=\frac{1}{n}\sum_{i}\bar{s}_{i}$ with 95% confidence intervals via the $t$-distribution on $n{-}1$ degrees of freedom. All 394 evaluations are equally weighted; there is no upweighting by task category or platform. Per-evaluation means are approximately independent because evaluations use different datasets and different prompts; the shared model is the only common factor and is constant within a model’s column. For stratified breakdowns (by task or platform), we apply the same procedure to the relevant subset, recomputing $n$, $\hat{\mu}$, and the corresponding $t$ critical value.