ExpertCrowd-SWE is a November 2026 technical report about a coding-agent benchmark built from 100 expert-authored, multi-file engineering tasks. The tasks sit in 87 open-source projects and 11 languages. That makes the benchmark interesting for a reason beyond its leaderboard: it exposes how much a score depends on the work being asked for and the way acceptance is judged.
The task authors start with short tickets in codebases they know. Each task includes a Docker environment pinned to a repository commit, a sealed reference solution and a grader. The agent cannot read the reference solution, hidden grader, specification files or pristine checkout. The benchmark therefore evaluates a change made inside a controlled repository state, rather than asking a model to produce an isolated code fragment.
Acceptance is part of the measurement
EC-SWE's graders use weighted criteria that sum to 1.0. A failed blocker caps the score below full credit, while a failed pass/fail gate forces a zero. Only 10 tasks use any LLM-judged criteria, accounting for 0.85% of the total benchmark weight; the other 90 tasks use deterministic checks. The paper also reports calibration controls: the reference solution scores 1.0, while untouched and inert changes score 0. Alternative-valid-solution, adversarial and lazy/no-op control patches were included across subsets of the tasks.
The reported model comparison is correspondingly specific. Seven models were run five times per model-task in HUD, using each provider's native tool interface, extended-reasoning high-effort defaults, no additional system prompt and a 1,000-step limit. The report puts Claude Opus 5.5 at 67.3% and GPT-6 Astra at 63.7%. Their paired comparison has a 95% interval from -1.4 to +9.7 percentage points with p=0.20, so the paper does not treat those two results as statistically separable. Task-sampling intervals are wider than run-to-run intervals, and the authors caution against reading much into adjacent scores less than roughly five points apart.
A useful boundary around the claim
Twenty-four tasks are released with environments, graders and results; 76 are held out. That split gives the benchmark a public working surface and a private evaluation portion, but it does not make the tasks universally unseen. The authors note that code from public repositories may have appeared in training data, and that the benchmark tests problem solving in familiar-looking code rather than code no model has ever encountered.
EC-SWE therefore measures how seven agents performed on this task population, with this harness, these provider-native tools and these grading rules. It is not a ranking of coding products or a general measure of model quality. The paper also leaves important variables unswept: tool interfaces and agent loops differ by provider, reasoning settings were not systematically varied, and 100 tasks cannot represent every engineering repository or workflow.
That is the practical value of the report. It makes a coding-agent score easier to interrogate: inspect the ticket, the acceptance criteria, the controls, the held-out portion and the uncertainty before treating a percentage as a capability claim.
Source: ExpertCrowd-SWE: Measuring Frontier Coding Agents on Expert-Authored Engineering Tasks in Unfamiliar Codebases, November 2026 technical report.