Code 80 tasks

Leaderboard排行榜

How models rank on this subset. Scores below are from the CodeBuddy Code harness; Claude Code harness runs on this subset — where Claude Opus 4.8 leads at 77.90% — are on the home leaderboard’s Claude Code tab. The second, lighter number on each row is the LLM-judge score (7-dim weighted rubric) — a reference signal only, it never affects ranking. Switching the tabs below re-slices the same 80 tasks by role / repo complexity / edit surface and recomputes the hidden-test reward within that slice; the judge column is hidden in sliced views since judge scores are only computed at the overall level.各模型在该子集上的排名。以下为 CodeBuddy Code harness 成绩;该子集下 Claude Code harness 的成绩(Claude Opus 4.8 以 77.90% 领先)见首页排行榜的 Claude Code 分栏。每行第二个、较浅的数字是 LLM Judge 分(7 维加权评分)— 仅作参考信号,不影响排名。切换下方 tab 可按 角色 / 仓库复杂度 / 编辑面 对同一批 80 题重新切片并计算隐藏测试 reward;切片视图下不显示 judge 列,因为 judge 分数目前只有整体聚合值,没有按切片拆分。

Model settings: reasoning effort defaults to high, and each model runs with its provider-default inference hyperparameters.模型设置：reasoning effort 默认为 high，各模型使用其默认推理超参。
Harness settings: the context window is unified at 200k with a unified auto-compaction threshold; the WebSearch and AskUserQuestion tools are disabled.harness 设置：上下文窗口统一为 200k，自动压缩阈值统一；禁用 WebSearch 与 AskUserQuestion 工具。
Claude Opus 4.8's Code score under Claude Code comes from a modified-instruction run: on top of the disabled AskUserQuestion tool, an explicit do-not-ask / complete-in-one-pass instruction was added, so its setup differs slightly from the other runs.Claude Opus 4.8 在 Claude Code 下的 Code 分数来自一次调整指令的运行：在禁用 AskUserQuestion 工具的基础上，额外加入了不要询问、一次性完成的指令，因此其设置与其他运行略有不同。
Evaluation is pinned to specific harness builds — codebuddy-code:2.109.3 and claude-code:2.1.187; metrics may shift as harness versions evolve.评测固定在特定 harness 构建上：codebuddy-code:2.109.3 与 claude-code:2.1.187；随 harness 版本演进，指标可能变化。
The HY (Hunyuan) endpoint in this evaluation is served first-party by its provider; the other models are accessed through third-party serving endpoints, whose parameter configuration and request handling may affect metrics.本次评测中 HY（混元）端点由其提供方第一方部署；其余模型通过第三方推理端点接入，第三方的参数配置与请求处理可能影响指标。

Overview概览

What makes Code differentCode 的核心亮点

The goal is not simply more tasks — it is to move the benchmark from “fix a bug / add a feature” toward the workflows real Copilot users hit: a PM asks for a report from exported data, an algo engineer asks for an evaluation script or a feature pipeline, QA asks for regression tests, Ops cares about reliability and production behaviour. Two things carry that: task diversity and scenario diversity.目标不是单纯增加题量,而是把评测集从「修 bug / 加 feature」推向更接近真实 Copilot 使用的工作流:PM 让你基于导出数据出报告,算法工程师让你补评估脚本或特征流水线,QA 让你加回归测试,Ops 关注可靠性与线上行为。任务多样性 与 场景多样性 共同支撑了这一点。

The 80 open tasks still explicitly cover all 18 categories. To read the coverage at a glance, we merge them into 6 usage domains named after real engineering collaboration — only ~1/8 of the suite is now plain bug-fix.80 道开源任务依然显式覆盖全部 18 类 category。为便于一眼看清覆盖面,我们把它们归并为按真实研发协作命名的 6 个使用大类，单纯的 bug-fix 现在只占约 1/8。

Every task also carries a role — developer, algo, PM, QA or ops — so the same model can be sliced by where the requirement comes from: 每题还带一个 role：开发、算法、PM、QA、运维，让同一模型可以按需求来自谁切片观察：

Placed against representative code benchmarks on the same axes (qualitative, relative coverage only), Code spreads across role diversity, task-type breadth, artifact variety and contamination resistance, where single-skill benchmarks concentrate on a few axes:与代表性代码 benchmark 放在同一组维度上做定性对比(仅表示相对覆盖面):Code 在 角色多样性、任务类型广度、产物多样性、防过拟合 上摊开了一整片,而以单点能力为主的 benchmark 集中在少数几个轴:

Code keeps the tone of natural-language requirements — it does not hand the model the root cause, the gold-patch idea, or the full failure context. Each of the 5 roles phrases the ask its own way: developers describe expected code behaviour, PMs describe the business result they want to see, algo engineers keep the metric and data details, QA cares about edge-case regressions, Ops cares about production stability — all in everyday, colloquial language.Code 保持自然语言需求的口吻，不把 root cause、gold patch 思路或完整失败上下文直接塞给模型。5 类 role 各按自己的方式提需求:开发者描述代码层面的预期行为,PM 描述想看的业务结果,算法工程师保留指标与数据细节,QA 关注边界回归,Ops 关注线上稳定性，但都保持日常协作语气。

Positioned on “instruction style × specification detail” against mainstream code / code-agent benchmarks (the SWE-bench family & SWE-Gym, Commit0, Aider Polyglot, RepoBench, Terminal-Bench, CommitPackFT; relative position only), Code lands in the most colloquial + most vague / underspecified region — those benchmarks hand the agent a detailed issue or task spec, whereas Code gives only a short, colloquial ask:在「指令风格 × 详细程度」上与主流 code / code-agent 基准做定性定位(SWE-bench 系列及 SWE-Gym、Commit0、Aider Polyglot、RepoBench、Terminal-Bench、CommitPackFT;仅表示相对位置):Code 落在 最口语化 + 最模糊/欠规约 区域。那些基准会给出详细的 issue 或任务规格,而 Code 只给一句简短、口语化的需求：

One representative query per role — note how differently the same “coding” work is phrased depending on who is asking. Representative in style; abridged/illustrative wording. 每类 role 各取一个典型需求，同样是写代码,提需求的人不同,说法差别很大。风格代表性示例,文字经改写。

Developerbug-fixa_crash_in_local

我这里偶发看到 Local 对象回收时从 __del__ 冒异常出来,通常是在退出或清理阶段。这个阶段不应该再把异常打到外面,麻烦把清理过程处理得稳一点。I occasionally see the Local object throw an exception from __del__ during collection, usually at exit or cleanup. Exceptions shouldn’t leak out at that stage — please make the cleanup process more robust.

Algomodel-evaluation

我这边离线评估推荐结果时发现 topK 指标不太可信,重复 item 和同分排序会把 recall、ndcg 带偏。麻烦把这份 ranking 评估脚本补完整,按用户算 precision/recall/ndcg/map,再汇总整体结果;排序和去重规则要稳定,别让同一批样本每次跑出来不一样。While running offline eval on our recommendation results, I noticed the topK metrics aren’t really trustworthy — duplicate items and same-score ties are throwing off recall and ndcg. Can you finish out this ranking eval script: compute precision/recall/ndcg/map per user, then roll up the overall numbers. The sort and de-dup rules need to be stable — don’t let the same batch of samples give different results on different runs.

PMproduct-analyticssignup_funnel

结账页文案实验跑完了,我想先知道新版有没有更好。数据里有曝光和购买事件,麻烦帮我算每组转化、收入和一个简单结论,别把很久以后的购买也算进去。The checkout-copy experiment finished; I want to know first whether the new version is better. The data has impression and purchase events — please compute per-group conversion, revenue, and a simple conclusion, and don’t count purchases that happen long afterward.

QAtesting

我给这个小版 request builder 补回归时,看到 headers、cookies、params 和 body 互斥这些地方很容易被改坏。麻烦只补测试,覆盖大小写 header 覆盖、params 顺序/list 展开/None 跳过,以及 json/data/content 互斥这些边界。测试文件按 unittest 写,别动 request_model.py,也别临时引 pytest,尽量补够十五到二十个小场景。While adding regression tests for this small request builder, I noticed the headers / cookies / params / body mutual-exclusion bits are really easy to break by accident. Please just add tests — cover case-insensitive header overrides, params ordering / list expansion / None-skipping, and the json/data/content mutual-exclusion edge cases. Write the test file in unittest, don’t touch request_model.py, and don’t sneak in pytest — try to get fifteen to twenty small scenarios in.

Opsreliability

线上 worker 偶尔遇到任务失败但消息已经 ack 掉,后面就没法重试了。帮我把 ack/requeue/dead-letter 的顺序理顺,成功才 ack,重试和失败路径也别重复确认。Our prod worker occasionally hits a task failure after the message’s already been acked, so there’s no way to retry it afterward. Help me straighten out the ack/requeue/dead-letter order — only ack on success, and don’t double-confirm on the retry or failure paths either.

Task categories任务类别

Code spans all 18 categories. As the task space broadened, plain bug-fix is just 10 of 80 (non-bug-fix now 70/80) — the bulk of the suite is feature, refactor, testing and a long tail of engineering-workflow types:Code 完整覆盖 18 类。随着任务空间拓宽,单纯的 bug-fix 仅占 10/80(非 bug-fix 已占 70/80)，主体转向 feature、refactor、testing 以及一条工程工作流的长尾:

Every task is additionally tagged along four axes — role, intent, edit-mode and surface — so slices can be scored independently:每题还沿四个维度打标签：role、intent、edit-mode、surface，便于分片独立评分:

Difficulty & scale难度与规模

Tasks are graded L2–L5 by repository complexity, centered on L4 (L2 ×4, L3 ×27, L4 ×40, L5 ×9). The ladder itself grades repository complexity; every task in this open set sits in the L2–L5 range — none falls below L2. The difficulty split runs easy / medium / hard — skewed toward medium and hard — and gold patches stay deliberately small: genuine reasoning over a big codebase, not large mechanical edits:任务按仓库复杂度分为 L2–L5,以 L4 为中心(L2 ×4、L3 ×27、L4 ×40、L5 ×9)。该阶梯按仓库复杂度分级,本次开源任务全部落在 L2–L5 区间，没有低于 L2 的任务。难度按 易 / 中 / 难 划分,偏向 中 / 难;gold patch 刻意保持精简，考验的是在大型代码库中的真实推理,而非大规模机械改动:

• Difficulty scales with repo complexity (L2–L5), not patch length.
• As complexity rises, the demand on the model's exploration ability grows: the hard part is not how to edit but finding where to edit across a large codebase.
• L1–L5 denote increasing repository complexity; L1 is a single-directory structure and is not covered in this test set (which starts at L2).

• 难度随仓库复杂度(L2–L5)增长,而非随补丁长度。
• 随着 complexity 升高,对模型 探索能力 的要求越来越高:要在大型代码库里找到该改哪里,而怎么改反而是相对简单的部分。
• L1–L5 表示复杂度依次提升;其中 L1 为单目录结构,本测试集不覆盖(从 L2 起)。

Representative repositories (real upstream-commit snapshots, ~34 of 80 tasks) include: asgiref, django, black, pytest, click, httpx, jsonschema, marshmallow, pydantic, werkzeug, flask, tornado, celery, aiohttp, urllib3, poetry and sqlfluff. A further ~28 tasks (api-contract, schema-behavior, performance, reliability, security-hardening, testing) instead ship de-identified clean-room *_like reimplementations of a target project’s public API design — e.g. fastapi_like/openapi.py — with no original code copied; the remaining ~22 tasks run on fully synthetic algo/PM workspaces.代表性仓库(携带真实上游 commit 快照,约 80 题中的 34 题)包括:asgiref、django、black、pytest、click、httpx、jsonschema、marshmallow、pydantic、werkzeug、flask、tornado、celery、aiohttp、urllib3、poetry、sqlfluff。另有约 28 题(api-contract、schema-behavior、performance、reliability、security-hardening、testing)改为去标识化的洁净室 *_like 重实现，参考目标项目的公开 API 设计独立重写,例如 fastapi_like/openapi.py,未拷贝任何原始代码;其余约 22 题运行在完全合成的算法/PM workspace 上。

Example task示例任务

# illustrative pseudocode — what the task requires per checkout-copy variant:
conversion_rate = purchases_in_window / impressions
revenue         = sum(amount for p in purchases_in_window)
# "in_window" excludes purchases that happen long after the impression —
# this implicit conversion-attribution window is the constraint most agents miss

# the agent produces a patch like:
--- a/fastapi_like/openapi.py
+++ b/fastapi_like/openapi.py
@@ def _param_schema(field):
-    schema = {"name": field.name, "in": field.location}
+    schema = {"name": field.alias or field.name, "in": field.location}
+    if field.location == "path":
+        schema["required"] = True
+    for k in ("nullable", "examples", "deprecated", "style", "explode"):
+        if (v := getattr(field, k, None)) is not None:
+            schema[k] = v
     return schema

Scoring评测口径