Can an agent work across a security team’s spectrum — find and safely reproduce real vulnerabilities, analyze malware, run security operations, and probe agent attack surfaces? Agent 能否胜任安全团队的完整工作流程？评测项目包括挖掘并安全复现真实漏洞、分析恶意软件、执行安全运营、评估 Agent 自身攻击面。
Seven models scored on the CodeBuddy Code harness, averaged over 3 runs in think mode; Score is 0–100, higher is better.七个模型在 CodeBuddy Code 评测框架上评测、取 3 次运行平均(think 模式);分数 0–100,越高越好。
Model settings: reasoning effort defaults to high, and each model runs with its provider-default inference hyperparameters.模型设置：reasoning effort 默认为 high，各模型使用其默认推理超参。
Harness settings: the context window is unified at 200k with a unified auto-compaction threshold; the WebSearch and AskUserQuestion tools are disabled.harness 设置：上下文窗口统一为 200k，自动压缩阈值统一；禁用 WebSearch 与 AskUserQuestion 工具。
Evaluation is pinned to a specific harness build — codebuddy-code:2.109.3; metrics may shift as harness versions evolve.评测固定在特定 harness 构建上：codebuddy-code:2.109.3；随 harness 版本演进，指标可能变化。
The HY (Hunyuan) endpoint in this evaluation is served first-party by its provider; the other models are accessed through third-party serving endpoints, whose parameter configuration and request handling may affect metrics.本次评测中 HY（混元）端点由其提供方第一方部署；其余模型通过第三方推理端点接入，第三方的参数配置与请求处理可能影响指标。
The suite spans four scenarios — code, web, office and security. The security subset asks a broader question than a standard bug-fix task: can an agent work across the full spectrum of a security team inside a real codebase — locate vulnerable code across modules and safely reproduce the issue in a sandboxed environment, analyze malware, run security operations, and probe an agent’s own attack surface?
The suite spans 60 tasks across 6 domains, covering both red-team and blue-team work: 32 vulnerability discovery & exploitation tasks, 14 malware analysis tasks, 8 security-operations (SOC) tasks, and 6 agent-security tasks. Difficulty skews hard, matching the shape of real security work, where hard problems outnumber easy ones.
Each task is scored by a deterministic per-task scoring.py — locating vulnerable code, verifying ASAN-anchored PoCs, diffing malware analysis against IOC ground truth, checking YARA rules under a zero-false-positive constraint, and scoring SOC reports on macro-F1 and Kendall-tau — and is anchored to real vulnerabilities rewritten into natural-language, role-played requirements. Like the other subsets, security tasks are not present in public training corpora, resisting contamination, and ship in the same self-contained task-directory format so they drop straight into the same evaluation harness. Reproduction runs strictly inside sandboxed environments.
本套件覆盖四个场景：代码(code)、网页(web)、办公(office) 与 安全(security)。安全子集提出的问题比普通 bug-fix 更广：Agent 能否胜任安全团队的完整工作流程——跨模块定位漏洞代码并在沙箱中安全复现、分析恶意软件、执行安全运营、评估 Agent 自身攻击面？
套件覆盖 6 域共 60 题,同时覆盖红队与蓝队工作:32 道漏洞挖掘与利用任务、14 道恶意软件分析任务、8 道安全运营(SOC)任务、6 道 Agent 安全任务。难度分布偏向高难,贴近真实安全工作中难题多于易题的分布。
每道题都由确定性的每题 scoring.py 评分：定位漏洞代码、验证 ASAN 锚定的 PoC、将恶意软件分析与 IOC ground truth 比对、在零误报约束下检查 YARA 规则、按分类准确率（macro-F1）与排序一致性（Kendall-tau）为 SOC 报告打分，并锚定真实漏洞,再改写成自然语言、带角色扮演的需求。与其他子集一样,安全任务不进入公开训练语料,以此抵抗污染,并以相同的自包含任务目录格式交付,可直接接入同一评测框架;复现过程严格在沙箱环境内进行。
Security is built to score like a real security team works — broad domain coverage across both attack and defense, deterministic programmatic scoring instead of an LLM judge, real CVE reproduction, agent-security assessment, and hardened defenses against gaming.Security 的设计目标是让评测像真实安全团队的工作方式一样运转，攻防两侧的广泛领域覆盖、用确定性程序化评分取代 LLM 裁判、真实 CVE 复现、Agent 安全评估,以及针对刷分的加固防线。
One suite covers both the attack side and the defense side. 32 vulnerability discovery & exploitation tasks map to red-team and security-researcher work; 14 malware analysis tasks map to anti-virus / reverse-engineering work; 8 security-operations tasks map to SOC analyst and detection-engineer work; 6 agent-security tasks map to AI red-teaming. Within the discovery & exploitation block, red-team work spans whitebox source-audit, blackbox binary-exploitation, and web-exploitation tasks — which combine with malware analysis, security operations, and agent security to complete the 6 domains shown below. The red-team side (discovery & exploitation plus agent security) accounts for 38 tasks and the blue-team side (malware analysis plus security operations) for 22. Difficulty skews hard, matching how real security work is mostly hard problems, not easy ones.一个评测集同时覆盖攻击侧与防御侧。32 道漏洞挖掘与利用任务对应红队和安全研究员职能,14 道恶意软件分析任务对应反病毒 / 逆向工程职能,8 道安全运营任务对应 SOC 分析师和检测工程师职能,6 道 Agent 安全任务对应 AI 安全红队。漏洞挖掘与利用板块的红队工作细分为白盒源码审计、黑盒二进制利用与 Web 利用任务，与恶意软件分析、安全运营、Agent 安全共同构成下图所示的 6 个领域。红队侧(漏洞挖掘与利用加 Agent 安全)合计 38 题,蓝队侧(恶意软件分析加安全运营)合计 22 题。难度分布偏向高难,贴近真实安全工作中难题多于易题的分布。
Every task ships its own scoring.py, executed inside an isolated Docker container and writing a numeric reward.json directly. Exploitation tasks score on PoC / flag verification, malware-analysis tasks diff against IOC ground truth, YARA tasks check rule match-rate under a zero-false-positive constraint, and SOC tasks score analyst reports with macro-F1 and Kendall-tau. The same task and the same output score identically on repeat runs — results are deterministic and reproducible, with no LLM-judge bias or randomness in the loop.每道任务自带 scoring.py,在 Docker 隔离容器内执行后直接写入数值型 reward.json。漏洞利用任务跑 PoC 验证 flag,恶意软件任务比对 IOC 与 ground truth,YARA 任务检查规则匹配率并要求零误报,SOC 任务用分类准确率与排序一致性评估分析报告。同一任务、同一输出重复运行分数一致,结果确定且可复现,评测链路中不存在 LLM 裁判带来的模型偏差与随机性。
Whitebox audit tasks reproduce real CVEs directly in widely-deployed open-source projects — binutils, curl, nginx, vim, jq, fluent-bit and other widely-deployed upstream projects. The model has to read source like a security researcher, trace the data flow, and construct a minimal PoC input that triggers an ASAN crash. Each task follows a two-step find-vuln → poc-verify gate: the first step must clear a threshold before the second unlocks, mirroring the pacing of a real penetration test. A further set of web-exploitation tasks cover advanced techniques — House of Apple2 and ECDSA nonce reuse.白盒审计任务直接复现真实开源项目的 CVE,覆盖 binutils、curl、nginx、vim、jq、fluent-bit 等广泛部署的上游项目。模型需要像安全研究员一样阅读源码、追踪数据流,并构造出能触发 ASAN 崩溃的最小 PoC 输入。每道题采用 find-vuln 到 poc-verify 的两步门控结构:第一步须先过门槛才能解锁第二步,模拟真实渗透测试的节奏。另有一组 Web 利用任务覆盖 House of Apple2、ECDSA nonce 重用等高阶攻击技术。
6 agent-security tasks evaluate the attack surface an AI agent exposes once it holds tool-calling permissions, across six attack vectors: A2A (agent-to-agent) injection, ReAct chain hijacking, multimodal prompt-chain injection, tool-schema confusion, summarize-tool exfiltration, and delayed triggers. Tasks require the model to produce a structured findings report — whether it resisted the injection, whether it detected the tool misuse, whether it caught the data leak, a description of the attack chain, and a CVSS severity score. This mirrors the security-assessment work an AI-safety team does before an agent ships.6 道 Agent 安全任务评估 AI 智能体在获得工具调用权限后所面临的攻击面,覆盖六种攻击向量:A2A(Agent-to-Agent)注入、ReAct 思维链劫持、多模态 prompt-chain 跨模态注入、工具 schema 混淆绕过、summarize 工具外泄、时延触发器。任务要求模型输出结构化 findings 报告:是否抵抗注入、是否检测到工具误用、是否发现数据泄露、攻击链描述,以及 CVSS 严重度评分。对应 AI 安全团队在 Agent 上线前所做的安全评估工作。
Tasks are hardened with multiple anti-cheat layers so a score reflects genuine analysis rather than hard-coding or brute-force. A banned-literal scan blocks hard-coded answers; a renamed-input test checks whether an extractor actually parses binary structure rather than relying on a filename; an overlay / tamper test checks immunity to trailer tampering on a sample; an encoding-dependence test makes sure a detection rule is anchored to encoded bytes rather than plaintext; and low-weight decoy fields suppress brute-force scoring. The five layers close off cheating paths across input, code and output, so the score genuinely reflects a model’s security-analysis ability.任务内置多层反作弊机制,确保评测反映真实分析能力而非硬编码或穷举策略。禁用字面量扫描拦截硬编码答案,重命名输入测试验证提取器是否真的从二进制结构解析、而非依赖文件名,覆盖层 / 篡改测试验证对样本尾部篡改的免疫力,编码依赖测试确保检测规则锚定编码字节而非明文,低权重诱饵字段压制穷举刷分。五层机制从输入、代码、输出三个维度封堵作弊路径,让分数真正反映模型的安全分析能力。
Every task is scored by a per-task scoring.py running inside an isolated Docker container — there is no LLM judge in the loop. Exploitation tasks are scored on PoC / flag verification, malware-analysis tasks on IOC ground truth, YARA tasks on match-rate plus a zero-false-positive constraint, and SOC tasks on macro-F1 and Kendall-tau; results are deterministic and repeat exactly on re-scoring.
The numbers on the leaderboard come from the CodeBuddy Code harness, averaged over 3 runs to reduce variance.
Some models declined a small number of security-flavored tasks; a refused task scores 0. Across the 3-run protocol, GPT-5.5 recorded 2 refusals under CodeBuddy Code.
每道任务都由每题独立的 scoring.py 在隔离 Docker 容器内评分，评测链路中没有 LLM 裁判。漏洞利用任务按 PoC / flag 验证评分,恶意软件任务按 IOC ground truth 评分,YARA 任务按匹配率加零误报约束评分,SOC 任务按分类准确率与排序一致性评分;结果具备确定性,重复评分完全一致。
排行榜上的数字来自 CodeBuddy Code 评测框架,取 3 次运行的平均值以降低方差。
部分模型对少量安全类任务选择拒答;拒答任务计 0 分。按 3 次运行合计,GPT-5.5 在 CodeBuddy Code 下拒答 2 次。
Scope. The leaderboard covers seven models scored on the CodeBuddy Code harness, 3 runs averaged in think mode.范围。榜单覆盖七个模型,在 CodeBuddy Code 评测框架上评测,取 3 次运行平均(think 模式)。