Open-weight AI models can perform as well as leading proprietary systems on a demanding cybersecurity-agent benchmark when they are allowed enough time to finish, according to a new preprint from researchers Sahil Garg, Ruhi Gottumukkala, Vishal Agarwal and Shubham Vishwakarma. Their study does not establish that open models are broadly superior at cyber defense. It does, however, challenge a simpler assumption: that the most expensive closed model will necessarily be the most capable choice for a long, tool-using security workflow.
The researchers evaluated 26 models—19 open-weight and seven closed—on CTI-REALM, a benchmark built around 50 tasks spanning Linux, cloud and Kubernetes environments. Each task asks an agent to turn a cyber threat-intelligence report into a working detection. That requires more than recalling security terminology: the agent must interpret the report, map behaviors to MITRE ATT&CK techniques, identify relevant telemetry, query a live data store, and produce both a Kusto Query Language detection and a Sigma rule.
The published benchmark imposes a limit of 70 agent messages. Under that rule, an agent that reaches the cap without submitting receives zero credit for every checkpoint, even if it completed much of the investigation. The authors argue that this mixes two different qualities—whether a model can solve the task and whether it can solve it quickly enough. To separate them, they reran the evaluation with a budget of up to 300 messages and used a continuation procedure for interrupted trajectories. They report that no evaluated sample was truncated at the higher limit.
With that additional runway, the leading open-weight systems moved into the same performance band as the strongest closed models. GLM-5.3-flash recorded a reported reward of 0.74 at an estimated cost of $2, compared with 0.71 for Grok-4.6 at $28 and 0.69 for Claude Opus 5 at $50. A separate full 300-message run of GLM-5.3-flash scored 0.703, which the authors say would rank third. The paper cautions that the gaps among the leaders are small, and its uncertainty intervals frequently include zero, so the results support a shared top tier rather than a definitive winner.
Cost is one of the study’s sharpest findings, but also one of the easiest to overstate. Among the ten highest-scoring models, rewards ranged from 0.66 to 0.74 while estimated costs ranged from $2 to $50. The correlation between price and reward within that group was only 0.14, although it was stronger—0.64—across all 26 models. The estimates use published list prices, and the authors label costs for the GPT-5.6 family as lower bounds because a cache-read price was not available.
The study also revisits what the benchmark should count as success. Its stage-level metrics execute submitted queries against the telemetry store and compare their results with answer keys. The authors contrast that with published scoring signals that may reward mentioning the right action or returning pasted rows without demonstrating that a detection actually runs. Under the refined measures, closed models lead on some detection stages, yet the best open and closed systems remain within 0.04 of each other on every metric the paper reports.
No single model dominated the entire workflow. Different systems led at different stages, from interpreting the threat report to writing the final detection artifacts. The researchers therefore suggest that a production system could route separate stages to models suited to each one instead of selecting one model for the whole job. Their measurements also found little relationship between final reward and either total output volume or the share of tokens devoted to reasoning; the reported correlation for reasoning share was 0.07.
Those conclusions come with meaningful limits. The evaluation covers one benchmark and 50 tasks, and the paper reports a single measurement for each task rather than repeated trials. The authors explicitly warn against treating narrow score differences as a stable ordering. The work is also an arXiv preprint, so its methods and conclusions should be read as research findings awaiting broader scrutiny rather than settled evidence about real-world security operations.
Even with those constraints, the paper makes a useful point for teams choosing AI systems for cyber work: evaluation rules can change the apparent winner. A short cutoff may favor agents that submit quickly, while a longer budget can reveal models that reason more slowly but still reach a valid detection. On this test, once that constraint was relaxed, access to model weights and a much lower estimated price no longer implied a large capability penalty.
Comments
Loading comments…