LLM benchmark: intelligence, price and AI pentesting performance.
Two independent benchmarks mapped to our catalog: the Artificial Analysis Intelligence Index for general capability and BB Arena for autonomous vulnerability discovery. We maintain both datasets by hand and state source and date.
More intelligence per euro.
Each point is a model from our catalog: higher means a stronger index score, further left means cheaper. The price is the current 2342.ai catalog price, not the vendor's list price.
- EU or Germany data path
- External data path
Chart price: 75% input and 25% output at the current catalog price, without caching. Logarithmic axis: each grid step doubles the price. Index scores at the reasoning level Artificial Analysis lists by default. More reasoning changes token usage, not the price per token.
Benchmark: Artificial Analysis Intelligence Index v4.3.2, checked 2026-10-14. Prices: 2342.ai live catalog. Artificial Analysis ↗
| Model | Index | Reasoning | Price 3:1 | Input | Output | Data path | Source |
|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 58 | max | €11.44 | €5.72 | €28.60 | EU | Details ↗ |
| Claude Sonnet 5.5 | 56 | max | €5.72 | €2.86 | €14.30 | EU | Details ↗ |
| ChatGPT 6 Astra | 53 | max | €28.60 | €14.30 | €71.50 | External | Details ↗ |
| ChatGPT 6.1 Sol | 52 | max | €5.72 | €2.86 | €14.30 | EU | Details ↗ |
| Claude Opus 5 | 51 | max | €14.30 | €7.15 | €35.75 | EU | Details ↗ |
| ChatGPT 6 Sol | 48 | max | €5.72 | €2.86 | €14.30 | EU | Details ↗ |
| ChatGPT 5.6 Sol | 47 | max | €10.40 | €5.20 | €26.00 | EU | Details ↗ |
| GLM 5.3 | 45 | max | €3.07 | €2.00 | €6.29 | EU | Details ↗ |
| Kimi K3 | 44 | max | €7.80 | €3.90 | €19.50 | EU | Details ↗ |
| GLM 5.3 Flash | 42 | reasoning | €0.284 | €0.140 | €0.715 | EU | Details ↗ |
| ChatGPT 5.6 Terra | 42 | max | €5.85 | €2.60 | €15.60 | EU | Details ↗ |
| Claude Opus 4.8 | 42 | max | €14.30 | €7.15 | €35.75 | EU | Details ↗ |
| Gemini 3.7 Flash | 39 | high | €1.95 | €0.975 | €4.88 | EU | Details ↗ |
| ChatGPT 6 Luna | 38 | max | €0.286 | €0.143 | €0.715 | EU | Details ↗ |
| Claude Sonnet 5 | 38 | max | €5.72 | €2.86 | €14.30 | EU | Details ↗ |
| ChatGPT 5.6 Luna | 37 | max | €0.585 | €0.260 | €1.56 | EU | Details ↗ |
| DeepSeek V4 Pro | 36 | max | €2.69 | €2.08 | €4.52 | External | Details ↗ |
| Qwen 3.8 27B | 34 | xhigh | €0.949 | €0.312 | €2.86 | External | Details ↗ |
| Gemini 3.6 Flash | 34 | high | €1.95 | €0.975 | €4.88 | EU | Details ↗ |
| GLM 5.2 | 34 | max | €2.61 | €1.70 | €5.35 | External | Details ↗ |
| MiniMax M3 | 29 | reasoning | €0.682 | €0.390 | €1.56 | External | Details ↗ |
| Kimi K2.5 | 23 | reasoning | €1.56 | €0.780 | €3.90 | EU | Details ↗ |
AI pentesting benchmark: which models find and prove real vulnerabilities?
BB Arena runs each model as an autonomous agent against 100 private web and API labs, without source code or hints. A lab only counts when a hidden verifier confirms all four rungs up to the complete exploit.
| Rank | Model | Solved | Labs with refusals | On 2342.ai |
|---|---|---|---|---|
| 1 | Grok 4.6 high · xAI · Proprietary | 0 | – | |
| 2 | Claude Opus 5 high · Anthropic · Proprietary | 0 | EU | |
| 3 | Grok 4.7 high · xAI · Proprietary | 0 | – | |
| 4 | DeepSeek V4.1 Flash max · DeepSeek · Open weights | 0 | – | |
| 5 | GLM 5.3 max · Zhipu AI · Open weights | 0 | EU | |
| 6 | MiMo V2.6 Pro high · Xiaomi · Open weights | 0 | – | |
| 7 | Muse Spark 1.3 max · Meta · Proprietary | 0 | – | |
| 8 | Mistral Large 4.0 high · Mistral · Proprietary | 0 | – | |
| 9 | SWE-2 max · Cognition · Proprietary | 0 | – | |
| 10 | Step-5 Preview high · StepFun · Proprietary | 0 | – | |
| 11 | MiMo V2.6 Flash high · Xiaomi · Open weights | 0 | – | |
| 12 | Claude Mythos 5.1 high · Anthropic · Proprietary | 52 | – | |
| 13 | GLM 5.3 Flash max · Zhipu AI · Open weights | 0 | EU | |
| 14 | ChatGPT 5.6 Sol high · OpenAI · Proprietary | 0 | EU | |
| 15 | Claude Sonnet 5 high · Anthropic · Proprietary | 1 | EU | |
| 16 | Gemini 3.8 Flash high · Google · Proprietary | 71 | – | |
| 17 | Claude Sonnet 5.5 high · Anthropic · Proprietary | 26 | EU | |
| 18 | Claude Opus 5.5 high · Anthropic · Proprietary | 51 | EU | |
| 19 | ChatGPT 6 Sol high · OpenAI · Proprietary | 0 | EU | |
| 20 | ChatGPT 6 Sol Daybreak Blue high · OpenAI · Proprietary | 0 | – | |
| 21 | Qwen3.8 Max max · Alibaba · Proprietary | 97 | – | |
| 22 | Qwen3.8 Flash max · Alibaba · Proprietary | 97 | – | |
| 23 | Claude Haiku 5.5 high · Anthropic · Proprietary | 0 | – | |
| 24 | ChatGPT 6.1 Sol high · OpenAI · Proprietary | 67 | EU | |
| 25 | ChatGPT 6 Astra high · OpenAI · Proprietary | 35 | External |
"Labs with refusals": labs where the provider's safeguards stopped at least one attempt, by refusing or by answering with a different model. Stopped attempts score zero. BB Arena tests through the vendor API at a fixed reasoning level; through 2342.ai, behavior can differ by endpoint and setting. We do not reproduce BB Arena's cost figures: they are based on vendor list prices on the test date, not on our prices.
Source: du Preez, M. (2026). BB Arena: a black-box cyber benchmark for AI models (bbarena v1.0). MDP SEC. Accessed 2026-10-14. blackboxarena.ai ↗ · BB Arena methodology ↗
The security LLMs and abliterated models in our catalog have not been tested by BB Arena yet. We only show published results.
Measure it yourself, not just compare.
Benchmarks point in a direction. Whether a model reliably solves your task is best tested on your own data, with budget and data path under control.