Test results
What the runs actually show.
Real fleet eval runs from the skill-matrix suite — 547 tasks across 12 models and 8 machines. Each task runs under a fixed Hermes harness; we report median speed, time to first token where available, and averaged quality and accuracy.
Total runs
547
Models
12
Machines
8
V3 results
20
Filter by hardware
Summary by model
Aggregated scores across all machines and runs (40 rows shown).
| Tier | Model | Machine | Tests | Capable | Avg Quality | Avg Accuracy | Median Speed | TTFT | Last tested |
|---|---|---|---|---|---|---|---|---|---|
| Large — 32B+ class | anthropic/claude-sonnet-5 | Cloud instance | 1 | 0/1 | 2.0 | 1.0 | 46.0 | — | 2026-08-26 |
| Tiny — 1B–8B class | gemma3:4b | Mac Mini (M2 Pro, 16 GB)Mac | 19 | 17/19 | 4.4 | 4.4 | 6.4 | — | 2026-08-30 |
| Tiny — 1B–8B class | gemma3:4b | Custom PC (64 GB RAM + NVIDIA RTX 4070, 16 GB VRAM) | 13 | 12/13 | 4.0 | 4.1 | 3.3 | — | 2026-08-26 |
| Tiny — 1B–8B class | gemma3:4b | MacBook Pro (M1 Pro, 16 GB)Mac | 13 | 12/13 | 4.0 | 4.1 | 6.9 | — | 2026-08-25 |
| Tiny — 1B–8B class | gemma3:4b | Mac Mini (M4 Pro, 24 GB)Mac | 13 | 12/13 | 4.2 | 4.2 | 3.5 | — | 2026-08-25 |
| Tiny — 1B–8B class | gemma3:4b | MacBook Pro 16" (M4 Pro, 24 GB)Mac | 13 | 12/13 | 4.2 | 4.2 | 3.2 | — | 2026-08-25 |
| Tiny — 1B–8B class | gemma3:4b | Mac Studio (M4 Max, 36 GB)Mac | 13 | 12/13 | 4.0 | 4.1 | 2.4 | — | 2026-08-25 |
| Untiered | gemma4:12b-it-qat | Mac Mini (M2 Pro, 16 GB)Mac | 19 | 18/19 | 4.7 | 4.7 | 39.2 | — | 2026-08-30 |
| Untiered | gemma4:12b-it-qat | Custom PC (64 GB RAM + NVIDIA RTX 4070, 16 GB VRAM) | 13 | 12/13 | 4.5 | 4.5 | 23.1 | — | 2026-08-26 |
| Untiered | gemma4:12b-it-qat | Mac Mini (M2 Pro, 16 GB)Mac | 2 | 2/2 | 5.0 | 5.0 | 55.7 | — | 2026-08-25 |
| Untiered | gemma4:12b-it-qat | Mac Mini (M4 Pro, 24 GB)Mac | 13 | 13/13 | 4.5 | 4.6 | 57.2 | — | 2026-08-25 |
| Untiered | gemma4:12b-it-qat | MacBook Pro 16" (M4 Pro, 24 GB)Mac | 13 | 13/13 | 4.4 | 4.5 | 40.7 | — | 2026-08-25 |
| Untiered | gemma4:12b-it-qat | Mac Studio (M4 Max, 36 GB)Mac | 13 | 12/13 | 4.7 | 4.8 | 34.4 | — | 2026-08-25 |
| Untiered | gemma4:12b | Mac Mini (M4 Pro, 24 GB)Mac | 19 | 18/19 | 4.4 | 4.5 | 45.7 | — | 2026-08-27 |
| Untiered | gemma4:12b | Mac Studio (M4 Max, 36 GB)Mac | 19 | 17/19 | 4.7 | 4.7 | 32.0 | — | 2026-08-27 |
| Untiered | latest | Mac Studio (M4 Max, 36 GB)Mac | 1 | 0/1 | — | — | 0.1 | — | 2026-09-01 |
| Tiny — 1B–8B class | llama3.1:8b | Mac Mini (M2 Pro, 16 GB)Mac | 19 | 16/19 | 4.2 | 4.3 | 5.0 | — | 2026-08-30 |
| Tiny — 1B–8B class | llama3.1:8b | Mac Mini (M4 Pro, 24 GB)Mac | 2 | 0/2 | 2.0 | 1.0 | 3.2 | — | 2026-08-25 |
| Tiny — 1B–8B class | llama3.1:8b | MacBook Pro 16" (M4 Pro, 24 GB)Mac | 13 | 12/13 | 4.2 | 4.2 | 3.1 | — | 2026-08-25 |
| Tiny — 1B–8B class | mistral-small:24b | Mac Mini (M4 Pro, 24 GB)Mac | 19 | 16/19 | 4.1 | 4.2 | 12.7 | — | 2026-08-27 |
| Tiny — 1B–8B class | mistral-small:24b | MacBook Pro 16" (M4 Pro, 24 GB)Mac | 19 | 16/19 | 4.3 | 4.4 | 11.8 | — | 2026-08-27 |
| Tiny — 1B–8B class | qwen3.5:4b | Mac Mini (M2 Pro, 16 GB)Mac | 19 | 12/19 | 4.8 | 4.5 | 216.2 | — | 2026-08-30 |
| Tiny — 1B–8B class | qwen3.5:4b | Custom PC (64 GB RAM + NVIDIA RTX 4070, 16 GB VRAM) | 13 | 7/13 | 5.0 | 4.7 | 35.9 | — | 2026-08-26 |
| Tiny — 1B–8B class | qwen3.5:4b | MacBook Pro (M1 Pro, 16 GB)Mac | 19 | 16/19 | 4.7 | 4.5 | 166.7 | — | 2026-08-27 |
| Tiny — 1B–8B class | qwen3.5:4b | Mac Mini (M4 Pro, 24 GB)Mac | 19 | 17/19 | 4.6 | 4.4 | 84.7 | — | 2026-08-27 |
| Tiny — 1B–8B class | qwen3.5:4b | MacBook Pro 16" (M4 Pro, 24 GB)Mac | 13 | 8/13 | 5.0 | 4.6 | 64.9 | — | 2026-08-25 |
| Untiered | qwen3.5:9b | Mac Mini (M2 Pro, 16 GB)Mac | 19 | 15/19 | 4.2 | 4.5 | 158.2 | — | 2026-08-30 |
| Untiered | qwen3.5:9b | Custom PC (64 GB RAM + NVIDIA RTX 4070, 16 GB VRAM) | 13 | 8/13 | 4.4 | 4.1 | 50.1 | — | 2026-08-26 |
| Untiered | qwen3.5:9b | Mac Mini (M4 Pro, 24 GB)Mac | 13 | 8/13 | 4.4 | 4.1 | 129.3 | — | 2026-08-25 |
| Untiered | qwen3.5:9b | MacBook Pro 16" (M4 Pro, 24 GB)Mac | 13 | 9/13 | 4.5 | 4.2 | 80.2 | — | 2026-08-25 |
| Untiered | qwen3.5:9b | Mac Studio (M4 Max, 36 GB)Mac | 13 | 11/13 | 4.6 | 4.3 | 65.0 | — | 2026-08-25 |
| Untiered | qwen3.6:latest | Custom PC (64 GB RAM + NVIDIA RTX 4070, 16 GB VRAM) | 19 | 15/19 | 4.6 | 4.4 | 80.4 | — | 2026-09-02 |
| Untiered | qwen3.6:latest | MacBook Pro (M1 Pro, 16 GB)Mac | 1 | 0/1 | — | — | 600.3 | — | 2026-09-02 |
| Untiered | qwen3.6:latest | Mac Mini (M4 Pro, 24 GB)Mac | 16 | 10/16 | 5.0 | 4.8 | 313.6 | — | 2026-09-02 |
| Untiered | qwen3.6:latest | MacBook Pro 16" (M4 Pro, 24 GB)Mac | 19 | 17/19 | 4.7 | 4.7 | 88.3 | — | 2026-09-02 |
| Untiered | qwen3.6:latest | Mac Studio (M4 Max, 36 GB)Mac | 14 | 14/14 | 5.0 | 4.8 | 46.0 | — | 2026-09-02 |
| Tiny — 1B–8B class | qwen3.8:27b | Custom PC (64 GB RAM + NVIDIA RTX 4070, 16 GB VRAM) | 19 | 17/19 | 4.8 | 4.8 | 66.5 | — | 2026-09-02 |
| Tiny — 1B–8B class | qwen3.8:27b | Mac Mini (M4 Pro, 24 GB)Mac | 19 | 16/19 | 4.6 | 4.8 | 75.8 | — | 2026-08-27 |
| Tiny — 1B–8B class | qwen3.8:27b | Mac Studio (M4 Max, 36 GB)Mac | 14 | 10/14 | 4.6 | 4.7 | 92.6 | — | 2026-08-27 |
| Untiered | qwen3.8 | Mac Studio (M4 Max, 36 GB)Mac | 1 | 0/1 | — | — | 0.1 | — | 2026-09-01 |
All runs (V3)
Every individual V3 long-context task result (20 rows shown).
| Tier | Model | Machine | Category | Test | Capable | Quality (1-5) | Accuracy (1-5) | Speed (sec) | TTFT (ms) | Out tokens | Tested |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Tiny — 1B–8B class | gemma3:4b | Mac Mini (M2 Pro, 16 GB)Mac | long_context | V3 | ✓ | 5.0 | 5.0 | 11.6 | — | 80 | 2026-08-30 |
| Untiered | gemma4:12b-it-qat | Mac Mini (M2 Pro, 16 GB)Mac | long_context | V3 | ✓ | 5.0 | 5.0 | 49.3 | — | 391 | 2026-08-30 |
| Untiered | gemma4:12b | Mac Mini (M4 Pro, 24 GB)Mac | long_context | V3 | ✓ | 5.0 | 5.0 | 34.4 | — | 376 | 2026-08-27 |
| Untiered | gemma4:12b | Mac Studio (M4 Max, 36 GB)Mac | long_context | V3 | ✓ | 5.0 | 5.0 | 8.5 | — | 270 | 2026-08-27 |
| Untiered | latest | Mac Studio (M4 Max, 36 GB)Mac | long_context | V3 | ✗ | — | — | 0.1 | — | — | 2026-09-01 |
| Tiny — 1B–8B class | llama3.1:8b | Mac Mini (M2 Pro, 16 GB)Mac | long_context | V3 | ✓ | 5.0 | 5.0 | 17.1 | — | 9 | 2026-08-30 |
| Tiny — 1B–8B class | mistral-small:24b | Mac Mini (M4 Pro, 24 GB)Mac | long_context | V3 | ✓ | 1.0 | 1.0 | 43.4 | — | 107 | 2026-08-27 |
| Tiny — 1B–8B class | mistral-small:24b | MacBook Pro 16" (M4 Pro, 24 GB)Mac | long_context | V3 | ✓ | 5.0 | 5.0 | 31.0 | — | 132 | 2026-08-27 |
| Tiny — 1B–8B class | qwen3.5:4b | Mac Mini (M2 Pro, 16 GB)Mac | long_context | V3 | ✗ | — | — | 206.3 | — | 4094 | 2026-08-30 |
| Tiny — 1B–8B class | qwen3.5:4b | MacBook Pro (M1 Pro, 16 GB)Mac | long_context | V3 | ✓ | 5.0 | 5.0 | 573.4 | — | 4949 | 2026-08-27 |
| Tiny — 1B–8B class | qwen3.5:4b | Mac Mini (M4 Pro, 24 GB)Mac | long_context | V3 | ✓ | 5.0 | 5.0 | 64.5 | — | 2929 | 2026-08-27 |
| Untiered | qwen3.5:9b | Mac Mini (M2 Pro, 16 GB)Mac | long_context | V3 | ✓ | 5.0 | 5.0 | 182.9 | — | 3980 | 2026-08-30 |
| Untiered | qwen3.6:latest | Custom PC (64 GB RAM + NVIDIA RTX 4070, 16 GB VRAM) | long_context | V3 | ✓ | 5.0 | 5.0 | 20.0 | — | 663 | 2026-09-02 |
| Untiered | qwen3.6:latest | MacBook Pro (M1 Pro, 16 GB)Mac | long_context | V3 | ✗ | — | — | 600.3 | — | — | 2026-09-02 |
| Untiered | qwen3.6:latest | Mac Mini (M4 Pro, 24 GB)Mac | long_context | V3 | ✓ | 5.0 | 5.0 | 256.8 | — | 948 | 2026-09-02 |
| Untiered | qwen3.6:latest | MacBook Pro 16" (M4 Pro, 24 GB)Mac | long_context | V3 | ✓ | 5.0 | 5.0 | 73.2 | — | 574 | 2026-09-02 |
| Untiered | qwen3.6:latest | Mac Studio (M4 Max, 36 GB)Mac | long_context | V3 | ✓ | 5.0 | 5.0 | 83.6 | — | 818 | 2026-09-02 |
| Tiny — 1B–8B class | qwen3.8:27b | Custom PC (64 GB RAM + NVIDIA RTX 4070, 16 GB VRAM) | long_context | V3 | ✓ | 5.0 | 5.0 | 104.8 | — | 284 | 2026-09-02 |
| Tiny — 1B–8B class | qwen3.8:27b | Mac Mini (M4 Pro, 24 GB)Mac | long_context | V3 | ✓ | 5.0 | 5.0 | 128.9 | — | 298 | 2026-08-27 |
| Untiered | qwen3.8 | Mac Studio (M4 Max, 36 GB)Mac | long_context | V3 | ✗ | — | — | 0.1 | — | — | 2026-09-01 |