LLLoki's Lab
News
News
◆The Den Builds the Den→
next up
AI Radar · Follow the people behind the progress.Explore people & new signals ↗

The Agent Leaderboard

What can your machine run?

Reproducible agent benchmarks, useful setup guides, and AI news for builders who would rather test the claim than repeat it.

See all results →See all newsRun the test yourself

Live · 547 eval runs

Updated from the fleet skill-matrix feed.

Best right now·from the latest runs
Best overallgemma4:12b-it-qatm2pro2/2 capable tasks · avg quality 5.0/See the result →Best Mac modelgemma4:12b-it-qatm2proLocal AI on Apple Silicon · 2/2 capable tasksSee the result →Fastest usefulqwen3.5:4bheimdall35.9s median · quality 5.0/See the result →

The Agent Leaderboard

Local models, judged by the work.

Live · 5 rows shown · top 5 by quality

See all results →
Tier:Tiny — 1B–8B classMedium — 14B–30B classLarge — 32B+ classMac rows: 1
01

gemma4:12b-it-qat

tier?

fleet-skill-matrix v2 · auto-scored

Mac mini M2 Pro

local inference

Mac
2/2capable
55.7smedian
85Q 5/5
02

qwen3.6:latest

medium

fleet-skill-matrix v2 · auto-scored

Mac Studio M4 Max

local inference

14/14capable
46smedian
85Q 5/5
03

qwen3.6:latest

medium

fleet-skill-matrix v2 · auto-scored

Mac mini M4 Pro (24GB)

local inference

10/16capable
313.6smedian
76Q 5/5
04

qwen3.5:4b

tiny

fleet-skill-matrix v2 · auto-scored

Mac mini M2 Pro

local inference

8/13capable
64.9smedian
75Q 5/5
05

qwen3.5:4b

tiny

fleet-skill-matrix v2 · auto-scored

Linux RTX 4070

local inference

7/13capable
35.9smedian
73Q 5/5

Ranked by averaged quality (1–5) then capability rate across the skill-matrix suite. Score blends quality (60%), capability rate (25%), and inference speed (15%). Lower median time = higher speed score. Tiers are a rough practical-memory grouping for orientation; they are not a model benchmark.
Full per-task data and all the fleet eval runs live on the results page.

Methodology

The test is the test.

Sponsors can fund hardware or segments. They cannot purchase a score, placement, or conclusion.

01

Fixed harness

The same versioned Hermes profile and tool configuration for every comparable run.

02

Three runs

Each task runs three times. We report the median and retain best, worst, and failures.

03

Public evidence

Privacy-safe raw JSON, runner scripts, scoring rules, and moderation history stay inspectable.

Community testing

Run it. Inspect it. Send the evidence.

The public runner will support macOS, Linux, Windows PowerShell, and WSL. Every submission receives a unique ID and a visible verification status.

Standard JSON schema

Privacy-safe raw output

Verified, unverified, or under review

Submit a benchmark result

Google sign-in is required for verified email and secure JSON upload.

Loki's Lab 101

Useful from the first command.

Free, practical courses for local models and agents—without a paywall.

01 / COMING NEXT

Choose the right local model

Match memory, hardware, context, and workload before downloading.

02 / COMING NEXT

Set up Ollama + Hermes

Build a clean, repeatable first local agent environment.

03 / COMING NEXT

Read a benchmark honestly

Understand coverage, speed, failures, and what one score leaves out.