Local LLM Benchmarks: Open Dataset

Every number on this page was measured on a machine sitting in my house. No vendor slides, no extrapolation from a card I’ve never touched, no averaging of Reddit comments. Two rigs, a handful of models, and the exact flags each run used.

The whole thing is also a JSON file you can pull down and use however you want.

Download benchmarks.json →

License and citation

This dataset is licensed CC BY 4.0. Use it in a paper, a blog post, a model card, a spreadsheet, a training set. Sell something built on top of it. The only thing I ask is attribution.

Bartlett, Mark. “InsiderLLM Local LLM Benchmark Dataset.” InsiderLLM, 2026. Licensed CC BY 4.0. https://insiderllm.com/benchmarks.json

If you’re citing a single row rather than the dataset, each row carries a source_article URL pointing at the write-up where that measurement first appeared, with the reasoning and caveats around it.

How we bench

Everything here is batch=1 single-user inference, because that’s how people actually run models at home. Nothing is served, nothing is batched across concurrent requests.

Rows come from two harness families, and they don’t mix:

• llama-bench rows report generation and prompt-processing rates at fixed context depths. Good for isolating a flag.
• Nine-prompt harness rows (am17an’s gist, bench_llm.py) report an aggregate plus a per-prompt breakdown, kept in the JSON. On this class of workload throughput swings more between prompts than between backends, so an aggregate on its own hides the story.

A number from one family is not comparable to a number from the other. The same RTX 3090 running the same Qwen 3.6-27B at the same quant clocks about 33 tok/s under bench_llm.py and about 42 tok/s under the gist, purely because the prompts are shaped differently. Both are honest; neither is the model’s “real” speed.

What we exclude

The rule is narrow on purpose: a row exists only if I measured it on my own hardware and published it in an article first.

That means a lot of perfectly reasonable numbers are not here. Community reports, vendor benchmarks, figures extrapolated from a similar card, anything I read in a thread and found plausible. Several of my own articles carry those figures where they’re useful, clearly labeled as someone else’s work. They stay out of this file, because the point of the file is that every row has one person standing behind it.

If you want the wider picture including community numbers, the individual articles are the place to look.

Why the rig spec matters

There’s a full hardware block on every row — CPU, system RAM type and speed, PCIe generation, whether a display was attached — and it’s not padding.

The moment any part of a model lives in system RAM, memory bandwidth becomes the thing you’re measuring. Rushuna runs DDR4-2133, roughly 34 GB/s, which is slow by current standards. Every expert-offload number from that box would be higher on a DDR5 machine. Quoting “38 tok/s on a 3060” without the RAM spec attached is close to meaningless.

Same for the display. Rushuna is headless, so all 12 GB of VRAM is available. A 3060 also driving a monitor gives up somewhere around 0.5-1 GB, which is enough to turn a config that fits into a config that throws an out-of-memory error.

The data

Reading the table

A few things worth knowing before you use these:

Speedup figures compare against a baseline in this same dataset. Speculative-decoding rows carry a baseline_ref in the JSON pointing at the autoregressive run they were measured against, on the same box, same day. No cross-referencing someone else’s baseline.

The -ncmoe 20 row is a trap and it’s in here deliberately. It’s the fastest 3060 config that loads, at 42.6 tok/s, and it falls over as soon as real context arrives. A sweep that only shows the settings that worked isn’t a sweep.

Some fields are null and stay null. I didn’t instrument host memory during these runs, so system_ram_used_gb is empty on all 50 rows. driver is empty on 17 of them — every Miu row measured before 2026-05-21, when that box moved to 580.159.03. The version it ran before that was never written down, and a null is worth more than a plausible guess backfilled from memory. repetitions is null wherever the repetition count wasn’t recorded at the time; where it was, it’s there, and the two sweeps don’t match (-r 5 on the Miu rows, -r 3 on the 3060 rows). A couple of rows record a flag as published prose because the exact value wasn’t written down either. Guessing would make the file look more complete and be worth less.

Engine builds move fast. Every row pins an engine build or commit hash. Speculative decoding in particular changes behavior commit to commit: the MTP flag got renamed on merge, and a bad default value made some configs slower for three days in May before anyone caught it. A number from a specific build is a fact about that build and nothing more.

Corrections

If you reproduce something here and get a materially different result, I want to hear it. Numbers that don’t replicate are more useful to me than numbers that do. hello@insiderllm.com.

The dataset carries a version and a last_updated field. New rows get appended as new benches run; existing rows change only to fix an error, and the version moves when they do.