{ "_meta": { "name": "InsiderLLM Local LLM Benchmark Dataset", "description": "Firsthand local-inference measurements taken on hardware owned and operated by InsiderLLM. Every row traces to a published article and, where one exists, to the original bench log.", "version": "1.2.0", "last_updated": "2026-07-28", "maintainer": { "name": "Mark Bartlett", "site": "https://insiderllm.com", "email": "hello@insiderllm.com" }, "canonical_url": "https://insiderllm.com/benchmarks.json", "human_readable_url": "https://insiderllm.com/benchmarks/", "license": "CC BY 4.0", "license_url": "https://creativecommons.org/licenses/by/4.0/", "citation": "Bartlett, Mark. \"InsiderLLM Local LLM Benchmark Dataset.\" InsiderLLM, 2026. Licensed CC BY 4.0. https://insiderllm.com/benchmarks.json", "row_count": 50, "methodology": { "scope": "Single-user local inference on consumer hardware we own. Generation speed is batch=1 unless a row says otherwise.", "inclusion_rule": "A row exists only if the number was measured on our hardware AND appears in a published InsiderLLM article. Community-reported, vendor-reported, and extrapolated figures are excluded from this file by design.", "rig_spec_matters": "System RAM type and speed are recorded because they set the ceiling on any expert-offload or layer-offload configuration. A DDR4-2133 result and a DDR5-6000 result on the same GPU are not the same measurement. PCIe generation and whether a display is attached both move the numbers too: a card driving a monitor loses roughly 0.5-1 GB of usable VRAM, and on the miu rig a desktop session was enough to turn two working configs into load failures.", "two_result_shapes": "Rows come from two harness families. llama-bench rows report generation and prompt-processing rates at fixed context depths. Nine-prompt harness rows (am17an's gist, bench_llm.py) report an aggregate plus a per_prompt or per_bench breakdown, because throughput on this class of workload varies more by prompt than by any flag.", "precision": "Values are recorded as published. Where a source article and a server log disagree in precision, the article value is used and the log is cited as corroboration in provenance.raw_log.", "units": "File sizes carry an explicit unit field because upstream sources report a mix of GB and GiB. No unit conversion has been applied to them. VRAM is the one place a conversion happens: result.vram_peak_mib is the raw MiB integer from nvidia-smi and result.vram_used_gb is that same figure in decimal GB. Rows predating 2026-07-28 carry only vram_used_gb, as published.", "known_gaps": "system_ram_used_gb is null across every row: host memory was never instrumented during these runs. Driver version is null on the miu rows dated before 2026-05-21, because the rig was on a different driver then and the version was not written down. Two values are inferred rather than stated: display_attached on the 21 miu rows measured before 2026-07-28, which the source articles imply by noting that a browser and mail client were resident during the run, and baseline_ref on the rushuna speculative-decoding row, where the article presents the run as a flag tried on top of its -ncmoe 24 base config without naming that config outright. Headless status is stated outright on the rushuna rows and on the 2026-07-28 miu rows. The display-attached miu rows from 2026-07-28 do not record a repetition count; the harness field says so.", "repetition_counts": "Repetition count is not uniform across rigs. The rushuna llama-bench rows ran -r 3; the miu 2026-07-28 sweep ran -r 5. More repeats is strictly better and the difference shows: tg128 error bars on the -r 5 sweep run 0.02 to 0.97 tok/s. Every llama-bench row carries its repetition count in provenance.harness, and the miu rows carry it as an explicit provenance.repetitions integer as well.", "cross_rig_comparison": "The two rigs do not have matched memory. Miu is DDR4-2667 (about 43 GB/s dual channel), rushuna is DDR4-2133 (about 34 GB/s), a ratio of 1.25x. That matters for any comparison involving expert offload, because the more of a model that lives in system RAM the more of the result is RAM bandwidth rather than GPU. At full offload (-ncmoe 40) the two curves narrow toward the RAM ratio, 36.11 against 28.1 tok/s or 1.28x, but they do not converge, and the residual gap is mostly the faster DIMMs. Read cross-rig offload comparisons as directional. Fully-resident configs, where nothing crosses the bus, are the ones that isolate the GPU." }, "schema": { "measurement_type": "measured | estimated. Every row released so far is measured. The field exists so estimated rows can be added later without ambiguity.", "date_measured_precision": "day | month. Month precision means the source article dates the run to a month, not a day.", "hardware": "Inlined in full on every row so each row is independently citable without a lookup.", "config.speculative_decoding": "null for plain autoregressive rows. Otherwise the method, draft budget, and draft model.", "result.generation_tok_s": "Token generation rate. For nine-prompt harness rows this is the mean across prompts.", "result.prompt_tok_s": "Prompt processing (prefill) rate. Null where the harness did not report it.", "baseline_ref": "On a speculative-decoding row, the id of the autoregressive row it was measured against.", "result.status": "ok | failed_to_load_model | failed_to_create_context. Present from v1.2.0. A failed row has null throughput and exists because a config that does not load is a result.", "result.generation_tok_s_stddev": "Standard deviation reported by llama-bench, in tok/s. Present only where the source log recorded it. Same for result.prompt_tok_s_stddev.", "result.vram_peak_mib": "Peak VRAM in MiB sampled from nvidia-smi across the whole run, raw and unconverted.", "telemetry": "Peak GPU temperature, SM clock and power draw for the run. Sampled per config, so the three depth rows of one config share one telemetry block. Present from v1.2.0 on rows whose log captured it.", "provenance.repetitions": "llama-bench -r value as an integer. Null where the run did not record it.", "hardware.motherboard": "Board model, recorded where it explains a platform limit. On the miu rig the B365 chipset and Coffee Lake CPU are why a Gen4-capable RTX 3090 negotiates PCIe 3.0.", "hardware.system_ram_detail": "DIMM count, size, part number and rank, where dmidecode captured it." }, "changelog": [ { "version": "1.2.0", "date": "2026-07-28", "changes": [ "Added 19 rows from a Qwen3.6-35B-A3B UD-Q4_K_M sweep on the miu RTX 3090, llama.cpp b10088 (67b9b0e), -r 5. Thirteen are headless: one row per -ncmoe value at empty context for all eleven values swept (0, 1, 2, 3, 4, 6, 8, 16, 24, 32, 40), plus 4K and 8K depth rows for -ncmoe 0, the recommended config. That is the same granularity the rushuna 3060 rows use. The remaining 33 measured points are in the raw log but are not published rows, because a row only exists here if it also appears in the source article.", "Six of those 19 rows come from an earlier display-attached run of the same sweep and are marked display_attached true. Two are load failures with null throughput: -ncmoe 0 could not load the model at all and -ncmoe 2 loaded the weights and then failed to create a context, both because a desktop session was holding about 4.4 GiB of VRAM. Both configs run fine headless, at 157.66 and 136.03 tok/s. The -ncmoe 24, 32 and 40 points from that run are deliberately excluded: swap was down to 248 kB free by then and the numbers are contaminated.", "Backfilled the rig spec onto all 21 pre-existing miu rows, which previously shipped with cpu, system_ram_gb, system_ram_type, system_ram_channels, system_ram_bandwidth_gbs and pcie all null. They now read i7-8086K (6c/12t), 64GB DDR4-2667 dual channel (about 43 GB/s), PCIe 3.0 x16. Two new hardware fields carry the detail behind those: system_ram_detail (4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank) and motherboard (ASRock B365M Pro4, which is why a Gen4 card runs at Gen3). No measured value changed on any of those rows. If you pulled an earlier version and filtered miu rows out for lacking a RAM spec, re-pull.", "Driver was not backfilled onto the miu rows dated before 2026-05-21. The rig was upgraded to 580.159.03 on that date and the previous version was never recorded, so those rows keep a null rather than an assumed value.", "New fields: result.status, result.generation_tok_s_stddev, result.prompt_tok_s_stddev, result.vram_peak_mib, a telemetry block (peak temperature, SM clock, power draw), provenance.repetitions and provenance.methodology_note. All are additive and present only on the new rows. Existing rows are unchanged apart from the hardware backfill.", "methodology gains two keys. repetition_counts records that rushuna ran -r 3 and the 2026-07-28 miu sweep ran -r 5. cross_rig_comparison records that the two rigs do not have matched memory, so miu-against-rushuna offload comparisons are directional: at full offload the gap is 1.28x against a 1.25x RAM-speed ratio.", "row_count 31 to 50." ] }, { "version": "1.1.0", "date": "2026-07-27", "changes": [ "Added one row: rushuna-qwen36-35b-a3b-udq4km-specdec-qwen35-08b, a Qwen 3.5 0.8B draft model paired with the 35B-A3B on the 3060. 11 tok/s at a 65% acceptance rate, against a 38.9 tok/s baseline. It is a negative result and it was already published in the source article; it was missing here because the original import took only the -ncmoe sweep. row_count 30 to 31. No existing row changed.", "known_gaps now names two inferred values rather than one, adding the baseline_ref on the new row." ] }, { "version": "1.0.1", "date": "2026-07-27", "changes": [ "Corrected hardware on the three 2026-05-19 MTP rows (miu-qwen36-27b-mtp-q4km-baseline-20260519, -n3-20260519, -n2-20260519). They read 'RTX 3090 24GB + RTX 3060 12GB', gpu_count 2, vram_gb 36. They now read 'RTX 3090 24GB', gpu_count 1, vram_gb 24. The source article's hardware line was wrong: MTP carries its draft head inside the target checkpoint, so there is no separate draft model and nothing to offload to a second card. The 3060 12GB belongs to the rushuna rig, a different machine. If you pulled v1.0.0 and grouped by GPU string or summed VRAM, re-pull. No measured value changed: every tok/s, acceptance rate, and wall-clock figure on those rows is as originally published.", "Relabelled file_size_unit from 'GB' to 'GiB' on the six rushuna Qwen3.6-35B-A3B UD-Q4_K_M rows. 20.6 is llama.cpp's figure, which is GiB; the same file is 22.1 GB in decimal units. The number was not converted, only the unit label corrected, and the file is now internally consistent with the 27B rows that already carried GiB." ] }, { "version": "1.0.0", "date": "2026-07-27", "changes": [ "Initial public release. 30 measured rows across two rigs." ] } ] }, "benchmarks": [ { "id": "rushuna-qwen36-35b-a3b-udq4km-ncmoe40-d0", "measurement_type": "measured", "date_measured": "2026-07", "date_measured_precision": "month", "hardware": { "rig": "rushuna", "gpu": "NVIDIA GeForce RTX 3060 12GB", "gpu_count": 1, "vram_gb": 12, "driver": "580.159.03", "cpu": "Intel Core i7-7700 (4c/8t)", "system_ram_gb": 32, "system_ram_type": "DDR4-2133", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 34, "pcie": "3.0 x16", "display_attached": false }, "model": { "name": "Qwen3.6-35B-A3B", "repo": "unsloth/Qwen3.6-35B-A3B-GGUF", "architecture": "moe", "params_total_b": 35, "params_active_b": 3, "quant": "UD-Q4_K_M", "file_size_value": 20.6, "file_size_unit": "GiB" }, "config": { "flags": "-ngl 99 -ncmoe 40 -fa 1", "n_gpu_layers": 99, "n_cpu_moe": 40, "flash_attention": true, "context_depth_tokens": 0, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 28.1, "prompt_tok_s": null, "vram_used_gb": 2.5, "system_ram_used_gb": null }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench -p 512 -n 128 -d 0,4096,8192 -r 3", "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": null }, "notes": "All 40 layers of routed experts held in system RAM. The floor of the offload sweep, and the point of it: a model whose weights never fully fit the card still holds 28 tok/s because only the selected experts move per token." }, { "id": "rushuna-qwen36-35b-a3b-udq4km-ncmoe32-d0", "measurement_type": "measured", "date_measured": "2026-07", "date_measured_precision": "month", "hardware": { "rig": "rushuna", "gpu": "NVIDIA GeForce RTX 3060 12GB", "gpu_count": 1, "vram_gb": 12, "driver": "580.159.03", "cpu": "Intel Core i7-7700 (4c/8t)", "system_ram_gb": 32, "system_ram_type": "DDR4-2133", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 34, "pcie": "3.0 x16", "display_attached": false }, "model": { "name": "Qwen3.6-35B-A3B", "repo": "unsloth/Qwen3.6-35B-A3B-GGUF", "architecture": "moe", "params_total_b": 35, "params_active_b": 3, "quant": "UD-Q4_K_M", "file_size_value": 20.6, "file_size_unit": "GiB" }, "config": { "flags": "-ngl 99 -ncmoe 32 -fa 1", "n_gpu_layers": 99, "n_cpu_moe": 32, "flash_attention": true, "context_depth_tokens": 0, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 31.8, "prompt_tok_s": null, "vram_used_gb": 6.1, "system_ram_used_gb": null }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench -p 512 -n 128 -d 0,4096,8192 -r 3", "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": null }, "notes": "Experts of 8 of 40 layers kept on the card." }, { "id": "rushuna-qwen36-35b-a3b-udq4km-ncmoe24-d0", "measurement_type": "measured", "date_measured": "2026-07", "date_measured_precision": "month", "hardware": { "rig": "rushuna", "gpu": "NVIDIA GeForce RTX 3060 12GB", "gpu_count": 1, "vram_gb": 12, "driver": "580.159.03", "cpu": "Intel Core i7-7700 (4c/8t)", "system_ram_gb": 32, "system_ram_type": "DDR4-2133", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 34, "pcie": "3.0 x16", "display_attached": false }, "model": { "name": "Qwen3.6-35B-A3B", "repo": "unsloth/Qwen3.6-35B-A3B-GGUF", "architecture": "moe", "params_total_b": 35, "params_active_b": 3, "quant": "UD-Q4_K_M", "file_size_value": 20.6, "file_size_unit": "GiB" }, "config": { "flags": "-ngl 99 -ncmoe 24 -fa 1", "n_gpu_layers": 99, "n_cpu_moe": 24, "flash_attention": true, "context_depth_tokens": 0, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 38.9, "prompt_tok_s": 413, "vram_used_gb": 9.8, "system_ram_used_gb": null }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench -p 512 -n 128 -d 0,4096,8192 -r 3", "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": null }, "notes": "The recommended config on this card. Leaves enough headroom that the KV cache can grow under real load, which -ncmoe 20 does not. On a 3060 that is also driving a display, expect to need 26 or 28 instead." }, { "id": "rushuna-qwen36-35b-a3b-udq4km-ncmoe24-d4096", "measurement_type": "measured", "date_measured": "2026-07", "date_measured_precision": "month", "hardware": { "rig": "rushuna", "gpu": "NVIDIA GeForce RTX 3060 12GB", "gpu_count": 1, "vram_gb": 12, "driver": "580.159.03", "cpu": "Intel Core i7-7700 (4c/8t)", "system_ram_gb": 32, "system_ram_type": "DDR4-2133", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 34, "pcie": "3.0 x16", "display_attached": false }, "model": { "name": "Qwen3.6-35B-A3B", "repo": "unsloth/Qwen3.6-35B-A3B-GGUF", "architecture": "moe", "params_total_b": 35, "params_active_b": 3, "quant": "UD-Q4_K_M", "file_size_value": 20.6, "file_size_unit": "GiB" }, "config": { "flags": "-ngl 99 -ncmoe 24 -fa 1", "n_gpu_layers": 99, "n_cpu_moe": 24, "flash_attention": true, "context_depth_tokens": 4096, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 38.5, "prompt_tok_s": 394, "vram_used_gb": null, "system_ram_used_gb": null }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench -p 512 -n 128 -d 0,4096,8192 -r 3", "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": null }, "notes": "VRAM at depth was not reported separately by the harness; the 9.8 GB figure belongs to the depth-0 row." }, { "id": "rushuna-qwen36-35b-a3b-udq4km-ncmoe24-d8192", "measurement_type": "measured", "date_measured": "2026-07", "date_measured_precision": "month", "hardware": { "rig": "rushuna", "gpu": "NVIDIA GeForce RTX 3060 12GB", "gpu_count": 1, "vram_gb": 12, "driver": "580.159.03", "cpu": "Intel Core i7-7700 (4c/8t)", "system_ram_gb": 32, "system_ram_type": "DDR4-2133", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 34, "pcie": "3.0 x16", "display_attached": false }, "model": { "name": "Qwen3.6-35B-A3B", "repo": "unsloth/Qwen3.6-35B-A3B-GGUF", "architecture": "moe", "params_total_b": 35, "params_active_b": 3, "quant": "UD-Q4_K_M", "file_size_value": 20.6, "file_size_unit": "GiB" }, "config": { "flags": "-ngl 99 -ncmoe 24 -fa 1", "n_gpu_layers": 99, "n_cpu_moe": 24, "flash_attention": true, "context_depth_tokens": 8192, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 38.2, "prompt_tok_s": 386, "vram_used_gb": null, "system_ram_used_gb": null }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench -p 512 -n 128 -d 0,4096,8192 -r 3", "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": null }, "notes": "0.7 tok/s lost across 8K of context. Expert-streaming cost per token is constant and flash attention keeps KV growth cheap, so depth barely registers here." }, { "id": "rushuna-qwen36-35b-a3b-udq4km-ncmoe20-d0", "measurement_type": "measured", "date_measured": "2026-07", "date_measured_precision": "month", "hardware": { "rig": "rushuna", "gpu": "NVIDIA GeForce RTX 3060 12GB", "gpu_count": 1, "vram_gb": 12, "driver": "580.159.03", "cpu": "Intel Core i7-7700 (4c/8t)", "system_ram_gb": 32, "system_ram_type": "DDR4-2133", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 34, "pcie": "3.0 x16", "display_attached": false }, "model": { "name": "Qwen3.6-35B-A3B", "repo": "unsloth/Qwen3.6-35B-A3B-GGUF", "architecture": "moe", "params_total_b": 35, "params_active_b": 3, "quant": "UD-Q4_K_M", "file_size_value": 20.6, "file_size_unit": "GiB" }, "config": { "flags": "-ngl 99 -ncmoe 20 -fa 1", "n_gpu_layers": 99, "n_cpu_moe": 20, "flash_attention": true, "context_depth_tokens": 0, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 42.6, "prompt_tok_s": null, "vram_used_gb": 11.7, "system_ram_used_gb": null }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench -p 512 -n 128 -d 0,4096,8192 -r 3", "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": null }, "notes": "Do not reproduce this one as a working config. It is the fastest setting that loads, but at 11.7 GB of 12 GB there is no room for the KV cache and it threw an out-of-memory error as soon as real context arrived. Included because the sweep is only honest with the failure point in it." }, { "id": "rushuna-qwen36-35b-a3b-udq4km-specdec-qwen35-08b", "measurement_type": "measured", "date_measured": "2026-07", "date_measured_precision": "month", "baseline_ref": "rushuna-qwen36-35b-a3b-udq4km-ncmoe24-d0", "hardware": { "rig": "rushuna", "gpu": "NVIDIA GeForce RTX 3060 12GB", "gpu_count": 1, "vram_gb": 12, "driver": "580.159.03", "cpu": "Intel Core i7-7700 (4c/8t)", "system_ram_gb": 32, "system_ram_type": "DDR4-2133", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 34, "pcie": "3.0 x16", "display_attached": false }, "model": { "name": "Qwen3.6-35B-A3B", "repo": "unsloth/Qwen3.6-35B-A3B-GGUF", "architecture": "moe", "params_total_b": 35, "params_active_b": 3, "quant": "UD-Q4_K_M", "file_size_value": 20.6, "file_size_unit": "GiB" }, "config": { "flags": null, "n_gpu_layers": null, "n_cpu_moe": null, "flash_attention": null, "context_depth_tokens": null, "batch": 1, "speculative_decoding": { "method": "draft model (conventional speculative decoding)", "draft_model": "Qwen3.5-0.8B", "draft_vram_gib": null, "draft_budget": null } }, "result": { "generation_tok_s": 11, "prompt_tok_s": null, "vram_used_gb": null, "system_ram_used_gb": null, "acceptance_rate": 0.65, "speedup_vs_baseline": 0.28 }, "provenance": { "engine": "llama.cpp", "engine_build": null, "engine_commit": null, "compute": "CUDA sm_86", "harness": null, "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": null }, "notes": "A negative result, and in the file on purpose: the configs that did not work are the ones nobody publishes. Bolting a Qwen 3.5 0.8B draft model onto the 35B-A3B dropped generation from 38.9 tok/s to 11, a 3.5x regression, and it did that while the drafter was guessing right 65% of the time. The acceptance rate is the point. A good drafter does not rescue this, because the cost sits on the verification side: verifying a draft window in a batch pulls from up to 64 experts per layer, which thrashes memory on a card already streaming experts across PCIe from system RAM, and the state-space layers cannot parallelize across the window anyway. The model's own in-checkpoint MTP head is a different mechanism and is not what this row measures. One inference is recorded rather than hidden: the article presents this as a flag tried on top of its -ncmoe 24 base config, so baseline_ref points at that row, but the pairing is not stated outright. The exact command line, engine build, and harness for this run were never published, so those fields stay null." }, { "id": "rushuna-qwen3-14b-q4km-ngl41", "measurement_type": "measured", "date_measured": "2026-07", "date_measured_precision": "month", "hardware": { "rig": "rushuna", "gpu": "NVIDIA GeForce RTX 3060 12GB", "gpu_count": 1, "vram_gb": 12, "driver": "580.159.03", "cpu": "Intel Core i7-7700 (4c/8t)", "system_ram_gb": 32, "system_ram_type": "DDR4-2133", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 34, "pcie": "3.0 x16", "display_attached": false }, "model": { "name": "Qwen3-14B", "repo": null, "architecture": "dense", "params_total_b": 14, "params_active_b": 14, "quant": "Q4_K_M", "file_size_value": 8.4, "file_size_unit": "GB" }, "config": { "flags": "-ngl 41", "n_gpu_layers": 41, "n_cpu_moe": null, "flash_attention": null, "context_depth_tokens": 0, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 35.9, "prompt_tok_s": null, "vram_used_gb": null, "system_ram_used_gb": null }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench", "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": null }, "notes": "Dense control for the offload comparison: every layer on the card, nothing left over. Roughly a tie with the 35B MoE at -ncmoe 24, which is carrying 2.5x the parameters from a position this model cannot survive." }, { "id": "rushuna-qwen3-14b-q4km-half-ram", "measurement_type": "measured", "date_measured": "2026-07", "date_measured_precision": "month", "hardware": { "rig": "rushuna", "gpu": "NVIDIA GeForce RTX 3060 12GB", "gpu_count": 1, "vram_gb": 12, "driver": "580.159.03", "cpu": "Intel Core i7-7700 (4c/8t)", "system_ram_gb": 32, "system_ram_type": "DDR4-2133", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 34, "pcie": "3.0 x16", "display_attached": false }, "model": { "name": "Qwen3-14B", "repo": null, "architecture": "dense", "params_total_b": 14, "params_active_b": 14, "quant": "Q4_K_M", "file_size_value": 8.4, "file_size_unit": "GB" }, "config": { "flags": "half of 41 layers offloaded to system RAM", "n_gpu_layers": null, "n_cpu_moe": null, "flash_attention": null, "context_depth_tokens": 0, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 5.7, "prompt_tok_s": null, "vram_used_gb": null, "system_ram_used_gb": null }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench", "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": null }, "notes": "Reproduction gap: the source article states 'half in RAM' but does not publish the exact -ngl value, so n_gpu_layers is null rather than guessed. A 6.3x drop from full-GPU. This is what offload costs a dense model, and why the MoE's 1.4x dip across the same move is the whole argument." }, { "id": "rushuna-qwen3-14b-q4km-all-ram", "measurement_type": "measured", "date_measured": "2026-07", "date_measured_precision": "month", "hardware": { "rig": "rushuna", "gpu": "NVIDIA GeForce RTX 3060 12GB", "gpu_count": 1, "vram_gb": 12, "driver": "580.159.03", "cpu": "Intel Core i7-7700 (4c/8t)", "system_ram_gb": 32, "system_ram_type": "DDR4-2133", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 34, "pcie": "3.0 x16", "display_attached": false }, "model": { "name": "Qwen3-14B", "repo": null, "architecture": "dense", "params_total_b": 14, "params_active_b": 14, "quant": "Q4_K_M", "file_size_value": 8.4, "file_size_unit": "GB" }, "config": { "flags": "all layers offloaded to system RAM", "n_gpu_layers": 0, "n_cpu_moe": null, "flash_attention": null, "context_depth_tokens": 0, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 3.05, "prompt_tok_s": null, "vram_used_gb": null, "system_ram_used_gb": null }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench", "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": null }, "notes": "11.8x down from full-GPU. Same card, same DDR4-2133, same PCIe 3.0 as the MoE rows — the only variable is how many bytes cross the bus per token, and that is set by active parameters, not total size." }, { "id": "miu-qwen36-27b-q4km-ar-humaneval", "measurement_type": "measured", "date_measured": "2026-04-30", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": null, "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "display_attached": true }, "model": { "name": "Qwen3.6-27B", "repo": "unsloth/Qwen3.6-27B-GGUF", "architecture": "dense", "params_total_b": 27, "params_active_b": 27, "quant": "Q4_K_M", "file_size_value": 16, "file_size_unit": "GB" }, "config": { "flags": "bench_llm.py defaults, batch=1, greedy", "n_gpu_layers": null, "n_cpu_moe": null, "flash_attention": null, "context_depth_tokens": null, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 32.95, "prompt_tok_s": null, "vram_used_gb": null, "system_ram_used_gb": null }, "provenance": { "engine": "Luce-Org lucebox-hub (DFlash fork)", "engine_build": null, "engine_commit": "April 30 2026 HEAD", "compute": "CUDA 12, sm_86", "harness": "scripts/bench_llm.py — HumanEval", "source_article": "https://insiderllm.com/guides/dflash-rtx-3090-bench-both-qwens/", "raw_log": "docs/bench-results/qwen36_bench_results.json" }, "notes": "Autoregressive baseline for the DFlash comparison. Target file size published as approximately 16 GB. Raw log records 32.952." }, { "id": "miu-qwen36-27b-q4km-ar-gsm8k", "measurement_type": "measured", "date_measured": "2026-04-30", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": null, "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "display_attached": true }, "model": { "name": "Qwen3.6-27B", "repo": "unsloth/Qwen3.6-27B-GGUF", "architecture": "dense", "params_total_b": 27, "params_active_b": 27, "quant": "Q4_K_M", "file_size_value": 16, "file_size_unit": "GB" }, "config": { "flags": "bench_llm.py defaults, batch=1, greedy", "n_gpu_layers": null, "n_cpu_moe": null, "flash_attention": null, "context_depth_tokens": null, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 32.86, "prompt_tok_s": null, "vram_used_gb": null, "system_ram_used_gb": null }, "provenance": { "engine": "Luce-Org lucebox-hub (DFlash fork)", "engine_build": null, "engine_commit": "April 30 2026 HEAD", "compute": "CUDA 12, sm_86", "harness": "scripts/bench_llm.py — GSM8K", "source_article": "https://insiderllm.com/guides/dflash-rtx-3090-bench-both-qwens/", "raw_log": "docs/bench-results/qwen36_bench_results.json" }, "notes": "Raw log records 32.857." }, { "id": "miu-qwen36-27b-q4km-ar-math500", "measurement_type": "measured", "date_measured": "2026-04-30", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": null, "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "display_attached": true }, "model": { "name": "Qwen3.6-27B", "repo": "unsloth/Qwen3.6-27B-GGUF", "architecture": "dense", "params_total_b": 27, "params_active_b": 27, "quant": "Q4_K_M", "file_size_value": 16, "file_size_unit": "GB" }, "config": { "flags": "bench_llm.py defaults, batch=1, greedy", "n_gpu_layers": null, "n_cpu_moe": null, "flash_attention": null, "context_depth_tokens": null, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 32.84, "prompt_tok_s": null, "vram_used_gb": null, "system_ram_used_gb": null }, "provenance": { "engine": "Luce-Org lucebox-hub (DFlash fork)", "engine_build": null, "engine_commit": "April 30 2026 HEAD", "compute": "CUDA 12, sm_86", "harness": "scripts/bench_llm.py — Math500", "source_article": "https://insiderllm.com/guides/dflash-rtx-3090-bench-both-qwens/", "raw_log": "docs/bench-results/qwen36_bench_results.json" }, "notes": "Raw log records 32.841." }, { "id": "miu-qwen36-27b-q4km-dflash-humaneval", "measurement_type": "measured", "date_measured": "2026-04-30", "date_measured_precision": "day", "baseline_ref": "miu-qwen36-27b-q4km-ar-humaneval", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": null, "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "display_attached": true }, "model": { "name": "Qwen3.6-27B", "repo": "unsloth/Qwen3.6-27B-GGUF", "architecture": "dense", "params_total_b": 27, "params_active_b": 27, "quant": "Q4_K_M", "file_size_value": 16, "file_size_unit": "GB" }, "config": { "flags": "bench_llm.py defaults, batch=1, greedy", "n_gpu_layers": null, "n_cpu_moe": null, "flash_attention": null, "context_depth_tokens": null, "batch": 1, "speculative_decoding": { "method": "DFlash + DDTree", "draft_model": "z-lab/Qwen3.6-27B-DFlash (BF16)", "draft_vram_gib": 3.5, "draft_budget": null } }, "result": { "generation_tok_s": 92.62, "prompt_tok_s": null, "vram_used_gb": null, "system_ram_used_gb": null, "acceptance_length": 7.48, "speedup_vs_baseline": 2.81 }, "provenance": { "engine": "Luce-Org lucebox-hub (DFlash fork)", "engine_build": null, "engine_commit": "April 30 2026 HEAD", "compute": "CUDA 12, sm_86", "harness": "scripts/bench_llm.py — HumanEval", "source_article": "https://insiderllm.com/guides/dflash-rtx-3090-bench-both-qwens/", "raw_log": "docs/bench-results/qwen36_bench_results.json" }, "notes": "Fastest single result in the DFlash suite. Raw log records 92.621 tok/s, acceptance length 7.476, speedup 2.8108. DDTree at temp=1.0 is non-deterministic; single runs swing a few percent, so run twice before trusting a delta." }, { "id": "miu-qwen36-27b-q4km-dflash-gsm8k", "measurement_type": "measured", "date_measured": "2026-04-30", "date_measured_precision": "day", "baseline_ref": "miu-qwen36-27b-q4km-ar-gsm8k", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": null, "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "display_attached": true }, "model": { "name": "Qwen3.6-27B", "repo": "unsloth/Qwen3.6-27B-GGUF", "architecture": "dense", "params_total_b": 27, "params_active_b": 27, "quant": "Q4_K_M", "file_size_value": 16, "file_size_unit": "GB" }, "config": { "flags": "bench_llm.py defaults, batch=1, greedy", "n_gpu_layers": null, "n_cpu_moe": null, "flash_attention": null, "context_depth_tokens": null, "batch": 1, "speculative_decoding": { "method": "DFlash + DDTree", "draft_model": "z-lab/Qwen3.6-27B-DFlash (BF16)", "draft_vram_gib": 3.5, "draft_budget": null } }, "result": { "generation_tok_s": 73.9, "prompt_tok_s": null, "vram_used_gb": null, "system_ram_used_gb": null, "acceptance_length": 5.97, "speedup_vs_baseline": 2.25 }, "provenance": { "engine": "Luce-Org lucebox-hub (DFlash fork)", "engine_build": null, "engine_commit": "April 30 2026 HEAD", "compute": "CUDA 12, sm_86", "harness": "scripts/bench_llm.py — GSM8K", "source_article": "https://insiderllm.com/guides/dflash-rtx-3090-bench-both-qwens/", "raw_log": "docs/bench-results/qwen36_bench_results.json" }, "notes": "Weakest bench in the 3.6 DFlash suite. Raw log records 73.897 tok/s, acceptance length 5.967." }, { "id": "miu-qwen36-27b-q4km-dflash-math500", "measurement_type": "measured", "date_measured": "2026-04-30", "date_measured_precision": "day", "baseline_ref": "miu-qwen36-27b-q4km-ar-math500", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": null, "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "display_attached": true }, "model": { "name": "Qwen3.6-27B", "repo": "unsloth/Qwen3.6-27B-GGUF", "architecture": "dense", "params_total_b": 27, "params_active_b": 27, "quant": "Q4_K_M", "file_size_value": 16, "file_size_unit": "GB" }, "config": { "flags": "bench_llm.py defaults, batch=1, greedy", "n_gpu_layers": null, "n_cpu_moe": null, "flash_attention": null, "context_depth_tokens": null, "batch": 1, "speculative_decoding": { "method": "DFlash + DDTree", "draft_model": "z-lab/Qwen3.6-27B-DFlash (BF16)", "draft_vram_gib": 3.5, "draft_budget": null } }, "result": { "generation_tok_s": 85.86, "prompt_tok_s": null, "vram_used_gb": null, "system_ram_used_gb": null, "acceptance_length": 6.94, "speedup_vs_baseline": 2.61 }, "provenance": { "engine": "Luce-Org lucebox-hub (DFlash fork)", "engine_build": null, "engine_commit": "April 30 2026 HEAD", "compute": "CUDA 12, sm_86", "harness": "scripts/bench_llm.py — Math500", "source_article": "https://insiderllm.com/guides/dflash-rtx-3090-bench-both-qwens/", "raw_log": "docs/bench-results/qwen36_bench_results.json" }, "notes": "Raw log records 85.856 tok/s, acceptance length 6.941. Suite mean across the three benches: 84.13 tok/s, 2.56x." }, { "id": "miu-qwen35-27b-q4km-ar-humaneval", "measurement_type": "measured", "date_measured": "2026-04-30", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": null, "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "display_attached": true }, "model": { "name": "Qwen3.5-27B", "repo": "unsloth/Qwen3.5-27B-GGUF", "architecture": "dense", "params_total_b": 27, "params_active_b": 27, "quant": "Q4_K_M", "file_size_value": null, "file_size_unit": null }, "config": { "flags": "bench_llm.py defaults, batch=1, greedy", "n_gpu_layers": null, "n_cpu_moe": null, "flash_attention": null, "context_depth_tokens": null, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 33.65, "prompt_tok_s": null, "vram_used_gb": null, "system_ram_used_gb": null }, "provenance": { "engine": "Luce-Org lucebox-hub (DFlash fork)", "engine_build": null, "engine_commit": "April 30 2026 HEAD", "compute": "CUDA 12, sm_86", "harness": "scripts/bench_llm.py — HumanEval", "source_article": "https://insiderllm.com/guides/dflash-rtx-3090-bench-both-qwens/", "raw_log": "docs/bench-results/qwen35_bench_results.json" }, "notes": "Raw log records 33.647. File size for the 3.5 target was not published." }, { "id": "miu-qwen35-27b-q4km-ar-gsm8k", "measurement_type": "measured", "date_measured": "2026-04-30", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": null, "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "display_attached": true }, "model": { "name": "Qwen3.5-27B", "repo": "unsloth/Qwen3.5-27B-GGUF", "architecture": "dense", "params_total_b": 27, "params_active_b": 27, "quant": "Q4_K_M", "file_size_value": null, "file_size_unit": null }, "config": { "flags": "bench_llm.py defaults, batch=1, greedy", "n_gpu_layers": null, "n_cpu_moe": null, "flash_attention": null, "context_depth_tokens": null, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 33.66, "prompt_tok_s": null, "vram_used_gb": null, "system_ram_used_gb": null }, "provenance": { "engine": "Luce-Org lucebox-hub (DFlash fork)", "engine_build": null, "engine_commit": "April 30 2026 HEAD", "compute": "CUDA 12, sm_86", "harness": "scripts/bench_llm.py — GSM8K", "source_article": "https://insiderllm.com/guides/dflash-rtx-3090-bench-both-qwens/", "raw_log": "docs/bench-results/qwen35_bench_results.json" }, "notes": "Raw log records 33.661." }, { "id": "miu-qwen35-27b-q4km-ar-math500", "measurement_type": "measured", "date_measured": "2026-04-30", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": null, "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "display_attached": true }, "model": { "name": "Qwen3.5-27B", "repo": "unsloth/Qwen3.5-27B-GGUF", "architecture": "dense", "params_total_b": 27, "params_active_b": 27, "quant": "Q4_K_M", "file_size_value": null, "file_size_unit": null }, "config": { "flags": "bench_llm.py defaults, batch=1, greedy", "n_gpu_layers": null, "n_cpu_moe": null, "flash_attention": null, "context_depth_tokens": null, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 33.69, "prompt_tok_s": null, "vram_used_gb": null, "system_ram_used_gb": null }, "provenance": { "engine": "Luce-Org lucebox-hub (DFlash fork)", "engine_build": null, "engine_commit": "April 30 2026 HEAD", "compute": "CUDA 12, sm_86", "harness": "scripts/bench_llm.py — Math500", "source_article": "https://insiderllm.com/guides/dflash-rtx-3090-bench-both-qwens/", "raw_log": "docs/bench-results/qwen35_bench_results.json" }, "notes": "Raw log records 33.690." }, { "id": "miu-qwen35-27b-q4km-dflash-humaneval", "measurement_type": "measured", "date_measured": "2026-04-30", "date_measured_precision": "day", "baseline_ref": "miu-qwen35-27b-q4km-ar-humaneval", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": null, "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "display_attached": true }, "model": { "name": "Qwen3.5-27B", "repo": "unsloth/Qwen3.5-27B-GGUF", "architecture": "dense", "params_total_b": 27, "params_active_b": 27, "quant": "Q4_K_M", "file_size_value": null, "file_size_unit": null }, "config": { "flags": "bench_llm.py defaults, batch=1, greedy", "n_gpu_layers": null, "n_cpu_moe": null, "flash_attention": null, "context_depth_tokens": null, "batch": 1, "speculative_decoding": { "method": "DFlash + DDTree", "draft_model": "z-lab/Qwen3.5-27B-DFlash (BF16)", "draft_vram_gib": 3.5, "draft_budget": null } }, "result": { "generation_tok_s": 92.74, "prompt_tok_s": null, "vram_used_gb": null, "system_ram_used_gb": null, "acceptance_length": 7.66, "speedup_vs_baseline": 2.76 }, "provenance": { "engine": "Luce-Org lucebox-hub (DFlash fork)", "engine_build": null, "engine_commit": "April 30 2026 HEAD", "compute": "CUDA 12, sm_86", "harness": "scripts/bench_llm.py — HumanEval", "source_article": "https://insiderllm.com/guides/dflash-rtx-3090-bench-both-qwens/", "raw_log": "docs/bench-results/qwen35_bench_results.json" }, "notes": "The upstream README headlines 3.43x on this bench. We got 2.76x on stock bench_llm.py defaults. Raw log records 92.737 tok/s, acceptance length 7.658." }, { "id": "miu-qwen35-27b-q4km-dflash-gsm8k", "measurement_type": "measured", "date_measured": "2026-04-30", "date_measured_precision": "day", "baseline_ref": "miu-qwen35-27b-q4km-ar-gsm8k", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": null, "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "display_attached": true }, "model": { "name": "Qwen3.5-27B", "repo": "unsloth/Qwen3.5-27B-GGUF", "architecture": "dense", "params_total_b": 27, "params_active_b": 27, "quant": "Q4_K_M", "file_size_value": null, "file_size_unit": null }, "config": { "flags": "bench_llm.py defaults, batch=1, greedy", "n_gpu_layers": null, "n_cpu_moe": null, "flash_attention": null, "context_depth_tokens": null, "batch": 1, "speculative_decoding": { "method": "DFlash + DDTree", "draft_model": "z-lab/Qwen3.5-27B-DFlash (BF16)", "draft_vram_gib": 3.5, "draft_budget": null } }, "result": { "generation_tok_s": 83.33, "prompt_tok_s": null, "vram_used_gb": null, "system_ram_used_gb": null, "acceptance_length": 6.75, "speedup_vs_baseline": 2.48 }, "provenance": { "engine": "Luce-Org lucebox-hub (DFlash fork)", "engine_build": null, "engine_commit": "April 30 2026 HEAD", "compute": "CUDA 12, sm_86", "harness": "scripts/bench_llm.py — GSM8K", "source_article": "https://insiderllm.com/guides/dflash-rtx-3090-bench-both-qwens/", "raw_log": "docs/bench-results/qwen35_bench_results.json" }, "notes": "Raw log records 83.326 tok/s, acceptance length 6.749." }, { "id": "miu-qwen35-27b-q4km-dflash-math500", "measurement_type": "measured", "date_measured": "2026-04-30", "date_measured_precision": "day", "baseline_ref": "miu-qwen35-27b-q4km-ar-math500", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": null, "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "display_attached": true }, "model": { "name": "Qwen3.5-27B", "repo": "unsloth/Qwen3.5-27B-GGUF", "architecture": "dense", "params_total_b": 27, "params_active_b": 27, "quant": "Q4_K_M", "file_size_value": null, "file_size_unit": null }, "config": { "flags": "bench_llm.py defaults, batch=1, greedy", "n_gpu_layers": null, "n_cpu_moe": null, "flash_attention": null, "context_depth_tokens": null, "batch": 1, "speculative_decoding": { "method": "DFlash + DDTree", "draft_model": "z-lab/Qwen3.5-27B-DFlash (BF16)", "draft_vram_gib": 3.5, "draft_budget": null } }, "result": { "generation_tok_s": 85.08, "prompt_tok_s": null, "vram_used_gb": null, "system_ram_used_gb": null, "acceptance_length": 6.91, "speedup_vs_baseline": 2.53 }, "provenance": { "engine": "Luce-Org lucebox-hub (DFlash fork)", "engine_build": null, "engine_commit": "April 30 2026 HEAD", "compute": "CUDA 12, sm_86", "harness": "scripts/bench_llm.py — Math500", "source_article": "https://insiderllm.com/guides/dflash-rtx-3090-bench-both-qwens/", "raw_log": "docs/bench-results/qwen35_bench_results.json" }, "notes": "Raw log records 85.076 tok/s, acceptance length 6.913. Suite mean across the three benches: 87.05 tok/s, 2.59x." }, { "id": "miu-qwen36-27b-mtp-q4km-baseline-20260506", "measurement_type": "measured", "date_measured": "2026-05-06", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": null, "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "display_attached": true }, "model": { "name": "Qwen3.6-27B-MTP", "repo": "RDson/Qwen3.6-27B-MTP-Q4_K_M-GGUF", "architecture": "dense", "params_total_b": 27, "params_active_b": 27, "quant": "Q4_K_M", "file_size_value": 15.35, "file_size_unit": "GiB" }, "config": { "flags": "-fa on -c 10000 -np 1 -ngl 99 --no-mmap --no-cache-prompt", "n_gpu_layers": 99, "n_cpu_moe": null, "flash_attention": true, "context_depth_tokens": null, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 42.44, "prompt_tok_s": null, "vram_used_gb": null, "system_ram_used_gb": null, "wall_time_s": 35.82, "tokens_predicted": 1421, "per_prompt": [ { "prompt": "code_python", "tok_s": 42.42 }, { "prompt": "code_cpp", "tok_s": 42.41 }, { "prompt": "explain_concept", "tok_s": 42.39 }, { "prompt": "summarize", "tok_s": 41.83 }, { "prompt": "qa_factual", "tok_s": 42.36 }, { "prompt": "translation", "tok_s": 43.73 }, { "prompt": "creative_short", "tok_s": 42.36 }, { "prompt": "stepwise_math", "tok_s": 42.36 }, { "prompt": "long_code_review", "tok_s": 42.09 } ] }, "provenance": { "engine": "llama.cpp (am17an mtp-clean branch)", "engine_build": "b9030", "engine_commit": "267f8afe857b7bd1a49e4fde9138ab0f7be36625", "compute": "CUDA 12, sm_86", "harness": "am17an gist, 9 prompts, n_predict=192, temperature=0.0, seed=42", "source_article": "https://insiderllm.com/guides/dflash-vs-mtp-rtx-3090-head-to-head/", "raw_log": null }, "notes": "Autoregressive baseline on the MTP-capable GGUF, spec flags removed. Runs about 42 tok/s where the bench_llm.py suite on the same GPU runs about 33 — the gist prompts are shorter and structurally different, so the two harnesses are not interchangeable baselines." }, { "id": "miu-qwen36-27b-mtp-q4km-n3-20260506", "measurement_type": "measured", "date_measured": "2026-05-06", "date_measured_precision": "day", "baseline_ref": "miu-qwen36-27b-mtp-q4km-baseline-20260506", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": null, "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "display_attached": true }, "model": { "name": "Qwen3.6-27B-MTP", "repo": "RDson/Qwen3.6-27B-MTP-Q4_K_M-GGUF", "architecture": "dense", "params_total_b": 27, "params_active_b": 27, "quant": "Q4_K_M", "file_size_value": 15.35, "file_size_unit": "GiB" }, "config": { "flags": "--spec-type mtp --spec-draft-n-max 3 -fa on -c 10000 -np 1 -ngl 99 --no-mmap --no-cache-prompt", "n_gpu_layers": 99, "n_cpu_moe": null, "flash_attention": true, "context_depth_tokens": null, "batch": 1, "speculative_decoding": { "method": "MTP (PR #22673)", "draft_model": "in-checkpoint MTP head", "draft_vram_gib": null, "draft_budget": 3 } }, "result": { "generation_tok_s": 63.47, "prompt_tok_s": null, "vram_used_gb": null, "system_ram_used_gb": null, "wall_time_s": 25.19, "acceptance_rate": 0.69, "speedup_vs_baseline": 1.5, "wall_speedup_vs_baseline": 1.42, "per_prompt": [ { "prompt": "code_python", "tok_s": 68.74, "speedup": 1.62, "acceptance_rate": 0.764 }, { "prompt": "code_cpp", "tok_s": 65.1, "speedup": 1.53, "acceptance_rate": 0.705 }, { "prompt": "explain_concept", "tok_s": 61.98, "speedup": 1.46, "acceptance_rate": 0.646 }, { "prompt": "summarize", "tok_s": 64.85, "speedup": 1.55, "acceptance_rate": 0.726 }, { "prompt": "qa_factual", "tok_s": 66.44, "speedup": 1.57, "acceptance_rate": 0.722 }, { "prompt": "translation", "tok_s": 57.02, "speedup": 1.3, "acceptance_rate": 0.542 }, { "prompt": "creative_short", "tok_s": 57.11, "speedup": 1.35, "acceptance_rate": 0.576 }, { "prompt": "stepwise_math", "tok_s": 67.28, "speedup": 1.59, "acceptance_rate": 0.764 }, { "prompt": "long_code_review", "tok_s": 62.73, "speedup": 1.49, "acceptance_rate": 0.694 } ] }, "provenance": { "engine": "llama.cpp (am17an mtp-clean branch)", "engine_build": "b9030", "engine_commit": "267f8afe857b7bd1a49e4fde9138ab0f7be36625", "compute": "CUDA 12, sm_86", "harness": "am17an gist, 9 prompts, n_predict=192, temperature=0.0, seed=42", "source_article": "https://insiderllm.com/guides/dflash-vs-mtp-rtx-3090-head-to-head/", "raw_log": null }, "notes": "Lower bound, not a ceiling: server logs showed 'draft size 3 exceeds max 2, truncating' on several prompts, so the effective draft budget was 2 rather than the requested 3 on part of the run. The flag was renamed to --spec-type draft-mtp when PR #22673 merged to mainline on May 16 2026; --spec-type mtp is branch-era syntax and silently does nothing on current builds." }, { "id": "miu-qwen36-27b-mtp-q4km-baseline-20260519", "measurement_type": "measured", "date_measured": "2026-05-19", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": null, "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "display_attached": true }, "model": { "name": "Qwen3.6-27B-MTP", "repo": "RDson/Qwen3.6-27B-MTP-Q4_K_M-GGUF", "architecture": "dense", "params_total_b": 27, "params_active_b": 27, "quant": "Q4_K_M", "file_size_value": 15.35, "file_size_unit": "GiB" }, "config": { "flags": "-ngl 99 -c 10000 --no-mmap --no-cache-prompt -fa on -np 1", "n_gpu_layers": 99, "n_cpu_moe": null, "flash_attention": true, "context_depth_tokens": null, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 38.3, "prompt_tok_s": null, "vram_used_gb": null, "system_ram_used_gb": null, "wall_time_s": 50.64, "per_prompt": null }, "provenance": { "engine": "llama.cpp (am17an mtp-clean branch)", "engine_build": null, "engine_commit": "2dff7ff8f", "compute": "CUDA 12, sm_86", "harness": "am17an gist, 9 prompts, n_predict=192, temperature=0.0, seed=42", "source_article": "https://insiderllm.com/guides/wicked-fast-qwen-3-6-27b-mtp-rtx-3090/", "raw_log": null }, "notes": "Rig correction, applied 2026-07-27. The source article originally described this run as a 3090 paired with a 3060 12GB \"for the draft offload.\" That was wrong. MTP carries its draft head inside the target checkpoint, so there is no separate draft model and nothing to offload to a second card; the article's own model line says as much. The May 6 and May 22 runs on this box both state single-3090 outright, and the 3060 12GB belongs to a different machine (rushuna). The dual-GPU detail most likely bled in from cturan's community 3090+3060 bench, discussed nearby in the same article. Hardware fields now read single 3090, 24 GB, matching every other miu row; the article prose has been corrected to match. Per-prompt baseline figures were not published for this date." }, { "id": "miu-qwen36-27b-mtp-q4km-n3-20260519", "measurement_type": "measured", "date_measured": "2026-05-19", "date_measured_precision": "day", "baseline_ref": "miu-qwen36-27b-mtp-q4km-baseline-20260519", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": null, "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "display_attached": true }, "model": { "name": "Qwen3.6-27B-MTP", "repo": "RDson/Qwen3.6-27B-MTP-Q4_K_M-GGUF", "architecture": "dense", "params_total_b": 27, "params_active_b": 27, "quant": "Q4_K_M", "file_size_value": 15.35, "file_size_unit": "GiB" }, "config": { "flags": "--spec-type draft-mtp --spec-draft-n-max 3 -ngl 99 -c 10000 --no-mmap --no-cache-prompt -fa on -np 1", "n_gpu_layers": 99, "n_cpu_moe": null, "flash_attention": true, "context_depth_tokens": null, "batch": 1, "speculative_decoding": { "method": "MTP (PR #22673)", "draft_model": "in-checkpoint MTP head", "draft_vram_gib": null, "draft_budget": 3 } }, "result": { "generation_tok_s": 61.4, "prompt_tok_s": null, "vram_used_gb": null, "system_ram_used_gb": null, "wall_time_s": 27.2, "acceptance_rate": 0.711, "speedup_vs_baseline": 1.6, "wall_speedup_vs_baseline": 1.86, "per_prompt": [ { "prompt": "code_python", "tok_s": 64.0 }, { "prompt": "code_cpp", "tok_s": 62.7 }, { "prompt": "explain_concept", "tok_s": 58.8 }, { "prompt": "summarize", "tok_s": 64.4 }, { "prompt": "qa_factual", "tok_s": 64.9 }, { "prompt": "translation", "tok_s": 55.1 }, { "prompt": "creative_short", "tok_s": 56.0 }, { "prompt": "stepwise_math", "tok_s": 68.6, "acceptance_rate": 0.808 }, { "prompt": "long_code_review", "tok_s": 57.8, "wall_time_s": 5.77 } ] }, "provenance": { "engine": "llama.cpp (am17an mtp-clean branch)", "engine_build": null, "engine_commit": "2dff7ff8f", "compute": "CUDA 12, sm_86", "harness": "am17an gist, 9 prompts, n_predict=192, temperature=0.0, seed=42", "source_article": "https://insiderllm.com/guides/wicked-fast-qwen-3-6-27b-mtp-rtx-3090/", "raw_log": null }, "notes": "Wins wall clock over n=2 despite losing per-prompt throughput on 8 of 9 prompts. The entire margin is prefill on long_code_review: 5.77s here against 13.26s at n=2. Per-prompt acceptance was published only for stepwise_math. Same rig correction as the baseline row: this ran on a single 3090, not the 3090+3060 pair the source article first claimed." }, { "id": "miu-qwen36-27b-mtp-q4km-n2-20260519", "measurement_type": "measured", "date_measured": "2026-05-19", "date_measured_precision": "day", "baseline_ref": "miu-qwen36-27b-mtp-q4km-baseline-20260519", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": null, "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "display_attached": true }, "model": { "name": "Qwen3.6-27B-MTP", "repo": "RDson/Qwen3.6-27B-MTP-Q4_K_M-GGUF", "architecture": "dense", "params_total_b": 27, "params_active_b": 27, "quant": "Q4_K_M", "file_size_value": 15.35, "file_size_unit": "GiB" }, "config": { "flags": "--spec-type draft-mtp --spec-draft-n-max 2 -ngl 99 -c 10000 --no-mmap --no-cache-prompt -fa on -np 1", "n_gpu_layers": 99, "n_cpu_moe": null, "flash_attention": true, "context_depth_tokens": null, "batch": 1, "speculative_decoding": { "method": "MTP (PR #22673)", "draft_model": "in-checkpoint MTP head", "draft_vram_gib": null, "draft_budget": 2 } }, "result": { "generation_tok_s": 63.8, "prompt_tok_s": null, "vram_used_gb": null, "system_ram_used_gb": null, "wall_time_s": 34.44, "acceptance_rate": 0.805, "speedup_vs_baseline": 1.67, "wall_speedup_vs_baseline": 1.47, "per_prompt": [ { "prompt": "code_python", "tok_s": 67.9 }, { "prompt": "code_cpp", "tok_s": 65.7 }, { "prompt": "explain_concept", "tok_s": 63.3 }, { "prompt": "summarize", "tok_s": 65.5 }, { "prompt": "qa_factual", "tok_s": 65.9 }, { "prompt": "translation", "tok_s": 60.8 }, { "prompt": "creative_short", "tok_s": 63.1 }, { "prompt": "stepwise_math", "tok_s": 58.1 }, { "prompt": "long_code_review", "tok_s": 64.3, "wall_time_s": 13.26 } ] }, "provenance": { "engine": "llama.cpp (am17an mtp-clean branch)", "engine_build": null, "engine_commit": "2dff7ff8f", "compute": "CUDA 12, sm_86", "harness": "am17an gist, 9 prompts, n_predict=192, temperature=0.0, seed=42", "source_article": "https://insiderllm.com/guides/wicked-fast-qwen-3-6-27b-mtp-rtx-3090/", "raw_log": null }, "notes": "Higher acceptance and better per-prompt throughput than n=3, but a larger prefill penalty that costs it the wall clock on long inputs. Pick n=2 for short prompts, n=3 for long ones. Same rig correction as the baseline row: this ran on a single 3090, not the 3090+3060 pair the source article first claimed." }, { "id": "miu-shootout-mainline-20260522", "measurement_type": "measured", "date_measured": "2026-05-22", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": "580.159.03", "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "os": "Xubuntu 24.04, kernel 6.8.0-111", "display_attached": true }, "model": { "name": "Qwen3.6-27B", "repo": "unsloth/Qwen3.6-27B-GGUF", "architecture": "dense", "params_total_b": 27, "params_active_b": 27, "quant": "UD-Q4_K_XL", "file_size_value": 17, "file_size_unit": "GB" }, "config": { "flags": "-ngl 99 -c 10000 --no-mmap -fa on -np 1", "n_gpu_layers": 99, "n_cpu_moe": null, "flash_attention": true, "context_depth_tokens": null, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 40.16, "prompt_tok_s": null, "vram_used_gb": null, "system_ram_used_gb": null, "wall_time_s": 37.22, "per_prompt": [ { "prompt": "code_python", "tok_s": 40.3 }, { "prompt": "code_cpp", "tok_s": 40.2 }, { "prompt": "explain_concept", "tok_s": 40.2 }, { "prompt": "summarize", "tok_s": 39.3 }, { "prompt": "qa_factual", "tok_s": 40.2 }, { "prompt": "translation", "tok_s": 41.5 }, { "prompt": "creative_short", "tok_s": 40.0 }, { "prompt": "stepwise_math", "tok_s": 40.1 }, { "prompt": "long_code_review", "tok_s": 39.6 } ] }, "provenance": { "engine": "llama.cpp (mainline)", "engine_build": "b9079", "engine_commit": "f9cd456ea", "compute": "CUDA 12.8.93, sm_86", "harness": "am17an gist, 9 prompts, n_predict=192, temperature=0.0, seed=42", "source_article": "https://insiderllm.com/guides/best-24gb-backend-shootout-ik-llama-beellama-llamacpp/", "raw_log": "mainline_server.log (server log corroborates to 2dp: 40.28, 40.21, 40.20, 39.96, 40.13, 39.62)" }, "notes": "No-speculative-decoding baseline for the three-way backend shootout. This May 8 binary predates the MTP merge, so it is also a fair proxy for what a stock distro package or prebuilt release gives you." }, { "id": "miu-shootout-ik-llama-mtp-20260522", "measurement_type": "measured", "date_measured": "2026-05-22", "date_measured_precision": "day", "baseline_ref": "miu-shootout-mainline-20260522", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": "580.159.03", "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "os": "Xubuntu 24.04, kernel 6.8.0-111", "display_attached": true }, "model": { "name": "Qwen3.6-27B", "repo": "ubergarm/Qwen3.6-27B-GGUF", "architecture": "dense", "params_total_b": 27, "params_active_b": 27, "quant": "IQ4_KS-MTP", "file_size_value": 16, "file_size_unit": "GB" }, "config": { "flags": "--spec-stage mtp --draft 3 -ngl 99 -c 10000 --no-mmap -fa on -np 1", "n_gpu_layers": 99, "n_cpu_moe": null, "flash_attention": true, "context_depth_tokens": null, "batch": 1, "speculative_decoding": { "method": "MTP", "draft_model": "in-checkpoint MTP heads (ik_llama-exclusive quant)", "draft_vram_gib": null, "draft_budget": 3 } }, "result": { "generation_tok_s": 74.08, "prompt_tok_s": null, "vram_used_gb": null, "system_ram_used_gb": null, "wall_time_s": 22.38, "acceptance_rate": 0.885, "draft_tokens_generated": 981, "draft_tokens_accepted": 868, "speedup_vs_baseline": 1.66, "per_prompt": [ { "prompt": "code_python", "tok_s": 79.6, "acceptance_rate": 0.913 }, { "prompt": "code_cpp", "tok_s": 75.1, "acceptance_rate": 0.91 }, { "prompt": "explain_concept", "tok_s": 78.3, "acceptance_rate": 0.932 }, { "prompt": "summarize", "tok_s": 73.7, "acceptance_rate": 0.821 }, { "prompt": "qa_factual", "tok_s": 73.9, "acceptance_rate": 0.85 }, { "prompt": "translation", "tok_s": 68.0, "acceptance_rate": 0.75 }, { "prompt": "creative_short", "tok_s": 69.6, "acceptance_rate": 0.865 }, { "prompt": "stepwise_math", "tok_s": 84.0, "acceptance_rate": 0.963 }, { "prompt": "long_code_review", "tok_s": 64.5, "acceptance_rate": 0.822 } ] }, "provenance": { "engine": "ik_llama.cpp", "engine_build": "4530", "engine_commit": "48a55f74", "compute": "CUDA 12.8.93, sm_86", "harness": "am17an gist, 9 prompts, n_predict=192, temperature=0.0, seed=42", "source_article": "https://insiderllm.com/guides/best-24gb-backend-shootout-ik-llama-beellama-llamacpp/", "raw_log": "ik_llama_server.log (build=4530 commit=48a55f74; corroborates 83.98 and 64.50 to 2dp)" }, "notes": "Acceptance rates came from grepping the server log, not the HTTP timings response — ik_llama leaves draft_n empty in its API output, which matters if you are building bench tooling against it. One MTP warmup error fired during long_code_review ('Source hidden state size mismatch'); acceptance still landed at 82.2% on that prompt, so the data holds. Lowest variance of any backend here: every prompt between 64.5 and 84.0." }, { "id": "miu-shootout-beellama-dflash-v01x-20260522", "measurement_type": "measured", "date_measured": "2026-05-22", "date_measured_precision": "day", "baseline_ref": "miu-shootout-mainline-20260522", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": "580.159.03", "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "os": "Xubuntu 24.04, kernel 6.8.0-111", "display_attached": true }, "model": { "name": "Qwen3.6-27B", "repo": "unsloth/Qwen3.6-27B-GGUF", "architecture": "dense", "params_total_b": 27, "params_active_b": 27, "quant": "Q5_K_S", "file_size_value": 18, "file_size_unit": "GB" }, "config": { "flags": "-md -ctk turbo4 -ctv turbo3_tcq -ub 256 -ngl 99 -c 10000 --no-mmap -fa on -np 1", "n_gpu_layers": 99, "n_cpu_moe": null, "flash_attention": true, "context_depth_tokens": null, "batch": 1, "speculative_decoding": { "method": "DFlash", "draft_model": "Ardenzard/Qwen3.6-27B-DFlash-GGUF (1.7B, 986 MB)", "draft_vram_gib": null, "draft_budget": "adaptive (controller auto-scales during the run)" } }, "result": { "generation_tok_s": 73.31, "prompt_tok_s": null, "vram_used_gb": null, "system_ram_used_gb": null, "wall_time_s": 23.0, "acceptance_rate": 0.374, "draft_tokens_generated": 2809, "draft_tokens_accepted": 1050, "speedup_vs_baseline": 1.62, "per_prompt": [ { "prompt": "code_python", "tok_s": 118.9, "acceptance_rate": 0.658 }, { "prompt": "code_cpp", "tok_s": 93.3, "acceptance_rate": 0.45 }, { "prompt": "explain_concept", "tok_s": 76.5, "acceptance_rate": 0.327 }, { "prompt": "summarize", "tok_s": 65.8, "acceptance_rate": 0.319 }, { "prompt": "qa_factual", "tok_s": 59.0, "acceptance_rate": 0.333 }, { "prompt": "translation", "tok_s": 34.2, "acceptance_rate": 0.106 }, { "prompt": "creative_short", "tok_s": 61.0, "acceptance_rate": 0.391 }, { "prompt": "stepwise_math", "tok_s": 89.0, "acceptance_rate": 0.425 }, { "prompt": "long_code_review", "tok_s": 62.1, "acceptance_rate": 0.267 } ] }, "provenance": { "engine": "BeeLlama", "engine_build": "b9353", "engine_commit": "ba1fdce59", "compute": "CUDA 12.8.93, sm_86", "harness": "am17an gist, 9 prompts, n_predict=192, temperature=0.0, seed=42", "source_article": "https://insiderllm.com/guides/best-24gb-backend-shootout-ik-llama-beellama-llamacpp/", "raw_log": "beellama_server.log (corroborates 118.85, 93.34, 76.50, 59.02, 60.96, 88.98, 62.06 to 2dp)" }, "notes": "The counterexample to reading acceptance rate as a quality score. Less than half ik_llama's acceptance, and effectively the same wall clock, because it drafted 2,809 tokens to ik_llama's 981 and landed more accepted tokens in absolute terms (1,050 vs 868). Variance is the trade: 34.2 to 118.9 tok/s across the harness, over 4x ik_llama's spread." }, { "id": "miu-shootout-beellama-dflash-v020-20260525", "measurement_type": "measured", "date_measured": "2026-05-25", "date_measured_precision": "day", "baseline_ref": "miu-shootout-mainline-20260522", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": "580.159.03", "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "os": "Xubuntu 24.04, kernel 6.8.0-111", "display_attached": true }, "model": { "name": "Qwen3.6-27B", "repo": "unsloth/Qwen3.6-27B-GGUF", "architecture": "dense", "params_total_b": 27, "params_active_b": 27, "quant": "Q5_K_S", "file_size_value": 18, "file_size_unit": "GB" }, "config": { "flags": "-md -ctk turbo4 -ctv turbo3_tcq -ub 256 -ngl 99 -c 10000 --no-mmap -fa on -np 1", "n_gpu_layers": 99, "n_cpu_moe": null, "flash_attention": true, "context_depth_tokens": null, "batch": 1, "speculative_decoding": { "method": "DFlash", "draft_model": "Ardenzard/Qwen3.6-27B-DFlash-GGUF (1.7B, 986 MB)", "draft_vram_gib": null, "draft_budget": "adaptive (controller auto-scales during the run)" } }, "result": { "generation_tok_s": 75.42, "prompt_tok_s": null, "vram_used_gb": null, "system_ram_used_gb": null, "wall_time_s": 23.82, "acceptance_rate": 0.3073, "draft_tokens_generated": 3466, "draft_tokens_accepted": null, "speedup_vs_baseline": 1.56, "per_prompt": [ { "prompt": "code_python", "tok_s": 128.5 }, { "prompt": "code_cpp", "tok_s": 99.4 }, { "prompt": "explain_concept", "tok_s": 75.2 }, { "prompt": "summarize", "tok_s": 69.2 }, { "prompt": "qa_factual", "tok_s": 63.5 }, { "prompt": "translation", "tok_s": 31.1 }, { "prompt": "creative_short", "tok_s": 63.2 }, { "prompt": "stepwise_math", "tok_s": 89.1 }, { "prompt": "long_code_review", "tok_s": 59.6 } ] }, "provenance": { "engine": "BeeLlama v0.2.0", "engine_build": null, "engine_commit": "285933ed2", "compute": "CUDA 12.8.93, sm_86", "harness": "am17an gist, 9 prompts, n_predict=192, temperature=0.0, seed=42", "source_article": "https://insiderllm.com/guides/best-24gb-backend-shootout-ik-llama-beellama-llamacpp/", "raw_log": "beellama_v020_server.log (corroborates 128.54, 99.38, 75.19, 63.54, 63.17, 89.09, 59.57 to 2dp)" }, "notes": "Re-bench three days later, same harness and hardware, only the binary changed. A mixed result rather than the across-the-board speedup the community reported: +8.1% peak on code_python, −9.1% on translation, and a slightly slower wall clock. The controller explores wider after the v0.2.0 hardening work — 23.4% more drafts generated, a smaller fraction accepted. Reddit reports of 164 tok/s on this build do not replicate against this harness; our peak is 128.5. Accepted-token count was not published for this run." }, { "id": "miu-qwen36-35b-a3b-udq4km-ncmoe0-d0", "measurement_type": "measured", "date_measured": "2026-07-28", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": "580.159.03", "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "os": "Xubuntu 24.04, kernel 6.8.0-134", "display_attached": false }, "model": { "name": "Qwen3.6-35B-A3B", "repo": "unsloth/Qwen3.6-35B-A3B-GGUF", "architecture": "moe", "params_total_b": 35, "params_active_b": 3, "quant": "UD-Q4_K_M", "file_size_value": 20.6, "file_size_unit": "GiB" }, "config": { "flags": "-ngl 99 -fa 1", "n_gpu_layers": 99, "n_cpu_moe": null, "flash_attention": true, "context_depth_tokens": 0, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 157.66, "generation_tok_s_stddev": 0.97, "prompt_tok_s": 3674.27, "prompt_tok_s_stddev": 22.52, "status": "ok", "vram_peak_mib": 21652, "vram_used_gb": 22.7, "system_ram_used_gb": null }, "telemetry": { "peak_temp_c": 73, "sm_clock_mhz": 1905, "power_draw_w": 382.39 }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench -p 512 -n 128 -d 0,4096,8192 -r 5", "repetitions": 5, "methodology_note": "Five repetitions per data point here, against three on the rushuna rows. More repeats is strictly better, but it is a difference between the two rigs and not one to leave silent: tg128 error bars on this sweep run 0.02 to 0.97 tok/s, where the earlier display-attached run on the same box, whose repetition count was not recorded, produced bars of 0.64 to 13.47.", "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": "raw.log (miu headless sweep 2026-07-28T09:41:26-07:00 to 09:52:12-07:00)" }, "notes": "The headline config and the one to use on a 24GB card. The whole model is resident: 21,652 MiB of peak VRAM against the 24,123 MiB the card reports free headless, so 2,471 MiB (just under 2.4 GiB) is still spare at 8K depth. No expert offload, no -ncmoe flag at all. Headless: lightdm stopped and zero Xorg processes, so the full 24,123 MiB was free at start. MemAvailable held near 60 GiB and swap stayed 100% free for the whole sweep, so nothing here is a paging artifact." }, { "id": "miu-qwen36-35b-a3b-udq4km-ncmoe0-d4096", "measurement_type": "measured", "date_measured": "2026-07-28", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": "580.159.03", "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "os": "Xubuntu 24.04, kernel 6.8.0-134", "display_attached": false }, "model": { "name": "Qwen3.6-35B-A3B", "repo": "unsloth/Qwen3.6-35B-A3B-GGUF", "architecture": "moe", "params_total_b": 35, "params_active_b": 3, "quant": "UD-Q4_K_M", "file_size_value": 20.6, "file_size_unit": "GiB" }, "config": { "flags": "-ngl 99 -fa 1", "n_gpu_layers": 99, "n_cpu_moe": null, "flash_attention": true, "context_depth_tokens": 4096, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 156.25, "generation_tok_s_stddev": 0.66, "prompt_tok_s": 3469.25, "prompt_tok_s_stddev": 30.38, "status": "ok", "vram_peak_mib": 21652, "vram_used_gb": 22.7, "system_ram_used_gb": null }, "telemetry": { "peak_temp_c": 73, "sm_clock_mhz": 1905, "power_draw_w": 382.39 }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench -p 512 -n 128 -d 0,4096,8192 -r 5", "repetitions": 5, "methodology_note": "Five repetitions per data point here, against three on the rushuna rows. More repeats is strictly better, but it is a difference between the two rigs and not one to leave silent: tg128 error bars on this sweep run 0.02 to 0.97 tok/s, where the earlier display-attached run on the same box, whose repetition count was not recorded, produced bars of 0.64 to 13.47.", "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": "raw.log (miu headless sweep 2026-07-28T09:41:26-07:00 to 09:52:12-07:00)" }, "notes": "Same fully-resident config at 4K depth. Generation gives up 1.4 tok/s against empty context, prefill gives up about 5%. Headless: lightdm stopped and zero Xorg processes, so the full 24,123 MiB was free at start. MemAvailable held near 60 GiB and swap stayed 100% free for the whole sweep, so nothing here is a paging artifact." }, { "id": "miu-qwen36-35b-a3b-udq4km-ncmoe0-d8192", "measurement_type": "measured", "date_measured": "2026-07-28", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": "580.159.03", "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "os": "Xubuntu 24.04, kernel 6.8.0-134", "display_attached": false }, "model": { "name": "Qwen3.6-35B-A3B", "repo": "unsloth/Qwen3.6-35B-A3B-GGUF", "architecture": "moe", "params_total_b": 35, "params_active_b": 3, "quant": "UD-Q4_K_M", "file_size_value": 20.6, "file_size_unit": "GiB" }, "config": { "flags": "-ngl 99 -fa 1", "n_gpu_layers": 99, "n_cpu_moe": null, "flash_attention": true, "context_depth_tokens": 8192, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 153.24, "generation_tok_s_stddev": 0.67, "prompt_tok_s": 3340.47, "prompt_tok_s_stddev": 26.73, "status": "ok", "vram_peak_mib": 21652, "vram_used_gb": 22.7, "system_ram_used_gb": null }, "telemetry": { "peak_temp_c": 73, "sm_clock_mhz": 1905, "power_draw_w": 382.39 }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench -p 512 -n 128 -d 0,4096,8192 -r 5", "repetitions": 5, "methodology_note": "Five repetitions per data point here, against three on the rushuna rows. More repeats is strictly better, but it is a difference between the two rigs and not one to leave silent: tg128 error bars on this sweep run 0.02 to 0.97 tok/s, where the earlier display-attached run on the same box, whose repetition count was not recorded, produced bars of 0.64 to 13.47.", "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": "raw.log (miu headless sweep 2026-07-28T09:41:26-07:00 to 09:52:12-07:00)" }, "notes": "Same fully-resident config at 8K depth. Down 4.4 tok/s from empty across the full 8K, and the 21,652 MiB peak includes this depth's KV cache. Headless: lightdm stopped and zero Xorg processes, so the full 24,123 MiB was free at start. MemAvailable held near 60 GiB and swap stayed 100% free for the whole sweep, so nothing here is a paging artifact." }, { "id": "miu-qwen36-35b-a3b-udq4km-ncmoe1-d0", "measurement_type": "measured", "date_measured": "2026-07-28", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": "580.159.03", "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "os": "Xubuntu 24.04, kernel 6.8.0-134", "display_attached": false }, "model": { "name": "Qwen3.6-35B-A3B", "repo": "unsloth/Qwen3.6-35B-A3B-GGUF", "architecture": "moe", "params_total_b": 35, "params_active_b": 3, "quant": "UD-Q4_K_M", "file_size_value": 20.6, "file_size_unit": "GiB" }, "config": { "flags": "-ngl 99 -ncmoe 1 -fa 1", "n_gpu_layers": 99, "n_cpu_moe": 1, "flash_attention": true, "context_depth_tokens": 0, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 146.45, "generation_tok_s_stddev": 0.98, "prompt_tok_s": 2766.34, "prompt_tok_s_stddev": 42.93, "status": "ok", "vram_peak_mib": 21220, "vram_used_gb": 22.25, "system_ram_used_gb": null }, "telemetry": { "peak_temp_c": 72, "sm_clock_mhz": 1905, "power_draw_w": 364.15 }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench -p 512 -n 128 -d 0,4096,8192 -r 5", "repetitions": 5, "methodology_note": "Five repetitions per data point here, against three on the rushuna rows. More repeats is strictly better, but it is a difference between the two rigs and not one to leave silent: tg128 error bars on this sweep run 0.02 to 0.97 tok/s, where the earlier display-attached run on the same box, whose repetition count was not recorded, produced bars of 0.64 to 13.47.", "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": "raw.log (miu headless sweep 2026-07-28T09:41:26-07:00 to 09:52:12-07:00)" }, "notes": "One layer's experts pushed to RAM and generation already drops 11.2 tok/s. This is the cost of the offload path itself, not of the volume of data moved. Headless: lightdm stopped and zero Xorg processes, so the full 24,123 MiB was free at start. MemAvailable held near 60 GiB and swap stayed 100% free for the whole sweep, so nothing here is a paging artifact." }, { "id": "miu-qwen36-35b-a3b-udq4km-ncmoe2-d0", "measurement_type": "measured", "date_measured": "2026-07-28", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": "580.159.03", "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "os": "Xubuntu 24.04, kernel 6.8.0-134", "display_attached": false }, "model": { "name": "Qwen3.6-35B-A3B", "repo": "unsloth/Qwen3.6-35B-A3B-GGUF", "architecture": "moe", "params_total_b": 35, "params_active_b": 3, "quant": "UD-Q4_K_M", "file_size_value": 20.6, "file_size_unit": "GiB" }, "config": { "flags": "-ngl 99 -ncmoe 2 -fa 1", "n_gpu_layers": 99, "n_cpu_moe": 2, "flash_attention": true, "context_depth_tokens": 0, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 136.03, "generation_tok_s_stddev": 0.79, "prompt_tok_s": 2221.43, "prompt_tok_s_stddev": 36.4, "status": "ok", "vram_peak_mib": 20756, "vram_used_gb": 21.76, "system_ram_used_gb": null }, "telemetry": { "peak_temp_c": 71, "sm_clock_mhz": 1905, "power_draw_w": 346.11 }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench -p 512 -n 128 -d 0,4096,8192 -r 5", "repetitions": 5, "methodology_note": "Five repetitions per data point here, against three on the rushuna rows. More repeats is strictly better, but it is a difference between the two rigs and not one to leave silent: tg128 error bars on this sweep run 0.02 to 0.97 tok/s, where the earlier display-attached run on the same box, whose repetition count was not recorded, produced bars of 0.64 to 13.47.", "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": "raw.log (miu headless sweep 2026-07-28T09:41:26-07:00 to 09:52:12-07:00)" }, "notes": "Runs fine headless at 136.03. This config failed to create a context in the earlier display-attached run, which made it look like a hardware boundary; it is not one. See miu-qwen36-35b-a3b-udq4km-ncmoe2-d0-display. Headless: lightdm stopped and zero Xorg processes, so the full 24,123 MiB was free at start. MemAvailable held near 60 GiB and swap stayed 100% free for the whole sweep, so nothing here is a paging artifact." }, { "id": "miu-qwen36-35b-a3b-udq4km-ncmoe3-d0", "measurement_type": "measured", "date_measured": "2026-07-28", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": "580.159.03", "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "os": "Xubuntu 24.04, kernel 6.8.0-134", "display_attached": false }, "model": { "name": "Qwen3.6-35B-A3B", "repo": "unsloth/Qwen3.6-35B-A3B-GGUF", "architecture": "moe", "params_total_b": 35, "params_active_b": 3, "quant": "UD-Q4_K_M", "file_size_value": 20.6, "file_size_unit": "GiB" }, "config": { "flags": "-ngl 99 -ncmoe 3 -fa 1", "n_gpu_layers": 99, "n_cpu_moe": 3, "flash_attention": true, "context_depth_tokens": 0, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 127.25, "generation_tok_s_stddev": 0.66, "prompt_tok_s": 1844.76, "prompt_tok_s_stddev": 24.31, "status": "ok", "vram_peak_mib": 20290, "vram_used_gb": 21.28, "system_ram_used_gb": null }, "telemetry": { "peak_temp_c": 70, "sm_clock_mhz": 1905, "power_draw_w": 335.79 }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench -p 512 -n 128 -d 0,4096,8192 -r 5", "repetitions": 5, "methodology_note": "Five repetitions per data point here, against three on the rushuna rows. More repeats is strictly better, but it is a difference between the two rigs and not one to leave silent: tg128 error bars on this sweep run 0.02 to 0.97 tok/s, where the earlier display-attached run on the same box, whose repetition count was not recorded, produced bars of 0.64 to 13.47.", "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": "raw.log (miu headless sweep 2026-07-28T09:41:26-07:00 to 09:52:12-07:00)" }, "notes": "Third step down the offload curve. Headless: lightdm stopped and zero Xorg processes, so the full 24,123 MiB was free at start. MemAvailable held near 60 GiB and swap stayed 100% free for the whole sweep, so nothing here is a paging artifact." }, { "id": "miu-qwen36-35b-a3b-udq4km-ncmoe4-d0", "measurement_type": "measured", "date_measured": "2026-07-28", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": "580.159.03", "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "os": "Xubuntu 24.04, kernel 6.8.0-134", "display_attached": false }, "model": { "name": "Qwen3.6-35B-A3B", "repo": "unsloth/Qwen3.6-35B-A3B-GGUF", "architecture": "moe", "params_total_b": 35, "params_active_b": 3, "quant": "UD-Q4_K_M", "file_size_value": 20.6, "file_size_unit": "GiB" }, "config": { "flags": "-ngl 99 -ncmoe 4 -fa 1", "n_gpu_layers": 99, "n_cpu_moe": 4, "flash_attention": true, "context_depth_tokens": 0, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 118.9, "generation_tok_s_stddev": 0.49, "prompt_tok_s": 1581.93, "prompt_tok_s_stddev": 17.06, "status": "ok", "vram_peak_mib": 19826, "vram_used_gb": 20.79, "system_ram_used_gb": null }, "telemetry": { "peak_temp_c": 70, "sm_clock_mhz": 1920, "power_draw_w": 322.7 }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench -p 512 -n 128 -d 0,4096,8192 -r 5", "repetitions": 5, "methodology_note": "Five repetitions per data point here, against three on the rushuna rows. More repeats is strictly better, but it is a difference between the two rigs and not one to leave silent: tg128 error bars on this sweep run 0.02 to 0.97 tok/s, where the earlier display-attached run on the same box, whose repetition count was not recorded, produced bars of 0.64 to 13.47.", "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": "raw.log (miu headless sweep 2026-07-28T09:41:26-07:00 to 09:52:12-07:00)" }, "notes": "The direct comparison point against the display-attached run: 118.90 here against 107.12 with a desktop session resident, a 9.9% cost for having a monitor plugged in. Headless: lightdm stopped and zero Xorg processes, so the full 24,123 MiB was free at start. MemAvailable held near 60 GiB and swap stayed 100% free for the whole sweep, so nothing here is a paging artifact." }, { "id": "miu-qwen36-35b-a3b-udq4km-ncmoe6-d0", "measurement_type": "measured", "date_measured": "2026-07-28", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": "580.159.03", "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "os": "Xubuntu 24.04, kernel 6.8.0-134", "display_attached": false }, "model": { "name": "Qwen3.6-35B-A3B", "repo": "unsloth/Qwen3.6-35B-A3B-GGUF", "architecture": "moe", "params_total_b": 35, "params_active_b": 3, "quant": "UD-Q4_K_M", "file_size_value": 20.6, "file_size_unit": "GiB" }, "config": { "flags": "-ngl 99 -ncmoe 6 -fa 1", "n_gpu_layers": 99, "n_cpu_moe": 6, "flash_attention": true, "context_depth_tokens": 0, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 106.56, "generation_tok_s_stddev": 0.49, "prompt_tok_s": 1272.87, "prompt_tok_s_stddev": 14.31, "status": "ok", "vram_peak_mib": 18898, "vram_used_gb": 19.82, "system_ram_used_gb": null }, "telemetry": { "peak_temp_c": 67, "sm_clock_mhz": 1920, "power_draw_w": 292.64 }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench -p 512 -n 128 -d 0,4096,8192 -r 5", "repetitions": 5, "methodology_note": "Five repetitions per data point here, against three on the rushuna rows. More repeats is strictly better, but it is a difference between the two rigs and not one to leave silent: tg128 error bars on this sweep run 0.02 to 0.97 tok/s, where the earlier display-attached run on the same box, whose repetition count was not recorded, produced bars of 0.64 to 13.47.", "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": "raw.log (miu headless sweep 2026-07-28T09:41:26-07:00 to 09:52:12-07:00)" }, "notes": "Crosses back under the ~101 tok/s figure the community had for this card, while still keeping the experts of 34 of 40 layers on the GPU. Headless: lightdm stopped and zero Xorg processes, so the full 24,123 MiB was free at start. MemAvailable held near 60 GiB and swap stayed 100% free for the whole sweep, so nothing here is a paging artifact." }, { "id": "miu-qwen36-35b-a3b-udq4km-ncmoe8-d0", "measurement_type": "measured", "date_measured": "2026-07-28", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": "580.159.03", "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "os": "Xubuntu 24.04, kernel 6.8.0-134", "display_attached": false }, "model": { "name": "Qwen3.6-35B-A3B", "repo": "unsloth/Qwen3.6-35B-A3B-GGUF", "architecture": "moe", "params_total_b": 35, "params_active_b": 3, "quant": "UD-Q4_K_M", "file_size_value": 20.6, "file_size_unit": "GiB" }, "config": { "flags": "-ngl 99 -ncmoe 8 -fa 1", "n_gpu_layers": 99, "n_cpu_moe": 8, "flash_attention": true, "context_depth_tokens": 0, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 94.54, "generation_tok_s_stddev": 0.57, "prompt_tok_s": 1061.75, "prompt_tok_s_stddev": 19.81, "status": "ok", "vram_peak_mib": 17970, "vram_used_gb": 18.84, "system_ram_used_gb": null }, "telemetry": { "peak_temp_c": 67, "sm_clock_mhz": 1920, "power_draw_w": 285.7 }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench -p 512 -n 128 -d 0,4096,8192 -r 5", "repetitions": 5, "methodology_note": "Five repetitions per data point here, against three on the rushuna rows. More repeats is strictly better, but it is a difference between the two rigs and not one to leave silent: tg128 error bars on this sweep run 0.02 to 0.97 tok/s, where the earlier display-attached run on the same box, whose repetition count was not recorded, produced bars of 0.64 to 13.47.", "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": "raw.log (miu headless sweep 2026-07-28T09:41:26-07:00 to 09:52:12-07:00)" }, "notes": "Offloading 8 of 40 layers costs 40% of generation speed and saves 3.6 GiB of VRAM. Headless: lightdm stopped and zero Xorg processes, so the full 24,123 MiB was free at start. MemAvailable held near 60 GiB and swap stayed 100% free for the whole sweep, so nothing here is a paging artifact." }, { "id": "miu-qwen36-35b-a3b-udq4km-ncmoe16-d0", "measurement_type": "measured", "date_measured": "2026-07-28", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": "580.159.03", "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "os": "Xubuntu 24.04, kernel 6.8.0-134", "display_attached": false }, "model": { "name": "Qwen3.6-35B-A3B", "repo": "unsloth/Qwen3.6-35B-A3B-GGUF", "architecture": "moe", "params_total_b": 35, "params_active_b": 3, "quant": "UD-Q4_K_M", "file_size_value": 20.6, "file_size_unit": "GiB" }, "config": { "flags": "-ngl 99 -ncmoe 16 -fa 1", "n_gpu_layers": 99, "n_cpu_moe": 16, "flash_attention": true, "context_depth_tokens": 0, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 66.96, "generation_tok_s_stddev": 0.61, "prompt_tok_s": 679.5, "prompt_tok_s_stddev": 9.91, "status": "ok", "vram_peak_mib": 14260, "vram_used_gb": 14.95, "system_ram_used_gb": null }, "telemetry": { "peak_temp_c": 64, "sm_clock_mhz": 1920, "power_draw_w": 244.23 }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench -p 512 -n 128 -d 0,4096,8192 -r 5", "repetitions": 5, "methodology_note": "Five repetitions per data point here, against three on the rushuna rows. More repeats is strictly better, but it is a difference between the two rigs and not one to leave silent: tg128 error bars on this sweep run 0.02 to 0.97 tok/s, where the earlier display-attached run on the same box, whose repetition count was not recorded, produced bars of 0.64 to 13.47.", "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": "raw.log (miu headless sweep 2026-07-28T09:41:26-07:00 to 09:52:12-07:00)" }, "notes": "Roughly the midpoint of the curve. VRAM is down to 14,260 MiB, which is the first point on this sweep that would also fit a 16GB card. Headless: lightdm stopped and zero Xorg processes, so the full 24,123 MiB was free at start. MemAvailable held near 60 GiB and swap stayed 100% free for the whole sweep, so nothing here is a paging artifact." }, { "id": "miu-qwen36-35b-a3b-udq4km-ncmoe24-d0", "measurement_type": "measured", "date_measured": "2026-07-28", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": "580.159.03", "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "os": "Xubuntu 24.04, kernel 6.8.0-134", "display_attached": false }, "model": { "name": "Qwen3.6-35B-A3B", "repo": "unsloth/Qwen3.6-35B-A3B-GGUF", "architecture": "moe", "params_total_b": 35, "params_active_b": 3, "quant": "UD-Q4_K_M", "file_size_value": 20.6, "file_size_unit": "GiB" }, "config": { "flags": "-ngl 99 -ncmoe 24 -fa 1", "n_gpu_layers": 99, "n_cpu_moe": 24, "flash_attention": true, "context_depth_tokens": 0, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 52.76, "generation_tok_s_stddev": 0.81, "prompt_tok_s": 502.29, "prompt_tok_s_stddev": 9.45, "status": "ok", "vram_peak_mib": 10550, "vram_used_gb": 11.06, "system_ram_used_gb": null }, "telemetry": { "peak_temp_c": 60, "sm_clock_mhz": 1800, "power_draw_w": 188.12 }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench -p 512 -n 128 -d 0,4096,8192 -r 5", "repetitions": 5, "methodology_note": "Five repetitions per data point here, against three on the rushuna rows. More repeats is strictly better, but it is a difference between the two rigs and not one to leave silent: tg128 error bars on this sweep run 0.02 to 0.97 tok/s, where the earlier display-attached run on the same box, whose repetition count was not recorded, produced bars of 0.64 to 13.47.", "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": "raw.log (miu headless sweep 2026-07-28T09:41:26-07:00 to 09:52:12-07:00)" }, "notes": "The config the rushuna 3060 runs as its recommended setting, measured here for a like-for-like cross-rig read: 52.76 on the 3090 against 38.9 on the 3060. Both rigs are offloading the same 24 layers, so most of what separates them is DDR4-2667 against DDR4-2133. Headless: lightdm stopped and zero Xorg processes, so the full 24,123 MiB was free at start. MemAvailable held near 60 GiB and swap stayed 100% free for the whole sweep, so nothing here is a paging artifact." }, { "id": "miu-qwen36-35b-a3b-udq4km-ncmoe32-d0", "measurement_type": "measured", "date_measured": "2026-07-28", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": "580.159.03", "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "os": "Xubuntu 24.04, kernel 6.8.0-134", "display_attached": false }, "model": { "name": "Qwen3.6-35B-A3B", "repo": "unsloth/Qwen3.6-35B-A3B-GGUF", "architecture": "moe", "params_total_b": 35, "params_active_b": 3, "quant": "UD-Q4_K_M", "file_size_value": 20.6, "file_size_unit": "GiB" }, "config": { "flags": "-ngl 99 -ncmoe 32 -fa 1", "n_gpu_layers": 99, "n_cpu_moe": 32, "flash_attention": true, "context_depth_tokens": 0, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 43.9, "generation_tok_s_stddev": 0.05, "prompt_tok_s": 412.29, "prompt_tok_s_stddev": 7.98, "status": "ok", "vram_peak_mib": 6838, "vram_used_gb": 7.17, "system_ram_used_gb": null }, "telemetry": { "peak_temp_c": 59, "sm_clock_mhz": 1800, "power_draw_w": 173.7 }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench -p 512 -n 128 -d 0,4096,8192 -r 5", "repetitions": 5, "methodology_note": "Five repetitions per data point here, against three on the rushuna rows. More repeats is strictly better, but it is a difference between the two rigs and not one to leave silent: tg128 error bars on this sweep run 0.02 to 0.97 tok/s, where the earlier display-attached run on the same box, whose repetition count was not recorded, produced bars of 0.64 to 13.47.", "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": "raw.log (miu headless sweep 2026-07-28T09:41:26-07:00 to 09:52:12-07:00)" }, "notes": "Down to 6,838 MiB of VRAM, well inside an 8GB card, at 43.90 tok/s. The 3060 running the same -ncmoe 32 manages 31.8. Headless: lightdm stopped and zero Xorg processes, so the full 24,123 MiB was free at start. MemAvailable held near 60 GiB and swap stayed 100% free for the whole sweep, so nothing here is a paging artifact." }, { "id": "miu-qwen36-35b-a3b-udq4km-ncmoe40-d0", "measurement_type": "measured", "date_measured": "2026-07-28", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": "580.159.03", "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "os": "Xubuntu 24.04, kernel 6.8.0-134", "display_attached": false }, "model": { "name": "Qwen3.6-35B-A3B", "repo": "unsloth/Qwen3.6-35B-A3B-GGUF", "architecture": "moe", "params_total_b": 35, "params_active_b": 3, "quant": "UD-Q4_K_M", "file_size_value": 20.6, "file_size_unit": "GiB" }, "config": { "flags": "-ngl 99 -ncmoe 40 -fa 1", "n_gpu_layers": 99, "n_cpu_moe": 40, "flash_attention": true, "context_depth_tokens": 0, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 36.11, "generation_tok_s_stddev": 0.02, "prompt_tok_s": 340.67, "prompt_tok_s_stddev": 3.55, "status": "ok", "vram_peak_mib": 3022, "vram_used_gb": 3.17, "system_ram_used_gb": null }, "telemetry": { "peak_temp_c": 59, "sm_clock_mhz": 1800, "power_draw_w": 165.02 }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench -p 512 -n 128 -d 0,4096,8192 -r 5", "repetitions": 5, "methodology_note": "Five repetitions per data point here, against three on the rushuna rows. More repeats is strictly better, but it is a difference between the two rigs and not one to leave silent: tg128 error bars on this sweep run 0.02 to 0.97 tok/s, where the earlier display-attached run on the same box, whose repetition count was not recorded, produced bars of 0.64 to 13.47.", "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": "raw.log (miu headless sweep 2026-07-28T09:41:26-07:00 to 09:52:12-07:00)" }, "notes": "Every expert in system RAM. At 36.11 the 3090 lands below the 3060's 38.9 at -ncmoe 24: a card costing four times as much loses when the config is wrong, because at full offload the number being measured is RAM bandwidth. Against the 3060's own -ncmoe 40 figure of 28.1 the ratio is 1.28x, close to the 1.25x separating DDR4-2667 from DDR4-2133. The curves narrow toward the RAM ratio but do not meet, and the residual gap is mostly the faster DIMMs rather than the faster GPU. Headless: lightdm stopped and zero Xorg processes, so the full 24,123 MiB was free at start. MemAvailable held near 60 GiB and swap stayed 100% free for the whole sweep, so nothing here is a paging artifact." }, { "id": "miu-qwen36-35b-a3b-udq4km-ncmoe0-d0-display", "measurement_type": "measured", "date_measured": "2026-07-28", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": "580.159.03", "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "os": "Xubuntu 24.04, kernel 6.8.0-134", "display_attached": true }, "model": { "name": "Qwen3.6-35B-A3B", "repo": "unsloth/Qwen3.6-35B-A3B-GGUF", "architecture": "moe", "params_total_b": 35, "params_active_b": 3, "quant": "UD-Q4_K_M", "file_size_value": 20.6, "file_size_unit": "GiB" }, "config": { "flags": "-ngl 99 -fa 1", "n_gpu_layers": 99, "n_cpu_moe": null, "flash_attention": true, "context_depth_tokens": 0, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": null, "generation_tok_s_stddev": null, "prompt_tok_s": null, "prompt_tok_s_stddev": null, "status": "failed_to_load_model", "vram_peak_mib": 4269, "vram_used_gb": null, "system_ram_used_gb": null }, "telemetry": { "peak_temp_c": 63, "sm_clock_mhz": 1800, "power_draw_w": 135.58 }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench -p 512 -n 128 -d 0,4096,8192 (repetition count not recorded)", "repetitions": null, "methodology_note": null, "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": "display-attached-raw.log" }, "notes": "Fails outright. llama.cpp could not load the model: the desktop session was already holding 4,269 MiB, which leaves less than the 21,652 MiB the weights need. The fully-resident config that gives 157.66 tok/s headless does not exist on a 3090 driving a monitor. Display-attached: a normal Xubuntu desktop session held roughly 4.4 GiB of VRAM throughout. Same box, same build, same model file as the headless sweep. The repetition count for this run was not recorded, and host memory was not instrumented, so treat it as a directional comparison rather than a like-for-like one." }, { "id": "miu-qwen36-35b-a3b-udq4km-ncmoe2-d0-display", "measurement_type": "measured", "date_measured": "2026-07-28", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": "580.159.03", "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "os": "Xubuntu 24.04, kernel 6.8.0-134", "display_attached": true }, "model": { "name": "Qwen3.6-35B-A3B", "repo": "unsloth/Qwen3.6-35B-A3B-GGUF", "architecture": "moe", "params_total_b": 35, "params_active_b": 3, "quant": "UD-Q4_K_M", "file_size_value": 20.6, "file_size_unit": "GiB" }, "config": { "flags": "-ngl 99 -ncmoe 2 -fa 1", "n_gpu_layers": 99, "n_cpu_moe": 2, "flash_attention": true, "context_depth_tokens": 0, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": null, "generation_tok_s_stddev": null, "prompt_tok_s": null, "prompt_tok_s_stddev": null, "status": "failed_to_create_context", "vram_peak_mib": 23908, "vram_used_gb": null, "system_ram_used_gb": null }, "telemetry": { "peak_temp_c": 63, "sm_clock_mhz": 1935, "power_draw_w": 158.76 }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench -p 512 -n 128 -d 0,4096,8192 (repetition count not recorded)", "repetitions": null, "methodology_note": null, "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": "display-attached-raw.log" }, "notes": "Fails one step later than -ncmoe 0 and for a different reason. The weights loaded, peaking at 23,908 MiB of 24,123, then context creation failed with nothing left for the KV cache. Headless the same config runs at 136.03, so this is a display artifact and not a card boundary. Display-attached: a normal Xubuntu desktop session held roughly 4.4 GiB of VRAM throughout. Same box, same build, same model file as the headless sweep. The repetition count for this run was not recorded, and host memory was not instrumented, so treat it as a directional comparison rather than a like-for-like one." }, { "id": "miu-qwen36-35b-a3b-udq4km-ncmoe4-d0-display", "measurement_type": "measured", "date_measured": "2026-07-28", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": "580.159.03", "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "os": "Xubuntu 24.04, kernel 6.8.0-134", "display_attached": true }, "model": { "name": "Qwen3.6-35B-A3B", "repo": "unsloth/Qwen3.6-35B-A3B-GGUF", "architecture": "moe", "params_total_b": 35, "params_active_b": 3, "quant": "UD-Q4_K_M", "file_size_value": 20.6, "file_size_unit": "GiB" }, "config": { "flags": "-ngl 99 -ncmoe 4 -fa 1", "n_gpu_layers": 99, "n_cpu_moe": 4, "flash_attention": true, "context_depth_tokens": 0, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 107.12, "generation_tok_s_stddev": 9.42, "prompt_tok_s": 1525.93, "prompt_tok_s_stddev": 29.19, "status": "ok", "vram_peak_mib": 23788, "vram_used_gb": 24.94, "system_ram_used_gb": null }, "telemetry": { "peak_temp_c": 68, "sm_clock_mhz": 1920, "power_draw_w": 313.56 }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench -p 512 -n 128 -d 0,4096,8192 (repetition count not recorded)", "repetitions": null, "methodology_note": null, "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": "display-attached-raw.log" }, "notes": "The first config that survives a desktop session, and the cleanest comparison in the set: 107.12 here against 118.90 headless. Note the error bar, 9.42 against 0.49 headless. A desktop session costs throughput and costs consistency. Display-attached: a normal Xubuntu desktop session held roughly 4.4 GiB of VRAM throughout. Same box, same build, same model file as the headless sweep. The repetition count for this run was not recorded, and host memory was not instrumented, so treat it as a directional comparison rather than a like-for-like one." }, { "id": "miu-qwen36-35b-a3b-udq4km-ncmoe6-d0-display", "measurement_type": "measured", "date_measured": "2026-07-28", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": "580.159.03", "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "os": "Xubuntu 24.04, kernel 6.8.0-134", "display_attached": true }, "model": { "name": "Qwen3.6-35B-A3B", "repo": "unsloth/Qwen3.6-35B-A3B-GGUF", "architecture": "moe", "params_total_b": 35, "params_active_b": 3, "quant": "UD-Q4_K_M", "file_size_value": 20.6, "file_size_unit": "GiB" }, "config": { "flags": "-ngl 99 -ncmoe 6 -fa 1", "n_gpu_layers": 99, "n_cpu_moe": 6, "flash_attention": true, "context_depth_tokens": 0, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 95.56, "generation_tok_s_stddev": 3.44, "prompt_tok_s": 1132.32, "prompt_tok_s_stddev": 39.48, "status": "ok", "vram_peak_mib": 23095, "vram_used_gb": 24.22, "system_ram_used_gb": null }, "telemetry": { "peak_temp_c": 67, "sm_clock_mhz": 1920, "power_draw_w": 248.85 }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench -p 512 -n 128 -d 0,4096,8192 (repetition count not recorded)", "repetitions": null, "methodology_note": null, "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": "display-attached-raw.log" }, "notes": "95.56 against 106.56 headless. Display-attached: a normal Xubuntu desktop session held roughly 4.4 GiB of VRAM throughout. Same box, same build, same model file as the headless sweep. The repetition count for this run was not recorded, and host memory was not instrumented, so treat it as a directional comparison rather than a like-for-like one." }, { "id": "miu-qwen36-35b-a3b-udq4km-ncmoe8-d0-display", "measurement_type": "measured", "date_measured": "2026-07-28", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": "580.159.03", "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "os": "Xubuntu 24.04, kernel 6.8.0-134", "display_attached": true }, "model": { "name": "Qwen3.6-35B-A3B", "repo": "unsloth/Qwen3.6-35B-A3B-GGUF", "architecture": "moe", "params_total_b": 35, "params_active_b": 3, "quant": "UD-Q4_K_M", "file_size_value": 20.6, "file_size_unit": "GiB" }, "config": { "flags": "-ngl 99 -ncmoe 8 -fa 1", "n_gpu_layers": 99, "n_cpu_moe": 8, "flash_attention": true, "context_depth_tokens": 0, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 90.88, "generation_tok_s_stddev": 2.82, "prompt_tok_s": 839.64, "prompt_tok_s_stddev": 213.55, "status": "ok", "vram_peak_mib": 22123, "vram_used_gb": 23.2, "system_ram_used_gb": null }, "telemetry": { "peak_temp_c": 66, "sm_clock_mhz": 1920, "power_draw_w": 278.66 }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench -p 512 -n 128 -d 0,4096,8192 (repetition count not recorded)", "repetitions": null, "methodology_note": null, "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": "display-attached-raw.log" }, "notes": "90.88 against 94.54 headless. The prefill error bar of 213.55 tok/s is the desktop session competing for the GPU mid-run; the generation figure is the usable one. Display-attached: a normal Xubuntu desktop session held roughly 4.4 GiB of VRAM throughout. Same box, same build, same model file as the headless sweep. The repetition count for this run was not recorded, and host memory was not instrumented, so treat it as a directional comparison rather than a like-for-like one." }, { "id": "miu-qwen36-35b-a3b-udq4km-ncmoe16-d0-display", "measurement_type": "measured", "date_measured": "2026-07-28", "date_measured_precision": "day", "hardware": { "rig": "miu", "gpu": "NVIDIA GeForce RTX 3090 24GB", "gpu_count": 1, "vram_gb": 24, "driver": "580.159.03", "cpu": "Intel Core i7-8086K (6c/12t, Coffee Lake)", "system_ram_gb": 64, "system_ram_type": "DDR4-2667", "system_ram_channels": 2, "system_ram_bandwidth_gbs": 43, "system_ram_detail": "4 x 16GB Samsung M378A2K43DB1-CTD, dual-rank, all four slots populated across two channels", "motherboard": "ASRock B365M Pro4", "pcie": "3.0 x16", "os": "Xubuntu 24.04, kernel 6.8.0-134", "display_attached": true }, "model": { "name": "Qwen3.6-35B-A3B", "repo": "unsloth/Qwen3.6-35B-A3B-GGUF", "architecture": "moe", "params_total_b": 35, "params_active_b": 3, "quant": "UD-Q4_K_M", "file_size_value": 20.6, "file_size_unit": "GiB" }, "config": { "flags": "-ngl 99 -ncmoe 16 -fa 1", "n_gpu_layers": 99, "n_cpu_moe": 16, "flash_attention": true, "context_depth_tokens": 0, "batch": 1, "speculative_decoding": null }, "result": { "generation_tok_s": 61.48, "generation_tok_s_stddev": 4.49, "prompt_tok_s": 613.51, "prompt_tok_s_stddev": 25.91, "status": "ok", "vram_peak_mib": 18492, "vram_used_gb": 19.39, "system_ram_used_gb": null }, "telemetry": { "peak_temp_c": 64, "sm_clock_mhz": 1920, "power_draw_w": 233.4 }, "provenance": { "engine": "llama.cpp", "engine_build": "b10088", "engine_commit": "67b9b0e", "compute": "CUDA sm_86", "harness": "llama-bench -p 512 -n 128 -d 0,4096,8192 (repetition count not recorded)", "repetitions": null, "methodology_note": null, "source_article": "https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/", "raw_log": "display-attached-raw.log" }, "notes": "61.48 against 66.96 headless. This is the last config in the display-attached run taken before swap ran down to 248 kB free. Everything after it in that run (-ncmoe 24, 32 and 40) is excluded from this dataset as contaminated. Display-attached: a normal Xubuntu desktop session held roughly 4.4 GiB of VRAM throughout. Same box, same build, same model file as the headless sweep. The repetition count for this run was not recorded, and host memory was not instrumented, so treat it as a directional comparison rather than a like-for-like one." } ] }