DDR4
An $8 chip, twice as fast, same story
esp32-ai's 28.9M-parameter story model on an $8 ESP32-S3: 10.34 to 22.59 tok/s from five patches, the same 200-token story byte for byte. Plus four new articles, an unpatched LMCache hole and six Quick Hits.
Does RAM Speed Matter for Local LLMs? 2133 vs 3000, Measured
DDR4-2133 vs 3000 on an RTX 3090 box: no change for models that fit on the GPU, 13 to 29% faster with MoE experts offloaded. Plus why 3200 wouldn't train.
Nineteen gigabytes of my 3090 sat empty. The 3060 kept up.
Stock llama.cpp put 5.4 GB of a 177B MoE on an RTX 3090 and a 3060 kept pace; -ncmoe 29 buys 34 to 40 percent and a real 32 GB box does 8.7 tok/s. Plus a 4 GB GTX 1650 at 20 tok/s on a 35B MoE, and the MoE primer corrected in public.
A weekly email with every new guide and measured benchmark.
Subscribe — free, no spam