Why Your Local AI Is Slow
Here's the thing nobody tells you about local AI speed: for every single token a model generates, it rereads itself entirely from memory. All 17GB, every token. That's why memory bandwidth sets a hard speed ceiling, and the formula is simple: bandwidth divided by model size.
On an M5 Max at 614 GB/s, a 17GB model predicts around 36 tokens per second, with real benchmarks closer to 31. It's not exact, but it gets you close to the max speed you'll hit before any optimization, useful to know before you buy.
#techwithtim #ai #machinelearning
On an M5 Max at 614 GB/s, a 17GB model predicts around 36 tokens per second, with real benchmarks closer to 31. It's not exact, but it gets you close to the max speed you'll hit before any optimization, useful to know before you buy.
#techwithtim #ai #machinelearning
Tech With Tim
Dive into the world of programming, software engineering, machine learning, and all things tech through my channel! I place a strong focus on Python and JavaScript, offering you an array of free resources to kickstart your coding journey and make your mar...