Field Note
Why unified memory changes edge AI
Memory capacity decides whether a model can run before compute decides how fast. Unified memory expands the models available to a single machine, but capacity, bandwidth, and workload still have to be measured separately.
The first local-inference calculation is not tokens per second. It is whether the weights and working state fit in memory.
A rough lower bound for weights is parameter count multiplied by bytes per parameter. At four bits, 27 billion parameters occupy about 13.5 GB before quantisation metadata, runtime buffers, the key-value cache, and the operating system take their share. Context length and concurrent requests expand the working set further. A model advertised as “27B” does not define one memory requirement.
On a conventional workstation, CPU memory and GPU memory are separate pools. The whole model may fit in system RAM while only part fits in VRAM, forcing slower transfers or partial offload. A unified-memory system gives the CPU and GPU access to one pool. That does not make every byte equally fast, but it removes a hard boundary that shapes which models a single machine can attempt.
Apple’s 2025 Mac Studio announcement made the shift unusually visible: configurations up to 512 GB of unified memory, with an Apple claim that models above 600B parameters can be held entirely in memory. “Held” is the operative word. The release does not establish useful speed for a particular model or task.
The practical ceiling has three parts
- Capacity: weights, cache, runtime, and the rest of the system fit without pathological swapping.
- Bandwidth and compute: prompt processing and generation meet the workflow’s latency requirement.
- Software support: the runtime can use the hardware efficiently at the required quantisation and context.
At the edge, those constraints sit beside power, heat, physical size, network availability, and serviceability. A desk-side system may be local without being deployable beside a camera or machine.
The useful consequence of larger unified memory is optionality. A small team can test larger models without beginning with a multi-GPU server or a cloud contract. The result still has to earn its place against smaller local models and hosted alternatives on finished work.