Measured where a miss hurts most.
NVIDIA H100 80 GB, against criteria registered before the runs. A mixture-of-experts model whose expert weights don't all fit on the accelerator, so every miss is a stall on a transfer that can't be recomputed.
Criteria first, results second.
Compared against no caching at batch 1, and against stock at batch 4 and 8 with the same cache size.
A mixture-of-experts model whose expert weights don't all fit on the accelerator, so every miss is a stall on a transfer that can't be recomputed.
Results are judged against criteria registered before the runs. Figures are medians of three runs, with output verified unchanged.
How is this measured on your own fleet?
The first step installs nothing in your request path: a couple of weeks of counters your serving engine already keeps, and a written answer on whether memory is costing you anything. If it isn't, we'll tell you so.
Find out whether memory is costing your fleet.
The first step puts nothing in your request path: a couple of weeks of counters your serving engine already keeps, and a written answer. Then we confirm it on your own traffic, on your own hardware, against your current configuration.