simd-anim
All strategy pages

03 Strategy · memory choreography

Memory in flight

A kernel is a schedule for hiding memory latency under compute. Three decisions do the hiding — and one slot pays for it.

Instruction
prefetch prfm pldl2keep, [pb] ; prfm pldl1strm, [pc]
store    st1w { za0h.s[w12, 0] }, p0, [pc]  →  aligned buffer
drain    stnt1w  (non-temporal write stream to C)

Illustrative model at the microkernel page's scale: one k block reads A + B = 64 B (two blocks are drawn, hence the second B prefetch) and the kernel writes C = 64 B; one slot = 8 cycles. Per-slot budgets come from unigemm_sme's measured tiers — L1 224 B, L2 192 B, HBM 64 B (≈ 28 / 24 / 8 bytes per cycle). In flight means an outstanding transfer that occupies the destination's resources before its bytes arrive — which is why the meters charge it.

Three moves, one collision

Every choice below is a trade. Read the cards, then watch each one land in the timeline underneath.

  1. Prefetch ahead, so the loop never waits

    pldl2keep starts the next B page on its way to L2 while the FMOPAs run. By the time the k1 load wants it the page is already two levels closer, so that load hits L2 instead of HBM — the latency was spent before anyone needed the bytes.

    Cost: one HBM read per k block, paid early.

    Watch: the pldl2keep slot, then the load after it — the fill lands in L2 first, and that load is tagged as a hit.

  2. Never read what you overwrite

    The store is non-temporal: stnt1w writes whole lines straight to memory instead of pulling them into cache first. C costs 64 B out and nothing in — no read-for-ownership, no doubled traffic.

    Cost: none in bytes; the store detours through the aligned buffer instead of sitting in cache.

    Watch: C's only HBM traffic is a single write. No read of C's lines appears anywhere on the page.

  3. Let the tail hide under the next kernel

    The drain is asynchronous. It leaves L1 as kernel n ends and lands while kernel n+1 is already packing and reading — so the store's tail costs this kernel nothing.

    Cost: for exactly one slot HBM is asked for 64 B of read and 64 B of write at once — two directions on one bus.

    Watch: the slot where the HBM row holds a read and a write together. Its meter reads 128 / 64 — the only over-budget slot here.

L1 / L2 / HBM · memory traffic across two kernels

Ready. Play the choreography across two kernels.

Arrow keys step, space plays or pauses.

Reading the picture

  1. The bet behind a prefetch

    A prefetch spends bandwidth now to save a stall later. pldl2keep pulls the next B page into L2 on HBM's budget while the FMOPAs run, so the k1 load that follows is a hit rather than a wait; pldl1strm streams C's lines through L1 without evicting what is resident. Neither is free — if the prefetched page were never used, that bandwidth would be wasted.

  2. A store should cost one write

    st1w writes 64 B into the aligned buffer that L1 already holds, so nothing leaves the core yet. stnt1w then drains that buffer non-temporally: whole lines, straight to memory, with no fetch of the bytes being replaced. C therefore costs 64 B out and 0 B in — the read-for-ownership a normal store would trigger never happens.

  3. The tail belongs to the next kernel

    A store is not finished when the instruction retires. The drain is still in flight as kernel n returns, and it lands during kernel n+1's pack — that overlap is the whole trick: nobody waits for the tail. The price is one slot where HBM must serve a 64 B read and a 64 B write together, 128 B against its 64 B budget. Real pipelines absorb that with multi-threaded phases, so this contention is what a scheduler is there to spread out.

  4. Compute is the quiet part

    FMOPA slots move zero bytes: once the operands sit in registers the compute leaves every meter at rest, which is why all the drama here is on the supply side. The budgets are unigemm_sme's measured tiers; on a real machine C is far larger than 64 B, so one drain spans many k blocks — the mechanics are exactly these.

Bytes are labelled per hop; budgets are per 8-cycle slot.