Inference on CPU

Derives the cache blocking every fast CPU GEMM still uses by refining a memory model until the packed kernel falls out.

Reduces a BLAS to one register-tile micro-kernel per architecture and states which loop owns which level of cache.

Builds a register-tile kernel step by step and shows where outer-loop unrolling wins and where a vendor BLAS still does.

Sets run-time dispatch for a BLAS, one kernel table per microarchitecture chosen as a DYNAMIC_ARCH build loads.

Fixes the data-type table, packed weight format and fused post-ops a CPU GEMM must expose to an inference graph.

Fixes the fused attention pattern, f32 accumulation under bf16 inputs and the shapes the fast CPU path accepts.

Generates VNNI and AMX kernels at run time under PyTorch, TensorFlow and OpenVINO, and calls Compute Library on Arm.

Defines the quantized block formats and the per-ISA dot-product kernels over them that every llama.cpp type rests on.

Where new quantization types, kernels and thread pools land first, each with the perplexity and speed table behind it.

Holds the CPU provider's GEMM, int8 and int4 MatMul kernels, dispatched per ISA at run time from SSE to AMX and SME.

States precision defaults per ISA, the int8 path through oneDNN and the streams model that turns cores into throughput.

Defines the affine scale and zero-point scheme with integer accumulation that every int8 CPU runtime still implements.

States where int8 saturates on pre-VNNI x86, the compensation that makes signed by signed work, and what VNNI removes.

Defines the QDQ form against QOperator and dynamic against static, and which sign choices are safe on which CPUs.

Defines super-block formats with quantized scales and the perplexity against size curve behind mixing types per tensor.

Replaces unpacking and multiplying low-bit weights with bit-sliced lookups, so a narrower weight costs less to apply.

Defines AMX tiles, tile configuration and TMUL, the VNNI and bf16 dot products, and the XSAVE state a tile load needs.

Also in §5§7

Defines the arch_prctl permission Linux requires for AMX tile data and the trap a tile instruction takes until granted.

Holds the design report for the ggml AMX path, whose double buffering hides applying scales between tile products.

Defines streaming SVE mode, the ZA tile array and SME2 multi-vector instructions, with pseudocode a kernel must match.

Also in §7

Walks through SME2 int8 and f32 matmul and gemv kernels and a lookup-table path, on a unit so far only in client parts.

Sets the physical-core default, the affinity it implies and the spin-wait controls that trade idle CPU for latency.

States the vendor defaults, one thread per core, SMT siblings off, core type by precision and one socket for latency.

Defines the explicit thread pool with CPU masks, strict placement, priority and polling that ggml runs without OpenMP.

Defines the prompt and generation tests, repetitions and mean with deviation behind any comparable llama.cpp number.

Traces poor decode scaling across sockets to remote NUMA access from weight placement, with numatop counts as evidence.

Origin of ggml's bandwidth argument, NEON threads saturate a laptop's memory bus, so a GPU on that bus gains nothing.

Holds the memory clock sweep with all else fixed, the token rate following it and flattening after a few threads.

Measures decode kernels on a Xeon as DRAM-bound, so AMX pays at batch one only once bytes per token shrink, with code.

Holds CPU-only entries whose logs show the batched Offline case lost to GPU entries at the same accuracy floor.