Three performance-engineering projects at three levels of parallelism: SIMD vectorization on a single ARM core, thousands of threads on an NVIDIA GPU, and hundreds of processes across supercomputer nodes with MPI.
We wrote a cache-blocked DGEMM in the style of BLIS. The matrices are split into large tiles that fit in the L2 and L3 caches (MC=256, KC=128, NC=256). Each tile is packed into contiguous memory so the kernel reads it in order. A macro-kernel then hands each pair of packed sub-panels to a micro-kernel written with ARM SVE intrinsics.
perf hardware counters to explain the dips in performance. For example, going from N=512 to N=513 adds about 9M L1 and 1.7M L2 cache refills because of the partial panels.| Stage | Approx. GFLOP/s |
|---|---|
| Naive triple loop | ~1 |
| Packing (BLIS-style, row-major) | ~6.5 |
| First SVE micro-kernel | ~9–12 |
| SVE, 4 then 8 rows of C at once | 16 → 18 |
| 8×8 register block in two passes | ~22–23 |
| Parameter tuning (final) | up to 23.1 |
Each 16×16 block of 256 threads computes one 64×64 tile of C. The block loads tiles of A and B into shared memory, and neighboring threads read neighboring addresses so global-memory loads coalesce. Each thread then keeps a 4×4 block of C in registers, which gives it 16 independent accumulators (instruction-level parallelism). The inner loops are fully unrolled.
| N | Naive | 8×8 | 16×16 | 32×16 | 32×8 |
|---|---|---|---|---|---|
| 256 | 23.6 | 1,229 | 1,518 | 881 | 827 |
| 512 | 31.7 | 1,552 | 2,759 | 2,412 | 2,125 |
| 1024 | 52.5 | 1,405 | 3,274 | 2,487 | 2,199 |
| 1025 | – | 1,056 | 2,613 | 2,385 | 2,119 |
| 2047 | – | 1,352 | 3,109 | 2,911 | 2,620 |
| 2048 | 52.5 | 1,357 | 3,298 | 2,969 | 2,576 |
A 16×16 block (256 threads, 8 full warps) was best at every size. An 8×8 block has only 2 warps and too little shared-memory reuse. 32×16 has 512 threads, which raises register pressure and lowers occupancy.
The solver advances the 2D wave equation with a 5-point stencil and absorbing boundary conditions. The grid is split across a 2D Cartesian grid of MPI processes (MPI_Cart_create), and no process gets more than one extra row or column of work than any other.
MPI_Isend/MPI_Irecv. A strided MPI_Type_vector sends columns, which aren't contiguous in memory, without manual packing.| Cores | Geometry | Mpts/s | GFLOP/s | Speedup | Efficiency |
|---|---|---|---|---|---|
| 128 | 8×16 | 3,259 | 29.3 | 91× | 71% |
| 192 | 8×24 | 5,452 | 49.1 | 152× | 79% |
| 256 | 8×32 | 6,771 | 60.9 | 189× | 74% |
| 384 | 8×48 | 9,777 | 88.0 | 273× | 71% |
With the same 256 processes, the shape of the process grid changed throughput by 30%. C++ stores the grid row by row, so north–south ghost rows are contiguous while east–west ghost columns are strided and slower to send. Layouts that keep the strided edge short won, even over the square layout that minimizes total communication.
| Geometry (256 cores, N=4000) | Sub-block | Mpts/s |
|---|---|---|
| 16×16 (square) | 250 × 250 | 4,307 |
| 64×4 | 63 × 1000 | 4,939 |
| 8×32 | 500 × 125 | 5,466 |
| 32×8 | 125 × 500 | 5,603 |
We also ran the solver on a 27-core Raspberry Pi cluster. Tripling the core count from 9 to 27 gave a 2.6× speedup at N=2000, but only 2.0× at N=1000: the cluster's Gigabit Ethernet needs larger per-process workloads to hide its latency.