LingQi, the network

Two sources describe the node side differently. Lockwood and ServeTheHome put one 800 Gb/s NIC inside each LX2, so two per node, one per plane. The NSCS release says "eight network interfaces" per node and a "dual-plane, four-rail" fat tree.

Both can be true. Eight interfaces of 200 Gb/s, four rails on each of two planes, also make 1.6 Tb/s. An 800G NIC built as four 200G ports would look like one NIC to Lockwood and four interfaces to NSCS. That reconciliation is mine; nobody has published the NIC.

Lockwood reconstructed the tree from the ISC slides. Read these as an informed estimate:

2,944 is exactly the number of blades in 23,552 nodes, so one leaf switch per blade, which matches Lockwood's node count. Only the L2 to L3 hop is optical, so copper does everything inside a frame and inside the core. The switch ASICs look like 32 ports of 100G. The tree tapers: 2:1 at L2, about 7.7:1 overall by Lockwood's estimate.

200 GB/s per node against 120.6 TFlop/s of FP64 is 0.0017 bytes per flop. Within a node, a cluster reads its HBM at 450 GB/s, more than twice what the whole node can send to the network.

Take a 3D stencil with one rank per cluster, 16 ranks per node, each owning a $256^3$ block of doubles. A one-cell halo is $6times256^{2}times8textBapprox3.1$ MB per rank per step, about 50 MB per node. At 200 GB/s that is 0.25 ms. Updating the block reads and writes about $256^{3}times16textBapprox270$ MB, which takes about 0.6 ms at 450 GB/s from the cluster's HBM. Same order of magnitude: the halo can hide behind the update if the code overlaps them, and it cannot if it does not. The taper starts to matter for all-to-all patterns such as FFTs and MoE dispatch, which is why one arXiv team compressed its MoE all-to-all to FP16 and halved the volume.