Model sharding, and what it does not divide
Sharding splits one model across several cards. It lowers what each card has to hold and does not divide the requirement by the number of cards, because activations and buffers are duplicated while only part of the load is split. As of 2026-09-22.
| Part of the statement | Why it matters |
|---|---|
| The card count | The split is over that many devices |
| The card model | It fixes the interconnect as well as the memory |
| The output | The frame and the frames are still the workload |
| The per-card figure | A total tells a reader nothing actionable |
Inclusion rule. Components of a reproducible multi-card statement. A component whose absence makes the figure unreproducible is listed as required. Order. From the count outward to the figure itself.
1The naive arithmetic fails badly
One release in this register states 60.3GB on one card and 44.3GB across four or more. A reader dividing by four would budget about fifteen gigabytes per card and be short by nearly thirty.
That is the single most useful thing a vendor can publish about sharding, and almost nobody does. The register keeps both figures in the cell rather than choosing.
2A card count is itself a hardware statement
Another release states eight named data-centre cards for its large model and one consumer card for its small one. Read as gigabytes that is a large number and a useless one; read as a node it specifies the interconnect too.
Sharding across cards is bandwidth-bound, so the same figures on four cards without a fast link between them would not reproduce.
3The one measured answer in the register
One family publishes a single-card figure and a multi-card figure for the same output, and the second is about a quarter lower rather than a quarter of the first. Another names a count of eight data-centre cards without a per-card figure.
Between them those are the only multi-card statements here. Four families say explicitly that their figure is for one GPU, which is cheap precision and worth having.
A reading note, not an entry: no vendor value appears on this page. Where the register records this term for a particular release, it is on the hardware column and on that family's own page. Nearby terms: GPU architecture, Batch size.