Datacenter GPU generations: V100 → H100 → H200 → B100/B200/B300

This page is a working operator's view of the NVIDIA datacenter GPU lineup, not a marketing comparison. The questions it answers: what changed between generations, what does it mean for me as someone running these in production, and which one do I actually need.

Quick reference table

(Numbers are peak / sparse / on-paper. Real workload throughput depends heavily on your kernel, memory bandwidth, and interconnect.)

What actually changes between generations

Compute density

Each generation roughly doubles useful FP-x compute. The headline "X PFLOPS" means little; what matters is the format mix supported by the Tensor Cores. Hopper introduced FP8 with dynamic scaling via the Transformer Engine — that's where the 4× LLM training speedup over A100 came from, not raw silicon. Blackwell added FP4 (B200) and improved FP6/FP8 throughput with second-generation Transformer Engine, which is the basis of B100/B200/B300's headline numbers.

Memory

This is usually the bottleneck for inference and big model training:

• A100 → H100: 80 GB HBM2e/HBM3, similar capacity but 1.6 → 3.35 TB/s bandwidth.
• H100 → H200: same compute, but 141 GB HBM3e at 4.8 TB/s. A drop-in for serving bigger context windows or KV cache without changing the rest of the stack.
• H200 → B100/B200: jump to 192 GB HBM3e at 8 TB/s.
• B200 → B300: 288 GB, same 8 TB/s. Aimed at very-long-context inference and reasoning models.

Interconnect

• A100: NVLink 3 (600 GB/s per GPU), NVSwitch optional.
• H100: NVLink 4 (900 GB/s), 18 links per GPU. NVSwitch v3 enables a flat all-to-all 900 GB/s within the 8-GPU baseboard.
• Blackwell (B100/B200/B300): NVLink 5 (1.8 TB/s per GPU) and NVSwitch v4. NVL72 packages 72 GPUs into a single rack-level NVLink domain — meaning all 72 GPUs talk to each other at 1.8 TB/s as if they were on the same baseboard. Big deal for trillion-parameter model training.

Power & cooling

• H100 SXM: 700 W, mostly air-cooled in DGX/HGX form factors.
• B200: 1000 W per GPU, liquid cooling effectively required at scale.
• B300: 1400 W per GPU, exclusively liquid.
• GB200 NVL72: 120 kW per rack. This forces datacenters to support direct-to-chip liquid cooling, larger PSU budgets, and CDU plumbing.

MIG (Multi-Instance GPU)

• A100: up to 7 MIG instances per GPU.
• H100: up to 7, with isolated SM partitions (better for multi-tenant inference).
• Blackwell: continues MIG with refreshed slicing, plus Confidential Computing MIG (encrypted memory boundaries between tenants).

Confidential Computing

H100 introduced Confidential VMs with memory encryption between the GPU and host. Blackwell extends it (TEE-IO, encrypted NVLink). Relevant if you run multi-tenant inference where customers don't trust each other and you want to prove no host-side memory inspection is possible.

Operational impact (what changes for you as an operator)

What you actually want, by use case

• Training big LLMs (≥70B params, multi-node) — H100 / H200 today, B200 / GB200 NVL72 if you can get them. The fabric (NVLink 5 + NVSwitch v4) matters more than raw FLOPS once you go past 1024 GPUs.
• Inference of large models (open-weight 70B-405B) — H200 is the sweet spot in 2025: 141 GB lets you fit Llama-3.1-405B with KV cache without TP-shard pain. B200/B300 if available.
• Inference of very large reasoning models — B300 (288 GB) becomes meaningful for o1-class workloads with long chain-of-thought KV.
• Multi-tenant inference — A100 with MIG is still cost-efficient. H100 if confidential computing matters.
• Mixed HPC / scientific compute (FP64) — H100 still has the edge over Blackwell on pure FP64, since Blackwell deemphasized FP64 in favor of inference-friendly low-precision.

"I have nvidia-smi output, how do I tell which generation"

Name                                  Architecture  Generation
NVIDIA A100-SXM4-40GB / -80GB         Ampere        A100
NVIDIA H100 80GB HBM3                  Hopper        H100 SXM5
NVIDIA H100 PCIe                       Hopper        H100 PCIe (lower NVLink, no NVSwitch)
NVIDIA H200                            Hopper        H200
NVIDIA B100                            Blackwell     B100 (air-cooled-friendly)
NVIDIA B200                            Blackwell     B200 (liquid cooled, 1000 W)
NVIDIA B300                            Blackwell Ultra B300 (1400 W)
NVIDIA GB200                           Grace+Blackwell GB200 superchip

If nvidia-smi -q | grep "Compute Capability" shows:

• 7.0 → V100
• 8.0 → A100
• 9.0 → H100 / H200
• 10.0 → B100 / B200 / B300 (different SKUs share the same CC; check name + memory size to disambiguate)

"I'm being offered X — is it real H100 or a degraded SKU"

Three things to check before signing a contract for H-class:

1. Memory: H100 SXM5 = 80 GB HBM3, H200 SXM5 = 141 GB HBM3e. Anything else is either H100-PCIe (limited NVLink), L40S (Ada, not Hopper), or fraudulent.
2. NVLink: nvidia-smi nvlink -s should show 18 active links at 26.562 GB/s each on real SXM5. PCIe variants have at most 4-6 NVLink lanes.
3. NVSwitch: nvidia-smi -q | grep "NVLink Reduction" and dcgmi diag -r 2 should report a healthy fabric. If GPU-to-GPU bandwidth via nccl-tests caps at ~50 GB/s instead of ~450 GB/s busbw, NVSwitch is missing or degraded.

See also

• NVLink / NVSwitch deep dive
• PCIe topology and P2P constraints
• DCGM monitoring
• MIG partitioning
• NCCL multi-node tuning