Choosing a GPU: B200, H100 and A100

Which workload belongs on which card? A plain decision guide across the memory, bandwidth and cost triangle.

GPU selection in most teams starts with "let's get the newest one" and ends when the invoice arrives. The right question is different: does the model you are running fit in memory, and how fast does it have to answer?

Once you answer those two, the card picks itself.

Start by naming your workload

AI workloads fall into roughly three groups, and each one stresses a different resource:

• Inference: Sensitive to memory bandwidth. As long as the model fits in memory, response speed is governed by how fast memory can be read, not by raw compute.
• Fine-tuning: Sensitive to memory size. During training the model, the gradients and the optimiser state all sit in memory at once, so the requirement roughly triples.
• Image and video generation: Stresses memory and compute together. In batch generation, card count directly determines output volume.

Where each card fits

A100 is a mature, balanced card. For running mid-sized models, generating embeddings and building evaluation environments it still offers the best price-performance ratio. It is the right starting point for teams new to this.

H100 offers markedly faster memory and stronger compute than the A100. If you want to serve large language models in production at low latency, this is the card. For most enterprise inference workloads, H100 is the correct answer.

B200 is the newest generation, and its real advantage shows on very large models. Running a model that does not fit on one card with fewer cards lowers both cost and inter-card communication latency. If you are not working with very large models, you will not get your money's worth out of that power.

A simple decision rule

1. How many GB are the model weights? For inference you need roughly 1.2× that in memory.
2. If you are fine-tuning, multiply the same number by three.
3. If the resulting requirement fits on one card, start with one card. If it does not, look at a card with more memory before you look at more cards.
4. If your latency target is under a second, choose the card with higher bandwidth.

The three most common mistakes

Buying big up front. Most teams commit to a year of capacity before measuring what they need. Two weeks of measurement on a single pod makes that decision far cheaper.

Looking only at the card price. Total cost is determined not by the hourly rate but by how long the workload takes on that card. A more expensive card that finishes the job three times faster is cheaper.

Idle capacity. Keeping intermittent workloads on an always-on card is the most common waste. Running batch jobs on a schedule noticeably lowers the bill.

If you are unsure

Describe your workload and let us measure it together. Two days of testing on a single GPU pod with your real data gives a more reliable answer than any estimate.