Cloud & Infrastructure

Choosing a GPU Server for Model Training and Inference

Card memory, not core count, determines which model can run on a GPU server. A practical guide to choosing between training and inference and deciding whether to rent or buy.

Cloud & Infrastructure

The CUDA out of memory error in the middle of training a model usually shows up after two hours of waiting, right when the loss is going down. If you're stuck between a few graphics cards right now, or you don't know whether to rent a GPU server or buy one, this text is for that exact moment. The first thing you need to understand is that most wrong decisions come from a single number: card memory, not core count.

Why card memory is the real ceiling of your GPU server

Model parameters, gradients, optimizer state, and intermediate activations all fit into VRAM. A rule of thumb that works in practice: full training with Adam at mixed precision needs about 16 to 20 bytes per parameter. That means a 7-billion-parameter model needs roughly 120 to 140 gigabytes of VRAM. On a 24-gigabyte card this number simply doesn't add up, and that's exactly where you turn to distributed methods.

Inference is a different story. The fp16 weights of a 7B model are about 14 gigabytes, and at 8 bits they come down to 7 gigabytes. A 24-gigabyte card is enough to serve such a model with a medium context. It's this difference that makes many people buy an expensive card for training and then find they've paid far more than necessary for everyday serving.

The numbers you should calculate before ordering

  • Model weights in fp16: roughly 2 bytes per parameter.
  • Adam optimizer state: two extra copies, i.e. about 8 bytes per parameter in fp32.
  • Activations and temporary buffers: depends on batch size and sequence length, and often accounts for at least 20 to 30 percent of total usage.
  • Safety margin: never fill a card right up to the last gigabyte; random crashes usually come from exactly this.

Training and inference want two different GPU servers

For training, bandwidth between cards matters more than single-card power. If you plan to spread the model across four cards with torchrun --nproc_per_node=4, without NVLink or at least full-lane fourth-generation PCIe, communication becomes the bottleneck and you won't get linear efficiency. In practice, on four cards without a high-speed interconnect, you may see only 2.5 to 3 times the speed of a single card, not four times.

For inference, the criterion changes: latency and throughput of concurrent requests. Here mid-range cards with enough memory and lower power consumption make more sense. If your load fluctuates, combining a fixed GPU server with horizontal or vertical autoscaling costs less than reserving peak capacity all month long.

CriterionTrainingInference
Main constraintTotal memory and bandwidth between cardsSingle-card memory and latency
Card typeData-center cards with NVLinkMid-range with enough memory
Usage patternSeveral continuous days, then idleContinuous and fluctuating
Price sensitivityHigh, because the load is shortMedium, because the monthly cost is fixed

Renting or buying a GPU server: a decision solved by usage hours

Buying a card makes sense when your usage is above roughly 60 to 70 percent of the month's hours and your model stays unchanged for more than a year. Below this threshold, renting wins; not because of the hourly price, but because the card generation changes every 18 to 24 months and the card you bought last year comes up short on memory for this year's new model.

Also factor in the hidden cost of buying: electricity, cooling, rack space, and a replacement card when one fails. A data-center card produces several hundred watts of heat, and if your server room isn't ready for it, the card temperature rises and the card throttles itself; the result is a noticeable drop in throughput, not a clear error.

If you decide to rent, for a temporary training load a machine with a dedicated card, and for a permanent service a cloud infrastructure (IaaS) whose size you can change, is a more realistic combination. More details about the differences between cards are in the guide to choosing a graphics card for artificial intelligence.

This is where they go wrong

A common mistake I've seen a lot: someone pushes the batch size all the way up on a 24-gigabyte card to make training faster, training goes fine for a few hours, and then it crashes with CUDA out of memory in the middle of an epoch. The sign is that memory usage in the nvidia-smi log slowly climbs and bursts at a random point; because the activations stored for backward grow with longer sequence lengths. The solution is gradient accumulation, not a bigger batch.

Mistake number two: buying a card based on an inference benchmark and then using the same card for fine-tuning. The tokens per second number in the vendor's table says nothing about training capacity.

Practical configuration: from counting memory to stable execution

First of all, measure actual usage. A small script with torch.cuda.memory_allocated() after one forward step and one backward step gives you the real number; far more accurate than guessing. Then watch the behavior under real load with nvidia-smi --query-gpu=memory.used,utilization.gpu --format=csv -l 5.

For multi-card training, torchrun --nproc_per_node=4 --master_port=29500 train.py is the starting point. If the model doesn't fit on one card, move to FSDP or DeepSpeed with ZeRO stage 3; both spread parameters and optimizer state across cards and demand more communication in return.

export CUDA_VISIBLE_DEVICES=0,1,2,3
torchrun --nproc_per_node=4 --master_port=29500 \
  train.py --per-device-batch-size 8 --grad-accum 4

Don't keep training data on the card's local disk. If the dataset is large, reading from object storage with a local cache prevents an I/O bottleneck. Also write checkpoints somewhere they won't be lost after you release the machine.

For the inference service, bring the model up with vLLM or TensorRT-LLM and put it behind a load balancer; read the principles of load distribution in the load balancer guide. Take monitoring of card temperature and memory usage seriously, because gradual performance degradation is harder to spot than an explicit error; the alerting pattern is in the monitoring and alerting guide.

Frequently asked questions

How many graphics cards are needed to train a 7-billion-parameter model?

With full training and the Adam optimizer at mixed precision, you need about 120 to 140 gigabytes of VRAM. That means at least four 40-gigabyte cards or several 80-gigabyte cards with a high-speed interconnect between them. With LoRA or QLoRA, the same model can be fine-tuned even on a single 24-gigabyte card, because only a small portion of the parameters are trained.

Do I also need the same expensive training card for inference?

No. Inference needs enough memory for the weights and the context buffer, not bandwidth between cards. A mid-range card with 24 gigabytes of memory is enough to serve a quantized 7B model and consumes less power too.

Should I rent a GPU server or buy a card?

If you use the card less than about two-thirds of the month's hours, or your model might change in the next six months, renting makes more sense. Buying pays off when usage is continuous and long-term and you also have the ability to maintain the hardware and cooling.

Why doesn't training on multiple cards achieve linear speedup?

Because communication between cards becomes the bottleneck. Without NVLink, gradient exchange happens over PCIe, and in large models the communication share goes up. Using gradient accumulation, reducing the number of synchronizations, and choosing an appropriate batch size makes efficiency more realistic.

The next step is simple: before any order, calculate your model's memory number with the same bytes-per-parameter formula and then decide. If that number exceeds the memory of a single card, your problem is no longer choosing a card; it's designing distributed training.

ServerNet Support

ServerNet engineering & editorial team — specialists in infrastructure, networking and web hosting.

Cloud Infrastructure (IaaS)
Share:

Comments 0

No comments yet — be the first!

Leave a comment

Related service

Cloud Infrastructure (IaaS)

Servers, private networks, firewalls and storage — all API-driven and billed hourly. Infrastructure as code that scales with you.