LLM Infrastructure Sizing: From Hardware Requirements to Production Capacity

Most VRAM calculators answer the wrong question. They tell you whether a model will load. They don't tell you whether it will serve your production traffic.

A Llama 3.1 70B model needs about 35GB to load at 4-bit quantization. But serving 50 concurrent users with 8K context windows? That pushes memory requirements past 80GB. The difference is KV cache, and most sizing guides ignore it.

This guide covers the complete memory equation, from model weights through KV cache and batch overhead. You'll learn to calculate throughput capacity, understand when self-hosting breaks even against APIs, and size infrastructure for actual production workloads rather than demo deployments.

The Complete Memory Equation

GPU memory during LLM inference splits into four components:

Total VRAM = Model Weights + KV Cache + Activations + Framework Overhead

Most discussions stop at model weights. Production deployments can't afford to.

Model Weights

The baseline. Every parameter needs storage, and the amount depends on precision.

The formula: VRAM (GB) = Parameters (B) × Bytes per Parameter

A 70B model at FP16 precision needs 140GB just for weights. That's two A100-80GB cards before you process a single token. Quantization to 4-bit cuts that to 35GB, fitting on a single A100 with room for inference overhead.

Quantization quality has improved dramatically. Modern GPTQ and AWQ methods preserve 95%+ of model quality at INT4. For most production use cases, the quality difference is imperceptible while memory savings are substantial.

KV Cache: The Hidden Memory Consumer

KV cache stores attention key-value pairs for every token in the context window. It grows with sequence length and batch size, and it's where most sizing estimates go wrong.

The formula:

KV Cache (bytes) = 2 × num_layers × hidden_size × sequence_length × batch_size × bytes_per_element

For Llama 3.1 70B (80 layers, 8192 hidden size) at FP16:

At batch size 1, KV cache is manageable. At production batch sizes serving concurrent users, it dominates memory consumption.

This is why a model that loads fine for testing fails under production load. The model weights fit. The KV cache for 32 concurrent requests doesn't.

KV Cache Optimization:

PagedAttention (used by vLLM) reduces KV cache waste from 60-80% to under 4% by managing memory in fixed-size blocks rather than pre-allocating for maximum sequence length. This optimization alone can double or triple the number of concurrent requests a given GPU configuration supports.

Activations

Intermediate computation results during the forward pass. These depend on batch size, sequence length, and model architecture. Typically 1-5GB for inference workloads, more for training.

Activations are usually the smallest component for inference, but they scale with batch size. At very large batch sizes (64+), activation memory becomes significant.

Framework Overhead

CUDA contexts, memory allocators, graph compilation, and runtime buffers. Expect 5-15% overhead on top of theoretical requirements. vLLM and TensorRT-LLM are more memory-efficient than naive HuggingFace implementations.

Sizing by Model Class

Practical recommendations based on real deployment patterns:

Small Models (1B-8B)

Hardware: RTX 4060 Ti 16GB handles most small models comfortably. RTX 4090 24GB provides headroom for larger batch sizes or longer contexts.

Use cases: Classification, extraction, simple Q&A, edge deployment, high-volume low-complexity tasks.

Medium Models (13B-34B)

Hardware: RTX 4090 24GB for smaller medium models. Dual RTX 4090s or single A100-40GB for 32B class.

Use cases: General assistants, coding, document analysis, moderate-complexity reasoning.

Large Models (65B-72B)

Hardware: A100-80GB minimum for Q4 at production batch sizes. Dual A100-80GB or H100-80GB for FP16 or higher throughput.

Use cases: Complex reasoning, nuanced generation, tasks where quality matters more than latency.

Frontier Models (100B+)

Hardware: Multi-node GPU clusters. H100 NVLink configurations. Enterprise infrastructure.

Throughput: From Memory to Capacity

Knowing a model fits tells you nothing about how many users it can serve. Throughput depends on batch efficiency and memory bandwidth.

Tokens Per Second Estimation

LLM inference is memory-bandwidth bound. The theoretical maximum throughput:

Max Tokens/Second ≈ Memory Bandwidth (GB/s) / Model Size (GB)

These are theoretical maximums at batch size 1. Real-world throughput depends heavily on batching efficiency.

Batch Size Impact

Larger batches amortize the cost of loading model weights across more requests, dramatically improving throughput per GPU:

Numbers vary by framework. vLLM with continuous batching achieves 2-4× higher throughput than static batching at the same batch size.

Users to Throughput Math

Convert business requirements to infrastructure needs:

Assumptions for a typical chat application:

• Average request: 200 input tokens + 300 output tokens
• Target latency: 5 seconds for complete response
• Concurrent users: peak simultaneous requests

Required Throughput (tok/s) = Concurrent Users × Output Tokens / Target Latency

Example: 50 concurrent users, 300 output tokens, 5-second target = 3,000 tok/s required

For smaller models at lower precision, these numbers improve dramatically. A 7B model at Q4 on a single RTX 4090 can handle 50+ concurrent users for typical chat workloads.

Self-Hosting Economics

The decision between self-hosting and APIs comes down to utilization and volume.

Cost Per Token Comparison

Self-hosted costs assume amortized hardware or spot instance pricing at high utilization.

Break-Even Analysis

The crossover point depends on daily token volume and utilization rate.

Self-hosting 7B model (single H100 spot instance at $2/hour):

• Monthly infrastructure cost: ~$1,500 (including overhead)
• Throughput capacity: ~500,000 tokens/hour at batch size 8
• Break-even vs GPT-4o mini: ~2.5M tokens/day
• Break-even vs GPT-4o: ~150K tokens/day

Self-hosting 70B model (2× A100-80GB at $4/hour):

• Monthly infrastructure cost: ~$3,000
• Throughput capacity: ~100,000 tokens/hour at batch size 8
• Break-even vs GPT-4o: ~300K tokens/day
• Break-even vs Claude Sonnet: ~200K tokens/day

For detailed guidance on the self-hosting decision, including setup walkthroughs and tool comparisons, see our complete self-hosted LLM guide.

Hidden Costs

Infrastructure cost isn't just hardware:

Compare against equivalent API spend. If your projected API bill exceeds $15K/month with consistent utilization, self-hosting likely makes sense. Below $5K/month, APIs are almost always more cost-effective when accounting for engineering overhead.

Multi-GPU Scaling

When single-GPU capacity isn't enough, you have two scaling strategies.

Tensor Parallelism (Model Sharding)

Split the model across GPUs. Each GPU holds a portion of every layer, and GPUs communicate during forward passes.

• Use when: Model doesn't fit on single GPU (70B+ at FP16)
• Scaling: Near-linear up to 4-8 GPUs, diminishing returns beyond
• Requirement: High-bandwidth interconnect (NVLink recommended)
• Latency impact: Minimal per-request, communication overhead ~5-15%

# vLLM with tensor parallelism
from vllm import LLM

llm = LLM(
    model="meta-llama/Llama-3.1-70B",
    tensor_parallel_size=4,  # Split across 4 GPUs
    dtype="float16"
)

Pipeline Parallelism (Layer Sharding)

Split layers across GPUs sequentially. GPU 1 processes layers 1-20, GPU 2 processes layers 21-40, etc.

• Use when: Memory is the bottleneck, throughput is secondary
• Scaling: Works across nodes, tolerates lower interconnect bandwidth
• Latency impact: Higher per-request (sequential dependency)
• Throughput: Lower than tensor parallelism for single requests, but scales batches well

Data Parallelism (Replicas)

Run multiple model copies, route requests across them.

• Use when: Throughput is the bottleneck, model fits on single GPU
• Scaling: Linear with GPU count
• Management: Load balancer required, simpler than model parallelism
• Cost: Most efficient for high-volume deployments

For most production deployments, data parallelism (replicas) combined with efficient single-GPU inference (vLLM) provides the best throughput per dollar.

Production Checklist

Before deploying:

Memory Validation

• [ ] Model loads with 20% VRAM headroom
• [ ] KV cache calculated for max batch size × max context
• [ ] Tested under sustained load for 30+ minutes
• [ ] OOM errors handled gracefully (request rejection, not crash)

Throughput Verification

• [ ] Benchmarked at expected batch sizes
• [ ] Latency P50/P95/P99 meet SLA targets
• [ ] Throughput sustains during peak traffic simulation
• [ ] Continuous batching enabled (vLLM, TensorRT-LLM)

Reliability

• [ ] Health checks configured
• [ ] Auto-restart on failure
• [ ] Monitoring for VRAM usage, throughput, latency
• [ ] Alerting for degraded performance

Cost Controls

• [ ] Utilization monitoring (target 60%+ for cost efficiency)
• [ ] Auto-scaling policies if using cloud
• [ ] Spot instance handling (checkpointing, graceful migration)

Quick Reference Tables

VRAM by Model Size (Q4 Quantization)

GPU Recommendations by Use Case

Serving Framework Comparison

For teams that want production-grade infrastructure without managing the stack, Prem's self-hosted platform handles optimization, scaling, and monitoring while keeping data on your infrastructure.

Summary

Sizing LLM infrastructure requires thinking beyond model weights:

1. Calculate complete memory: Weights + KV cache + activations + overhead
2. Size for production batch sizes: Single-request memory is misleading
3. Match throughput to users: Convert concurrent users to required tokens/second
4. Factor total cost: Hardware, engineering, power, not just GPU hours
5. Break-even analysis: Self-hosting wins above ~2M tokens/day for most configurations

The common mistake is sizing for "can it run" rather than "can it serve." A model that loads fine in testing may fail under production load when KV cache for concurrent requests exceeds available VRAM.

Start with clear requirements: concurrent users, target latency, acceptable cost per request. Work backward to infrastructure. Benchmark under realistic load before committing to hardware purchases or cloud reservations.

For managed infrastructure that handles the optimization complexity, Prem Studio provides fine-tuning, evaluation, and deployment as an integrated pipeline. You focus on the model; the platform handles the infrastructure math.