Modern AI systems are not just models. They are full-stack machines: silicon, memory, kernels, precision formats, interconnects, distributed systems, and serving engines all working together.
When someone asks, “Why is this LLM slow?” the real answer is rarely “not enough parameters.” More often, the answer involves memory bandwidth, KV cache, batching, quantization, GPU interconnects, or the difference between prefill and decode.
This article builds a complete mental model of GPU-powered AI infrastructure. It starts from basic computer architecture and ends with production LLM inference. The goal is not just to define terms, but to help you understand how the whole stack fits together.
If you finish this article, you should be able to answer questions like:
- Why are GPUs so good for AI?
- What is a warp, and why does it matter?
- Why is LLM inference often memory-bound?
- How much VRAM does a 7B model really need?
- What is KV cache?
- Why do Tensor Cores matter?
- What is the difference between FP16, BF16, FP8, and INT8?
- When should you use data parallelism, tensor parallelism, or pipeline parallelism?
- Why are systems like vLLM, PagedAttention, and FlashAttention so important?
Let’s build the picture layer by layer.
1. The CPU: A Brilliant Specialist
Before understanding GPUs, it helps to understand CPUs.
A CPU is designed to finish individual tasks quickly. It has a small number of powerful cores, large caches, sophisticated control logic, and many features aimed at reducing latency.
A modern CPU does things like:
- Fetch instructions
- Decode instructions
- Execute instructions
- Predict branches
- Reorder operations
- Cache data close to the cores
The CPU is optimized for a question like:
“How fast can I finish this one task?”
This is why CPUs are excellent for general-purpose software, operating systems, databases, and workloads with complex control flow.
The cache hierarchy
One of the most important ideas in computer architecture is the memory hierarchy.
From fastest to slowest, a typical hierarchy looks like this:
- Registers
- L1 cache
- L2 cache
- L3 cache
- Main memory
- Storage
Each level trades capacity for speed. Registers are tiny but extremely fast. Main memory is much larger but slower.
The goal of high-performance code is often to keep data close to the compute unit. This is called locality.
There are two main types:
- Temporal locality: recently used data is likely to be used again.
- Spatial locality: nearby data is likely to be used soon.
This is why sequential array access is usually faster than random access. Sequential access uses cache lines efficiently.
The memory wall
Over time, processors became much faster than memory access. This created the so-called memory wall.
The processor can often compute faster than data can be supplied. When that happens, the bottleneck is not compute. It is data movement.
This idea becomes even more important in GPUs and AI systems.
On the upshot, a CPU is optimized for low latency and not necessarily for total throughput.
2. The GPU: A Massive Parallel Workforce
A GPU is designed differently.
Instead of a few extremely powerful cores, a GPU has many smaller cores. It is optimized for throughput.
A useful analogy:
- A CPU is like a few expert professors solving hard problems quickly.
- A GPU is like thousands of workers doing simple tasks simultaneously.
If your problem is complex and sequential, the professors win. If your problem is massive and parallel, the workforce wins.
AI workloads are often highly parallel. Matrix multiplication, convolution, attention, and elementwise operations can be split into many small pieces. That is why GPUs became the engine of modern deep learning.
Streaming Multiprocessors
Inside a GPU, there are many Streaming Multiprocessors, usually abbreviated as SMs.
Each SM contains compute resources such as:
- CUDA cores
- Tensor cores
- registers
- shared memory
- warp schedulers
- cache resources
You can think of an SM as a small processing unit. A GPU contains many SMs working together.
CUDA cores vs Tensor cores
A CUDA core is a general-purpose compute unit. It can perform operations such as floating-point addition and multiplication.
A Tensor core is specialized for matrix multiply-and-accumulate operations. These operations are central to deep learning.
A typical Tensor core operation looks like:
D = A * B + C
Because neural networks are filled with matrix multiplications, Tensor cores are extremely important for AI performance.
Warps and SIMT
In CUDA programming, you often think in terms of threads. But on NVIDIA GPUs, the hardware executes threads in groups called warps.
A warp is usually 32 threads.
1 warp = 32 threads
The GPU executes these threads using a model called SIMT: Single Instruction, Multiple Threads.
In simple terms, threads inside a warp often execute the same instruction at the same time, but on different data.
This is perfect for workloads like:
Each thread can work on a different index i.
Warp divergence
Warp divergence happens when threads inside the same warp take different branches.
Example:
if (idx % 2 == 0) {
do_a();
} else {
do_b();
}
If some threads in the same warp go one way and others go another way, the GPU may need to execute both paths, reducing efficiency.
This does not mean branches are forbidden. It means that GPU performance depends on how well your workload matches parallel execution.
A GPU is optimized for throughput, and its real execution unit is usually the warp, not the individual thread.
3. GPU Memory: The Quiet Bottleneck
Many people focus on GPU compute performance, but in practice, memory is often the real bottleneck.
The GPU has its own memory, usually called VRAM. This is where model weights, activations, KV cache, and temporary tensors live.
GDDR vs HBM
Two common GPU memory families are GDDR and HBM.
- GDDR is common in consumer GPUs and is cost-effective.
- HBM is common in data-center AI accelerators and provides much higher bandwidth.
Bandwidth is the amount of data that can be moved per second.
For example, if a GPU has 2 TB/s of memory bandwidth, it can theoretically move about 2 terabytes of data per second.
That sounds enormous, but AI workloads move enormous amounts of data too.
Memory bandwidth matters
Consider a simple operation:
For each element, the GPU must:
- Read a[i]
- Read b[i]
- Write c[i]
- Perform one addition
The compute is tiny. The memory movement is large. This kind of kernel is usually memory-bound.
Now consider a large matrix multiplication. There is much more compute per byte moved. That kind of kernel can become compute-bound.
Coalesced memory access
GPUs perform best when threads access memory in a coalesced pattern.
Good pattern:
Thread 0 reads a[0]
Thread 1 reads a[1]
Thread 2 reads a[2]
Thread 3 reads a[3]
Poor pattern:
Thread 0 reads a[0]
Thread 1 reads a[1024]
Thread 2 reads a[2048]
Thread 3 reads a[3072]
The first pattern allows efficient memory transactions. The second creates scattered access and reduces performance.
Shared memory
GPUs also have faster memory close to the compute units. One important example is shared memory.
Shared memory is memory that threads inside the same block can share. It is much faster than global memory.
A common optimization pattern is:
- Load data from global memory into shared memory.
- Reuse the data from shared memory many times.
- Write results back to global memory.
This reduces global memory traffic.
The key insight
In AI infrastructure, performance is often not about raw FLOPS. It is about data movement.
A good phrase to remember:
Compute is loud, but memory is usually the real bottleneck.
GPU performance is often limited by how fast data can move, not by how fast arithmetic can be performed.
4. CUDA: How We Program GPUs
CUDA is NVIDIA’s programming model for writing code that runs on GPUs.
The basic idea is simple:
- The CPU controls the overall program.
- The GPU executes massively parallel functions called kernels.
- Work is organized into threads, blocks, and grids.
Threads, blocks, and grids
CUDA uses a hierarchy:
Thread < Block < Grid
- A thread is the basic unit of work.
- A block is a group of threads that can cooperate.
- A grid is a collection of blocks.
A kernel is launched like this:
kernel_name<<<grid_size, block_size>>>(arguments);
Inside the kernel, each thread computes its own index:
int idx = blockIdx.x * blockDim.x + threadIdx.x;
Example: vector addition
Here is a simple CUDA kernel:
__global__ void vector_add(const float* a, const float* b, float* c, int n) {
int idx = blockIdx.x * blockDim.x + threadIdx.x;
if (idx < n) {
c[idx] = a[idx] + b[idx];
}
}
Launch it like this:
int block_size = 256;
int grid_size = (n + block_size - 1) / block_size;
vector_add<<<grid_size, block_size>>>(d_a, d_b, d_c, n);
The formula for grid_size ensures that enough threads are launched even if n is not divisible by block_size.
Shared memory and synchronization
Threads inside a block can share data using shared memory.
But if multiple threads read and write shared data, they may need synchronization:
This makes threads in the block wait until they all reach the same point.
Streams
CUDA streams allow operations to be asynchronous. For example, one stream can run a kernel while another copies data. This can improve GPU utilization.
Why CUDA matters
Even if you mostly use PyTorch or JAX, CUDA concepts still matter. They help you understand:
- kernel launches
- GPU memory usage
- synchronization overhead
- asynchronous execution
- profiling results
- custom operators
- inference engine internals
CUDA organizes GPU work into threads, blocks, and grids, enabling massive parallelism.
5. GPU Performance: FLOPS, Roofline, and Bottlenecks
To understand AI performance, you need a few core metrics.
FLOPS
FLOPS means floating-point operations per second.
1 TFLOPS = 1 trillion floating-point operations per second
GPU specifications often list theoretical peak FLOPS. But real applications rarely reach the theoretical maximum.
Why?
Because performance depends on:
- memory bandwidth
- cache behavior
- kernel efficiency
- precision format
- Tensor core utilization
- occupancy
- data movement
- CPU overhead
- synchronization
Peak vs achieved performance
Peak performance is the theoretical maximum.
Achieved performance is what your workload actually gets.
A GPU may have enormous peak FLOPS, but if your workload is memory-bound, you may use only a small fraction of that compute.
Arithmetic intensity
A crucial concept is arithmetic intensity.
Arithmetic Intensity = FLOPs / Bytes moved
If your workload does very little compute per byte moved, it is likely memory-bound.
If it does a lot of compute per byte moved, it may be compute-bound.
Memory-bound vs compute-bound
Here are some rough examples:
| |
| |
| |
| |
| |
| |
LLM prefill with large batch | |
This distinction is extremely important.
If a workload is memory-bound, adding more compute may not help much. You need better memory bandwidth, better data reuse, or lower precision.
If a workload is compute-bound, you may benefit from faster compute units, Tensor cores, or lower-precision arithmetic.
Roofline model
The roofline model gives a simple way to think about performance limits:
Performance = min(peak compute, memory bandwidth * arithmetic intensity)
In other words, your kernel is limited either by compute capacity or by memory bandwidth.
Occupancy
Occupancy is the ratio of active warps on an SM to the maximum possible warps.
Higher occupancy can help hide memory latency. If one warp is waiting for memory, another warp can run.
But occupancy is not everything. Sometimes using fewer warps with more registers or shared memory performs better.
Profiling
Common profiling tools include:
- NVIDIA Nsight Systems
- NVIDIA Nsight Compute
- PyTorch Profiler
- nvidia-smi
- DCGM tools
When profiling, ask:
- Is the GPU actually busy?
- Is the kernel memory-bound or compute-bound?
- Are memory accesses efficient?
- Is CPU overhead high?
- Are data transfers blocking the GPU?
Before optimizing, determine whether your workload is limited by memory bandwidth or by compute capacity.
6. Precision Formats: FP32, FP16, BF16, FP8, and Quantization
AI systems rarely need full FP32 precision everywhere. Lower precision can reduce memory usage and increase throughput.
But precision is not free. It affects numerical stability, training quality, and inference accuracy.
FP32
FP32 is 32-bit floating point.
Pros:
- high precision
- wide compatibility
- stable numerically
Cons:
- uses more memory
- requires more bandwidth
- may not fully exploit modern AI accelerators
FP16
FP16 is 16-bit floating point.
Pros:
- half the memory of FP32
- higher throughput on many GPUs
- common for inference
Cons:
- narrower dynamic range than FP32
- can suffer from overflow or underflow in some cases
BF16
BF16 is also 16-bit, but it has a different bit layout.
Its main advantage is that it keeps a dynamic range closer to FP32. This makes it attractive for training.
In simple terms:
- FP16 often gives more precision in the fraction bits.
- BF16 gives more range.
FP8
FP8 is an 8-bit floating-point format used in newer AI hardware and software stacks.
It can provide higher throughput and lower memory usage. Modern training and inference systems increasingly use FP8 where supported.
FP8 variants include:
INT8 and INT4
INT8 and INT4 are integer formats often used for quantization.
Quantization means representing weights and/or activations with fewer bits.
Benefits:
- smaller model size
- lower memory bandwidth requirements
- potentially faster inference
- lower serving cost
Risks:
- accuracy degradation
- calibration sensitivity
- hardware/software support constraints
Tensor cores and precision
Tensor cores are designed to accelerate matrix operations, especially at lower precision.
This is one reason modern AI systems use formats like:
The lower the precision, the more data can fit in memory and the more operations can sometimes be performed per second. But the system must be designed carefully.
Mixed precision
Mixed precision training combines different precisions. For example:
- forward pass in FP16 or BF16
- loss scaling to preserve gradient information
- master weights or optimizer state in higher precision
PyTorch example:
with torch.amp.autocast(device_type="cuda", dtype=torch.float16):
output = model(input)
loss = loss_fn(output, target)
Mixed precision can:
- reduce memory usage
- increase Tensor core utilization
- speed up training
But it must be monitored carefully.
Lower precision can dramatically improve AI performance, but it always introduces a numerical trade-off.
7. LLM Internals: Attention, KV Cache, and VRAM
Now we arrive at one of the most important topics: how large language models actually use GPU resources.
Transformer inference
LLMs generate text one token at a time. At each step, the model predicts the next token based on previous tokens.
Inside a transformer, attention uses three main tensors:
A simplified attention expression is:
Attention(Q, K, V) = softmax(QK^T / sqrt(d)) V
During generation, recomputing all past Key and Value tensors for every new token would be wasteful.
So systems cache them.
This is called the KV cache.
What is KV cache?
The KV cache stores the Key and Value tensors from previous tokens.
Without KV cache:
- every new token would require recomputing attention over all previous tokens
- inference would become extremely slow
With KV cache:
- previous K and V values are reused
- each new token only needs to compute its own new contribution
- generation becomes much faster
But the KV cache grows with:
- number of layers
- number of KV heads
- head dimension
- batch size
- sequence length
- precision
This is why long-context LLMs can consume enormous amounts of VRAM.
Prefill vs decode
LLM inference has two major phases.
Prefill
Prefill processes the input prompt.
Characteristics:
- many tokens are processed together
- large matrix multiplications happen
- often compute-bound
- GPU compute capacity matters a lot
Decode
Decode generates new tokens one by one.
Characteristics:
- often only one token per request is generated at a time
- model weights and KV cache must be read from memory
- often memory-bound
- memory bandwidth matters a lot
This distinction is crucial.
A system can have good prefill performance but poor decode performance, or vice versa.
VRAM math
The memory used by an LLM is not only the model weights.
The major components are:
- Model weights
- KV cache
- Activations
- Framework overhead
- CUDA context
- Memory fragmentation
Model weight memory
The basic formula is:
weights memory = parameter_count * bytes_per_parameter
Example: a 7B parameter model in FP16.
7,000,000,000 * 2 bytes = 14 GB
In INT8:
7,000,000,000 * 1 byte = 7 GB
But that is only the weights.
KV cache memory
A simplified formula for KV cache per token is:
kv_per_token =2 * num_layers * num_kv_heads * head_dim * bytes_per_value
Then total KV cache:
total_kv_cache = batch_size * sequence_length * kv_per_token
Example:
num_layers = 32
num_kv_heads = 8
head_dim = 128
bytes_per_value = 2
KV cache per token:
2 * 32 * 8 * 128 * 2 = 131,072 bytes
That is about:
128 KB per token
Now suppose:
batch_size = 16
sequence_length = 4096
Total KV cache:
16 * 4096 * 131,072 bytes ≈ 8.59 GB
If the model is 7B FP16:
weights ≈ 14 GB
KV cache ≈ 8.59 GB
So you are already around:
22.6 GB + overhead
This is why a “7B model” often needs much more than 14 GB in practice.
MHA, MQA, and GQA
Different attention designs affect KV cache size.
Multi-Head Attention, MHA
Each attention head has its own K and V projections. This can create a large KV cache.
Multi-Query Attention, MQA
All query heads share the same K and V. This greatly reduces KV cache size.
Grouped-Query Attention, GQA
Query heads are grouped and share K/V within groups. This is a middle ground between MHA and MQA.
GQA has become popular because it reduces KV cache while preserving much of the model quality.
Quantization for LLMs
Quantization can reduce LLM memory usage.
Common approaches include:
- INT8 weight quantization
- FP8 inference
- INT4 weight-only quantization
- AWQ
- GPTQ
- SmoothQuant
Quantization can:
- reduce VRAM
- increase throughput
- lower cost
But it can also reduce quality. Always evaluate accuracy after quantization.
LLM inference is dominated not only by model weights, but also by KV cache, batching, and memory bandwidth.
8. GPU Interconnect: PCIe, NVLink, and NVSwitch
Once you use more than one GPU, communication becomes part of the performance story.
GPUs need to exchange data such as:
- gradients
- activations
- model shards
- KV cache data
- expert routing information
The speed of this communication depends on the interconnect.
PCIe
PCIe is a common connection between CPUs and GPUs, and sometimes between GPUs.
It is widely available and cost-effective, but it is not always fast enough for large-scale AI training.
A rough example:
PCIe Gen5 x16 ≈ 64 GB/s in one direction
Actual bandwidth depends on system implementation.
NVLink
NVLink is NVIDIA’s high-bandwidth GPU-to-GPU interconnect.
It typically provides much higher bandwidth than PCIe. This is important for multi-GPU training and inference.
If GPUs are connected with NVLink, they can exchange data faster, which can improve scaling.
NVSwitch
NVSwitch connects multiple GPUs in a high-bandwidth fabric.
It is used in systems where many GPUs need to communicate efficiently with each other.
You can think of NVSwitch as a dedicated high-speed network inside a GPU server.
GPU topology
Not all GPUs in a system are connected equally. Some GPU pairs may communicate faster than others.
You can inspect topology with:
Topology matters because communication cost affects scaling.
Node-level vs cluster-level communication
There are two levels:
- Inside one server
- PCIe
- NVLink
- NVSwitch
- Between servers
- InfiniBand
- RoCE
- Ethernet
Communication between servers is usually slower and higher latency than communication inside one server.
This affects distributed training design.
Communication overlap
Good distributed systems overlap communication with computation.
For example:
- one part of the model computes
- another part transfers gradients or activations
This hides some of the communication cost.
Multi-GPU performance depends not only on GPU speed, but also on how fast GPUs can talk to each other.
9. Multi-GPU: Data, Tensor, Pipeline, Sequence, and Expert Parallelism
Large models often cannot fit on one GPU. Even if they can fit, you may want multiple GPUs for higher throughput.
There are several major parallelism strategies.
Data parallelism
In data parallelism, each GPU keeps a copy of the model. The data batch is split across GPUs.
Each GPU:
- processes its own data shard
- computes gradients
- synchronizes gradients with other GPUs
- updates the model
This is common for training.
Pros:
- relatively simple
- scales well for many workloads
Cons:
- model must fit on one GPU, or you need memory-sharding techniques
- gradient synchronization creates communication overhead
ZeRO and FSDP
ZeRO and FSDP improve data parallelism by sharding model state across GPUs.
Instead of every GPU storing all parameters, gradients, and optimizer states, these states are divided.
This reduces memory usage and enables training of larger models.
Tensor parallelism
Tensor parallelism splits individual layers or matrices across GPUs.
For example, a large matrix multiplication:
Y = XW
can be split so that different GPUs compute different parts of the result.
This is useful when a single layer is too large for one GPU.
Pros:
- handles very large layers
- reduces per-GPU memory pressure
Cons:
- can require heavy communication
- often most effective inside a node with fast interconnect
Pipeline parallelism
Pipeline parallelism splits model layers across GPUs.
Example:
GPU 0: layers 0-10
GPU 1: layers 11-20
GPU 2: layers 21-30
GPU 3: layers 31-40
Data flows through the pipeline.
Pros:
- useful for very deep models
- allows models that do not fit on one GPU
Cons:
- pipeline bubbles can reduce efficiency
- scheduling micro-batches is important
Sequence parallelism
Sequence parallelism splits along the sequence dimension. It is useful for very long context workloads.
Long context increases attention and KV cache costs. Sequence parallelism can distribute those costs.
Expert parallelism
Expert parallelism is used for Mixture-of-Experts models.
Different experts are placed on different GPUs. Tokens are routed to the appropriate experts.
This often involves all-to-all communication.
NCCL and collective operations
NCCL is NVIDIA’s collective communications library. It is widely used for multi-GPU communication.
Important collective operations include:
All-Reduce
Each GPU contributes data, and the reduced result is shared with all GPUs.
Common use:
average gradients across GPUs
All-Gather
Each GPU contributes a piece, and all GPUs receive the combined result.
Reduce-Scatter
Data is reduced and scattered across GPUs.
All-to-All
Each GPU sends different data to each other GPU.
Common in expert parallelism.
Choosing a strategy
A practical guide:
| |
Model fits on one GPU, need more throughput | |
Model does not fit, but optimizer state is large | |
Single layer is too large | |
| |
| |
| |
In real systems, these strategies are often combined.
There is no universal parallelism strategy; the right choice depends on model size, hardware topology, and workload.
10. Modern LLM Inference Infrastructure
Training is only half the story. In production, inference cost and latency often dominate.
Modern LLM serving systems solve several hard problems:
- request scheduling
- batching
- GPU memory management
- KV cache management
- token streaming
- quantization
- latency optimization
- throughput optimization
Static batching and its limits
In static batching, a fixed batch of requests is processed together.
Problem:
- some requests are short
- some are long
- short requests may wait for long requests
- GPU resources may be underused
This is why modern systems use more dynamic approaches.
Continuous batching
Continuous batching allows requests to enter and leave the batch dynamically.
Benefits:
- better GPU utilization
- shorter requests do not unnecessarily wait
- higher throughput
- improved serving efficiency
This is one of the key ideas behind modern LLM engines.
vLLM
vLLM is a high-performance LLM inference engine known for efficient memory management and high throughput.
One of its major innovations is PagedAttention.
PagedAttention
PagedAttention manages KV cache memory in pages, similar to virtual memory paging.
Traditional KV cache allocation can waste memory due to fragmentation. You may reserve large contiguous blocks even if they are not fully used.
PagedAttention helps by:
- splitting KV cache into pages
- reducing fragmentation
- improving memory utilization
- allowing more concurrent requests
This can significantly increase serving throughput.
FlashAttention
FlashAttention is an optimized attention implementation focused on reducing memory traffic.
Classic attention can materialize large intermediate attention matrices. This creates heavy memory movement, especially for long sequences.
FlashAttention improves this by:
- tiling computation
- reducing reads and writes to high-bandwidth memory
- recomputing some values instead of storing them
- improving cache usage
The result is often faster attention with lower memory overhead.
CUDA Graphs
CUDA Graphs can reduce kernel launch overhead.
This is especially useful in decode phases where many small kernels may be launched repeatedly.
Instead of launching kernels one by one every time, a graph of operations can be captured and replayed.
This can reduce CPU overhead and improve performance.
Speculative decoding
Speculative decoding tries to speed up token generation by guessing multiple tokens ahead and verifying them.
The basic idea:
- A smaller draft model proposes tokens.
- The larger model verifies them.
- Accepted tokens are emitted quickly.
This can improve perceived generation speed in some settings.
Prefix caching
If many requests share the same prompt prefix, the system can cache the KV states for that prefix.
This is useful for:
- chatbots with system prompts
- RAG applications with repeated instructions
- agents with repeated context
Prefix caching avoids recomputing the same prefill work repeatedly.
Important inference metrics
When serving LLMs, several metrics matter.
- Time to First Token, TTFT: How long until the first token appears? This strongly affects user experience.
- Time Per Output Token, TPOT: How long does each generated token take? This affects streaming speed.
- Throughput: How many tokens per second can the system generate across all requests? This affects cost and capacity.
- VRAM usage: How much memory is consumed by weights, KV cache, and overhead? This affects scalability.
The production view
A production LLM system is not just “run model.generate()”.
It is a system that manages:
- queues
- batching
- memory pages
- GPU kernels
- precision
- networking
- scaling
- monitoring
- failure recovery
That is why inference infrastructure has become its own engineering discipline.
Modern LLM serving is a systems problem as much as a modeling problem.
11. Putting It All Together: An End-to-End Example
Let’s imagine you want to serve a 7B-parameter chat model on a single GPU.
Step 1: Model weights
If the model is FP16:
7B * 2 bytes = 14 GB
So you need at least 14 GB for weights alone.
Step 2: KV cache
Suppose:
layers = 32
kv_heads = 8
head_dim = 128
precision = FP16
KV cache per token:
2 * 32 * 8 * 128 * 2 = 131,072 bytes
About 128 KB per token.
If you serve:
batch_size = 16
sequence_length = 4096
KV cache is roughly:
16 * 4096 * 131 KB ≈ 8.6 GB
Step 3: Total memory
Now you have:
weights: 14 GB
KV cache: 8.6 GB
overhead: several GB
So you may need around 24 GB or more, depending on the system.
Step 4: Precision choice
You could use INT8 quantization:
weights ≈ 7 GB
This reduces memory pressure and may improve throughput.
But you must check model quality.
Step 5: Inference engine
You choose an inference engine such as vLLM.
It provides:
- continuous batching
- PagedAttention
- efficient scheduling
- KV cache management
- better GPU utilization
Step 6: Metrics
You measure:
- TTFT
- TPOT
- throughput
- VRAM usage
- request latency
Then you tune:
- batch size
- max sequence length
- precision
- caching
- concurrency
This is the real workflow of AI infrastructure engineering.
12. The Mental Model You Should Keep
If you remember only a few things from this article, remember these:
1. AI performance is usually a data movement problem
Compute is important, but moving data efficiently is often the real challenge.
2. GPUs optimize throughput, not latency
GPUs win when you have massive parallelism. They are not magic accelerators for every workload.
3. Tensor cores matter for AI
Matrix multiplication is central to deep learning. Tensor cores are built for exactly that.
4. Precision is a powerful lever
FP16, BF16, FP8, INT8, and INT4 can dramatically change memory and performance behavior.
5. LLM inference is not just model weights
KV cache, batch size, sequence length, and memory fragmentation matter enormously.
6. Prefill and decode are different
Prefill is often compute-bound. Decode is often memory-bound.
7. Multi-GPU systems are communication systems
Adding GPUs is not enough. Interconnect and parallelism strategy determine whether scaling works.
8. Modern inference engines are systems breakthroughs
Techniques like continuous batching, PagedAttention, FlashAttention, and CUDA Graphs can matter as much as hardware.
Final Thoughts
The AI stack looks intimidating at first because it spans many layers:
- transistors
- caches
- cores
- memory
- CUDA
- kernels
- precision formats
- transformers
- attention
- KV cache
- interconnects
- distributed training
- inference engines
But once you see the structure, it becomes much clearer.
The CPU is optimized for low latency. The GPU is optimized for throughput. Memory bandwidth often limits real performance. Tensor cores accelerate matrix math. Lower precision can save memory and increase speed. LLM inference is shaped by KV cache and batching. Multi-GPU systems depend on communication. Production serving engines turn all of this into usable, cost-effective systems.
That is the real lesson:
A large language model is not just a neural network. It is a workload running on a carefully engineered hardware and software stack.
And the better you understand that stack, the better you can build, debug, optimize, and scale AI systems.
Data Scientist
Building and researching end-to-end machine learning and LLM systems, from model training to deployment.
