Run a Local LLM on 8GB VRAM: Practical Setup

What 8GB of VRAM can realistically do

Eight gigabytes of video memory is the most common configuration among enthusiast graphics cards, and it sits at an awkward point for language models. It is comfortably enough to run capable models well, and not nearly enough to run the largest ones at all. Knowing exactly where that boundary falls saves a great deal of wasted downloading.

The practical summary is straightforward. With 8GB you can run 7B and 8B parameter models comfortably at good quality, 13B and 14B models with tighter quantisation and reduced context, and anything substantially larger only by offloading layers to system memory, which slows generation to the point of frustration for interactive use.

This is genuinely useful territory. Modern 7B and 8B models handle summarisation, drafting, code assistance, classification, extraction and conversational tasks at a quality that would have required a data centre a few years ago.

Understanding quantisation, briefly but properly

Model weights are stored at some numeric precision. Full precision uses sixteen bits per parameter, so a 7B model needs roughly 14GB of memory for weights alone — already beyond your budget before accounting for context or overhead.

Quantisation reduces that precision. Four-bit quantisation stores each weight in roughly a quarter of the space, bringing the same 7B model down to approximately 4GB. The trade is a small loss of accuracy, and modern quantisation methods make that loss remarkably small.

The common formats, in the order most people encounter them:

• Q4_K_M — the standard recommendation. Roughly four and a half bits per weight, with a well-judged balance of quality and size. Start here unless you have a reason not to.
• Q5_K_M — noticeably better fidelity at around 25% more memory. Worth it for a 7B model on an 8GB card, where the headroom exists.
• Q8_0 — near-lossless, but roughly double the size of Q4. Generally impractical here except for very small models.
• Q3 and below — use only when the alternative is not running the model at all. Quality degradation becomes obvious, particularly in reasoning and instruction following.

A useful mental model: dropping from Q8 to Q4 costs far less quality than dropping from a 13B model to a 7B model. Prefer a larger model at Q4 over a smaller model at Q8 when both fit.

Budgeting your 8GB honestly

Model weights are not the only consumer of video memory, and this is where most first attempts fail.

• Model weights — the figure quoted on the download page.
• KV cache — memory holding the attention state for the current context. This scales linearly with context length and is frequently underestimated. A long context window can consume more than a gigabyte on its own.
• Compute overhead — activations and working buffers during inference, typically a few hundred megabytes.
• Your desktop environment — on Linux with a desktop running, expect 400MB to 1GB already occupied before you start. On Windows the figure is often higher.

A workable budget for an 8GB card looks like this: reserve roughly 1GB for the system, allow around 1GB for cache and overhead at moderate context, and target model weights of approximately 5.5GB or less. That comfortably accommodates a 7B or 8B model at Q4_K_M or Q5_K_M.

If a model loads but generation is dramatically slower than expected, you have almost certainly exceeded VRAM and silently spilled into system memory.

Choosing your runtime

Three approaches cover nearly all practical use, and they differ mainly in how much control they expose.

Ollama

The simplest path. It manages downloads, quantisation selection and serving behind a single command, and exposes an HTTP API compatible with common client libraries. Sensible defaults, minimal configuration, and by far the fastest route from nothing to a working model. The trade-off is less granular control over inference parameters.

llama.cpp

The engine underneath much of the ecosystem. Running it directly gives complete control over layer offloading, context size, batch size, cache quantisation and threading. If you are trying to squeeze a slightly-too-large model onto the card, this is where the knobs are. It requires reading documentation and accepting some trial and error.

LM Studio or a similar GUI

A graphical interface over the same engines, with model browsing, an estimate of whether a given file will fit, and a chat interface. Excellent for exploration and for sharing a setup with someone who does not use a terminal.

A reasonable progression is to start with Ollama, confirm the hardware works and the quality is acceptable, then move to llama.cpp if you find yourself needing finer control.

Getting more out of the same card

Several adjustments materially improve throughput without new hardware.

• Reduce context length to what you actually use. This is the single largest lever. Many defaults allocate a very large window; if your prompts are short, reducing it frees substantial memory for the model itself.
• Quantise the KV cache. Storing cache at eight bits rather than sixteen roughly halves its footprint with minimal quality impact, which can be the difference between fitting and not fitting.
• Offload deliberately, not accidentally. If a model slightly exceeds VRAM, explicitly place a specific number of layers on the GPU and the remainder on the CPU. Controlled offloading is far better than an automatic fallback.
• Enable flash attention where supported. It reduces memory use during attention computation and typically improves speed.
• Close the browser. Genuinely. A modern browser with hardware acceleration can hold a gigabyte or more of VRAM, and reclaiming it sometimes lets a model fit that otherwise does not.
• On Linux, consider stopping the desktop session for a long batch job and running from a console. This can free close to a full gigabyte.

What to expect in terms of speed

Generation speed is measured in tokens per second. For interactive use, anything above roughly fifteen tokens per second feels acceptable, since it comfortably exceeds normal reading speed.

On a typical 8GB consumer card with a 7B model fully resident in VRAM at four-bit quantisation, generation speeds in the range of thirty to sixty tokens per second are common. Prompt processing is usually much faster than generation, so long inputs cost less time than long outputs.

The moment any layers spill to system memory, expect a severe drop — frequently to single-digit tokens per second, because data must cross the PCIe bus for every token. This cliff is why staying within VRAM matters far more than any other optimisation.

Measure your own numbers rather than trusting published benchmarks. Report the model, quantisation, context length, and whether the entire model was resident in VRAM, since a benchmark missing those details is not reproducible.

Choosing which model to run

Rather than naming specific models, which change constantly, use selection criteria that stay valid:

• Prefer recent releases. The rate of improvement at small parameter counts remains high; a current 7B model often outperforms a considerably larger model from eighteen months earlier.
• Match the model to the task. Code-specialised models outperform general models on code by a wide margin at the same size. The same applies to instruction-tuned versus base models — use instruction-tuned unless you are building something unusual.
• Check the licence if there is any chance of commercial use. Open weights and open licences are not the same thing.
• Test on your actual workload. Public leaderboards measure aggregate ability on academic benchmarks. Assemble twenty prompts representative of your real use and compare candidates on those. The result frequently disagrees with the leaderboard.

When local stops making sense

Running locally is compelling for privacy, offline operation, zero marginal cost and freedom to experiment. It stops making sense in identifiable situations, and recognising them early saves frustration.

• You need frontier-level reasoning quality. No 8GB-class model matches the largest hosted models on genuinely hard tasks, and quantisation cannot close that gap.
• You need to serve many concurrent users. A single consumer card handles one conversation well and several poorly.
• Your workload is occasional. Hosted API costs for light, intermittent use are frequently lower than the electricity and hardware amortisation.
• You need very long context windows. Memory requirements scale with context, and this is exactly where 8GB runs out first.

A common and sensible arrangement is hybrid: run a local model for high-volume, privacy-sensitive or routine work, and route genuinely difficult requests to a hosted model. If you are building retrieval on top of a local model, the architectural considerations are covered in RAG versus fine-tuning.