How Do You Benchmark Local LLMs for Speed, Quality, and Cost in 2026?

What Local LLM Benchmarking Actually Measures

Local LLM benchmarking is the controlled process of measuring how a model performs on a specific computer, runtime, quantization, context length, and workload. It is not a single model score: a 7B model running through llama.cpp on a desktop processor can produce different results from the same weights running through Ollama on a laptop GPU. A useful benchmark therefore records the model revision, file format, quantization, runtime version, accelerator, batch settings, prompt format, and test data. As of September 26, 2026, popular local runtimes include llama.cpp, Llamafile, LM Studio, and Ollama, but none is automatically “faster” or “better” than the others.

Also worth reading: How Do You Build a Local LLM Benchmark Setup That Fits Your Hardware in 2026? · What Is the Best Hardware for Running Local LLMs in 2026? · Which RAG Benchmark Metrics Actually Matter for Retrieval Systems?

Quality testing asks whether the model returns correct, relevant, and consistently formatted answers. Speed testing measures time to first token, tokens per second, total completion time, and behavior under concurrent requests. System testing evaluates memory use, prompt-processing throughput, long-context behavior, thermal throttling, and compatibility with tools. Cost testing estimates electricity, hardware amortization, maintenance, and hosted-API substitution. These categories must be kept separate: 80 tokens per second is impressive for generation but says little about factual accuracy, while a perfect reasoning score says little about whether the machine can keep up with interactive use.

Choosing a Representative Test Suite

A benchmark should reflect the task you expect to perform, not just a popular public leaderboard. For mathematics, AIME-style problems can provide a demanding, numerically scored test, but they do not represent coding, extraction, conversation, or instruction following. AIME normally consists of 15 questions in each of two contests, so a complete 30-question run is more informative than claiming success from a small subset. For coding, use a fixed set of documented programming problems and execute tests rather than judging only the visual appearance of generated code. For production workflows, include structured JSON extraction, function calling, retrieval passages, refusal behavior, and long documents.

A practical local suite can reserve 20 to 30 representative prompts, run each three times, and report the median. Three repetitions help expose random variation, while five may be justified for borderline decisions. Keep temperature, top-p, seed, context-window size, and system prompt fixed. Record a 95th-percentency latency if the application serves users, because an average can hide slow prompts. Compare quality and speed only after confirming that the test configuration is equivalent. The direct answer is to benchmark a small workload you understand deeply, then add a broader public suite as a secondary reference rather than treating a leaderboard score as a purchase decision.

Running the Four Main Local Server Options

Llama.cpp is a native C/C++ runtime with broad hardware support and strong quantization options. It is a sensible baseline because many GGUF conversions and community performance comparisons use it, although command-line operation may require more setup. Llamafile packages model execution into a portable executable, which is useful for testing whether a computer can run a model without installing a conventional application. LM Studio provides a desktop interface, model discovery, and local OpenAI-compatible serving, making it approachable for interactive testing and application development. Ollama emphasizes simple model management and an HTTP API, which makes it convenient for repeatable scripts and tutorials.

No option has a universal performance advantage. Portability, supported accelerators, build flags, model conversions, and server defaults can dominate the result. Compare identical weights and quantization wherever possible, and report cold-start time separately from steady-state throughput.

Measuring Speed Without Misleading Results

Start with three speed measurements: time to first token, generation throughput, and prompt-processing throughput. Time to first token determines perceived responsiveness, while generation tokens per second describes how quickly the answer expands after it starts. Prompt processing matters when loading a large document or sending a long chat history. A concise answer of 100 tokens can feel faster at 50 tokens per second than a 1,000-token answer at 120 tokens per second, even though the latter has higher raw generation throughput.

Run a warm-up request before recording results, because initial inference may compile kernels, load weights, and populate caches. Then test at least 128, 512, and 1,024 output tokens, plus a realistic input such as 2,000 or 8,000 prompt tokens. Repeat each case three times and calculate the median. For interactive use, a first-token latency below roughly 1 second is often desirable, while sustained rates above 20 tokens per second usually feel responsive for many chat tasks. Coding and tool-driven systems benefit from faster responses, but thresholds depend on hardware and model size; a 3B model at 60 tokens per second can still produce poor code.

Hardware acceleration changes the result substantially. Measure whether the runtime offloads all layers to the GPU, splits layers between GPU and CPU, or falls back to unified memory. Monitor temperature over a 15-to-30-minute run because thermal throttling can reduce a benchmark that begins at high speed. Do not combine CPU-only and GPU-accelerated runs in one ranking without labeling them separately.

Testing Accuracy, Reasoning, and Instruction Following

Accuracy benchmarks need exact scoring rules. Integer-answer mathematics permits objective comparison, code tasks can be checked with unit tests, and extraction tasks can use field-level precision and recall. For classification or yes/no answers, always include the expected label set and define whether invalid extra prose counts as failure. For open-ended writing, automated similarity scores are weak because a different answer can still be correct; use a documented rubric or blinded human review instead.

AIME-style scores should not be confused with general intelligence. A model may solve competition mathematics but fail a simple structured extraction task, especially after aggressive quantization. Coding evaluation should compile and execute the result with 10 to 30 hidden tests, not merely ask the model to claim that its program works. For agentic applications, test the complete model-and-runtime combination because tool-call formatting, context handling, and error recovery affect the final score.

Report uncertainty. If 30 test questions are divided evenly between two models and their scores are close, do not describe the difference as decisive without repeated runs. Public test sets can also become training data, so add private examples or recently created tasks. A local model that scores 90% on an easy private suite but 35% on AIME is not necessarily weak; it may be optimized for a different task. The meaningful result is the gap between expected and observed performance on your workload.

Quantization, LoRA, and the 8% Problem

Quantization reduces memory use and often improves throughput, but it can change output quality. Common formats represent weights with 4, 5, 6, or 8 bits, while a higher bit depth generally preserves more information at greater memory cost. A 7B model stored in 4-bit arithmetic may require roughly 3.5 billion parameters’ worth of weight storage before format overhead, while Q8 requires about twice that raw weight storage. Actual RAM use is higher because the runtime also needs metadata, the KV cache, buffers, and application memory.

The “8% problem” should be treated as a warning rather than a universal rule. A change from one quantization to another may leave most simple tasks within several percentage points while causing a larger loss on mathematics, code, or long-context reasoning. That is why aggregate accuracy can hide regressions. Compare FP16 or BF16 as a quality reference when memory permits, then repeat the same prompts at Q8, Q6, Q5, and Q4. Include at least 20 difficult cases where wrong answers are easy to detect.

LoRA adapters add task-specific behavior with less storage than retraining the base model, but they introduce another variable. Keep the base model, adapter revision, merge state, and inference settings fixed, and compare adapted output with the unadapted model. An adapter can improve one narrow task while harming general instructions. Do not attribute a speed change to the adapter alone unless it changes the executed model or adds meaningful computation; many LoRA implementations mainly change output behavior.

Memory Capacity and the Right Model Size

Choose a model that fits with room for the KV cache rather than barely fitting its weights. Unified-memory systems and dedicated GPUs share the same practical constraint: enough capacity must remain for a 4,096- or 8,192-token context. If the operating system and runtime leave only 5% free, long prompts can fail or force lower context lengths. For reliable experimentation, keep at least 15% to 20% of available memory available after loading the model.

A 7B or 8B model is often a practical starting point for quality testing on 16 GB systems, while 12B to 14B models are more demanding and may need 24 GB or more depending on format and context. These are ranges, not guarantees. A 24 GB machine may run a larger quantized model but still deliver low prompt throughput if the workload spills onto the CPU. Compare three points: the smallest model that meets quality requirements, the largest model that fits comfortably, and a mid-sized fallback for latency.

Context length is not equivalent to usable context. A model can advertise a 32,768-token window yet lose information near the beginning, especially at low quantization. Test the beginning, middle, and end of a controlled document. Measure retrieval accuracy and generation latency separately. A model that accepts 8,192 tokens but returns incorrect details from 6,000 tokens away is not a successful long-context deployment, regardless of the advertised window.

Cost, Power, and Production Alternatives

Local software is often free, but inference is not free. Electricity cost can be estimated as device power in kilowatts multiplied by hours, then multiplied by the local electricity rate. A workstation averaging 250 W for two hours consumes about 0.5 kWh, which costs approximately $0.05 to $0.15 at $0.10 to $0.30 per kWh. A laptop using 60 W for the same period consumes 0.12 kWh, usually costing less than $0.04 at those rates. These figures exclude heat, backup hardware, and the purchase price.

For amortized cost, divide the machine’s net value by its expected productive life. A $1,500 workstation retained for three years represents about $41.67 per day when used every day, before power and maintenance; used half the time, the daily allocation becomes about $83.34. Compare that figure with the per-token value of a hosted model, not merely the provider’s advertised API price. Local inference becomes economically attractive when usage is regular, privacy matters, hardware is already owned, or predictable offline operation has practical value.

Hosted APIs can still win on quality, fleet management, and burst capacity. A small local model routed between instances may outperform a larger model forced onto inadequate hardware. Remote fallback, batching, caching, and routing often improve production systems more than a marginal model change. Benchmarks should therefore include the local-only configuration, the API-only configuration, and the hybrid configuration under the same workload.

When to Benchmark, Change Models, or Take Action

Benchmark before buying additional hardware, publishing a production claim, or standardizing on one model. The test should be completed early enough to influence a decision but after basic correctness is established; changing models and runtimes simultaneously makes it impossible to identify the cause. A model is worth adopting if it exceeds the application’s quality threshold, meets the chosen latency target, fits with memory headroom, and lowers total operating cost under realistic use. Examples of useful thresholds are 95% valid JSON on extraction, at least 90% unit-test passage on a bounded coding set, and a 95th-percentile first-token time below 2 seconds for an interactive service.

Wait rather than act on a small score difference. Differences below 2 percentage points across only 30 examples may reflect sampling noise, prompt ordering, or runtime variation. Re-run the benchmark, expand the test set, and check whether the conclusion survives a longer context. Act quickly when a model consistently fails safety requirements, produces invalid tool calls, exceeds memory limits, or fails a core business criterion. A fast incorrect model is still a bad system, while a slightly slower model that passes all required tests may be the rational choice.

The definitive method is controlled comparison: hold the workload constant, vary one factor at a time, measure quality and performance separately, and retain the configuration details. Local benchmarking is most useful when it answers a concrete operational question such as whether a 7B Q4 model can serve a 4,000-token coding request with sub-2-second first-token latency. Popularity, parameter count, and an impressive AIME result cannot answer that question by themselves.