Interactive Troubleshooter

Published: Sep 3, 2026

Local LLM Running Slow: Ollama/LM Studio Optimization

Time: 15 mins Level: Beginner

By StepDiag Engineering Team

Reviewed by Ibraheem Taofeeq Opeyemi

An interactive, step-by-step diagnostic guide to troubleshoot slow token generation, high prompt evaluation latency, and CPU bottlenecks when running local LLMs in Ollama or LM Studio. Follow guided workflows to optimize GPU layer offloading, adjust context sizes, and select proper model quantization levels.

Full Diagnostic Tree & Step-by-Step Overview

Which local LLM execution framework or application is experiencing slow response times?

Ollama (Command Line Interface or Background Service API)
LM Studio (Graphical Desktop Interface)

What is the specific nature of the performance lag in Ollama?

Token generation is extremely slow (low tokens per second) during entire output streaming.
Prompt evaluation phase stalls for a long time before initial token generation begins.

Resolve VRAM Spilling & Enforce Full GPU Offloading in Ollama

Solution:

Root Cause: VRAM Capacity Exceeded / CPU Layer Fallback

When an Ollama model exceeds your GPU's dedicated Video RAM (VRAM), llama.cpp automatically offloads surplus model layers to system RAM and processing to CPU. Because system RAM bandwidth is significantly slower than VRAM, token generation speed degrades dramatically.

Step-by-Step Fix:

1.Check GPU Memory Allocation During Inference:

Open Terminal or PowerShell while Ollama is generating text.

Run nvidia-smi (for NVIDIA GPUs) or rocm-smi (for AMD GPUs) to inspect VRAM usage.

Execute ollama ps in a separate terminal window to view the GPU/CPU offload ratio (e.g., 100% GPU vs 70%/30% CPU/GPU).

2.Switch to a Lower Quantization or Smaller Parameter Model:

If offloading shows partial CPU usage, download a higher-quantized or smaller parameter variant (e.g., switch from llama3:8b-instruct-q8_0 to llama3:8b-instruct-q4_K_M).

Command: ollama run llama3:8b-instruct-q4_K_M

3.Update GPU Drivers & CUDA Toolkit:

Verify that your GPU drivers support the latest CUDA runtime by installing updates directly from the NVIDIA Official Driver Downloads page.

4.Force Full VRAM Offload via Environment Variables:

Set OLLAMA_NUM_PARALLEL=1 to prevent multiple concurrent requests from splitting available VRAM across session instances.

Optimize Ollama Context Window Size & Thread Count

Solution:

Root Cause: Excessive Context Allocation (`num_ctx`) Overhead

Processing massive prompt context windows (e.g., 32k or 128k tokens) requires significant compute resources during the initial prompt evaluation phase (KV cache creation). High default context allocations force CPU memory paging before generation starts.

Step-by-Step Fix:

1.Restrict Context Window in Custom Modelfile:

Create a text file named Modelfile containing:

dockerfile
     FROM llama3:8b
     PARAMETER num_ctx 4096

Build the optimized model variant: ollama create llama3-4k -f ./Modelfile

2.Adjust Runtime Parameters in API Requests:

When sending requests via API, explicitly declare num_ctx in the options block:

json
     {
       "model": "llama3",
       "options": {
         "num_ctx": 4096,
         "num_thread": 8
       }
     }

3.Set CPU Thread Limits for System Hardware:

Set num_thread to match your CPU's physical core count (not logical hyperthreads) to avoid CPU thread thrashing. Further setup details can be found in the Ollama Official Documentation.

What performance bottleneck are you encountering in LM Studio?

GPU acceleration is disabled or GPU Offload sliders are set incorrectly.
Text generation drops in speed as conversation length increases over time.

Enable Hardware Acceleration & Adjust GPU Offload Slider in LM Studio

Solution:

Root Cause: Unconfigured Acceleration Runtime or Partial CPU Fallback

LM Studio requires manual configuration of the GPU Offload setting for GGUF models. If the GPU Acceleration toggle is disabled or the offload slider is set to 0, inference runs exclusively on the system CPU.

Step-by-Step Fix:

1.Access Model Loader Settings:

Open LM Studio and select a model from your local library.

On the right-hand sidebar under My Models / Preset Settings, locate the GPU Acceleration panel.

2.Configure GPU Layer Offload:

Enable the GPU Acceleration toggle switch.

Drag the GPU Offload (n_gpu_layers) slider all the way to maximum (Max).

*Note:* If LM Studio crashes upon loading, reduce the layer slider by 5-10 layers until the model fits into VRAM without exceeding memory limits.

3.Select Optimal Hardware Runtime:

Under Advanced Settings, ensure the selected compute backend corresponds to your hardware: Apple Metal (Mac Apple Silicon), CUDA (NVIDIA GPUs), or Vulkan / ROCm (AMD GPUs).

Enable Flash Attention & KV Cache Quantization in LM Studio

Solution:

Root Cause: Uncompressed Key-Value (KV) Cache Saturation

As conversational context grows during multi-turn chats, storing full-precision KV cache matrices consumes substantial VRAM and increases memory bandwidth latency per token.

Step-by-Step Fix:

1.Enable Flash Attention:

In the right-hand panel under Model Initialization, toggle Use Flash Attention to On.

Flash Attention significantly reduces memory bandwidth requirements during long-context prompt processing.

2.Enable KV Cache Quantization:

Locate Context Overflow Policy / K/V Cache Precision in LM Studio settings.

Change KV Cache Precision from FP16 to Q8_0 or Q4_0 to cut memory usage for chat history in half.

3.Reduce Context Buffer Limit:

Lower the Context Length setting (e.g., from 32768 to 8192 or 4096) to cap memory allocation for context storage.

Related Diagnostic Guides

Ubuntu Black Screen After Nvidia Driver Install - Secure Boot Diagnostic

SSH Connection Refused on Port 22 But Service Running - Firewall Tree

Linux Boot Stuck at GRUB Rescue After Windows Update - Dual Boot Fix

SmartScreen Can't Be Reached Right Now Error Fix - Windows Diagnostic Guide

Account Locked After Too Many Attempts in Microsoft 365 - Admin Unlock Tree

Subscribe to StepDiag

Embed Guide on Your Website