Ollama Context Length: Configuration, VRAM Limits, and Scaling

Ollama context length defines the maximum token capacity allocated in memory for prompt input and completion output, dynamically scaled by hardware VRAM tiers or manually overridden via the OLLAMA_CONTEXT_LENGTH environment variable and num_ctx parameter. When developers run local models for retrieval tasks, coding assistance, or multi-turn agent conversations, token boundaries dictate whether an interaction succeeds or silently truncates. Understanding how Ollama allocates memory across hardware tiers eliminates unexpected prompt clipping and prevents GPU out-of-memory crashes.

Ollama default context length allocates 4,096 tokens under 24 GiB VRAM, 32,768 tokens from 24 to 48 GiB VRAM, and 256,000 tokens above 48 GiB VRAM. Many legacy guides claim that Ollama strictly caps all models at 2,048 tokens out of the box. That 2,048-token limit was the historical default inherited from early llama.cpp builds, intended to keep small models functional on low-memory personal computers. Modern Ollama versions dynamically inspect available GPU video memory at server startup and assign active context windows according to available hardware capacity.

Modern open-weights models feature expansive native context architectures designed to ingest hundreds of pages of documentation. Meta Llama 3.1, Llama 3.2, and Llama 3.3 support 128,000 tokens natively, Mistral NeMo supports 128,000 tokens, and Qwen 2.5 models range from 32,768 to 131,072 tokens. However, pulling and running these models in Ollama without explicit configuration locks them into Ollama hardware-tier default. If you send a 15,000-token prompt to Llama 3.1 running on a 16 GB GPU, Ollama restricts the execution context to 4,096 tokens, discarding the beginning of your prompt without returning an error.

As documented in official Ollama guidance on context length, tasks that require large context, including agents, web search, and coding tools, should be configured to at least 64,000 tokens when hardware allows. Operating an agent at that scale requires knowing both the native capabilities of the model architecture and the exact configuration parameters that override local server defaults.

The following comparison table outlines native model context ceilings alongside Ollama default and recommended agent allocations:

Deploying models for agent workflows requires actively configuring these limits rather than relying on server defaults. When an agent reads file directories or processes complex tool schemas, exceeding the 4,096-token baseline happens quickly, making manual configuration mandatory.

• Ollama Max Tokens: Setting Generation Limits and Context BudgetsIn Ollama, setting generation limits requires configuring the num_predict parameter, which sets the maximum number of...
• LM Studio Context Length: Configuration, VRAM Tuning, and MCP RetrievalLM Studio context length dictates how much local GPU VRAM and system memory are allocated for the attention KV cache...
• Llama.cpp Context Size: How to Set -c, Calculate KV Cache, and Optimize VRAMIn llama.cpp, context size is the maximum sequence length in tokens allocated for prompt and response processing,...
• vLLM max-model-len: Tuning Context Length, VRAM, and MCP SearchIn vLLM, max-model-len defines the total token sequence length for prompts and outputs, directly governing GPU KV cache...
• Salesforce Agentforce Limits: Tokens, Context Windows, and Grounding File CapsAgentforce limits establish operational boundaries for autonomous AI across Salesforce, including a 1.5 million hourly...
• ChatGPT Conversation Too Long: Fixes, Limits, and Context WorkaroundsThe conversation too long error in ChatGPT occurs when an ongoing thread exceeds the model working memory context...

More on this subject: AI Agents: General Guides (218 guides)

Configuring Ollama context length requires choosing the appropriate control point for your operational setup. Developers can enforce limits server-wide, adjust values inside desktop interfaces, modify active terminal sessions, or bake parameters into custom model distributions.

Here are the four standard methods for configuring Ollama context length:

1. Setting the Server-Wide Environment Variable (OLLAMA_CONTEXT_LENGTH): Define the global context length default across all models loaded by the Ollama server daemon. When launching Ollama directly from a terminal or shell script, prepend the variable:

OLLAMA_CONTEXT_LENGTH=64000 ollama serve

For production Linux deployments managed by systemd, add the environment variable to the systemd override configuration:

sudo systemctl edit ollama.service

Add the variable assignment under the service unit definition:

[Service=unit]
Environment="OLLAMA_CONTEXT_LENGTH=64000"

Apply the changes and restart the service:

sudo systemctl daemon-reload
sudo systemctl restart ollama

On macOS installations running the background service, set the environment variable through launchctl before launching the application:

launchctl setenv OLLAMA_CONTEXT_LENGTH 64000

When deploying Ollama in containerized environments using Docker or Docker Compose, pass OLLAMA_CONTEXT_LENGTH within the environment block of your service configuration:

services:
  ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    environment:
      - OLLAMA_CONTEXT_LENGTH=64000
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

2. Adjusting the Desktop Application Slider: For users running the Ollama desktop application on macOS or Windows, context length can be controlled without terminal commands. Click the Ollama status icon in the operating system menu bar or taskbar, open Settings, locate the Context Length slider, and adjust the slider to your desired token ceiling. Changes applied through the desktop interface take effect on subsequent model loads.

3. Setting Session-Level Parameters in the Interactive CLI: When experimenting with models inside the terminal REPL using ollama run, update the context length dynamically for the active session:

>>> /set parameter num_ctx 32768
Set parameter 'num_ctx' to '32768'

Any subsequent prompt submitted in that interactive terminal session will use the 32,768-token window. To persist this configuration as a named model variant without editing text files, run /save immediately after setting the parameter:

>>> /save llama3.2-32k
Created new model 'llama3.2-32k'

4. Defining Parameters Permanently in a Modelfile: Creating a custom model file is the recommended approach for team consistency and automated pipelines. Create a plain text file named Modelfile:

FROM llama3.2:latest
PARAMETER num_ctx 32768
PARAMETER temperature 0.2

Build the custom model into your local Ollama library:

ollama create llama3.2-32k -f Modelfile

Launch the newly built model using its custom tag:

ollama run llama3.2-32k

In programmatic integrations, developers can also override context length per request by passing num_ctx inside the options object of the Ollama REST API. This approach allows client applications to allocate smaller 4,096-token windows for simple queries while reserving 32,768-token windows for heavy document analysis:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Summarize the technical constraints outlined in the system architecture specification.",
  "options": {
    "num_ctx": 32768
  }
}'

Choosing between these methods depends on deployment architecture. Environment variables provide clean global baselines for server instances, while Modelfile definitions provide reliable reproducibility across developer workstations.

Increasing Ollama context length does not increase the video memory required to store static model weights. An 8B parameter model quantized at 4 bits (Q4_K_M) requires approximately 4.9 GB of VRAM regardless of whether your prompt contains 10 tokens or 50,000 tokens. Instead, expanded context windows consume memory through the Key-Value (KV) cache.

The Key-Value cache stores intermediate attention representations for every processed token across all transformer layers and attention heads. Without this cache, the model would need to recalculate self-attention across the entire sequence history for every newly generated token, causing generation latency to scale quadratically. The memory required for the KV cache scales linearly with sequence length according to a specific formula:

KV Cache Memory (Bytes) = 2 * Layers * KV_Heads * Head_Dimension * Context_Tokens * Bytes_Per_Element

In this calculation:

• The multiplier 2 accounts for separate Key and Value vectors.
• Layers represents the total number of transformer blocks in the model architecture.
• KV_Heads represents the number of key-value attention heads.
• Head_Dimension represents the dimensionality of each attention head (typically 128).
• Context_Tokens is the configured num_ctx value.
• Bytes_Per_Element corresponds to numerical precision, which is 2 bytes for standard 16-bit floating-point (FP16) representations.

Modern architectures implement Grouped-Query Attention (GQA), which shares key-value heads across groups of query heads. In Meta Llama 3.1 8B, the architecture contains 32 query heads but only 8 KV heads. This architectural optimization reduces KV cache memory consumption substantially, by a factor of four compared to older Multi-Head Attention designs.

Calculating the KV cache memory for Llama 3.1 8B (32 layers, 8 KV heads, head dimension of 128, and FP16 precision) demonstrates how memory scales with token count:

Memory per Token = 2 * 32 * 8 * 128 * 2 = 131,072 bytes (128 KiB per token)

Applying this per-token footprint to common context lengths yields the following memory demands:

• 4,096 tokens: 128 KiB * 4,096 = 512 MiB (0.5 GiB)
• 16,384 tokens: 128 KiB * 16,384 = 2,048 MiB (2.0 GiB)
• 32,768 tokens: 128 KiB * 32,768 = 4,096 MiB (4.0 GiB)
• 64,000 tokens: 128 KiB * 64,000 = 8,000 MiB (~7.8 GiB)
• 128,000 tokens: 128 KiB * 128,000 = 16,000 MiB (~15.6 GiB)

The following memory allocation table details total VRAM requirements across model sizes and context lengths, combining static model weights with the active KV cache:

When the combined memory footprint of model weights and KV cache exceeds available GPU VRAM, Ollama attempts to offload remaining layers and memory blocks to system RAM via CPU inference. This offloading introduces immediate performance degradation. Processing prompt evaluation across the system PCIe bus reduces generation throughput from 45 tokens per second to less than 2 tokens per second.

If system memory is also exhausted, the local process terminates with a fatal out-of-memory failure. To mitigate memory pressure without sacrificing token capacity, developers can run quantized KV cache modes (q8_0 or q4_0) in supported builds, cutting KV cache footprint roughly in half with minimal loss in retrieval accuracy.

Prevent Ollama GPU Out of Memory Crashes with Remote MCP Workspaces

Offload reference corpora, technical manuals, and multi-file codebases to Fastio workspaces with automated hybrid indexing, connecting your local models and agents through remote Streamable HTTP MCP tools. Monthly plans start with a 14-day free trial, which requires a credit card.

Start 14-Day Trial

The most challenging failure mode in Ollama is silent context truncation. Unlike API errors that fail loudly with explicit HTTP status codes, Ollama processes prompts that exceed num_ctx by quietly dropping the earliest tokens from the context window. The model generates a grammatically sound, confident response, but it does so without having read the opening chapters of your document, the earliest messages of your chat history, or critical system instructions.

Detecting and resolving this behavior requires active runtime inspection.

Verifying Runtime Allocations with ollama ps

To inspect running models and their active context allocations, run the process status command in a separate terminal:

ollama ps

The terminal prints a formatted table displaying active memory allocations:

NAME               ID              SIZE      PROCESSOR        UNTIL              CONTEXT
llama3.2:latest    a80c4f17acd5    4.9 GB    100% GPU         4 minutes from now 4096

Key indicators to inspect include:

• CONTEXT: Confirms the exact token limit assigned to the model. If you intended to run at 32,768 tokens but the table reports 4096, the server is running on default configuration.
• PROCESSOR: Shows execution placement between GPU and CPU. A status indicating pure GPU execution shows optimal inference throughput. If the column displays a mixed split between GPU and CPU, the active context length has pushed the KV cache beyond available VRAM, causing layer offloading and slower response times.

Inspecting API Response Metrics

When communicating with Ollama through scripts or agent runtimes, check the operational telemetry included in the JSON response payload:

{
  "model": "llama3.2",
  "created_at": "2026-10-31T09:30:00Z",
  "response": "The architectural review indicates...",
  "done": true,
  "total_duration": 4820194833,
  "load_duration": 1823904,
  "prompt_eval_count": 4096,
  "prompt_eval_duration": 3410294000,
  "eval_count": 284,
  "eval_duration": 1404076000
}

The critical metric is prompt_eval_count. If you dispatched a prompt containing 12,500 tokens but prompt_eval_count returns 4096, Ollama silently truncated 8,404 tokens. Reviewing this value in automated pipelines provides immediate detection of truncated inputs.

Context Degradation and Rotary Position Embeddings

Expanding num_ctx from 4,096 tokens to 64,000 tokens ensures that text reaches the model, but it does not guarantee equal attention across the entire sequence. Modern transformers rely on Rotary Position Embeddings (RoPE) to encode positional information. While frequency base adjustments (rope_frequency_base and rope_frequency_scale) permit context window expansion, models experience attention dilution when processing massive contexts.

In long-context evaluations, models frequently exhibit the 'lost in the middle' phenomenon: retrieval accuracy is highest for facts placed at the extreme beginning and end of the prompt, while facts located throughout the middle portion suffer higher omission rates. When configuring large context windows for autonomous agents, relying entirely on extended context windows to parse large document collections introduces latency penalties and retrieval inconsistencies.

Stuffing entire document libraries, technical specifications, and historical chat logs directly into local model context creates severe operational bottlenecks. Even when high-end hardware permits setting num_ctx to 64,000 or 128,000 tokens, processing massive prompts degrades Time-to-First-Token latency, consumes excessive power, and increases the likelihood of retrieval hallucinations.

This architectural constraint mirrors file upload limits across major cloud AI platforms. In official Anthropic documentation, a Claude chat conversation accepts up to 20 files at up to 500MB each, while a project accepts files up to 30MB each with no fixed file count limit. As Anthropic notes, project file counts are unlimited, but the total content must fit within Claude's context window. Claude Projects has no fixed file-count cap, so any fixed project file count is false. The practical ceiling on a project is the context window, and reaching it is when developers with a large corpus look for another path.

Engineering teams running local LLMs and multi-agent workflows evaluate three primary architectures for handling large document collections:

• Raw Prompt Stuffing: Pasting entire source documents into prompt templates. While straightforward, this approach consumes substantial KV cache memory, triggers CPU offload bottlenecks, risks silent truncation, and causes severe response delays.
• Self-Hosted Local Vector Databases: Operating independent vector databases like Chroma, Qdrant, or Milvus. This pattern provides retrieval-augmented generation but forces teams to maintain chunking scripts, manage embedding model runtimes, handle index synchronization, and build custom query routing.
• Intelligent Cloud Workspaces via Model Context Protocol (MCP): Storing files in organized cloud workspaces that index content automatically and expose standardized search tools to local models over Streamable HTTP.

Fastio workspaces provide an intelligent workspace layer specifically built for teams using storage for agents. Instead of attaching large files to chat prompts or overloading local VRAM with extended token contexts, documents are stored in shared, persistent workspaces:

• Automated Indexing with Intelligence Mode: Once Intelligence Mode is enabled on a workspace, all uploaded files are automatically indexed for hybrid search, combining full-text keyword indexing, semantic meaning, and metadata retrieval without requiring external vector databases.
• Streamable HTTP MCP Integration: Local models, coding assistants, and agent runtimes connect to Fastio through remote Model Context Protocol endpoints. Clients connect with OAuth or scoped tokens at https://mcp.fast.io/mcp/tools (for general assistants) or https://mcp.fast.io/mcp/code (for coding agents). Detailed connection procedures and endpoint configurations are available in the Fastio MCP documentation.
• Multi-Source Synchronization: Cloud Sync connects Google Drive, Dropbox, Box and OneDrive folders to a workspace, one-way or two-way, on a schedule or on demand. Reference documents update in the workspace automatically without manual file transfers.
• Structured Data Extraction: For workflows requiring tabular analysis across contracts, financial reports, or technical specifications, Metadata Views extract typed schema fields directly from unstructured documents without manual data entry.

In this hybrid architecture, an agent running locally on Ollama maintains a lean context length of 4,096 to 8,192 tokens. When the user asks a question about an extensive policy manual or codebase, the agent executes an MCP search call against the Fastio workspace. The workspace returns the exact 300-token relevant passage with source citations.

The local model generates its response using full GPU acceleration, avoiding VRAM exhaustion, eliminating silent truncation, and preserving system responsiveness.