# Ollama Context Length: Configuration, VRAM Limits, and Scaling

Ollama context length defines the maximum token capacity allocated in memory for prompt input and completion output, dynamically scaled by hardware VRAM tiers or manually overridden via the OLLAMA_CONTEXT_LENGTH environment variable and num_ctx parameter. Increasing context expands Key-Value cache memory demands, risking GPU out-of-memory errors. Sizing VRAM accurately and offloading large files to external workspaces via MCP keeps local inference fast.

Source: https://fast.io/resources/ollama-context-length/
Author: [Tom Langridge](https://fast.io/authors/tom-langridge/)
Last reviewed: 2026-10-31

## What Are the Default Context Lengths in Ollama Across Hardware Tiers?

Ollama context length defines the maximum token capacity allocated in memory for prompt input and completion output, dynamically scaled by hardware VRAM tiers or manually overridden via the OLLAMA_CONTEXT_LENGTH environment variable and num_ctx parameter. When developers run local models for retrieval tasks, coding assistance, or multi-turn agent conversations, token boundaries dictate whether an interaction succeeds or silently truncates. Understanding how Ollama allocates memory across hardware tiers eliminates unexpected prompt clipping and prevents GPU out-of-memory crashes.

Ollama default context length allocates 4,096 tokens under 24 GiB VRAM, 32,768 tokens from 24 to 48 GiB VRAM, and 256,000 tokens above 48 GiB VRAM. Many legacy guides claim that Ollama strictly caps all models at `2,048` tokens out of the box. That `2,048`-token limit was the historical default inherited from early llama.cpp builds, intended to keep small models functional on low-memory personal computers. Modern Ollama versions dynamically inspect available GPU video memory at server startup and assign active context windows according to available hardware capacity.

Modern open-weights models feature expansive native context architectures designed to ingest hundreds of pages of documentation. Meta Llama `3.1`, Llama `3.2`, and Llama `3.3` support `128,000` tokens natively, Mistral NeMo supports `128,000` tokens, and Qwen `2.5` models range from `32,768` to `131,072` tokens. However, pulling and running these models in Ollama without explicit configuration locks them into Ollama hardware-tier default. If you send a `15,000`-token prompt to Llama `3.1` running on a `16 GB` GPU, Ollama restricts the execution context to `4,096` tokens, discarding the beginning of your prompt without returning an error.

As documented in official Ollama guidance on context length, tasks that require large context, including agents, web search, and coding tools, should be configured to at least 64,000 tokens when hardware allows. Operating an agent at that scale requires knowing both the native capabilities of the model architecture and the exact configuration parameters that override local server defaults.

The following comparison table outlines native model context ceilings alongside Ollama default and recommended agent allocations:

| Model Identifier | Parameter Count | Native Architecture Ceiling | Ollama Default (<24 GiB VRAM) | Recommended Agent Context | Attention Mechanism |
|---|---|---|---|---|---|
| Llama 3.2 | 3B | 128,000 tokens | 4,096 tokens | 32,768 tokens | Grouped-Query Attention (GQA) |
| Llama 3.1 | 8B | 128,000 tokens | 4,096 tokens | 64,000 tokens | Grouped-Query Attention (GQA) |
| Qwen 2.5 Coder | 7B | 131,072 tokens | 4,096 tokens | 64,000 tokens | Grouped-Query Attention (GQA) |
| Mistral NeMo | 12B | 128,000 tokens | 4,096 tokens | 32,768 tokens | Grouped-Query Attention (GQA) |
| Qwen 2.5 Coder | 32B | 131,072 tokens | 4,096 tokens | 32,768 tokens | Grouped-Query Attention (GQA) |
| Llama 3.3 | 70B | 128,000 tokens | 4,096 tokens | 32,768 tokens | Grouped-Query Attention (GQA) |
| DeepSeek-R1-Distill-Qwen | 14B | 64,000 tokens | 4,096 tokens | 32,768 tokens | Grouped-Query Attention (GQA) |

Deploying models for agent workflows requires actively configuring these limits rather than relying on server defaults. When an agent reads file directories or processes complex tool schemas, exceeding the `4,096`-token baseline happens quickly, making manual configuration mandatory.

## How to Configure Ollama Context Length Across Four Environments

Configuring Ollama context length requires choosing the appropriate control point for your operational setup. Developers can enforce limits server-wide, adjust values inside desktop interfaces, modify active terminal sessions, or bake parameters into custom model distributions.

Here are the four standard methods for configuring Ollama context length:

1. **Setting the Server-Wide Environment Variable (`OLLAMA_CONTEXT_LENGTH`):** Define the global context length default across all models loaded by the Ollama server daemon. When launching Ollama directly from a terminal or shell script, prepend the variable:

```bash
OLLAMA_CONTEXT_LENGTH=64000 ollama serve
```

For production Linux deployments managed by systemd, add the environment variable to the systemd override configuration:

```bash
sudo systemctl edit ollama.service
```

Add the variable assignment under the service unit definition:

```text
[Service=unit]
Environment="OLLAMA_CONTEXT_LENGTH=64000"
```

Apply the changes and restart the service:

```bash
sudo systemctl daemon-reload
sudo systemctl restart ollama
```

On macOS installations running the background service, set the environment variable through `launchctl` before launching the application:

```bash
launchctl setenv OLLAMA_CONTEXT_LENGTH 64000
```

When deploying Ollama in containerized environments using Docker or Docker Compose, pass `OLLAMA_CONTEXT_LENGTH` within the environment block of your service configuration:

```yaml
services:
  ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    environment:
      - OLLAMA_CONTEXT_LENGTH=64000
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
```

2. **Adjusting the Desktop Application Slider:** For users running the Ollama desktop application on macOS or Windows, context length can be controlled without terminal commands. Click the Ollama status icon in the operating system menu bar or taskbar, open **Settings**, locate the **Context Length** slider, and adjust the slider to your desired token ceiling. Changes applied through the desktop interface take effect on subsequent model loads.

3. **Setting Session-Level Parameters in the Interactive CLI:** When experimenting with models inside the terminal REPL using `ollama run`, update the context length dynamically for the active session:

```text
>>> /set parameter num_ctx 32768
Set parameter 'num_ctx' to '32768'
```

Any subsequent prompt submitted in that interactive terminal session will use the `32,768`-token window. To persist this configuration as a named model variant without editing text files, run `/save` immediately after setting the parameter:

```text
>>> /save llama3.2-32k
Created new model 'llama3.2-32k'
```

4. **Defining Parameters Permanently in a Modelfile:** Creating a custom model file is the recommended approach for team consistency and automated pipelines. Create a plain text file named `Modelfile`:

```dockerfile
FROM llama3.2:latest
PARAMETER num_ctx 32768
PARAMETER temperature 0.2
```

Build the custom model into your local Ollama library:

```bash
ollama create llama3.2-32k -f Modelfile
```

Launch the newly built model using its custom tag:

```bash
ollama run llama3.2-32k
```

In programmatic integrations, developers can also override context length per request by passing `num_ctx` inside the `options` object of the Ollama REST API. This approach allows client applications to allocate smaller `4,096`-token windows for simple queries while reserving `32,768`-token windows for heavy document analysis:

```bash
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Summarize the technical constraints outlined in the system architecture specification.",
  "options": {
    "num_ctx": 32768
  }
}'
```

Choosing between these methods depends on deployment architecture. Environment variables provide clean global baselines for server instances, while Modelfile definitions provide reliable reproducibility across developer workstations.

## Why Key-Value Cache Memory Determines Real-World VRAM Limits

Increasing Ollama context length does not increase the video memory required to store static model weights. An `8B` parameter model quantized at `4` bits (`Q4_K_M`) requires approximately `4.9 GB` of `VRAM` regardless of whether your prompt contains `10` tokens or `50,000` tokens. Instead, expanded context windows consume memory through the Key-Value (KV) cache.

The Key-Value cache stores intermediate attention representations for every processed token across all transformer layers and attention heads. Without this cache, the model would need to recalculate self-attention across the entire sequence history for every newly generated token, causing generation latency to scale quadratically. The memory required for the KV cache scales linearly with sequence length according to a specific formula:

```text
KV Cache Memory (Bytes) = 2 * Layers * KV_Heads * Head_Dimension * Context_Tokens * Bytes_Per_Element
```

In this calculation:
* The multiplier `2` accounts for separate Key and Value vectors.
* `Layers` represents the total number of transformer blocks in the model architecture.
* `KV_Heads` represents the number of key-value attention heads.
* `Head_Dimension` represents the dimensionality of each attention head (typically 128).
* `Context_Tokens` is the configured `num_ctx` value.
* `Bytes_Per_Element` corresponds to numerical precision, which is 2 bytes for standard 16-bit floating-point (FP16) representations.

Modern architectures implement Grouped-Query Attention (GQA), which shares key-value heads across groups of query heads. In Meta Llama `3.1` `8B`, the architecture contains 32 query heads but only 8 KV heads. This architectural optimization reduces KV cache memory consumption substantially, by a factor of four compared to older Multi-Head Attention designs.

Calculating the KV cache memory for Llama `3.1` `8B` (32 layers, 8 KV heads, head dimension of 128, and FP16 precision) demonstrates how memory scales with token count:

```text
Memory per Token = 2 * 32 * 8 * 128 * 2 = 131,072 bytes (128 KiB per token)
```

Applying this per-token footprint to common context lengths yields the following memory demands:
* **`4,096` tokens:** `128 KiB * 4,096 = 512 MiB (0.5 GiB)`
* **`16,384` tokens:** `128 KiB * 16,384 = 2,048 MiB (2.0 GiB)`
* **`32,768` tokens:** `128 KiB * 32,768 = 4,096 MiB (4.0 GiB)`
* **`64,000` tokens:** `128 KiB * 64,000 = 8,000 MiB (~7.8 GiB)`
* **`128,000` tokens:** `128 KiB * 128,000 = 16,000 MiB (~15.6 GiB)`

The following memory allocation table details total VRAM requirements across model sizes and context lengths, combining static model weights with the active KV cache:

| Target Context Length | 8B Q4_K_M Total VRAM | 14B Q4_K_M Total VRAM | 32B Q4_K_M Total VRAM | 70B Q4_K_M Total VRAM | Minimum Hardware Tier |
|---|---|---|---|---|---|
| 4,096 tokens | 5.5 GiB | 9.8 GiB | 20.2 GiB | 43.5 GiB | RTX 3060 12GB / Apple M-series 16GB |
| 16,384 tokens | 7.0 GiB | 11.5 GiB | 22.8 GiB | 47.8 GiB | RTX 4070 12GB / Apple M-series 24GB |
| 32,768 tokens | 9.0 GiB | 13.8 GiB | 26.2 GiB | 53.5 GiB | RTX 4080 16GB / Apple M-series 36GB |
| 64,000 tokens | 12.8 GiB | 18.2 GiB | 32.5 GiB | 64.0 GiB | RTX 4090 24GB / Apple M-series 48GB |
| 128,000 tokens | 20.5 GiB | 27.0 GiB | 45.0 GiB | 85.0 GiB | Dual RTX 3090 / Apple M-series 96GB+ |

When the combined memory footprint of model weights and KV cache exceeds available GPU VRAM, Ollama attempts to offload remaining layers and memory blocks to system RAM via CPU inference. This offloading introduces immediate performance degradation. Processing prompt evaluation across the system PCIe bus reduces generation throughput from `45` tokens per second to less than `2` tokens per second.

If system memory is also exhausted, the local process terminates with a fatal out-of-memory failure. To mitigate memory pressure without sacrificing token capacity, developers can run quantized KV cache modes (`q8_0` or `q4_0`) in supported builds, cutting KV cache footprint roughly in half with minimal loss in retrieval accuracy.

## How to Detect and Fix Silent Context Truncation in Ollama

The most challenging failure mode in Ollama is silent context truncation. Unlike API errors that fail loudly with explicit HTTP status codes, Ollama processes prompts that exceed `num_ctx` by quietly dropping the earliest tokens from the context window. The model generates a grammatically sound, confident response, but it does so without having read the opening chapters of your document, the earliest messages of your chat history, or critical system instructions.

Detecting and resolving this behavior requires active runtime inspection.

### Verifying Runtime Allocations with ollama ps

To inspect running models and their active context allocations, run the process status command in a separate terminal:

```bash
ollama ps
```

The terminal prints a formatted table displaying active memory allocations:

```text
NAME               ID              SIZE      PROCESSOR        UNTIL              CONTEXT
llama3.2:latest    a80c4f17acd5    4.9 GB    100% GPU         4 minutes from now 4096
```

Key indicators to inspect include:
* **CONTEXT:** Confirms the exact token limit assigned to the model. If you intended to run at `32,768` tokens but the table reports `4096`, the server is running on default configuration.
* **PROCESSOR:** Shows execution placement between GPU and CPU. A status indicating pure GPU execution shows optimal inference throughput. If the column displays a mixed split between GPU and CPU, the active context length has pushed the KV cache beyond available VRAM, causing layer offloading and slower response times.

### Inspecting API Response Metrics

When communicating with Ollama through scripts or agent runtimes, check the operational telemetry included in the JSON response payload:

```json
{
  "model": "llama3.2",
  "created_at": "2026-10-31T09:30:00Z",
  "response": "The architectural review indicates...",
  "done": true,
  "total_duration": 4820194833,
  "load_duration": 1823904,
  "prompt_eval_count": 4096,
  "prompt_eval_duration": 3410294000,
  "eval_count": 284,
  "eval_duration": 1404076000
}
```

The critical metric is `prompt_eval_count`. If you dispatched a prompt containing `12,500` tokens but `prompt_eval_count` returns `4096`, Ollama silently truncated `8,404` tokens. Reviewing this value in automated pipelines provides immediate detection of truncated inputs.

### Context Degradation and Rotary Position Embeddings

Expanding `num_ctx` from `4,096` tokens to `64,000` tokens ensures that text reaches the model, but it does not guarantee equal attention across the entire sequence. Modern transformers rely on Rotary Position Embeddings (RoPE) to encode positional information. While frequency base adjustments (`rope_frequency_base` and `rope_frequency_scale`) permit context window expansion, models experience attention dilution when processing massive contexts.

In long-context evaluations, models frequently exhibit the 'lost in the middle' phenomenon: retrieval accuracy is highest for facts placed at the extreme beginning and end of the prompt, while facts located throughout the middle portion suffer higher omission rates. When configuring large context windows for autonomous agents, relying entirely on extended context windows to parse large document collections introduces latency penalties and retrieval inconsistencies.

## How to Offload Document Collections with MCP to Prevent Context Bloat

Stuffing entire document libraries, technical specifications, and historical chat logs directly into local model context creates severe operational bottlenecks. Even when high-end hardware permits setting `num_ctx` to `64,000` or `128,000` tokens, processing massive prompts degrades Time-to-First-Token latency, consumes excessive power, and increases the likelihood of retrieval hallucinations.

This architectural constraint mirrors file upload limits across major cloud AI platforms. In official Anthropic documentation, a Claude chat conversation accepts up to 20 files at up to 500MB each, while a project accepts files up to 30MB each with no fixed file count limit. As Anthropic notes, project file counts are unlimited, but the total content must fit within Claude's context window. Claude Projects has no fixed file-count cap, so any fixed project file count is false. The practical ceiling on a project is the context window, and reaching it is when developers with a large corpus look for another path.

Engineering teams running local LLMs and multi-agent workflows evaluate three primary architectures for handling large document collections:

* **Raw Prompt Stuffing:** Pasting entire source documents into prompt templates. While straightforward, this approach consumes substantial KV cache memory, triggers CPU offload bottlenecks, risks silent truncation, and causes severe response delays.
* **Self-Hosted Local Vector Databases:** Operating independent vector databases like Chroma, Qdrant, or Milvus. This pattern provides retrieval-augmented generation but forces teams to maintain chunking scripts, manage embedding model runtimes, handle index synchronization, and build custom query routing.
* **Intelligent Cloud Workspaces via Model Context Protocol (MCP):** Storing files in organized cloud workspaces that index content automatically and expose standardized search tools to local models over Streamable HTTP.

[Fastio workspaces](/product/workspaces/) provide an intelligent workspace layer specifically built for teams using [storage for agents](/storage-for-agents/). Instead of attaching large files to chat prompts or overloading local VRAM with extended token contexts, documents are stored in shared, persistent workspaces:

* **Automated Indexing with Intelligence Mode:** Once Intelligence Mode is enabled on a workspace, all uploaded files are automatically indexed for hybrid search, combining full-text keyword indexing, semantic meaning, and metadata retrieval without requiring external vector databases.
* **Streamable HTTP MCP Integration:** Local models, coding assistants, and agent runtimes connect to Fastio through remote Model Context Protocol endpoints. Clients connect with OAuth or scoped tokens at `https://mcp.fast.io/mcp/tools` (for general assistants) or `https://mcp.fast.io/mcp/code` (for coding agents). Detailed connection procedures and endpoint configurations are available in the [Fastio MCP documentation](https://mcp.fast.io/docs).
* **Multi-Source Synchronization:** Cloud Sync connects Google Drive, Dropbox, Box and OneDrive folders to a workspace, one-way or two-way, on a schedule or on demand. Reference documents update in the workspace automatically without manual file transfers.
* **Structured Data Extraction:** For workflows requiring tabular analysis across contracts, financial reports, or technical specifications, [Metadata Views](/product/document-data-extraction/) extract typed schema fields directly from unstructured documents without manual data entry.

In this hybrid architecture, an agent running locally on Ollama maintains a lean context length of `4,096` to `8,192` tokens. When the user asks a question about an extensive policy manual or codebase, the agent executes an MCP search call against the Fastio workspace. The workspace returns the exact `300`-token relevant passage with source citations. 

The local model generates its response using full GPU acceleration, avoiding VRAM exhaustion, eliminating silent truncation, and preserving system responsiveness.

## Frequently asked questions

### What is the default context length in Ollama?

Ollama dynamically allocates default context length based on system GPU video memory: 4,096 tokens on systems with under 24 GiB VRAM, 32,768 tokens on systems with 24 to 48 GiB VRAM, and 256,000 tokens on systems with 48 GiB VRAM or more. Older versions used a static 2,048-token default.

### How do I increase Ollama context length?

You can increase Ollama context length by setting the OLLAMA_CONTEXT_LENGTH environment variable before starting the server, adjusting the Context Length slider in the desktop application settings, using the /set parameter num_ctx command during an interactive CLI session, or declaring PARAMETER num_ctx in a custom Modelfile.

### How much VRAM does 32k context length require in Ollama?

For an `8B` parameter model quantized at `4` bits (such as Llama `3.1` `8B` `Q4_K_M`), static model weights require roughly `4.9 GB` of `VRAM`, while a `32,768`-token Key-Value cache requires approximately `4.0 GB` of `VRAM` under `16-bit` precision. In total, running `32k` context on an `8B` model requires at least `9.0 GB` of dedicated `VRAM`.

### Why does Ollama silently truncate long documents instead of returning an error?

Ollama relies on the underlying llama.cpp inference engine, which handles sequence overflow by shifting the evaluation window and discarding the earliest prompt tokens. This design prevents hard application crashes during continuous chat conversations, but it results in silent prompt truncation if total input tokens exceed the configured num_ctx value.

### Does increasing context length slow down token generation speed?

Increasing context length slightly increases attention computation per token, but the primary latency impact occurs during prompt evaluation (Time-to-First-Token). If the combined memory of model weights and the Key-Value cache exceeds GPU VRAM, Ollama offloads layers to system RAM, which can reduce generation speed from over 40 tokens per second to under 2 tokens per second.

### How does connecting Fastio via MCP prevent Ollama out-of-memory crashes?

Instead of loading entire document collections into local Ollama context, files are housed in a Fastio workspace with Intelligence Mode enabled. The local model queries the workspace over Model Context Protocol using tools described in the Fastio MCP documentation, retrieving only the relevant text snippets needed to answer the query. This keeps local context under 8,192 tokens and prevents GPU VRAM exhaustion.

## Sources

- [Ollama Documentation: Context length](https://docs.ollama.com/context-length): Ollama default context length allocates 4,096 tokens under 24 GiB VRAM, 32,768 tokens from 24 to 48 GiB VRAM, and 256,000 tokens above 48 GiB VRAM, with official guidance recommending at least 64,000 tokens for agents.
- [Anthropic: Claude Help Center](https://support.claude.com/en/articles/8241126-upload-files-to-claude): A Claude chat conversation accepts up to 20 files at up to 500MB each, while a project accepts files up to 30MB each with no fixed file count limit.

## About Fast.io

Fast.io provides shared workspaces where people and AI agents work on the same files, with built-in semantic search and citation-backed chat over what they hold. Agents reach it through a remote MCP server, a REST API at https://api.fast.io/current/, and a command line client published on npm as @vividengine/fastio-cli. MCP setup is at https://mcp.fast.io/docs: Claude and most MCP clients connect to https://mcp.fast.io/mcp/tools, ChatGPT to https://mcp.fast.io/mcp/operations, and coding agents to https://mcp.fast.io/mcp/code.