Ollama Context Window: Default Limits, num_ctx Tuning, and MCP Search

By default, Ollama uses a context window size of 4096 tokens across modern model instances. While older releases defaulted to 2,048 tokens, current builds allocate a 4,096-token buffer unless explicitly overridden at runtime or configured in a custom model definition.

In natural language processing, a token represents roughly four characters of standard English text. In programming code, however, token density is much higher. Syntax elements such as brackets, indentation spaces, camelCase variable names, punctuation, and operator symbols are tokenized individually. A source file containing 1,000 lines of Python or TypeScript code can easily consume 3,000 to 4,000 tokens on its own. When you add system prompt instructions, agent role definitions, and conversation turn formatting, the remaining budget for user queries and file context diminishes quickly.

When an input prompt exceeds the allocated context window, Ollama does not abort execution or display an error message in the terminal. Instead, the runtime applies silent truncation. To keep the request within the configured token budget, the inference engine drops the earliest conversation turns or truncates the prompt prefix. Generation proceeds normally, but the model loses access to the discarded context. Developers often mistake this silent truncation for model degradation or hallucinations. A model that suddenly forgets instructions defined in the system prompt, invents non-existent variable names, or repeats previously generated code is frequently suffering from silent context truncation rather than inherent reasoning failure.

A common source of confusion is the distinction between a model's architectural context window and Ollama's runtime context setting. Modern open-weight foundation models often advertise context capabilities of 32,768, 65,536, or 128,000 tokens in their model cards. These figures represent the maximum sequence length the model was trained or fine-tuned to evaluate. However, Ollama does not automatically configure that full context length when instantiating a model. The runtime parameter num_ctx defines the actual memory buffer allocated in RAM and video memory (VRAM). Regardless of whether the underlying architecture supports extended sequence lengths, Ollama caps the active session at 4,096 tokens unless instructed otherwise.

To inspect the active context allocation and memory distribution of running models, Ollama provides the ollama ps command. When a model is loaded into memory during generation, executing ollama ps in your terminal displays the active process table:

ollama ps

The resulting output details the model name, process identifier, size in memory, processor allocation, and idle expiration window. The processor column reveals whether the model is running entirely on the GPU, entirely in system memory, or split across both processing tiers. If your model is running with a split allocation, increasing the context window will push additional memory requirements onto system RAM, slowing generation speeds.

Transformer architectures rely on self-attention mechanisms to evaluate relationships between tokens in a sequence. During inference, the engine caches intermediate key and value vectors for every processed token across all attention heads and layers. This structure, known as the Key-Value (KV) cache, persists in memory throughout the generation process to prevent redundant matrix recalculations for earlier tokens.

While model weights occupy a fixed amount of memory once loaded, the KV cache grows in direct proportion to sequence length and batch size. As you increase num_ctx, the memory reserved for key-value projections expands. On an 8-billion parameter model quantized to 4-bit precision, model weights consume approximately 5 gigabytes of memory. At a baseline 4,096-token context window, the KV cache demands a modest memory footprint. Expanding num_ctx to 32,768 or 65,536 tokens, however, inflates the KV cache requirements by several gigabytes. On consumer graphics cards equipped with limited video memory, this additional allocation quickly consumes all remaining VRAM.

When total memory demand exceeds available physical VRAM, one of two outcomes occurs depending on your hardware configuration and Ollama settings.

First, if the runtime is configured to run strictly on the graphics device, the operating system triggers a CUDA out-of-memory (OOM) error, terminating the inference process immediately.

Second, if Ollama detects insufficient VRAM during model loading, it offloads layers of the model or fragments of the KV cache to system RAM. While offloading avoids an outright crash, it introduces severe memory bandwidth bottlenecks. Graphics processors achieve memory bandwidths exceeding hundreds of gigabytes per second, whereas standard system memory operates at a fraction of that throughput. When layers are split across system RAM and PCIe buses, generation throughput collapses from dozens of tokens per second to single-digit tokens per second.

Beyond hardware memory limits, inflating the context window introduces cognitive degradation known as attention dilution. When you pack hundreds of pages of documentation, boilerplate code, and background logs into an oversized prompt, the model must distribute its attention weights across a sprawling token space. Research across modern language models shows that retrieval accuracy deteriorates when critical information is placed in the middle of long contexts. Models frequently miss relevant details buried in conversational filler or duplicate text.

Prompt processing latency also scales with the number of input tokens. Before the model can generate its first output token, it must compute self-attention across the entire input sequence. Submitting a prompt containing 25,000 tokens forces the local GPU to execute extensive matrix multiplications during the prefill phase. On local hardware, this prefill stage can introduce latency pauses of ten to thirty seconds before generation begins. For interactive coding assistants, terminal agents, and conversational workflows, this delay impairs usability.

Search Large Document Collections Without Exhausting Local VRAM

Stop forcing massive file corpuses into your local Ollama context window. Index your documents in an intelligent Fast.io workspace, connect your assistant through the remote MCP server, and retrieve exact context chunks on demand. Every organization starts with a 14-day free trial.

Start 14-Day Trial

When your hardware permits and your workload genuinely demands an expanded context window, you can tune num_ctx using several methods. For moderate context expansions, such as moving from 4,096 tokens to 8,192 or 16,384 tokens on cards with sufficient VRAM headroom, Ollama provides declarative configuration options.

Method 1: Creating a Custom Model via Modelfile

The Modelfile approach creates a persistent, reusable model configuration so you do not need to specify context flags on every invocation. Inspect your base model template first to review default parameters:

ollama show --modelfile llama3.2

Create a new text file named Modelfile in your working directory. Use the FROM directive to specify the base image and the PARAMETER num_ctx directive to define your target context length:

FROM llama3.2
PARAMETER num_ctx 8192
PARAMETER temperature 0.7
SYSTEM You are a focused engineering assistant.

Compile the new model definition using the ollama create command:

ollama create llama3.2-8k -f ./Modelfile

Run your newly configured model:

ollama run llama3.2-8k

Any session launched with llama3.2-8k will automatically allocate an 8,192-token context buffer.

Method 2: Supplying num_ctx in API Requests

If you integrate Ollama into external applications, scripts, or local agent runners, you can configure context size dynamically on a per-request basis. Both the /api/generate and /api/chat endpoints accept an options dictionary containing runtime parameters:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Explain how key-value caching affects GPU memory consumption.",
  "options": {
    "num_ctx": 8192
  },
  "stream": false
}'

Official client libraries in Python, TypeScript, and Go pass these same options, allowing your code to set a smaller context window for brief conversational turns and expand it only for tasks requiring larger prompt inputs.

Method 3: Setting Global Environment Variables and CLI Parameters

To adjust the baseline context window across all models served by an Ollama instance without rebuilding individual Modelfiles, set the OLLAMA_CONTEXT_LENGTH environment variable before starting the background daemon:

OLLAMA_CONTEXT_LENGTH=8192 ollama serve

For ad-hoc experiments within an active interactive terminal session, use the /set parameter command:

/set parameter num_ctx 8192

Context Window Sizing and Tradeoffs

Attempting to solve document scale by continually inflating num_ctx is an architectural antipattern for local inference. Instead of forcing an entire documentation archive, API specification collection, or codebase into a local prompt, modern agent architectures separate storage from inference. You keep the local model running with a lean, fast 4,096 or 8,192 token window, while storing documents in an external retrieval system.

By moving reference material into Fast.io workspaces, your documents become queryable without consuming GPU video memory. You can populate a workspace by uploading files directly, or by syncing files from cloud services including Dropbox, Box, and OneDrive (with Google Drive imports available today and sync coming soon).

Once uploaded, enabling Intelligence Mode activates background indexing. Fast.io processes incoming documents, generating a hybrid index that combines full-text keyword search with semantic vector embeddings. Documents become searchable by exact syntax, keywords, and conceptual meaning, without requiring you to configure or maintain an external vector database.

To connect your local assistant or agent runner to this knowledge store, Fast.io provides a remote Model Context Protocol (MCP) server over Streamable HTTP at https://mcp.fast.io/mcp (with legacy SSE available at https://mcp.fast.io/sse). The MCP server exposes a consolidated toolset that allows AI assistants to search workspaces, read document excerpts, inspect metadata, and retrieve structured context on demand.

When using an MCP-compatible agent or developer client (such as Claude Code, Cline, or custom agent runners interfacing with Ollama), configure the remote MCP endpoint in your client settings:

{
  "mcpServers": {
    "fastio": {
      "url": "https://mcp.fast.io/mcp"
    }
  }
}

Developers building custom agent applications can review the developer storage documentation for detailed authentication and tooling patterns.

The retrieval query loop proceeds as follows:

1. The user asks a detailed question: "What are the authentication requirements for our webhook receiver service?"
2. Rather than relying on static prompt context, the assistant invokes the Fast.io MCP search tool, passing the query to the remote server.
3. Fast.io executes a hybrid search across the workspace, retrieving only the specific paragraphs, code blocks, and markdown sections that answer the query, alongside exact source citations.
4. The assistant injects these targeted excerpts into the active Ollama prompt.
5. Ollama processes the response using a compact prompt context, delivering an accurate, cited answer in fractions of a second.

This approach does not raise Ollama's physical context window. The local engine still operates within its configured num_ctx ceiling. The advantage is that your assistant can access thousands of files and gigabytes of reference material without loading those files into local GPU VRAM.

Balancing local model inference speed with deep document access requires choosing the right mechanism for the task. The table below outlines the core architectural tradeoffs between inflating local num_ctx and relying on external workspace retrieval via MCP:

Active Context Hygiene Techniques

To keep local Ollama sessions responsive, apply these practical context management habits:

1. Keep system prompts concise. Avoid bloated system instructions that consume hundreds of tokens declaring obvious behavioral rules. Define role constraints directly and specifically.
2. Clear conversation buffers between tasks. In extended interactive sessions, accumulated chat turns consume context capacity. Clear the history regularly using /clear in the Ollama CLI or restarting agent threads when switching tasks.
3. Scope retrieval queries. When searching external workspaces via MCP, use folder path scopes and file type filters to narrow the candidate pool, ensuring the model receives only relevant content.
4. Select appropriate precision formats. If you must run moderate context expansions (such as 8,192 tokens) on a card with tight memory, choose 4-bit or 5-bit quantized models to preserve remaining VRAM for the KV cache.

Shared Team Context and Governance

Storing reference documentation on individual developer laptops leads to fragmented context and stale file versions. When an engineer updates an API specification or troubleshooting runbook locally, other team members and local agents continue referencing outdated copies.

In a shared Fast.io workspace, every file maintains per-file version history and an append-only audit log. Granular permissions at organization, workspace, folder, and file levels allow engineering teams to control exactly which repositories and documents an agent can inspect. When an assistant generates updated specifications, summaries, or meeting briefs, it can save them directly back to the workspace, creating an auditable record that human colleagues can review and verify.