# LangChain Rate Limits: Handling 429 Errors, Token Buckets, and Lean MCP Retrieval

LangChain rate limits occur when application chains exceed upstream provider quotas for requests per minute or tokens per minute. While InMemoryRateLimiter and exponential backoff control request frequency, document-heavy workflows trigger HTTP 429 errors by repeatedly sending oversized payloads. Offloading reference documents to an indexed MCP workspace provides targeted chunk retrieval, preventing token quota exhaustion.

Source: https://fast.io/resources/langchain-rate-limit/
Author: [Tom Langridge](https://fast.io/authors/tom-langridge/)
Last reviewed: 2026-10-29

## What Causes LangChain Rate Limits: Requests Per Minute Versus Tokens Per Minute

A LangChain agent processing multi-page documents will reliably crash against upstream API rate limits within minutes, and adding sleep delays between tool calls does nothing to fix it. The failure happens because LangChain chains repeatedly re-transmit entire document payloads across reasoning loops, exhausting the provider's per-minute token quota while total request counts remain well below allowable thresholds.

A LangChain rate limit is an API throttling error (typically HTTP 429) encountered when concurrent chains, tools, or agents exceed upstream LLM provider quotas for requests per minute (RPM) or tokens per minute (TPM). Understanding why these throttles trigger requires separating the raw count of HTTP transactions from the cumulative token volume carried within those transactions.

To resolve rate limits in production LangChain applications, engineers follow a three-part operational strategy:

1. Configure client-side pacing using LangChain `InMemoryRateLimiter` to regulate request frequency across chain invocations.

2. Implement reactive exponential backoff retries using `.with_retry()` or `max_retries` to absorb transient upstream throttling.

3. Offload static document context to an indexed MCP workspace, replacing full-file transmissions with targeted chunk retrieval.

### Requests Per Minute Versus Tokens Per Minute Ceilings

When developers inspect HTTP 429 responses, the error message indicates whether the breach involves request frequency or token volume. Upstream model providers enforce two distinct quotas simultaneously:

* **Requests Per Minute (RPM)**: Limits the raw count of HTTP POST requests dispatched to the API gateway within a rolling 60-second window. RPM limits protect provider infrastructure from connection floods and concurrent execution exhaustion.

* **Tokens Per Minute (TPM)**: Restricts the cumulative volume of input prompt tokens, system message tokens, and generated completion tokens processed across that same 60-second window. TPM limits protect compute resources during intensive inference runs.

In document-heavy agent workflows, teams encounter tokens-per-minute ceilings far more frequently than requests-per-minute limits. A chain that iterates multiple times over a dense reference document consumes prompt tokens rapidly. On entry tiers, a single workflow can exhaust the entire minute allocation even though the application generated only a handful of individual requests.

### Document Payloads and Context Window Ceilings Across Providers

API limits interact directly with file upload boundaries across AI platforms. Developers often prototype agent workflows by uploading entire project files directly into model interfaces before migrating the logic to LangChain.

In Anthropic's documented Claude mechanics, a chat conversation accepts up to 20 files at up to 500MB each. In Claude Projects, files can be up to 30MB each with an unlimited file count, provided the total content fits within Claude's context window. Claude Projects enforces no fixed file-count cap, making the context window the practical ceiling for large-corpus reasoning. Reaching that context ceiling forces developers to migrate from chat interfaces to programmatic chains.

The table below outlines rate limits, token quotas, and file boundaries across prominent foundation model providers and interfaces:

| Platform or Provider | Request Limit (RPM) | Token Limit (TPM) | File or Attachment Boundary | Source and Verification Date |
|---|---|---|---|---|
| OpenAI API (Tier 1) | 500 RPM | 200,000 TPM | Model context window (up to 128,000 tokens) | OpenAI API Documentation (checked 2026-10-29) |
| OpenAI API (Tier 2) | 5,000 RPM | 2,000,000 TPM | Model context window (up to 128,000 tokens) | OpenAI API Documentation (checked 2026-10-29) |
| Anthropic Claude Chat | Model tier RPM | Model tier TPM | 20 files at up to 500 MB per file | Anthropic Support (checked 2026-09-14) |
| Anthropic Claude Projects | Model tier RPM | Model tier TPM | 30 MB per file (unlimited count up to context window) | Anthropic Support (checked 2026-09-14) |

When an application moves from interactive prototypes to automated LangChain pipelines, sending complete files inside prompt strings rapidly triggers upstream throttling.

## How to Configure InMemoryRateLimiter for Client-Side Request Pacing

Proactive rate limiting controls the frequency of outgoing requests before they leave your application server. Rather than waiting for an HTTP 429 response from OpenAI or Anthropic, LangChain provides a native client-side pacing tool: `InMemoryRateLimiter`.

Introduced in `langchain-core` 0.2, `InMemoryRateLimiter` attaches directly to chat model instances. The component tracks execution velocity locally, blocking outgoing invocations until capacity becomes available. This proactive smoothing prevents sudden bursts of parallel tasks from tripping upstream gateway monitors. By smoothing out traffic peaks across a continuous timeline, client applications preserve upstream stability and eliminate unnecessary rejection cycles in batch pipelines and evaluation jobs. Implementing client-side pacing ensures your application behaves predictably under unpredictable spikes in workload demand.

### The Token Bucket Algorithm: requests_per_second and max_bucket_size

The `InMemoryRateLimiter` class implements a classic token bucket algorithm. In this algorithm, an internal bucket continuously accumulates authorization tokens at a fixed rate specified by `requests_per_second`.

The behavior of the token bucket depends on three primary configuration parameters:

* `requests_per_second` (float): Dictates the steady-state replenishment velocity. A setting of 2.0 adds two authorization tokens per second, allowing a steady throughput of 120 requests per minute. A setting of 0.5 allows one request every two seconds.

* `max_bucket_size` (int): Sets the maximum number of authorization tokens the bucket can hold simultaneously. This controls allowable burst capacity. If an application remains idle for several seconds, the bucket fills up to `max_bucket_size`, permitting an immediate burst of concurrent requests before throttling begins.

* `check_every_n_seconds` (float): Specifies how frequently the synchronization loop wakes up to check whether an authorization token has become available in the bucket. A value of 0.1 checks every 100 milliseconds.

In this rate-limiting context, an authorization token represents permission to dispatch a single HTTP call. It is completely distinct from the language model tokens used to measure prompt and completion sizes.

### Attaching InMemoryRateLimiter to Chat Models

LangChain chat models natively accept a `rate_limiter` parameter during initialization. Attaching `InMemoryRateLimiter` requires importing the class from `langchain_core.rate_limiters` and passing the configured instance into the chat model constructor.

The following Python example demonstrates configuring a rate limiter for `ChatOpenAI`:

```python
from langchain_core.rate_limiters import InMemoryRateLimiter
from langchain_openai import ChatOpenAI

rate_limiter = InMemoryRateLimiter(
    requests_per_second=1.0,
    check_every_n_seconds=0.1,
    max_bucket_size=5,
)

model = ChatOpenAI(
    model="gpt-4o",
    rate_limiter=rate_limiter,
)

response = model.invoke("Summarize the quarterly incident report.")
print(response.content)
```

When invoking `model.invoke()`, `model.batch()`, or `model.ainvoke()`, the chat model queries the rate limiter before sending the HTTP request. If the bucket holds an authorization token, execution proceeds immediately. If the bucket is empty, the model pauses execution, sleeping in intervals defined by `check_every_n_seconds` until replenishment occurs.

`InMemoryRateLimiter` operates in memory within a single Python runtime. When running multi-threaded worker pools in a single application process, threads share the same bucket instance safely. However, if your architecture distributes tasks across independent operating system processes, such as multiple Celery workers or containerized Kubernetes replicas, each process maintains an independent rate limiter. Distributed environments require coordinating global rate limits at the API gateway or through external stores such as Redis.

## Implementing Reactive Exponential Backoff Retries in LangChain

Client-side rate limiters smooth traffic velocity, but they cannot anticipate upstream server conditions. An external foundation model provider may reduce throughput temporarily during infrastructure maintenance, or concurrent calls from other departments may deplete your organization's shared tier allocation. When an unexpected HTTP 429 status code arrives, reactive retry handling ensures your application recovers automatically rather than terminating midway through an operational workflow.

LangChain provides built-in mechanisms to handle transient throttling without requiring custom loop boilerplate or external error-handling wrappers. Combining proactive token bucket pacing with reactive exponential backoff allows applications to absorb expected throughput ceilings and unforeseen upstream congestion. While the rate limiter prevents self-inflicted gateway spikes, the retry layer provides fault tolerance when upstream services temporarily downgrade throughput or enforce sudden traffic smoothing.

### Adding Retries with with_retry and max_retries

LangChain ChatModel instances implement the Runnable interface, exposing the `.with_retry()` method. This method wraps model invocations in exponential backoff logic that intercepts API connection errors and rate-limiting responses.

Developers configure retry policies using standard parameters:

```python
from langchain_core.rate_limiters import InMemoryRateLimiter
from langchain_openai import ChatOpenAI

rate_limiter = InMemoryRateLimiter(
    requests_per_second=0.5,
    check_every_n_seconds=0.1,
    max_bucket_size=3,
)

base_model = ChatOpenAI(
    model="gpt-4o",
    rate_limiter=rate_limiter,
)

model_with_retry = base_model.with_retry(
    stop_after_attempt=5,
    wait_exponential_jitter=True,
)
```

The `stop_after_attempt` parameter specifies the maximum number of dispatch attempts before re-raising the exception to the calling function. In evaluation pipelines and batch data processing, setting `stop_after_attempt=5` or `stop_after_attempt=6` provides sufficient resilience to absorb rolling 60-second quota resets.

Alternatively, model constructors accept a `max_retries` integer parameter. Setting `ChatOpenAI(max_retries=5)` configures the underlying SDK HTTP client to retry failed requests internally. Using `.with_retry()` is generally preferred in LangChain architectures because it applies consistently across arbitrary Runnable components, including tool nodes, output parsers, and custom chain sequences.

### Handling Jitter and Differentiating Transient 429 Errors

Implementing exponential backoff requires careful attention to backoff intervals. When multiple worker threads hit an upstream rate limit simultaneously, naive retries that wait a fixed duration cause synchronized retry bursts. This phenomenon, known as the thundering herd problem, repeatedly trips the provider's rate monitor.

Enabling `wait_exponential_jitter=True` introduces randomized variance to each backoff interval. The wait duration expands exponentially with each failed attempt, while the random jitter disperses retries across time, allowing the provider's token bucket to replenish smoothly.

Production error handlers must also distinguish between two fundamentally different types of HTTP 429 responses:

1. **Transient Rate Limit Throttling**: The provider's rolling RPM or TPM quota has been momentarily exceeded. The error payload includes headers such as `retry-after` indicating how many seconds remain until capacity replenishes. Exponential backoff successfully recovers from this condition once the time window elapses.

2. **Hard Account Quota Exhaustion**: The organization has depleted its monthly credit balance, reached its hard monthly spend ceiling, or breached maximum tier limits. Retrying this request will fail repeatedly regardless of backoff duration.

Automated pipelines should parse upstream error messages and inspect HTTP response headers. When error text indicates hard quota depletion or unverified billing status, the application should halt execution immediately, notify administrators, and avoid burning compute cycles on futile retries.

## Offloading Document Payloads to Fastio MCP Workspaces

While client-side rate limiters and exponential retries control request frequency, they do not resolve the primary cause of rate limiting in document-heavy agent applications: payload bloating. In multi-turn agent workflows, developers frequently discover that their systems breach tokens-per-minute limits while request counts remain negligible.

Traditional approaches attempt to circumvent token exhaustion by introducing arbitrary sleep intervals or batching jobs during off-peak hours. However, pausing an agent between steps does not reduce the token mass transmitted to the provider. The permanent solution requires decoupling document storage from agent context windows.

Fastio provides a cloud workspace platform designed for agentic teams where files remain persistent, structured, and accessible through standardized Model Context Protocol (MCP) tooling. Rather than re-transmitting entire documents across reasoning iterations, agents query indexed workspaces on demand, retrieving only the specific text passages required for the current execution step.

### Why Repeated File Transmissions Exhaust Tokens Per Minute

In autonomous agent architectures, such as ReAct loops and multi-step LangGraph workflows, the language model repeatedly inspects conversational context to select tools, evaluate intermediate outputs, and synthesize final answers.

When developers attach complete documents directly to the initial prompt payload, that entire volume of tokens is re-transmitted on every subsequent loop:

* If an agent references a single forty-page contract containing twenty-five thousand tokens, the initial invocation consumes twenty-five thousand input tokens.
* If the agent invokes a tool and receives an observation, the second reasoning step submits the conversation history, the tool output, and the original twenty-five thousand token contract.
* Over five reasoning cycles, that single user task consumes over one hundred thousand input tokens from your organization's per-minute allowance.

On standard foundation model tiers with a quota of 200,000 TPM, two concurrent agent executions will saturate the allocation within seconds. The application crashes with an HTTP 429 error despite dispatching fewer than ten total HTTP requests. Adding delays between iterations does not solve the underlying problem, because each individual iteration re-submits the full payload.

### Connecting LangChain Agents to Fastio Over Streamable HTTP

Fastio eliminates prompt payload saturation by replacing raw file attachments with a hosted, remote MCP server. Workspaces can be organized per client or project, with files uploaded directly or connected through Cloud Sync. Cloud Sync connects Google Drive, Dropbox, Box, and OneDrive folders to a workspace, one-way or two-way, on a schedule or on demand.

When [workspace intelligence](/product/ai/) is enabled, documents are automatically indexed on arrival for hybrid search, combining full-text indexing, semantic search, and search-by-metadata-value. No separate vector database infrastructure, embedding pipeline, or chunking script is required.

LangChain agents connect to Fastio through the remote MCP server at `https://mcp.fast.io/mcp/tools` (or `https://mcp.fast.io/mcp/code` for coding agents) over Streamable HTTP. The following example demonstrates connecting a LangChain agent to an indexed Fastio workspace using `langchain-mcp-adapters`:

```python
import os
from langchain_mcp_adapters.client import MultiServerMCPClient
from langchain_openai import ChatOpenAI
from langchain_core.rate_limiters import InMemoryRateLimiter

rate_limiter = InMemoryRateLimiter(
    requests_per_second=1.0,
    check_every_n_seconds=0.1,
    max_bucket_size=5,
)

model = ChatOpenAI(
    model="gpt-4o",
    rate_limiter=rate_limiter,
)

async def initialize_workspace_agent():
    """Connect to Fastio remote MCP server over Streamable HTTP."""
    client = MultiServerMCPClient({
        "fastio": {
            "url": "https://mcp.fast.io/mcp/tools",
            "transport": "http",
            "headers": {
                "Authorization": f"Bearer {os.environ['FASTIO_API_KEY']}"
            }
        }
    })
    tools = await client.get_tools()
    return tools
```

For client connection specifics, consult the [Fastio MCP setup guide](https://mcp.fast.io/docs) or the [storage for agents](/storage-for-agents/) documentation.

### Targeted Passage Retrieval Versus Full File Ingestion

Once connected via MCP, the LangChain agent does not load whole documents into prompt memory. Instead, the agent invokes Fastio search tools to find relevant excerpts dynamically.

When asked a question about a specific policy clause or financial metric, the agent submits a semantic query to the workspace. Fastio executes the search across indexed files and returns only the top matching passages, complete with document citations.

This architectural shift transforms prompt token economics:

* Rather than transmitting twenty-five thousand tokens of raw document text on every turn, the agent receives two or three concise passages totaling a few hundred tokens.
* Each subsequent reasoning step carries only the targeted excerpts, keeping total prompt size below three thousand tokens throughout multi-turn execution loops.
* Prompt token volume drops drastically compared to raw document loading, eliminating the primary driver of tokens-per-minute throttling.

Workspaces provide team-level infrastructure for human and agent collaboration:

* **Per-File Version History**: Every document modification maintains complete version records, allowing teams to review agent outputs and restore previous revisions.
* **Detailed Activity Log**: Organizations maintain an auditable record of changes across workspaces, tracking uploads, edits, moves, deletions, and permission updates.
* **Metadata Views**: Teams can extract structured data from unstructured files using Metadata Views ([document data extraction](/product/document-data-extraction/)), converting documents into typed databases without custom OCR scripts.
* **Role-Based Permissions**: Access controls are configured per organization, workspace, and share, ensuring agents only access files within authorized boundaries.

## Production Architecture: Combining Rate Limiters, Retries, and Lean Retrieval

Building enterprise-grade LangChain applications requires a defense-in-depth approach to API quota management. Relying on a single mechanism leaves applications vulnerable to traffic surges, upstream provider degradation, or prompt payload bloating. When any single safeguard fails, downstream users experience interrupted sessions, incomplete tool traces, or unexpected execution failures.

A resilient production architecture coordinates three distinct defensive layers:

1. **Client-Side Request Pacing (`InMemoryRateLimiter`)**: Governs request frequency (RPM), preventing worker pools from exceeding provider connection thresholds.
2. **Reactive Exponential Backoff (`.with_retry()`)**: Intercepts transient HTTP 429 status codes, applying randomized jitter to absorb rolling quota resets.
3. **External Workspace Context (Fastio MCP)**: Decouples large reference documents from conversational prompts, keeping per-minute token consumption (TPM) minimal through targeted passage retrieval.

Integrating these controls into standard agent patterns ensures that autonomous reasoning loops remain stable across multi-step execution graphs and peak workload demands.

### Complete Implementation for Resilient Agent Pipelines

The following Python script illustrates an end-to-end implementation uniting client-side pacing, exponential backoff, and MCP workspace retrieval into a single operational workflow:

```python
import os
import asyncio
from langchain_core.rate_limiters import InMemoryRateLimiter
from langchain_openai import ChatOpenAI
from langchain_mcp_adapters.client import MultiServerMCPClient

async def run_resilient_agent(query: str):
    # Step 1: Configure proactive token bucket rate limiter
    rate_limiter = InMemoryRateLimiter(
        requests_per_second=1.0,
        check_every_n_seconds=0.1,
        max_bucket_size=5,
    )
    # Step 2: Initialize chat model with rate limiter and exponential retries
    base_model = ChatOpenAI(
        model="gpt-4o",
        rate_limiter=rate_limiter,
    )
    model = base_model.with_retry(
        stop_after_attempt=5,
        wait_exponential_jitter=True,
    )
    # Step 3: Connect to Fastio MCP workspace for lean retrieval
    client = MultiServerMCPClient({
        "fastio": {
            "url": "https://mcp.fast.io/mcp/tools",
            "transport": "http",
            "headers": {
                "Authorization": f"Bearer {os.environ['FASTIO_API_KEY']}"
            }
        }
    })
    tools = await client.get_tools()
    # Bind tools to the resilient model
    model_with_tools = model.bind_tools(tools)
    response = await model_with_tools.ainvoke(query)
    return response

if __name__ == "__main__":
    result = asyncio.run(run_resilient_agent("Find the termination liability clause in the vendor agreements."))
    print(result)
```

In this architecture, outgoing calls are paced at a predictable velocity. If an upstream service blip occurs, the runnable automatically pauses and retries with jitter. When the agent needs contextual reference data, it queries the workspace through Fastio MCP tools instead of loading full files into the prompt.

### Token Budget Allocation and Operational Guardrails

To prevent rate limits before they occur, engineering teams should establish structured token budgets across application components. For a model with a 128,000-token context window, allocating token headroom prevents runaway cost and quota depletion:

* **System Persona and Instructions**: Allocate 1,500 to 2,500 tokens for system prompts, formatting rules, and guardrails.
* **Tool Schemas and Specifications**: Allocate 2,000 to 4,000 tokens for tool definitions and parameter signatures.
* **Retrieved Knowledge Excerpts**: Constrain retrieved context to 3,000 to 6,000 tokens, representing the top matching passages from the workspace.
* **Conversation History**: Allocate 4,000 to 8,000 tokens, trimming older turns dynamically with `trim_messages` to maintain lean prompts.
* **Completion Generation Reserve**: Reserve 2,048 to 4,096 tokens for generated output via `max_tokens` or `max_completion_tokens`.

Monitoring response headers provides early warning of quota saturation. Upstream providers return informative headers on every API response:

* `x-ratelimit-remaining-requests`: Number of remaining requests allowed within the current minute window.
* `x-ratelimit-remaining-tokens`: Number of remaining tokens allowed within the current minute window.
* `x-ratelimit-reset-requests`: Time remaining until the request counter resets.
* `x-ratelimit-reset-tokens`: Time remaining until the token bucket replenishes.

Tracking these headers in application monitoring systems allows engineering teams to detect capacity pressure well before users encounter HTTP 429 exceptions.

## Frequently asked questions

### How do I fix rate limit exceeded in LangChain?

To fix rate limit exceeded errors in LangChain, implement a three-part strategy: configure InMemoryRateLimiter to regulate client-side request velocity, attach .with_retry() to chat models for exponential backoff retries with jitter, and offload large documents to an indexed MCP workspace like Fastio to prevent tokens-per-minute quota exhaustion.

### What is InMemoryRateLimiter in LangChain?

InMemoryRateLimiter is a client-side rate-limiting class in langchain-core that implements a token bucket algorithm. It regulates the frequency of outgoing API requests by blocking invocations until capacity is available, controlled by parameters including requests_per_second and max_bucket_size.

### How do you handle 429 Too Many Requests in LangChain?

Handle HTTP 429 errors by wrapping LangChain models with .with_retry(stop_after_attempt=5, wait_exponential_jitter=True) or setting max_retries on the model constructor. Exponential backoff absorbs temporary provider throttling, while randomized jitter prevents concurrent worker threads from causing synchronized retry spikes.

### What is the difference between RPM and TPM rate limits in LangChain?

RPM (requests per minute) measures the raw count of HTTP calls sent to an API provider within a rolling 60-second window. TPM (tokens per minute) restricts the cumulative volume of prompt and completion tokens processed in that same period. Document-heavy agent loops typically exhaust TPM limits long before hitting RPM thresholds.

### How does offloading documents to MCP prevent LangChain token exhaustion?

In standard agent loops, re-transmitting multi-page documents on every iteration quickly exhausts tokens-per-minute quotas. By storing files in a Fastio workspace with Intelligence Mode enabled, LangChain agents query documents over remote MCP and retrieve only relevant excerpts, reducing prompt payload sizes to a fraction of raw document volume.

### Does LangChain InMemoryRateLimiter share state across multiple worker processes?

No. InMemoryRateLimiter maintains its token bucket state in local memory within a single Python process. If an application runs across multiple operating system processes, Celery workers, or container replicas, each worker maintains an independent rate limiter. Distributed systems require coordinating limits via an API gateway or Redis.

## Sources

- [Anthropic Help Center: Uploading files to Claude](https://support.claude.com/en/articles/8241126-upload-files-to-claude): In Anthropic's documented Claude mechanics, a chat conversation accepts up to 20 files at up to 500MB each.
- [Anthropic Help Center: Uploading files to Claude](https://support.claude.com/en/articles/8241126-upload-files-to-claude): In Claude Projects, files can be up to 30MB each with an unlimited file count, provided the total content fits within Claude's context window.

## About Fast.io

Fast.io provides shared workspaces where people and AI agents work on the same files, with built-in semantic search and citation-backed chat over what they hold. Agents reach it through a remote MCP server, a REST API at https://api.fast.io/current/, and a command line client published on npm as @vividengine/fastio-cli. MCP setup is at https://mcp.fast.io/docs: Claude and most MCP clients connect to https://mcp.fast.io/mcp/tools, ChatGPT to https://mcp.fast.io/mcp/operations, and coding agents to https://mcp.fast.io/mcp/code.