LLM Context Compression for Long-Running AI Agents

An agent that answers one prompt never worries about the context window. An agent that holds a support conversation for three weeks, processes a queue around the clock, or works through a forty-step task does. Conversation history only grows, every model has a hard token limit, and when a session crosses it the provider rejects the call and the run fails mid-task.

The hard limit is only half the problem. Model quality drops well before the window is full. Chroma's context-rot study (2025) evaluated 18 frontier models and found that output quality degrades as input length grows, even on simple tasks. Earlier, Lost in the Middle (Liu et al., 2023) showed that models retrieve information best from the start and end of a prompt, with accuracy dropping sharply when it sits in the middle. A bloated context does not only risk an error. It makes every answer worse.

Why Long-Running Agents Run Out of Context

Three things fill an agent's context window, and none of them shrink on their own:

Truncation, Sliding Windows, and Summarization

There are three common ways to keep an LLM context bounded, with very different failure modes:

Connic uses LLM summarization, with retention rules designed to avoid the failure modes of the other two.

How Connic Compresses Context

The default trigger is the moment things would otherwise break: a provider context-limit error.

When a model call fails, the runner classifies the error. It recognizes context-window errors across providers and explicitly ignores rate-limit and quota errors, so compression only fires on a genuine overflow. It then compresses the request and retries the call once, inside the same run. The user never sees a failed run, and no work is lost. If the retried call still exceeds the window, the run fails with the original error.

What stays, what gets summarized

The most recent messages stay verbatim: keep_recent_messages, 8 by default. The boundary is then adjusted backwards so that a tool result is never separated from the call that produced it, the malformed-request trap that naive windowing falls into.

Everything older is summarized by an LLM with strict instructions: preserve user goals, constraints, decisions, tool names and arguments, tool results, IDs, numbers, errors, open questions, and next steps, and invent nothing. History too long for a single pass is split into 12,000-character chunks, each chunk summarized, and the chunk summaries merged into one continuation summary.

The result is inserted as a single message that tells the model exactly what happened: earlier history was compressed, and the retained recent messages are the source of truth for the current turn. The model continues the task instead of hallucinating a fresh start.

Oversized tool results

Compression also targets the single biggest space consumer directly. When compression runs, oversized tool responses within the retained messages are summarized in place and replaced with a compact record that includes the summary and the original size. A bulky JSON payload becomes a paragraph, and the agent keeps the facts it extracted from it.

Compress before the error: token budgets

Waiting for the provider to complain means one failed call per compression. Set max_prompt_tokens and the runner compresses proactively instead: it records the model-reported prompt token count after every call, and when the last call meets the budget it compresses before sending the next one. No error, no retry, no wasted call. Because Connic is bring-your-own-key, smaller prompts also show up directly as lower input cost on your provider bill.

A cheaper model for summaries

Summaries do not need your best model. Set context_compression.model to run them on a small, fast model while the agent itself stays on its configured one. If you omit it, summaries use the agent model. The retried agent call always uses the agent model either way.

Configuration in YAML

Context compression is part of the agent YAML for any LLM agent. A complete configuration for a session-backed chat agent:

version: "1.0"

name: support-bot
type: llm
model: gemini/gemini-2.5-pro
description: "A support chatbot that remembers conversation history"
system_prompt: |
  You are a helpful support agent. Use the conversation history
  to provide contextual responses.

# Persistent sessions keyed by chat ID
session:
  key: context.chat_id
  ttl: 86400

# Keep long conversations within the model context window
context_compression:
  enabled: true
  # Optional model for compression summaries; defaults to the agent model
  model: gemini/gemini-2.5-flash
  keep_recent_messages: 12
  # Optional early compression based on model-reported prompt usage
  max_prompt_tokens: 100000
  # Optional compaction for stored session history between runs
  session_history:
    interval: 4
    keep_recent_runs: 1

Only the block itself is required. Add context_compression with enabled: true and you get error-triggered compression with the defaults. Everything else is tuning.

Compacting Stored Session History

In-request compression shapes what a single model call sees. Session-backed agents have a second growth problem: the history stored between runs. The session_history block compacts it. Every interval runs, older stored history is replaced with a summary, keeping the most recent keep_recent_runs runs untouched. A session that has been alive for months starts each new run from a compact summary plus recent turns, not from its entire life story.

Every Compression Is a Trace Span

Compression rewrites what your agent sees, so it should never be invisible. Each compression is captured as an OpenTelemetry trace span in the run:

In-request compression appears as a context_compression span, stored-history compaction as history_compression. Both also write log lines into the run. See the observability documentation for how traces work, or read the agent observability guide for the bigger picture.

Getting Started

Adding context compression to an existing LLM agent takes minutes:

• 1.Add a context_compression block to the agent's YAML configuration
• 2.Deploy. Compression activates on the next run, no code changes needed
• 3.Watch the Traces tab for context_compression spans as long sessions hit the window
• 4.Tune keep_recent_messages, add a token budget, or switch summaries to a cheaper model based on what you see

For the full option reference, including sessions and compaction, see the runtime controls documentation. New to Connic? Follow the quickstart guide to deploy your first agent, then give it a memory that does not run out.