LLM Context Compression for Long-Running AI Agents

An agent that answers one prompt never worries about the context window. An agent that holds a support conversation for three weeks, processes a queue around the clock, or works through a forty-step task does. Conversation history only grows, every model has a hard token limit, and when a session crosses it the provider rejects the call and the run fails mid-task.

The hard limit is only half the problem. Model quality drops well before the window is full. Chroma's context-rot study (2025) evaluated 18 frontier models and found that output quality degrades as input length grows, even on simple tasks. Earlier, Lost in the Middle (Liu et al., 2023) showed that models retrieve information best from the start and end of a prompt, with accuracy dropping sharply when it sits in the middle. A bloated context does not only risk an error. It makes every answer worse.

Why Long-Running Agents Run Out of Context

An agent's context window fills from:

Further reading: Chroma's context-rot study (2025), Lost in the Middle (Liu et al., 2023).

Truncation, Sliding Windows, and Summarization

There are three common ways to keep an LLM context bounded, with very different failure modes:

Our platform uses LLM summarization, with retention rules designed to avoid the failure modes of the other two.

How Connic Compresses Context

The default trigger is the moment things would otherwise break: a provider context-limit error.

When a model call fails, the runner classifies the error. It recognizes context-window errors across providers and explicitly ignores rate-limit and quota errors, so compression only fires on a genuine overflow. It then compresses the request and retries the call once, inside the same run. The user never sees a failed run, and no work is lost. If the retried call still exceeds the window, the run fails with the original error.

What stays, what gets summarized

The most recent messages stay verbatim: keep_recent_messages, 8 by default. The boundary is then adjusted backwards so that a tool result is never separated from the call that produced it, the malformed-request trap that naive windowing falls into.

Everything older is summarized by an LLM with strict instructions: preserve user goals, constraints, decisions, tool names and arguments, tool results, IDs, numbers, errors, open questions, and next steps, and invent nothing. History too long for a single pass is split into 12,000-character chunks, each chunk summarized, and the chunk summaries merged into one continuation summary.

The result is inserted as a single message that tells the model exactly what happened: earlier history was compressed, and the retained recent messages are the source of truth for the current turn. The model continues the task instead of hallucinating a fresh start.

Oversized tool results

Compression also targets the single biggest space consumer directly. When compression runs, oversized tool responses within the retained messages are summarized in place and replaced with a compact record that includes the summary and the original size. A bulky JSON payload becomes a paragraph, and the agent keeps the facts it extracted from it.

Compress before the error: token budgets

Waiting for the provider to complain means one failed call per compression. With max_prompt_tokens set, the runner compresses proactively instead: it records the model-reported prompt token count after every call, and when the last call meets the budget it compresses before sending the next one. The next call is skipped before it can fail or consume tokens. Because Connic is bring-your-own-key, smaller prompts also show up directly as lower input cost on the provider bill.

A cheaper model for summaries

Summaries do not need the agent's best model. The context_compression.model setting lets summaries run on a small, fast model while the agent itself stays on its configured one. When omitted, summaries use the agent model. The retried agent call always uses the agent model either way.

Configuration in YAML

Context compression is part of the agent YAML for any LLM agent. A complete configuration for a session-backed chat agent:

version: "1.0"

name: support-bot
type: llm
model: connic/gpt-5.6-terra
description: "A support chatbot that remembers conversation history"
system_prompt: |
  You are a helpful support agent. Use the conversation history
  to provide contextual responses.

# Persistent sessions keyed by chat ID
session:
  key: context.chat_id
  ttl: 86400

# Keep long conversations within the model context window
context_compression:
  enabled: true
  # Optional model for compression summaries; defaults to the agent model
  model: connic/gpt-5.6-luna
  keep_recent_messages: 12
  # Optional early compression based on model-reported prompt usage
  max_prompt_tokens: 100000
  # Optional compaction for stored session history between runs
  session_history:
    interval: 4
    keep_recent_runs: 1

Only the block itself is required. Add context_compression with enabled: true to enable error-triggered compression with the defaults. Everything else is tuning.

Compacting Stored Session History

In-request compression shapes what a single model call sees. Session-backed agents have a second growth problem: the history stored between runs. The session_history block compacts it. Every interval runs, older stored history is replaced with a summary, keeping the most recent keep_recent_runs runs untouched. A session that has been alive for months starts each new run from a compact summary plus recent turns, not from its entire life story.

Every Compression Is a Trace Span

Compression rewrites what the agent sees, so it should never be invisible. Each compression is captured as an OpenTelemetry trace span in the run:

In-request compression appears as a context_compression span, stored-history compaction as history_compression. Both also write log lines into the run. See the observability documentation for how traces work, or read the agent observability guide for the bigger picture.

Getting Started

Adding context compression to an existing LLM agent takes minutes:

1. 1

   Add a context_compression block to the agent's YAML configuration

2. 2

   Deploy. Compression activates on the next run, no code changes needed

3. 3

   Watch the Traces tab for context_compression spans as long sessions hit the window

4. 4

   Tune keep_recent_messages, add a token budget, or switch summaries to a cheaper model based on the trace data

For the full option reference, including sessions and compaction, see the runtime controls documentation. New to Connic? Follow the quickstart guide to deploy a first agent, then give it a memory that does not run out.