Runtime Controls

Concurrency rules

Concurrency rules ensure that only one run per unique key is active at a time. Use them when an agent processes events for a specific support process, customer, or order and parallel processing would conflict.

Queue mode (default)

When another event arrives for an active key, it waits until the first run finishes and then starts automatically.

version: "1.0"

name: support-processor
type: llm
model: gemini/gemini-2.5-pro
description: "Processes support tickets one at a time per process"
system_prompt: "You are a support agent. Process the incoming ticket."

# Only one run per process_id at a time. Additional runs wait in queue.
concurrency:
  key: "process_id"
  on_conflict: queue

Drop mode

When another event arrives for an active key, the new run is cancelled immediately. Use this when duplicate work would be wasteful.

version: "1.0"

name: notification-handler
type: tool
description: "Handles notifications, skipping duplicates"
tool_name: notifications.handle

# Drop duplicate runs for the same customer while one is active.
concurrency:
  key: "data.customer_id"
  on_conflict: drop

Key extraction

The key uses dot notation to read the trigger payload. If a Kafka payload is {"data": {"customer_id": "abc"}}, use data.customer_id. If the path does not exist, concurrency enforcement is skipped for that run.

Interaction with max_concurrent_runs

Both constraints apply. With max_concurrent_runs: 5 and concurrency.key: "process_id", five process IDs can run in parallel, but only one run per process ID can be active.

Persistent sessions

By default, every request creates an isolated session. Persistent sessions maintain conversation history across requests, restarts, and redeployments.

version: "1.0"

name: support-bot
type: llm
model: gemini/gemini-2.5-pro
description: "A support chatbot that remembers conversation history"
system_prompt: |
  You are a helpful support agent. Use the conversation history
  to provide contextual responses.

# Maintain persistent sessions keyed by chat ID from middleware
session:
  key: context.chat_id
  ttl: 86400  # Sessions expire after 24 hours of inactivity

# Keep long conversations within the model context window
context_compression:
  enabled: true
  # Optional model for compression summaries; defaults to the agent model
  model: gemini/gemini-2.5-flash
  keep_recent_messages: 12
  # Optional early compression based on model-reported prompt usage
  max_prompt_tokens: 100000

Session key

The key identifies the session and must use one of two prefixes:

• context.<path> resolves from the middleware context. This is recommended because middleware can derive a stable key from any part of the request.
• input.<path> resolves from the raw connector payload before middleware. It requires valid JSON and supports nested paths.

Set the key in middleware

async def before(content: dict, context: dict) -> dict:
    # Extract a stable identifier for this conversation from the connector payload.
    payload = context.get("payload", {})
    context["chat_id"] = payload.get("chat_id")
    return content

Use an input key

# When the connector payload is JSON like {"user_id": "u123", "message": "..."}
session:
  key: input.user_id

TTL and session management

The optional ttlexpires sessions after that many inactive seconds. The minimum is 60 seconds; without a TTL, sessions never expire. View and delete active sessions in the dashboard under Storage > Sessions. Sessions are scoped per environment.

Context compression

Add context_compression to keep long-running LLM sessions inside the model context window. Once configured, provider context-window errors trigger compression and an automatic retry. Set context_compression.model to use a separate summary model; normal agent calls and their retry still use the agent model. Set max_prompt_tokens to compress earlier using reported prompt usage. Use session_history.interval only when stored history should also be compacted between runs.