How to Trigger AI Agents from Kafka Topics

Your transactions, orders, and telemetry already flow through Kafka. The missing piece is an AI agent that reacts to them: scores the transaction, triages the alert, updates the recommendation. In Connic you do not write a consumer service for that. You point a Kafka inbound connector at the topic, and every message starts an agent run with the payload attached. This tutorial walks through the whole setup, from scaffolding a project to watching the first run.

Why Kafka Is a Strong Agent Trigger

Kafka is a durable, ordered log. Triggering agents from it gives you at-least-once delivery, ordering within a partition, and the ability to replay history by resetting a consumer to an earlier offset, guarantees a plain HTTP webhook cannot make. The connector joins your cluster as a consumer, so there is no polling loop, no middleware service, and no bot process to keep alive between Kafka and your agents. If you are still deciding between trigger styles, compare webhook, Kafka, Postgres, and SQS triggers first; this post assumes Kafka is the right fit and shows the hands-on path.

Step 1: Scaffold a Project from the Template

The fastest follow-along path is the Kafka Fraud Detector template from the Connic Marketplace: a working project that consumes transactions from a Kafka topic and scores each one for fraud risk, with an escalation agent for high-risk cases. Install the SDK and scaffold from it:

pip install connic-composer-sdk
connic init my-project --templates=kafka-fraud-detector
cd my-project

This creates a complete, deployable structure:

my-project/
  agents/
    kafka-fraud-detector/
      fraud-scorer.yaml
      fraud-escalator.yaml
  tools/
    fraud_tools.py
  middleware/
    fraud-scorer.py
  hooks/
    fraud-scorer.py
  schemas/
    fraud-assessment.json
  tests/
  requirements.txt

Two agents, plain Python tools, middleware that flags admin requests, a hook that enforces the admin override, an output schema, and a test suite you can run with connic test. Explore the Kafka Fraud Detector template for the architecture diagram, or browse the other agent templates in the Marketplace if fraud scoring is not your use case. Everything below works the same for an agent you write from scratch.

Step 2: Look at the Agent YAML

Each YAML file under agents/ defines one agent. Here is the scorer from the template, with the system prompt shortened:

version: "1.0"

name: fraud-scorer
type: llm
model: anthropic/claude-sonnet-4-6
description: "Scores transactions for fraud risk in real-time from a Kafka stream"

system_prompt: |
  You are a fraud detection specialist analyzing financial transactions in
  real-time from a Kafka stream.

  Use _kafka.key for customer correlation and _kafka.timestamp to detect
  processing latency.

temperature: 0.1
output_schema: fraud-assessment

concurrency:
  key: "data.customer_id"
  on_conflict: queue

tools:
  - fraud_tools.calculate_velocity
  - fraud_tools.check_geo_anomaly
  - fraud_tools.create_alert
  - fraud_tools.search_fraud_patterns
  - fraud_tools.store_fraud_pattern
  - fraud_tools.admin_override: context.is_admin == True

Three details matter for Kafka workloads. The prompt tells the model to use _kafka.key and _kafka.timestamp, metadata the connector attaches to every message. The concurrency block queues runs that share a customer_id, so two transactions from the same customer never race each other while the rest of the stream processes in parallel. And the last tool entry is conditional: admin_override only exists for the model when middleware has set is_admin in the run context. See how agent configuration works for the full set of options.

Step 3: Deploy

Push the project to the branch your environment deploys from, and Connic builds and deploys it automatically. Or deploy straight from the CLI:

connic login
connic deploy

While iterating, connic dev gives you a hot-reloading dev server so you can adjust the prompt and tools before wiring up production traffic.

Step 4: Create the Kafka Inbound Connector

Connectors are created in the dashboard and linked to agents there. Open the fraud-scorer agent's detail page, click Add inbound connector on the Connector Flow, choose Create New Connector, and select Apache Kafka. For the tutorial setup:

The connector consumes from Connic's side, so the brokers must be reachable from outside your network: a managed endpoint works out of the box, and a private cluster connects through Bridge, covered at the end of this post. For managed Kafka such as Confluent Cloud, Amazon MSK, or Aiven, use SASL_SSL with SCRAM-SHA-256 and your service credentials. Mutual TLS is supported by pasting the CA certificate, client certificate, and client key PEM contents directly into the form; the connector does not accept local file paths for certificates. Once created, the connector is linked to the agent automatically and starts consuming.

Consumer groups behave the way you expect from Kafka: connectors with different group IDs each receive every message, connectors sharing a group ID split the load. The connector tracks offsets, reconnects with exponential backoff, and picks up configuration changes without redeploying the agent.

What the Agent Receives

JSON messages arrive with their top-level fields intact, plus a _kafka block carrying topic, partition, offset, timestamp, and key:

{
  "transaction_id": "tx-9917",
  "customer_id": "cust-1042",
  "amount": 4899.0,
  "currency": "EUR",
  "merchant": "TechWorld Berlin",
  "country": "DE",
  "_kafka": {
    "topic": "transactions",
    "partition": 0,
    "offset": 1542,
    "timestamp": 1705312800000,
    "key": "cust-1042"
  }
}

Non-JSON messages are wrapped under a message key, and compaction tombstones still trigger runs with message: null, so an agent can react to deletions via _kafka.key. The full payload behavior is in the Kafka connector reference.

Step 5: Publish Results Back to a Topic

Downstream systems usually want the agent's verdict on a stream of their own. Repeat the connector setup on the same fraud-scorer agent, this time choosing Outbound (Producer) mode with fraud-alerts as the topic. Every time the agent completes a run, its structured assessment is published with the run ID, agent name, status, and output. Failed and cancelled runs are skipped. If the run was triggered by an inbound Kafka message with a key, the outbound message reuses the same key, so per-customer ordering survives the round trip through the agent.

Step 6: Send a Test Event and Watch the Run

Produce a keyed test message to the topic:

kafka-console-producer \
  --bootstrap-server kafka:9092 \
  --topic transactions \
  --property parse.key=true \
  --property key.separator=:
>cust-1042:{"transaction_id":"tx-9917","customer_id":"cust-1042","amount":4899.0,"currency":"EUR","merchant":"TechWorld Berlin","country":"DE"}

Within moments a new run appears in the Runs section of your project, with the full trace: the incoming payload, every tool call the scorer made, and the structured risk assessment it returned. See how run traces and observability work for reading those traces. From here the loop is tight: adjust the prompt or tools, push, and the next message on the topic exercises the new version.

Private Kafka Clusters

If your brokers live in a private network, you do not need to expose them: enable Connect via Bridge on the connector, run Connic Bridge inside your network, and read how Bridge reaches private infrastructure through a secure outbound tunnel.

Building something other than fraud detection? The connector works the same for content moderation, personalization, and anomaly detection streams; see what the Kafka connector handles in the Marketplace.