Apache Kafka
Subscribe an agent to a topic and every message becomes a run. Publish the agent's output back to another topic for downstream consumers. Consumer groups, offsets, and reconnection are handled.
Overview
The Kafka connector is a managed consumer and producer for your agents. Point an inbound connector at your brokers and a topic, and every message on that topic becomes an agent run: JSON object messages arrive with their top-level fields intact, anything else is wrapped under a message key, and each run carries the topic, partition, offset, timestamp, and message key as _kafka metadata.
You skip the consumer service you would otherwise operate. Connic manages the consumer group, tracks offsets, reconnects with exponential backoff, and picks up configuration changes. Add an outbound connector and completed results are produced to a destination topic, so a full consume, process, produce pipeline needs no code beyond the agent itself.
How it works
Create an inbound consumer
Add an Apache Kafka connector from your agent's Connector Flow in Inbound (Consumer) mode. Enter your bootstrap servers and the topic to consume, plus an optional consumer group ID and auto offset reset (latest for new messages only, earliest to start from the beginning).
Configure security
Choose PLAINTEXT, SSL, SASL_PLAINTEXT, or SASL_SSL. For managed Kafka like Confluent, MSK, or Aiven, use SASL_SSL with SCRAM-SHA-256. For TLS and mutual TLS, paste certificate and key PEM contents directly into the form.
Link agents, optionally produce results
Each message on the topic is dispatched to all linked agents. To publish results downstream, add an outbound connector with bootstrap servers and a destination topic; completed run outputs are produced there automatically.
What you can build
Patterns teams ship in production. No queues, workers, or schedulers to run.
Message posted
topic: user.content.createdAgent scores each post for toxicity and policy fit, and either publishes, queues for review, or hides it — all before the comment renders to other users.
Message posted
topic: user.content.createdAgent scores each post for toxicity and policy fit, and either publishes, queues for review, or hides it — all before the comment renders to other users.
Message payload and consumer semantics
JSON object messages are dispatched with their top-level fields as the payload, plus a _kafka block with topic, partition, offset, timestamp, and key. Non-JSON values arrive under a message key, and compaction tombstones still trigger runs with message: null, so your agent can react to deletions using _kafka.key. Consumer groups behave the way you expect: connectors with different group IDs each see every message, and connectors sharing a group ID split the load.
Every consumed message becomes a normal agent run, with full traces, token and cost tracking, and the same guardrails and approval rules as any other trigger. A high-volume topic does not become a black box.
Producing results back
Outbound connectors publish results when linked agents complete. Only runs with status completed are published; failed and cancelled runs are skipped. When a run was triggered by an inbound message with a key, the result is produced with the same key, preserving ordering within partitions across the whole pipeline; runs without an inbound key use the run ID. Messages are produced with full replication acknowledgment and automatic retries. See payload formats and an end-to-end pipeline in the Kafka docs.
Information
- Publisher
- By Connic
- Category
- Connectors
- Modes
- Inbound, Outbound
- Documentation
- Apache Kafka docs
Frequently Asked Questions
How do I trigger an AI agent from a Kafka topic?
_kafka. The Kafka connector reference covers payload formats and the end-to-end pipeline.Which Kafka security setups are supported?
Can the agent publish results back to a Kafka topic?
Bring the event source, payload shape, result destination, and any private-network or approval requirements. We will map Apache Kafka to the right Connic connector mode, deployment path, and observability setup.
Talk to Sales