Apache Kafka
Every message on a topic starts an agent run, and results go back to a topic under the same key. connic runs the consumer group and commits offsets for you.
- Modes
- Inbound, Outbound
- Set up in
- Dashboard or connectors/*.yaml
- Documentation
- Apache Kafka docs
- Publisher
- By connic
Turn every message into an agent run
An inbound connector consumes one topic and starts each linked agent once per message, with the message as its input.
- JSON keeps its shape
- A JSON object arrives with its top-level fields, plus _kafka with the topic, partition, offset, timestamp and key.
- Text and tombstones
- Any other value arrives under message. A tombstone still starts a run with message set to null, so the agent can act on the deleted key.
- Files in the message
- A JSON object can carry a files array with each file's name, MIME type and base64 data, up to 10 files and 10 MiB in total.
{
"order_id": "ord_8812",
"customer": "jana@example.com",
"total": 184.5
}{
"order_id": "ord_8812",
"customer": "jana@example.com",
"total": 184.5,
"_kafka": {
"topic": "orders",
"partition": 3,
"offset": 20417,
"timestamp": 1791882000000,
"key": "ord_8812"
}
}{
"message": null,
"_kafka": {
"topic": "customers",
"partition": 0,
"offset": 9182,
"timestamp": 1791882000000,
"key": "cus_4410"
}
}Commit each offset after its runs are stored
connic commits a message once a queued run exists for every linked agent, so a restart or a deploy doesn't drop work.
- One run per message
- A message that arrives again after a restart or a rebalance reuses its stored run. Each agent runs once per message and consumer group.
- Failed runs stay in Agent Runs
- A run that fails after it was stored doesn't redeliver the message. It keeps its full input in Agent Runs, where you can re-run it.
- Parallel consumers
- Add inbound connectors with the same Group ID to split a busy topic's partitions between them. Connectors with different Group IDs each read every message.
Read
The connector reads its partitions one message at a time.
Store
connic stores a queued run for every linked agent. Stored runs survive worker restarts and deployments.
Commit
The offset is committed. If storing a run fails, the same message is delivered again after a short delay, and later messages wait.
Offset committedRun
Runs execute in parallel up to max_concurrent_runs. A concurrency rule on the message key starts runs for the same key one at a time.
concurrency.key: _kafka.key
Keep the consumer next to your agents
Declare it as a YAML file in connectors/. Brokers and credentials live in a Connection, so the file names only the topic and the consumer group. A consumer declared in code starts reading when its deployment goes live.
- SASL and TLS
- Use PLAINTEXT, SSL, SASL_PLAINTEXT or SASL_SSL with PLAIN, SCRAM-SHA-256 or SCRAM-SHA-512. For mutual TLS, paste the certificate and key PEM contents into the Connection.
- Private brokers
- Direct connections need brokers that resolve to public IP addresses. Reach brokers inside your network through a bridge.
type: kafka
mode: inbound
name: Order events
connection: orders-kafka
config:
topic: orders
group_id: connic-orders
auto_offset_reset: earliest
agents:
- order-processorPublish results back to a topic
An outbound connector produces each completed run to a topic, keyed by the message that started it.
- Keys carry through
- A run started by a Kafka message publishes with that message's key. Other runs use the run ID as the key.
- Completed runs only
- Failed and cancelled runs are skipped. Messages are produced with acks=all and automatic retries.
- Let the agent publish
- Linked as an agent tool, the connector lets the model publish its own payload, with an optional key, during the run.
type: kafka
mode: outbound
name: Order results
connection: orders-kafka
config:
topic: order-results
agents:
- name: order-processor
outbound:
usage: automatic
source: [order-events]{
"run_id": "550e8400-e29b-41d4-a716-446655440000",
"agent_name": "order-processor",
"status": "completed",
"output": "ord_8812 approved for standard shipping.",
"error": null,
"started_at": "2026-10-13T09:00:01Z",
"ended_at": "2026-10-13T09:00:05Z",
"token_usage": {
"input_tokens": 642,
"output_tokens": 38,
"thinking_tokens": 0,
"cached_input_tokens": 0,
"total_tokens": 680
}
}Where Kafka fits
Pick a setup to see which topic starts the run, what the agent receives and where the result goes.
Each order event starts a run, and the result lands on order-results under the order's key.
{
"order_id": "ord_8812",
"customer": "jana@example.com",
"total": 184.5,
"_kafka": {
"topic": "orders",
"partition": 3,
"offset": 20417,
"timestamp": 1791882000000,
"key": "ord_8812"
}
}{
"run_id": "550e8400-e29b-41d4-a716-446655440000",
"agent_name": "order-processor",
"status": "completed",
"output": "ord_8812 approved for standard shipping.",
"error": null,
"started_at": "2026-10-13T09:00:01Z",
"ended_at": "2026-10-13T09:00:05Z",
"token_usage": {
"input_tokens": 642,
"output_tokens": 38,
"thinking_tokens": 0,
"cached_input_tokens": 0,
"total_tokens": 680
}
}The agent scores each transaction and publishes to fraud-alerts only when the risk is high.
{
"transaction_id": "txn_30571",
"customer_id": "cus_4410",
"amount": 950.0,
"country": "PT",
"_kafka": {
"topic": "transactions",
"partition": 1,
"offset": 88213,
"timestamp": 1791882000000,
"key": "cus_4410"
}
}{
"payload": {
"transaction_id": "txn_30571",
"risk": "high",
"reason": "Card used in two countries within 10 minutes."
},
"key": "cus_4410"
}Readings stream in from the factory floor, and the agent posts to Slack when a machine runs outside its limits.
{
"sensor_id": "press-07",
"temperature_c": 94.2,
"limit_c": 85,
"_kafka": {
"topic": "sensor-readings",
"partition": 5,
"offset": 513340,
"timestamp": 1791882000000,
"key": "press-07"
}
}{
"text": "press-07 at 94.2 °C, 9.2 °C over its limit.",
"channel_id": "C01234567"
}A tombstone on the compacted customers topic starts a run, and the agent deletes that customer's data through its tools.
{
"message": null,
"_kafka": {
"topic": "customers",
"partition": 0,
"offset": 9182,
"timestamp": 1791882000000,
"key": "cus_4410"
}
}Apache Kafka FAQ
More connectors
AWS S3
Run an agent the moment a file lands in a bucket, with the file itself when you turn on Include Content.
Inbound
MCP Server
Expose agents as MCP tools. MCP clients list and call them and get the agent's answer as the tool result.
Inbound · Sync
Shopify
Run an agent when an order in your Shopify store is created, paid, fulfilled or cancelled. It tags the order or adds a note.
Inbound · Outbound
Slack
Connect a Slack bot to connic. Trigger agents from mentions or send agent outputs to Slack channels.
Inbound · Outbound
Telegram
Power a Telegram bot with an agent. Read user messages, reply in the chat, push alerts to any channel.
Inbound · Outbound
Twilio Messaging
Receive SMS, MMS, WhatsApp, and RCS messages with an agent. Send replies, media, and WhatsApp templates through your Twilio account.
Inbound · Outbound