Skip to main content
connic

Apache Kafka

Every message on a topic starts an agent run, and results go back to a topic under the same key. connic runs the consumer group and commits offsets for you.

Modes
Inbound, Outbound
Set up in
Dashboard or connectors/*.yaml
Documentation
Apache Kafka docs
Publisher
By connic

Turn every message into an agent run

An inbound connector consumes one topic and starts each linked agent once per message, with the message as its input.

JSON keeps its shape
A JSON object arrives with its top-level fields, plus _kafka with the topic, partition, offset, timestamp and key.
Text and tombstones
Any other value arrives under message. A tombstone still starts a run with message set to null, so the agent can act on the deleted key.
Files in the message
A JSON object can carry a files array with each file's name, MIME type and base64 data, up to 10 files and 10 MiB in total.
message on orders
{
  "order_id": "ord_8812",
  "customer": "jana@example.com",
  "total": 184.5
}
context["payload"]
{
  "order_id": "ord_8812",
  "customer": "jana@example.com",
  "total": 184.5,
  "_kafka": {
    "topic": "orders",
    "partition": 3,
    "offset": 20417,
    "timestamp": 1791882000000,
    "key": "ord_8812"
  }
}
context["payload"] for a tombstone
{
  "message": null,
  "_kafka": {
    "topic": "customers",
    "partition": 0,
    "offset": 9182,
    "timestamp": 1791882000000,
    "key": "cus_4410"
  }
}

Commit each offset after its runs are stored

connic commits a message once a queued run exists for every linked agent, so a restart or a deploy doesn't drop work.

One run per message
A message that arrives again after a restart or a rebalance reuses its stored run. Each agent runs once per message and consumer group.
Failed runs stay in Agent Runs
A run that fails after it was stored doesn't redeliver the message. It keeps its full input in Agent Runs, where you can re-run it.
Parallel consumers
Add inbound connectors with the same Group ID to split a busy topic's partitions between them. Connectors with different Group IDs each read every message.
  1. Read

    The connector reads its partitions one message at a time.

  2. Store

    connic stores a queued run for every linked agent. Stored runs survive worker restarts and deployments.

  3. Commit

    The offset is committed. If storing a run fails, the same message is delivered again after a short delay, and later messages wait.

    Offset committed
  4. Run

    Runs execute in parallel up to max_concurrent_runs. A concurrency rule on the message key starts runs for the same key one at a time.

    concurrency.key: _kafka.key

Keep the consumer next to your agents

Declare it as a YAML file in connectors/. Brokers and credentials live in a Connection, so the file names only the topic and the consumer group. A consumer declared in code starts reading when its deployment goes live.

SASL and TLS
Use PLAINTEXT, SSL, SASL_PLAINTEXT or SASL_SSL with PLAIN, SCRAM-SHA-256 or SCRAM-SHA-512. For mutual TLS, paste the certificate and key PEM contents into the Connection.
Private brokers
Direct connections need brokers that resolve to public IP addresses. Reach brokers inside your network through a bridge.
connectors/order-events.yaml
type: kafka
mode: inbound
name: Order events
connection: orders-kafka
config:
  topic: orders
  group_id: connic-orders
  auto_offset_reset: earliest
agents:
  - order-processor
create: connectors/order-events.yaml (kafka)

Publish results back to a topic

An outbound connector produces each completed run to a topic, keyed by the message that started it.

Keys carry through
A run started by a Kafka message publishes with that message's key. Other runs use the run ID as the key.
Completed runs only
Failed and cancelled runs are skipped. Messages are produced with acks=all and automatic retries.
Let the agent publish
Linked as an agent tool, the connector lets the model publish its own payload, with an optional key, during the run.
connectors/order-results.yaml
type: kafka
mode: outbound
name: Order results
connection: orders-kafka
config:
  topic: order-results
agents:
  - name: order-processor
    outbound:
      usage: automatic
      source: [order-events]
message on order-results (key ord_8812)
{
  "run_id": "550e8400-e29b-41d4-a716-446655440000",
  "agent_name": "order-processor",
  "status": "completed",
  "output": "ord_8812 approved for standard shipping.",
  "error": null,
  "started_at": "2026-10-13T09:00:01Z",
  "ended_at": "2026-10-13T09:00:05Z",
  "token_usage": {
    "input_tokens": 642,
    "output_tokens": 38,
    "thinking_tokens": 0,
    "cached_input_tokens": 0,
    "total_tokens": 680
  }
}

Where Kafka fits

Pick a setup to see which topic starts the run, what the agent receives and where the result goes.

Each order event starts a run, and the result lands on order-results under the order's key.

Checkout service
Kafka consumer
order-processor
Result on order-results
context["payload"]
{
  "order_id": "ord_8812",
  "customer": "jana@example.com",
  "total": 184.5,
  "_kafka": {
    "topic": "orders",
    "partition": 3,
    "offset": 20417,
    "timestamp": 1791882000000,
    "key": "ord_8812"
  }
}
order-results.json
{
  "run_id": "550e8400-e29b-41d4-a716-446655440000",
  "agent_name": "order-processor",
  "status": "completed",
  "output": "ord_8812 approved for standard shipping.",
  "error": null,
  "started_at": "2026-10-13T09:00:01Z",
  "ended_at": "2026-10-13T09:00:05Z",
  "token_usage": {
    "input_tokens": 642,
    "output_tokens": 38,
    "thinking_tokens": 0,
    "cached_input_tokens": 0,
    "total_tokens": 680
  }
}

The agent scores each transaction and publishes to fraud-alerts only when the risk is high.

Payment service
Kafka consumer
fraud-scorer
Alert on fraud-alerts
context["payload"]
{
  "transaction_id": "txn_30571",
  "customer_id": "cus_4410",
  "amount": 950.0,
  "country": "PT",
  "_kafka": {
    "topic": "transactions",
    "partition": 1,
    "offset": 88213,
    "timestamp": 1791882000000,
    "key": "cus_4410"
  }
}
connector-payload.json
{
  "payload": {
    "transaction_id": "txn_30571",
    "risk": "high",
    "reason": "Card used in two countries within 10 minutes."
  },
  "key": "cus_4410"
}

Readings stream in from the factory floor, and the agent posts to Slack when a machine runs outside its limits.

Factory sensors
Kafka consumer
sensor-monitor
Alert in Slack
context["payload"]
{
  "sensor_id": "press-07",
  "temperature_c": 94.2,
  "limit_c": 85,
  "_kafka": {
    "topic": "sensor-readings",
    "partition": 5,
    "offset": 513340,
    "timestamp": 1791882000000,
    "key": "press-07"
  }
}
connector-payload.json
{
  "text": "press-07 at 94.2 °C, 9.2 °C over its limit.",
  "channel_id": "C01234567"
}

A tombstone on the compacted customers topic starts a run, and the agent deletes that customer's data through its tools.

Customer service
Kafka consumer
privacy-cleanup
Data deleted through tools
context["payload"]
{
  "message": null,
  "_kafka": {
    "topic": "customers",
    "partition": 0,
    "offset": 9182,
    "timestamp": 1791882000000,
    "key": "cus_4410"
  }
}

Apache Kafka FAQ

Add an Apache Kafka connector in Inbound (Consumer) mode, pick a Connection with your brokers, enter the topic and link the agent. Connectors created in the dashboard start paused, so click Resume when you're ready. Each message then starts a run for every linked agent, with Kafka metadata under _kafka. For a complete example, walk through a full Kafka pipeline.

PLAINTEXT, SSL, SASL_PLAINTEXT and SASL_SSL, with PLAIN, SCRAM-SHA-256 or SCRAM-SHA-512 for SASL. Mutual TLS works with the PEM contents of the CA, client certificate and client key. Brokers inside a private network connect through a bridge.

No. Each agent gets one run per message and consumer group, even across connectors that share a Group ID. When a message arrives again after a restart or a rebalance, connic reuses the run it already stored. A failed run isn't retried by redelivery, so re-run it from Agent Runs.

Runs execute in parallel by default. Add a concurrency rule to the agent with key: _kafka.key and on_conflict: queue, and runs for the same key start one at a time in arrival order. Messages without a key are not serialized.