Skip to main content
connic

S3 Document Pipeline

Auto-process documents uploaded to S3 with text extraction, classification, and routing. Handles PDFs, images, and text files with retries.

Talk to Sales
Install with the CLI
connic init my-project --templates=s3-document-pipeline
Difficulty
Intermediate
Works with
AWS S3, HTTP Webhook
Source
GitHub
Publisher
By connic

What the template does

Triggered by file uploads to an S3 bucket. The intake agent extracts text, metadata, and entities from PDFs, images, and text files. Middleware rejects unsupported file types with StopProcessing before the LLM runs. The classifier assigns categories, confidence scores, and routing destinations. Retries on transient failures.

Architecture
S3 Bucket
document-pipeline
document-intake
document-classifier
Team Routing
Document management
Automatically classify and route documents to the right team as they land in the configured S3 inbox.
Contract processing
Extract key entities (parties, dates, amounts) from uploaded contracts and flag for legal review.
Multi-format intake
Process PDFs, images, Word documents, and text files through a single unified pipeline.

Every file lands in your project

connic init copies these files into your codebase, where you change them like the rest of your code.

agents/s3-document-pipeline/document-classifier.yaml

LLM agent with its model, prompt and tools.

version: "1.0"

name: document-classifier
type: llm
model: connic/gemini-3.5-flash-lite
description: "Classifies extracted documents and routes them to the right workflow"

system_prompt: |
  You receive structured extraction output from the intake agent. Your job is
  to make a routing decision.

  Classification rules:
  - Assign a primary category and sub-category.
  - Set a confidence score (0-100). Below 70 means "needs human review".
  - Determine the routing destination:
    - invoice -> "accounts-payable"
    - contract -> "legal-review"
    - report -> "analytics-team"
    - letter/memo -> "general-inbox"
    - form -> "data-entry"
    - other -> "manual-review"
  - Add tags for searchability (e.g., "q4-2025", "vendor:acme", "urgent").

  If the extraction flagged truncation, mark confidence as maximum 60 and
  route to manual-review regardless of type.

temperature: 0.2
output_schema: classification-result
agents/s3-document-pipeline/document-intake.yaml

LLM agent with its model, prompt and tools.

version: "1.0"

name: document-intake
type: llm
model: connic/gemini-3.7-flash
description: "Processes files uploaded to S3 - extracts text, metadata, and structured content"

system_prompt: |
  You process documents uploaded to an S3 bucket.

  Your job:
  1. Analyze the document from the content object. Check content.encoding
     to know if it's raw text ("utf-8") or base64-encoded binary ("base64").
  2. Extract the title, author, creation date, and a concise summary.
  3. Identify the document type: contract, invoice, report, letter, form, memo, or other.
  4. Pull out key entities: people, organizations, dates, monetary amounts.
  5. If the file is an image, describe what you see and extract any visible text.

  Always include the S3 key in your output for downstream traceability.
  For large documents, focus on the first 10 pages and note if content was truncated.

temperature: 0.2
output_schema: intake-result

retry_options:
  attempts: 3
  max_delay: 30
agents/s3-document-pipeline/document-pipeline.yaml

Runs its agents in order, each on the output of the one before.

version: "1.0"

name: document-pipeline
type: sequential
description: "Full document processing: intake from S3, then classify and route"

agents:
  - document-intake
  - document-classifier
middleware/document-intake.py

Runs before and after the agent with the same name.

"""Middleware for document-intake.

Validates file size and type before running the LLM. Rejects unsupported
formats with StopProcessing to avoid wasted tokens.

The S3 inbound connector delivers a JSON payload with:
bucket, key, size, etag, event_name, event_time, _s3,
and optionally content: {text, content_type, size_bytes, encoding}
"""

import json
from typing import Any

from connic import StopProcessing

SUPPORTED_TYPES = {
    "application/pdf",
    "text/plain",
    "text/csv",
    "text/markdown",
    "text/html",
    "image/png",
    "image/jpeg",
    "image/webp",
    "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
}

MAX_SIZE_BYTES = 50 * 1024 * 1024


async def before(content: dict[str, Any], context: dict[str, Any]) -> dict[str, Any]:
    """Reject unsupported file types and oversized files before LLM runs."""
    parts = content.get("parts", [])
    for part in parts:
        if isinstance(part, dict) and "text" in part:
            try:
                payload = json.loads(part["text"])
            except (json.JSONDecodeError, TypeError):
                continue

            if not isinstance(payload, dict):
                continue

            context["s3_bucket"] = payload.get("bucket", "")
            context["s3_key"] = payload.get("key", "")
            context["pipeline_stage"] = "intake"

            file_size = payload.get("size", 0)
            if file_size > MAX_SIZE_BYTES:
                raise StopProcessing(
                    f"File too large: {file_size} bytes "
                    f"(max {MAX_SIZE_BYTES // (1024*1024)} MB). "
                    f"Key: {payload.get('key', 'unknown')}"
                )

            file_content = payload.get("content", {})
            if isinstance(file_content, dict):
                ct = file_content.get("content_type", "")
                if ct and ct not in SUPPORTED_TYPES:
                    raise StopProcessing(
                        f"Unsupported file type: {ct}. "
                        f"Supported: {', '.join(sorted(SUPPORTED_TYPES))}"
                    )

    return content


async def after(response: str, context: dict[str, Any]) -> str:
    """Track processing metrics."""
    context["pipeline_stage"] = "intake_complete"
    return response
schemas/classification-result.json

JSON schema for an agent's structured output.

{
  "type": "object",
  "description": "Document classification and routing decision",
  "properties": {
    "category": {
      "type": "string",
      "enum": ["contract", "invoice", "report", "letter", "form", "memo", "other"]
    },
    "sub_category": {
      "type": "string",
      "description": "More specific classification (e.g. 'nda', 'sow', 'quarterly-report')"
    },
    "confidence": {
      "type": "number",
      "minimum": 0,
      "maximum": 100,
      "description": "Classification confidence; below 70 triggers human review"
    },
    "route_to": {
      "type": "string",
      "enum": ["accounts-payable", "legal-review", "analytics-team", "general-inbox", "data-entry", "manual-review"],
      "description": "Destination team or workflow"
    },
    "tags": {
      "type": "array",
      "items": {"type": "string"},
      "description": "Searchable tags for the document"
    },
    "needs_human_review": {
      "type": "boolean",
      "description": "True if confidence is low or content was truncated"
    }
  },
  "required": ["category", "confidence", "route_to"]
}
schemas/intake-result.json

JSON schema for an agent's structured output.

{
  "type": "object",
  "description": "Document intake extraction result",
  "properties": {
    "s3_key": {
      "type": "string",
      "description": "Original S3 object key for traceability"
    },
    "title": {
      "type": "string",
      "description": "Document title"
    },
    "author": {
      "type": "string",
      "description": "Document author if identifiable"
    },
    "creation_date": {
      "type": "string",
      "description": "Document creation or issue date (YYYY-MM-DD)"
    },
    "document_type": {
      "type": "string",
      "enum": ["contract", "invoice", "report", "letter", "form", "memo", "other"]
    },
    "summary": {
      "type": "string",
      "description": "Concise summary of the document content"
    },
    "entities": {
      "type": "object",
      "properties": {
        "people": {"type": "array", "items": {"type": "string"}},
        "organizations": {"type": "array", "items": {"type": "string"}},
        "dates": {"type": "array", "items": {"type": "string"}},
        "amounts": {"type": "array", "items": {"type": "string"}}
      }
    },
    "truncated": {
      "type": "boolean",
      "description": "True if the document was too long and content was cut"
    },
    "page_count": {
      "type": "integer",
      "description": "Number of pages processed"
    }
  },
  "required": ["s3_key", "document_type", "summary"]
}
tests/document-intake.yaml

1 test case for connic test.

version: "1.0"

# Both agents are tool-less, so no mocks are needed. The payload mimics the S3
# inbound connector shape (bucket/key/size/content) that the middleware reads -
# the document text is supplied inline via content.text, since the middleware
# extracts content from the JSON payload rather than from attached files.
defaults:
  runs: 1
  timeout_s: 90

tests:
  - name: extracts_invoice_metadata
    payload: '{"bucket": "documents-inbox", "key": "inbox/INV-2025-0042.txt", "size": 612, "content": {"content_type": "text/plain", "encoding": "utf-8", "text": "INVOICE\n\nAcme Supplies GmbH\nInvoice Number: INV-2025-0042\nIssue Date: 2025-11-14\nBill To: Northwind Trading Ltd\n\nIndustrial floor mats  10  45.00 EUR  450.00 EUR\nSafety signage pack     4  22.50 EUR   90.00 EUR\n\nSubtotal   540.00 EUR\nVAT (19%)  102.60 EUR\nTotal Due  642.60 EUR\n\nPayment terms: net 30."}}'
    expected_result: output.title != "" or output.document_type != ""
tests/document-pipeline.yaml

1 test case for connic test.

version: "1.0"

# End-to-end through the sequential pipeline: intake extracts the document, then
# the classifier routes it. No custom tools, so no mocks. The invoice text is
# supplied inline via content.text the way the middleware reads it.
defaults:
  runs: 1
  timeout_s: 150

tests:
  - name: invoice_is_classified_and_routed
    payload: '{"bucket": "documents-inbox", "key": "inbox/INV-2025-0042.txt", "size": 612, "content": {"content_type": "text/plain", "encoding": "utf-8", "text": "INVOICE\n\nAcme Supplies GmbH\nInvoice Number: INV-2025-0042\nIssue Date: 2025-11-14\nBill To: Northwind Trading Ltd\n\nIndustrial floor mats  10  45.00 EUR  450.00 EUR\nSafety signage pack     4  22.50 EUR   90.00 EUR\n\nSubtotal   540.00 EUR\nVAT (19%)  102.60 EUR\nTotal Due  642.60 EUR\n\nPayment terms: net 30."}}'
    expected_result: output.category != "" and output.route_to != ""

Install it and deploy

Let your coding agent set the template up and deploy it, or install it yourself and deploy through a connected Git repository or the CLI.

Prerequisites
  • Python 3.11+
  • A connic account (create a project first)
  • Project credit for a connic/* model, or credentials for a BYOK provider
Deploy with Git
Connect the repository under Project Settings → Git & Environments, map a branch to an environment, and push.
Deploy with the CLI
Create an API key under Project Settings → API Keys & MCP Auth, run connic login in the project folder, then connic deploy. Try changes first with connic dev and connic test.
Connect and configure
Add an S3 inbound connector with the bucket name and AWS credentials. Configure S3 Event Notifications to point to the connector's webhook URL. Use a connic/* model, or add BYOK credentials under Project Settings → LLM Provider.
  1. Install the connic plugin. Skip this if your coding agent has it already. Paste this into Claude Code, Codex or Cursor:

    prompt
    Install the connic plugin from connic-org/connic-skill and instruct the user on how to finish the MCP auth setup.
  2. Start from the template. Paste this into a new session. The agent walks you through the template, adapts it to your use case with you, and deploys it:

    prompt
    I'd like to start from the connic s3-document-pipeline template. Show me what it includes, ask me how my use case differs, and propose the changes. Once we agree, set it up in this folder with `connic init --templates=s3-document-pipeline`, adapt it, validate it, and deploy it to my connic project.
~/my-project
Install
$ pip install connic-composer-sdk
Scaffold
$ connic init my-project --templates=s3-document-pipeline
connic Init
──────────────────────────────
→ Created directory: my-project
→ Creating project structure...
✓ agents/, tools/, middleware/, schemas/
→ Fetching templates from connic-awesome-agents...
✓ Fetched
→ Merging templates...
✓ Added template: s3-document-pipeline
→ Next steps:
1. Run `connic lint` to validate your project
2. Run `connic dev` to choose a dev environment with hot reload
3. Run `connic deploy` when ready
✓ Initialized with templates: s3-document-pipeline
Deploy
$ connic deploy
→ Validating project files...
✓ Lint passed: 3 agent(s) validated (document-intake, document-classifier, document-pipeline)
✓ Lint passed
✓ Deployment created.