S3 Document Pipeline

Overview

Triggered by file uploads to an S3 bucket. The intake agent extracts text, metadata, and entities from PDFs, images, and text files. Middleware rejects unsupported file types with StopProcessing before the LLM runs. The classifier assigns categories, confidence scores, and routing destinations. Retries on transient failures.

Use cases

Architecture

Main template files

Install these files with connic init my-project --templates=s3-document-pipeline. The CLI places agent definitions in a folder named after the template under agents/.

s3-document-pipeline/
  agents/
    document-intake.yaml
    document-classifier.yaml
    document-pipeline.yaml
  middleware/
    document-intake.py
  schemas/
    intake-result.json
    classification-result.json
  requirements.txt
  README.md

Get started

Install the template, create a Connic project, and deploy through a connected Git repository or the CLI.

1. 1

   Install and scaffold

   Install the SDK and create a project from this template.

   terminal

   Copy

   pip install connic-composer-sdk
   connic init my-project --templates=s3-document-pipeline

   Then cd my-project

2. 2

   Deploy

   Choose a deployment method. A connected Git repository deploys on push; use the CLI for projects without a connected Git repository.

   Git integration

   1. 1

      In Connic: Project Settings → Git & Environments, connect the GitHub repository

   2. 2

      On the same page, map a branch (e.g. main) to the Production environment

   3. 3

      Push the scaffolded project to that repository

   terminal

   Copy

   git add .
   git commit -m "Add S3 Document Pipeline template"
   git push origin main

   CLI deploy

   1. 1

      In Connic: Project Settings → API Keys & MCP Auth, create an API key and copy project ID

   2. 2

      Run connic login in the project folder

   3. 3

      Use connic dev to try with hot-reload, connic test to run suites, or connic deploy for production

   terminal

   Copy

   connic login
   connic dev     # Ephemeral dev env with hot-reload
   connic test    # Run declarative test suites
   connic deploy  # Deploy to production

3. 3

   Connect and configure

   Add an S3 inbound connector with the bucket name and AWS credentials. Configure S3 Event Notifications to point to the connector's webhook URL. Use a connic/* model, or add BYOK credentials under Project Settings → LLM Provider.

   Full connector setup guideQuickstart guideConnector docs