Skip to main content
connic

Website Crawler

By connic
Auto-sync

Make any website available to agents through retrieval

Connect Website CrawlerTalk to Sales
Publisher
By connic
Syncs
Pages, Linked pages, Sitemaps
Set up in
Retrieval → Sources
Documentation
Retrieval Docs

How the Website Crawler source works

A lot of the knowledge agents need is already public: documentation, help centers, changelogs, or product pages. The website source crawls this content. Enter a start URL, set the crawl boundaries, and connic keeps the content synced into retrieval without a separate scraper.

Each page is converted to clean markdown with navigation and boilerplate stripped, then chunked and embedded like any other retrieval entry. Re-crawls run on the configured schedule and only reprocess pages whose content changed.

What gets synced

Pages
Rendered page content converted to clean markdown, with menus and boilerplate stripped.
Linked pages
The crawler follows links inside the configured scope up to the configured depth.
Sitemaps
When a sitemap.xml exists, it is used to discover pages faster and more completely.
Website CrawlerActiveWebsite
Configuration
Crawl scope
BoundariesPages under the start URL path
Max pages500
Max link depth5
Use sitemap.xmlYes
Permission
Authorized to crawl these URLsNo
Sync behavior
Target namespacesources.website
Sync intervalDaily
When content is deleted at the sourceRemove it from the retrieval
Recent syncsSync now
  • Completed13 Oct, 09:00
    4 new · 11 updated · 1 removed
  • CompletedManual12 Oct, 21:00
    0 new · 2 updated · 0 removed
  • Completed12 Oct, 09:00
    0 new · 6 updated · 0 removed

Configuration, in depth

Every available control for connecting Website Crawler to a project.

Crawl scope

Start URL*
Where the crawl begins, e.g. the documentation root.
Boundaries
How far the crawler may follow links from the start URL.
Include paths
Only crawl URLs matching these path patterns, comma-separated. Leave empty for no restriction.
Exclude paths
Skip URLs matching these path patterns, comma-separated.
Max pages
Upper bound on pages per crawl.
Max link depth
How many links away from the start URL the crawler may go.
Use sitemap.xml
Discover pages through the site's sitemap when one exists.

Permission

Authorized to crawl these URLs*
Confirm ownership of these URLs or other authorization to crawl and ingest their content.

Sync behavior

Where synced content lands and how often it refreshes.

Target namespace
Retrieval namespace the synced entries are filed under. Use dots to nest, e.g. sources.notion.
Sync interval
How often the source is checked for changes. Each run only reprocesses content that changed.
When content is deleted at the source
Pages that disappear from the site can be removed from the retrieval on the next crawl, or kept as they were last crawled.

Search it from an agent

Agents search synced content with the retrieval_query tool. A small tool that pins the namespace keeps an agent to this source.

tools/knowledge.py
from connic.tools import retrieval_query


async def search_website(question: str) -> dict:
    return await retrieval_query(
        question, namespace="sources.website"
    )
agents/support-agent.yaml
version: "1.0"

name: support-agent
type: llm
model: connic/gemini-3.7-flash
system_prompt: |
  Answer only from what search_website returns.
tools:
  - knowledge.search_website

Website Crawler retrieval source FAQ

Starting from the configured URL, the crawler follows links inside the configured boundaries, converts each page to clean markdown, and hands the result to connic's ingestion pipeline for chunking and embedding.

Yes. Configure the crawl boundary (start path, domain, or subdomains), include and exclude path patterns, a page limit, and a maximum link depth.

No. The crawl runs inside connic on the configured schedule. There is nothing to host, and JavaScript-rendered pages are handled by the crawling engine.

On the configured schedule, from every 12 hours up to weekly. Re-crawls only reprocess pages whose content changed, so frequent schedules stay cheap.

More retrieval sources

Browse the marketplace