Website Crawler

By Connic

Make any website available to agents through retrieval

WebsiteAuto-sync

How the Website Crawler source works

A lot of the knowledge agents need is already public: documentation, help centers, changelogs, or product pages. The website source crawls this content. Enter a start URL, set the crawl boundaries, and Connic keeps the content synced into retrieval without a separate scraper.

Each page is converted to clean markdown with navigation and boilerplate stripped, then chunked and embedded like any other retrieval entry. Re-crawls run on the configured schedule and only reprocess pages whose content changed.

What gets synced

Pages

Rendered page content converted to clean markdown, with menus and boilerplate stripped.

Linked pages

The crawler follows links inside the configured scope up to the configured depth.

Sitemaps

When a sitemap.xml exists, it is used to discover pages faster and more completely.

Configuration, in depth

Every available control for connecting Website Crawler to a project.

Crawl scope

Start URL*: Where the crawl begins, e.g. the documentation root.
Boundaries: How far the crawler may follow links from the start URL.
Include paths: Only crawl URLs matching these path patterns, comma-separated. Leave empty for no restriction.
Exclude paths: Skip URLs matching these path patterns, comma-separated.
Max pages: Upper bound on pages per crawl.
Max link depth: How many links away from the start URL the crawler may go.
Use sitemap.xml: Discover pages through the site's sitemap when one exists.

Permission

Authorized to crawl these URLs*: Confirm ownership of these URLs or other authorization to crawl and ingest their content.

Sync behavior

Where synced content lands and how often it refreshes.

Target namespace: Retrieval namespace the synced entries are filed under. Use dots to nest, e.g. sources.notion.
Sync interval: How often the source is checked for changes. Each run only reprocesses content that changed.
When content is deleted at the source: Pages that disappear from the site can be removed from the retrieval on the next crawl, or kept as they were last crawled.

Information

Publisher: By Connic
Category: Retrieval Sources
Type: Website
Sync schedule: Every 12 hours to weekly
Documentation: Retrieval Docs

Website Crawler retrieval source FAQ

Starting from the configured URL, the crawler follows links inside the configured boundaries, converts each page to clean markdown, and hands the result to Connic's ingestion pipeline for chunking and embedding.

Yes. Configure the crawl boundary (start path, domain, or subdomains), include and exclude path patterns, a page limit, and a maximum link depth.

No. The crawl runs inside Connic on the configured schedule. There is nothing to host, and JavaScript-rendered pages are handled by the crawling engine.

On the configured schedule, from every 12 hours up to weekly. Re-crawls only reprocess pages whose content changed, so frequent schedules stay cheap.

Need Website Crawler content behind a production agent?

Our team maps content location, freshness requirements, and access rules to source scopes, namespaces, and the required agent Retrieval setup.

Talk to Sales

More retrieval sources

Browse the marketplace

Confluence

Turn Confluence spaces into retrievable agent knowledge

Superhuman Docs (Coda)

Keep Superhuman Docs and tables in sync for agents

Notion

Sync Notion pages and databases for agent retrieval