Website Crawler

By Connic

Crawl any website into your agents' retrieval

WebsiteAuto-sync

How the Website Crawler source works

A lot of the knowledge agents need is already public: your documentation, help center, changelog, or product pages. The website source crawls it for you. Enter a start URL, set the crawl boundaries, and Connic keeps the content synced into your retrieval without you running a scraper.

Each page is converted to clean markdown with navigation and boilerplate stripped, then chunked and embedded like any other retrieval entry. Re-crawls run on your schedule and only reprocess pages whose content changed.

What gets synced

Pages

Rendered page content converted to clean markdown, with menus and boilerplate stripped.

Linked pages

The crawler follows links inside your configured scope up to the depth you set.

Sitemaps

When a sitemap.xml exists, it is used to discover pages faster and more completely.

Configuration, in depth

Everything you control when connecting Website Crawler to a project.

Crawl scope

Start URL*: Where the crawl begins, e.g. your documentation root.
Boundaries: How far the crawler may follow links from the start URL.
Include paths: Only crawl URLs matching these path patterns, comma-separated. Leave empty for no restriction.
Exclude paths: Skip URLs matching these path patterns, comma-separated.
Max pages: Upper bound on pages per crawl.
Max link depth: How many links away from the start URL the crawler may go.
Use sitemap.xml: Discover pages through the site's sitemap when one exists.

Permission

I am allowed to crawl these URLs*: Confirm you own these URLs or are otherwise authorized to crawl and ingest their content.

Sync behavior

Where synced content lands and how often it refreshes.

Target namespace: Retrieval namespace the synced entries are filed under. Use dots to nest, e.g. sources.notion.
Sync interval: How often the source is checked for changes. Each run only reprocesses content that changed.
When content is deleted at the source: Pages that disappear from the site can be removed from the retrieval on the next crawl, or kept as they were last crawled.

Information

Publisher: By Connic
Category: Retrieval Sources
Type: Website
Sync schedule: Every 12 hours to weekly
Documentation: Retrieval Docs

Website Crawler retrieval source FAQ

Starting from your URL, the crawler follows links inside the boundaries you set, converts each page to clean markdown, and hands the result to Connic's ingestion pipeline for chunking and embedding.

Yes. You set the crawl boundary (start path, domain, or subdomains), include and exclude path patterns, a page limit, and a maximum link depth.

No. The crawl runs inside Connic on your schedule. There is nothing to host, and JavaScript-rendered pages are handled by the crawling engine.

On the schedule you pick, from every 12 hours up to weekly. Re-crawls only reprocess pages whose content changed, so frequent schedules stay cheap.

Need Website Crawler content behind a production agent?

Tell us where your content lives, how fresh it needs to be, and who is allowed to see it. We will map that to source scopes, namespaces, and the retrieval setup your agents need.

Talk to Sales

More retrieval sources

Browse the marketplace

Confluence

Turn Confluence spaces into retrievable agent knowledge

Superhuman Docs (Coda)

Keep Superhuman Docs and tables in sync with your agents

Notion

Sync Notion pages and databases into your agents' retrieval