Skip to content

Website Crawler

Use Website Crawler when a maintained website is the reference source and your Agent should search selected public pages without uploading each page by hand.

The goal is not to copy an entire site. It is to create a bounded, reviewable Knowledge Base folder for the questions the Agent is approved to answer.

Before you crawl

Confirm that:

  • your team is allowed to collect and use the pages
  • the pages do not require a sign-in or contain private customer data
  • the site section has a clear owner and update process
  • the content supports a verified scenario for this Agent

Do not use a broad crawl to compensate for unclear scope or outdated source material.

Create a website sync folder

  1. Open Knowledge Base and select the workspace.
  2. Choose New Website Crawl.
  3. Enter the Start URL.
  4. Set the crawl boundary, then save.
  5. Review the credit estimate and start the sync.

The crawler creates a dedicated folder. Connect that folder to the Agent only after the relevant pages finish processing.

Set a boundary you can explain

Setting What it controls Practical use
Max depth How many link levels the crawler may follow from the start URL Keep it low for one documentation or policy section.
Page limit The maximum pages requested from the crawler provider Use it as a cost and review boundary, not an exact final page count.
Include path Paths that may be collected Limit the crawl to sections such as /docs/* or /help/*.
Exclude path Paths that must stay out Exclude login, account, search, archive, or duplicate locale paths.
Allow subdomains Whether links to subdomains may be followed Leave it off unless the approved source spans subdomains.
Ignore query parameters Whether URL variants such as tracking parameters are treated as the same page Turn it on when parameters create duplicates.
Ignore sitemap Whether the crawler should avoid sitemap discovery Use only when the sitemap is stale or expands beyond the intended scope.
Sync main content only Whether navigation and repeated page chrome should be reduced Keep it on when menus and footers add retrieval noise.

Comma-separate multiple path patterns. * matches any text; a path without * matches that path and its child pages.

Understand credits before syncing

Starting a sync reserves credits based on the page limit and the displayed per-page rate. When the job finishes, Codeer charges only successfully downloaded pages and returns unused reserved credits.

The provider's final discovered or downloaded count can differ slightly from the page limit because of redirects, concurrency, or sitemap handling. Lower the page limit or narrow the paths if the estimate is too high.

Confirm the result before using it

The sync panel separates two stages:

  • Page Fetching: pages discovered, downloaded, or download-failed
  • Knowledge Processing: pages processed, failed, or skipped

Open Failure details when either stage reports a problem. Copy or open the affected URL, decide whether the page belongs in scope, then correct the boundary or source before syncing again.

A downloaded page is not ready for the Agent until knowledge processing succeeds.

Connect and test the folder

  1. Add the crawler folder to the Agent's Knowledge Base tool.
  2. Write a When to Use rule that names the questions this website should answer.
  3. Test one expected question, one nearby boundary, and one question whose answer should come from another source.
  4. Verify that the Agent retrieves the intended page and does not treat the whole website as authority for unrelated topics.

Website sync is operator-controlled. When important pages change, run another sync and retest the affected cases rather than assuming the connected content is already current.