Crawl Pipeline Overview

Crawl Pipeline Overview

The Answer Engine grounds its answers in your site's real content. That content gets there via a four-phase crawl pipeline.

Phases

1. Pending — crawl queued, not started
2. Running — scraping pages (handled by Firecrawl, batched 12 URLs at a time)
3. Embedding — chunking scraped pages and computing embeddings
4. Completed — answer engine ready

If something goes wrong: Failed with a classified reason (rate limit, auth, network, etc.).

How content stays fresh

• Every page is hashed (SHA-256). On recrawl, unchanged pages are skipped — fast and cheap.
• Scheduled refresh runs weekly per site
• Targeted refresh: re-crawl a single URL on demand from the site detail page

Live progress

While crawling, the site detail page shows:

• Pages crawled / total discovered
• Embedded chunks / total chunks
• Current phase and last activity timestamp

If a crawl stalls (no progress for 3+ minutes), a Resume button appears.

Limits and respectfulness

• We honor robots.txt
• One concurrent worker per site
• 30-minute stall watchdog auto-recovers stuck crawls

What gets indexed

Public HTML pages, including main content, headings, and visible text. We skip nav/footer chrome, hidden elements, and any URL the site excludes via robots.txt.

Related

• Answer Engine Setup
• Insights Audit Overview