Automated Website Content Crawling
Instead of copying and pasting web pages manually, ChatEaver features an automated web crawler that parses public website URLs and XML sitemaps, extracting clean text from product pages, documentation, and blog posts.
Step-by-Step Web Import Guide
Step 1: Open Dashboard > Knowledge (/dashboard/knowledge).
Step 2: Click '+ Add Source' and select 'Website Crawl'.
Step 3: Enter your Target URL (e.g. 'https://example.com/pricing') or Sitemap URL ('https://example.com/sitemap.xml').
Step 4: Configure Crawl Depth — Choose Single Page, Subdirectory, or Full Domain Crawl.
Step 5: Set Exclusions — Add URL patterns to skip (e.g. '/login/*', '/cart/*').
Step 6: Click 'Start Crawling'. Watch the progress indicator as pages are processed.
Step 7: Review the imported pages in your source list and inspect captured text preview.
Handling Web Ingestion Errors
Error Symptom | Possible Cause | Recommended Resolution |
|---|---|---|
403 Forbidden | Website firewall blocking crawler agent | Whitelist ChatEaver crawler IP or use sitemap import. |
Empty Content | Page requires JavaScript rendering | Ensure static HTML content is served or upload page PDF. |
Duplicate Pages | Multiple URLs pointing to identical content | Set canonical link filters or exclude query parameters. |