Crawl entry points · robots.txt and Sitemap configuration

robots.txt / Sitemap check

Check robots.txt rules plus the availability and basic structure of Sitemap and Sitemap Index data to find site-wide crawl blocks, cross-host URLs, invalid addresses, and sitemap configuration problems.

robots.txtCheck HTTP status, rule count, site-wide Disallow, and Sitemap declarations.
SitemapCheck sitemap.xml / Sitemap Index, a bounded set of child files, URL count, and cross-host or invalid URLs.
Safety boundaryReuse the unified outbound-safety policy, same-host restrictions, and fixed check limits.
Check boundaryPages listed in the Sitemap are not probed one by one; there is no recursive crawl or full-site/bulk scan.