Site audits: crawling a whole website

A site audit starts at a website’s homepage and crawls it page by page, following internal links and the sitemap, up to your plan’s page limit. Start one on Site audits.

How it crawls

• It identifies itself as SemhackBot, obeys robots.txt rules (anything disallowed for Googlebot is skipped too) and honors Crawl-delay up to 10 seconds.
• It makes at most two requests at a time, with a pause after each, so it doesn’t strain the site.
• Pages blocked by robots.txt are listed but not fetched, and don’t count toward the page limit.
• If the worker restarts, the crawl continues where it stopped. You can stop a crawl at any time.

What it finds

Every crawled page gets the same 40 checks as a single page audit, plus issues that only show across a site: broken internal links, redirect chains, duplicate titles and descriptions, possible soft 404s, orphan pages in the sitemap, noindex pages in the sitemap, links blocked by robots.txt, slow pages (over 3 seconds), thin content (under 300 words), pages more than 4 clicks from the homepage, and more.

Issues are grouped by severity (errors, warnings, notices), each with the affected pages and the pages that link to them. You can filter and search the crawled pages and export pages and issues as CSV. A new crawl of the same site shows what changed since the last one.