Site Audit: scan many pages of one site for SEO issues
Enter a site and we list its pages from the sitemap and the homepage links. Pick up to 8, run the full SEO report on each, and see which issues show up on one page and which repeat across the site.
We read robots.txt, the sitemap, and the homepage as HelloSEOToolsBot. Public hostnames only. Nothing is stored beyond a 5 minute cache.
How it works
- Discovery. We read the Sitemap lines in robots.txt and /sitemap.xml. If the sitemap is an index, we open at most 2 of its child sitemaps. Then we add the internal links found on the homepage. Up to 40 pages are listed: the homepage first, then sitemap order, then link order.
- Cleanup. Only pages on the same host as the homepage are kept. Query strings and fragments are removed, a trailing index.html is removed, duplicates are merged, and files such as PDFs, images, feeds, scripts, and archives are skipped.
- robots.txt is respected. Any page that robots.txt disallows for all crawlers is left out of the list, and the count of skipped pages is shown.
- The scan. Each page you select is checked with the same full SEO report, two pages at a time. Findings are then grouped by check, so an issue that appears on 6 of 8 pages shows up once with the 6 pages listed under it.
- A sample of 8. The free run scans up to 8 pages. Pages that share a template usually share issues, so choose one page of each kind.
Limitations
- No JavaScript rendering. Links that only appear after scripts run are not found, and each page is checked as the server sends it.
- Discovery reads at most 200 KB per sitemap, 2 child sitemaps from a sitemap index, and the links on the homepage only. Compressed sitemaps are not read. It stops after 15 seconds.
- Pages on other hosts, including www versus the bare domain and subdomains, are not included. Enter the host you want to audit.
- The report API allows 10 reports per hour per visitor. If the limit is reached during a run, the scan stops and keeps the pages already done. Pages scanned in the last 5 minutes are served from cache.
- Scores describe what we could fetch and check. They are not a measure of rankings or traffic.
Frequently asked questions
- Why does the site audit scan only 8 pages?
- Each page is a full report: the page itself, its robots.txt, its links, its resources, and more. The free report API allows 10 reports per hour per visitor, so 8 pages per run leaves room to re-check a page after a fix. Larger audits will come with accounts.
- How does the tool decide which pages to list?
- It reads the Sitemap lines in robots.txt and /sitemap.xml, following a sitemap index into at most 2 child sitemaps, then adds internal links from the homepage. It keeps only pages on the same host, removes query strings and fragments, skips files such as PDFs and images, and drops anything robots.txt disallows. The homepage is listed first, then sitemap order, then link order, up to 40 pages.
- Which pages should I pick?
- Pick one page of each kind: the homepage, a product or service page, a category or listing page, a blog post, and a contact or about page. Pages built from the same template usually share the same issues, so a sample across templates tells you more than 8 pages of the same type.
API
Page discovery is available as JSON, free, no key required, cached for 5 minutes:
GET https://helloseotools.com/api/site-urls?url=https://example.com/
Response: { ok, origin, urls[], total, sources: { sitemap, links }, skippedByRobots, sitemapFound, checkedAt }. Errors return { ok: false, error }. Each page can then be checked with /api/report?url=.
Related tools
- Full SEO Report: one page, an overall score, six category grades, and every fix with its evidence.
- Indexability Checker: whether Google can crawl and index a URL, and the exact rule that blocks it.
- Redirect Checker: status codes and redirect chains for up to 100 URLs at once.