Technical SEO is about making sure search engines can find, crawl and understand your pages. The tools in this section check the files and server responses that control crawling: your robots.txt file, your XML sitemaps, your redirects and the links between your pages.
Which tool to use
- Robots.txt Checker reviews your robots.txt file and the rules it sets for crawlers. Use it to catch rules that block important pages or crawlers by mistake. Keep in mind that robots.txt controls crawling, not indexing. Google notes that it is not a mechanism for keeping a web page out of Google.
- Sitemap Checker checks your XML sitemap, so you can confirm it loads and lists the pages you want search engines to find.
- Redirect Checker shows where a URL redirects and the status codes along the way. Use it to find redirect chains, loops and temporary redirects that should be permanent.
- Broken Link Checker finds links that lead to errors, so visitors and crawlers do not hit dead ends.
Common workflows
Before a launch or migration. Check robots.txt to make sure the live site is not still blocking crawlers from a staging setup, confirm the sitemap lists the new URLs, and test that old URLs redirect to the right new pages.
When a page drops out of search. Look for a blocking rule in robots.txt, confirm the page is in your sitemap, and run the SEO Checker on the page to look for a noindex directive.
Routine maintenance. Run the Broken Link Checker on your important pages, and fix or remove dead links.
For crawler rules that affect AI systems, see our guide on how to allow or block AI crawlers.
Frequently asked questions
Does blocking a page in robots.txt remove it from Google?
Not reliably. Robots.txt controls crawling. To keep a page out of Google's index, Google recommends other methods, such as a noindex rule or password protection.
How often should I check these files?
After any change to your site's structure, platform or hosting, and as part of a regular SEO audit.
Are redirects bad for SEO?
No. Redirects are the right way to move pages. Problems come from long chains, loops and redirects that point to the wrong page, which is what the Redirect Checker helps you find.