Robots.txt or sitemap: which one tells crawlers what?
Choose the right control for blocking paths, advertising URLs, or mapping old addresses to new ones.
A sitemap is an invitation and robots.txt is a restriction, while a redirect is a forwarding instruction. They solve different problems, and using a sitemap to hide a page or robots.txt to remove one from the index usually produces the opposite of the intended result.
Use robots.txt to restrict crawling, not to hide pages
The robots generator drafts crawler rules and warns about conflicts between groups. Remember that robots.txt controls crawling, not indexing: a blocked URL can still be indexed if other pages link to it, so removal needs a noindex response or an actual deletion.
Use a sitemap to list canonical URLs
The sitemap validator checks structure, locations and size limits without fetching external sites. A sitemap should contain final, indexable, canonical addresses only; including redirects or noindex URLs sends mixed signals.
Use redirects when an address has moved
The redirect rule generator writes Apache, Nginx, Netlify and Vercel snippets for permanent or temporary moves. Prefer one-hop permanent redirects for content that has moved, and keep the old path out of the sitemap once it redirects.
Tools covered in this guide
- Robots.txt Generator
Draft crawler rules with conflict warnings, locally and safely.
- XML Sitemap Validator
Check sitemap structure, locations, and limits without fetching.
- Redirect Rule Generator
Generate redirect snippets for Apache, Nginx, Netlify or Vercel.
Frequently asked questions about crawler directives
Does blocking a URL in robots.txt remove it from search?
No. Blocking stops crawling, so search engines may keep the URL indexed without reading the page. Use noindex or remove the page to take it out of results.
Should redirects appear in the sitemap?
No. List only final URLs; a sitemap full of redirects wastes crawl attention and weakens the signal about which address is canonical.
Can robots.txt and a sitemap contradict each other?
Yes, and that is a common mistake: a sitemap lists a path that robots.txt blocks. The generator warns about rule conflicts, but always compare it with the sitemap by hand.
More guides
- Canonical or hreflang: which duplicate problem are you fixing?
- Basic meta, Open Graph, or Twitter Card: which tags do you need?
- Bold, italic, cursive, or blackletter: which Unicode style?
- Check, find, or build a palindrome?
- Clean messy text in the right order: spaces, duplicates, accents
- Compare, replace, or filter text: which fixes the difference?