Two small files sit at the root of nearly every serious website: robots.txt and an XML sitemap. They’re easy to ignore until something goes wrong - Google missing half your pages, or indexing a staging path you meant to keep private. Both files are simple to read, but the assumptions people make about what they do are usually wrong in one direction or the other, which is exactly why they cause so much confusion during audits and migrations.
What robots.txt actually controls
robots.txt is a plain-text file at yourdomain.com/robots.txt that tells well-behaved crawlers which paths they’re allowed to request. It uses simple Allow and Disallow rules grouped by user-agent, so you can give different instructions to Googlebot, Bingbot, or specific bots you don’t want crawling at all. It’s a request, not an enforcement mechanism - reputable search engines respect it, but nothing stops a script from ignoring it entirely.
What robots.txt does not do
This is where most confusion starts. A Disallow rule stops crawling, not indexing. If other sites link to a disallowed URL, Google can still show that URL in results using the anchor text from those links, just without a proper description - because it was never allowed to fetch the page. robots.txt is also not a security boundary: anyone can read the file, so blocking /admin/ there just tells attackers where to look, and sensitive paths still need real authentication.
- Believing Disallow removes a page from the index (it doesn’t - use noindex instead).
- Blocking CSS or JavaScript folders, which stops Google from rendering the page properly.
- Leaving a blanket “Disallow: /” in place after moving from staging to production.
Testing a change before it goes live matters more than people expect, because a single misplaced slash in a Disallow rule can block an entire section by accident. Google retired its standalone robots.txt Tester, but URL Inspection in Search Console will confirm whether a specific page is currently blocked, and most third-party crawling tools (Screaming Frog and similar) can simulate a full crawl against a staged robots.txt file before it ever reaches production.
What an XML sitemap actually does
A sitemap is an XML file listing the URLs you want search engines to know about, optionally with metadata like last-modified dates. It speeds up discovery, which matters most for large sites, brand-new sites with few external links, or sites that publish frequently. Submitting it in Search Console gives Google a clear list to check against what it has already crawled and indexed.
What a sitemap does not guarantee
Being listed in a sitemap is a hint, not an instruction - Google can still decide not to index a URL if it looks thin, duplicate, or low quality. Stuffing a sitemap with every possible URL variant, filtered category pages, or redirected old URLs can waste the trust Google places in the file and slow down discovery of the pages that matter. A clean sitemap of canonical, indexable URLs works better than a large, messy one.
Sitemap size limits and splitting large catalogues
A single sitemap file is capped at 50,000 URLs and 50MB uncompressed. E-commerce catalogues, real estate listings, or content-heavy publishers that exceed this simply split their URLs across several sitemap files (often by section - products, categories, blog posts) and reference all of them from one sitemap index file. This is a plumbing detail rather than a redesign, and most frameworks and CMS platforms generate the index automatically once the URL count grows past the limit.
Small marketing sites usually ship a static robots.txt file that rarely changes. Larger platforms - marketplaces, multi-tenant SaaS products, sites with user-generated content - often generate it dynamically per environment, so staging, preview, and production each get the correct rules automatically instead of relying on someone remembering to swap a file. If your platform already generates the sitemap dynamically from the database, it’s worth checking whether robots.txt should follow the same pattern rather than living as a static file that quietly drifts out of date.
A real example: blocking the wrong thing after a redesign
A common failure mode after a site redesign: the new robots.txt is copied from the staging environment, which correctly disallowed everything so search engines wouldn’t index an unfinished build. Nobody swaps it out for a production version before launch. Weeks later, Search Console starts flagging “Indexed, though blocked by robots.txt”, or crawl stats drop to near zero. The sitemap might still be perfectly fine and submitted - but with the whole site disallowed, it doesn’t matter.
Practical setup that avoids most problems
- Keep robots.txt minimal: allow public content, and only disallow true junk like admin panels, internal search results, or cart internals.
- Never disallow CSS, JavaScript, or image folders that the page needs to render correctly.
- Generate a sitemap of canonical, indexable URLs only, using one consistent host (https, with or without www) and no redirected or parameterised URLs.
- Reference the sitemap’s full URL inside robots.txt (Sitemap: https://yourdomain.com/sitemap.xml) and also submit it directly in Search Console.
- Re-check both files immediately after every launch, redesign, or platform migration - stale Disallow rules from staging are the single most common cause of sudden traffic loss.
At Killer Click we wire sitemap generation into the site build itself where possible, so it updates automatically as pages are added, and we verify robots.txt manually before every launch instead of trusting whatever the previous platform, agency, or default CMS install happened to leave behind.