Robots.txt in 2026: how to create, test, and avoid breaking indexation
A practical robots.txt guide: what to block, what not to block, why robots does not remove pages from Google, and how to connect it with sitemap.
Robots.txt decision tree
Robots manages crawling. It is not a privacy or deindexing tool.
Robots.txt is a small file with large risk. One line can block important pages from crawling, while another can leave filters, parameters, and technical folders open for crawl waste.
The main rule: robots.txt controls crawling, not guaranteed removal from search. Google warns that robots.txt should not be used as a way to hide web pages from search results.
What robots.txt is good for
| Job | Use robots.txt? | Note |
|---|---|---|
| Reduce crawl waste | yes | filters, parameters, technical folders |
| Block CSS/JS | usually no | Google should understand the page |
| Remove a page from index | no | use noindex or remove the URL |
| Point to sitemap | yes | useful for search engines |
| Hide private data | no | use authentication, not robots |
Basic example
User-agent: *
Disallow: /wp-admin/
Disallow: /*?sort=
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xml
Before publishing, check that you are not blocking service pages, blog pages, images, CSS, or JavaScript needed for rendering.
Common competitor mistakes
- blocking the entire site with
Disallow: /; - blocking theme resources and hurting rendering;
- blocking URLs and expecting them to disappear from the index;
- forgetting sitemap;
- not checking robots after redesign or migration.
Next step: if you are unsure, start with a technical audit, check the site in UNmiss, and add sem.chat so users do not get stuck after search visits.
Sources
Pagination SEO: how to index list pages without breeding duplicates
Pagination SEO in 2026: canonicals on list pages, noindex, rel=next/prev, infinite scroll, crawl budget. Based on a check of 36,000 traffic-driving URLs across twelve Ukrainian retailers, plus a teardown of pagination at Foxtrot, Epicentr and our own blog.
Read →
URL Structure: The Full Guide to Addresses You Can't Change After Launch
How to get URLs right: transliteration or Cyrillic, slug length, duplicates from case and filters, pagination, UTM tags, redirect maps. A breakdown of Rozetka, Comfy, Prom and Makeup URLs plus 2026 research data.
Read →
Why AI Cites Your Site But Never Names Your Brand
I analysed 751 Google AI Overview answers for my own domain and found the factor that decides whether your brand gets named: where your link sits in the source list. Plus six prompts to audit your brand in ten minutes.
Read →Want to apply this to your site?
We will review the current situation, find the first growth levers, and suggest a practical working format.
