Robots.txt in 2026: how to create, test, and avoid breaking indexation
A practical robots.txt guide: what to block, what not to block, why robots does not remove pages from Google, and how to connect it with sitemap.
Robots.txt decision tree
Robots manages crawling. It is not a privacy or deindexing tool.
Robots.txt is a small file with large risk. One line can block important pages from crawling, while another can leave filters, parameters, and technical folders open for crawl waste.
The main rule: robots.txt controls crawling, not guaranteed removal from search. Google warns that robots.txt should not be used as a way to hide web pages from search results.
What robots.txt is good for
| Job | Use robots.txt? | Note |
|---|---|---|
| Reduce crawl waste | yes | filters, parameters, technical folders |
| Block CSS/JS | usually no | Google should understand the page |
| Remove a page from index | no | use noindex or remove the URL |
| Point to sitemap | yes | useful for search engines |
| Hide private data | no | use authentication, not robots |
Basic example
User-agent: *
Disallow: /wp-admin/
Disallow: /*?sort=
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xml
Before publishing, check that you are not blocking service pages, blog pages, images, CSS, or JavaScript needed for rendering.
Common competitor mistakes
- blocking the entire site with
Disallow: /; - blocking theme resources and hurting rendering;
- blocking URLs and expecting them to disappear from the index;
- forgetting sitemap;
- not checking robots after redesign or migration.
Next step: if you are unsure, start with a technical audit, check the site in UNmiss, and add sem.chat so users do not get stuck after search visits.
Sources
Google AI Max in 2026: Should You Switch From Performance Max
What Google AI Max is, how it differs from Performance Max, and whether you should switch in 2026. Google's real numbers vs independent data, brand and geo controls, DSA migration. A breakdown by SEOquick.
Read →
Perplexity Optimization in 2026: How to Earn Citations via Reddit, YouTube and Review Sites
Perplexity sends little traffic, but it converts far better than Google. Here's where Perplexity pulls its sources and how a business owner can earn citations via Reddit, YouTube and G2. SEOquick experience.
Read →
TikTok SEO in 2026: How to Rank in TikTok Search and Get Into Google
TikTok is now a second search engine: 49% of users search on it, 65% of Gen Z. Here is how TikTok indexes video (ASR, OCR, captions), how to optimize clips for search, and how TikTok content also surfaces in Google — with 2026 data and SEOquick experience.
Read →Want to apply this to your site?
We will review the current situation, find the first growth levers, and suggest a practical working format.
