Robots.txt in 2026: how to create, test, and avoid breaking indexation
A practical robots.txt guide: what to block, what not to block, why robots does not remove pages from Google, and how to connect it with sitemap.
Robots.txt decision tree
Robots manages crawling. It is not a privacy or deindexing tool.
Robots.txt is a small file with large risk. One line can block important pages from crawling, while another can leave filters, parameters, and technical folders open for crawl waste.
The main rule: robots.txt controls crawling, not guaranteed removal from search. Google warns that robots.txt should not be used as a way to hide web pages from search results.
What robots.txt is good for
| Job | Use robots.txt? | Note |
|---|---|---|
| Reduce crawl waste | yes | filters, parameters, technical folders |
| Block CSS/JS | usually no | Google should understand the page |
| Remove a page from index | no | use noindex or remove the URL |
| Point to sitemap | yes | useful for search engines |
| Hide private data | no | use authentication, not robots |
Basic example
User-agent: *
Disallow: /wp-admin/
Disallow: /*?sort=
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xml
Before publishing, check that you are not blocking service pages, blog pages, images, CSS, or JavaScript needed for rendering.
Common competitor mistakes
- blocking the entire site with
Disallow: /; - blocking theme resources and hurting rendering;
- blocking URLs and expecting them to disappear from the index;
- forgetting sitemap;
- not checking robots after redesign or migration.
Next step: if you are unsure, start with a technical audit, check the site in UNmiss, and add sem.chat so users do not get stuck after search visits.
Sources
Merchant Center for AI Mode: Getting Your Product Feed Ready for Conversational Shopping in 2026
Google AI Mode is reshaping product search and a plain feed no longer cuts it. Here are the new Merchant Center attributes, why stores get banned, and where to start.
Read →
Image SEO Optimization: alt, Dimensions and Weight for Core Web Vitals
How to optimize images for SEO in 2026: why alt text matters, how width and height stop layout shifts (CLS), which formats (WebP, AVIF) to use, and how not to break LCP with lazy loading. A guide from SEOquick.
Read →
Search Everywhere Optimization: Ranking on TikTok, YouTube, Reddit and AI Search
Your customer does not only search on Google: 49% of consumers search on TikTok, Reddit pulls 842M organic clicks a month, and YouTube is the source of roughly one in four AI Overview citations. Here is how a business owner picks 2–3 platforms without burning the budget.
Read →Want to apply this to your site?
We will review the current situation, find the first growth levers, and suggest a practical working format.
