CRAWLING
Robots.txt made simple
Robots.txt gives crawlers instructions about which URL paths they should or should not crawl.
It controls crawling, not guaranteed indexing. A blocked URL can still sometimes appear in search if other pages link to it.
Don't block assets search engines need. Blocking CSS or JavaScript can make pages harder to understand.
Use Disallow carefully. A broad rule can accidentally block an entire section of a site.
Add your sitemap. A Sitemap line can help crawlers discover the preferred XML sitemap location.