Robots.txtBy Umer Farooq / November 10, 2024 A file that tells search engines which parts of a website to crawl and index.