robots.txt

The robots.txt file is a text file placed at a site's root that tells crawlers which areas they may or may not browse. It is the first checkpoint governing AI crawlers' access to a catalog: a misconfigured rule can accidentally block the engines that feed generative shopping answers.