AI Crawler

An AI crawler is an automated program that browses web pages to gather content used to train or feed AI models and answer engines, such as OpenAI's GPTBot or Anthropic's ClaudeBot. Allowing or blocking these crawlers determines whether a brand's content can be read and then drawn upon in generative answers.