AI Crawlers and robots.txt: GPTBot, ClaudeBot, Google-Extended and a Policy You Can Defend
Which AI bots train on your pages, which cite them and which fetch on demand, plus robots.txt templates that block training, keep citations and survive verification.