Don't block the bots you want to be cited by
Somewhere in your robots.txt there may be a single line quietly disinviting the exact crawlers that decide whether ChatGPT, Claude, and Perplexity ever mention you.
A few months ago a client asked why they never appeared in AI answers about their category. Good content, clean site, real authority. We opened their robots.txt and found it: a line, copied from some defensive template, disallowing GPTBot. They had locked the front door on the way out and forgotten.
The reflex that costs you
For years the instinct was to keep crawlers out — protect the content, save the bandwidth, guard against scraping. That reflex made sense in an era when crawling fed only search results you might or might not win.
It makes far less sense now. AI referral traffic rose 357% year over year. When someone asks an assistant for the best boutique hotel in a neighbourhood, the answer is assembled from sources the model was allowed to read. Disallow the bot and you are not in the running — not ranked low, just absent.
You cannot be quoted by a reader you refused to let in.
Decide on purpose, not by default
This isn't an argument to fling every door open. It's an argument to make the choice deliberately. Know which crawlers matter — GPTBot, ClaudeBot, PerplexityBot among them — and decide, brand by brand, whether you want to be readable by the systems your guests now ask first.
For a resort betting on direct bookings, being absent from AI travel answers is a leak as real as any OTA. We audited their crawler policy the same week we touched their booking flow. Both were visibility problems.
Check your robots.txt today. The most expensive line in it might be the one nobody remembers adding.