llms.txt is an emerging convention — a plain-text file at your domain root — that gives AI tools a clean, curated overview of your site’s key content. Alongside it, robots.txt directives let you control which AI crawlers may access your site, distinguishing answer-engine bots (like GPTBot, ClaudeBot, and PerplexityBot) from training-only crawlers. Together they let a business invite the AI engines it wants to be cited by while declining the ones it does not.
robots.txt can explicitly allow answer-engine crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended) while blocking training-only bots, and it can point to an llms.txt file that gives AI tools a curated map of your key content. This lets a business be discoverable and citable by the AI engines that drive referrals without surrendering its content to every crawler indiscriminately.
What llms.txt does
llms.txt is a curated, machine-readable summary of your most important pages and context, placed at your domain root. It helps AI tools understand your site quickly and accurately.
It is a young convention, but cheap to publish and aligned with where AI discovery is heading.
Controlling crawlers with robots.txt
robots.txt lets you allow or disallow specific user agents. You can welcome answer-engine bots that drive citations and referrals while blocking crawlers that only harvest training data.
This is a deliberate choice: visibility in AI answers versus control over how your content is reused.
A sensible default for service businesses
Most local businesses want to be recommended by ChatGPT, Perplexity, and Google’s AI — so they allow those crawlers and publish an llms.txt.
Blocking the answer engines that send customers would be self-defeating; the nuance is in how you treat training-only bots.
Key takeaways
- llms.txt gives AI tools a curated map of your key content.
- robots.txt controls which AI crawlers can access your site.
- Allow answer-engine bots; decide separately on training-only bots.
- Most local businesses should stay open to citation-driving engines.
Common questions
Related guides
Related services
Want this done for your business?
I build fast, schema-rich websites for Massachusetts service businesses — engineered for local and AI search from the first line of code.