Decision

Should I block AI crawlers in robots.txt?

The publisher answer and the business answer are different. Most companies asking this are businesses, and for them the maths is not close.

By Robin OoiPublished 2026-04-16 Last reviewed 2026-08-266 min read

The short answer

For almost every business that sells something other than content, no. Blocking removes you from the generated answer without returning the click, because the answer still gets written from whoever else is available. The argument only genuinely applies to publishers whose revenue is page views. Far more common than a deliberate decision is an accidental block sitting in a robots.txt nobody has read in three years, which is worth checking today.

This question gets argued badly, usually by people with a position to defend. The publisher answer and the business answer are genuinely different, and most Malaysian companies asking it are businesses, not publishers.

Who is asking, and why it changes the answer

Whether to block AI crawlers, by business model
You areRecommendationReasoning
A service businessDo not blockYour content is marketing. Being quoted is the point.
A local or B2B companyDo not blockYou need to be in the shortlist. Blocking removes you from it.
An e-commerce retailerDo not blockProduct visibility inside answers is becoming a real channel.
A publisher selling ads or subscriptionsGenuinely arguableYour content is the product and the click is the revenue.
Holding proprietary researchSelective blockingBlock the specific paths, not the whole domain.

The argument for blocking, stated fairly

Your content took time and money. A model reads it, restates the substance, and the reader never arrives. You bore the cost and someone else captured the value. That is a real grievance, and for a business whose revenue is page views it is not a philosophical objection, it is an income statement.

Why it is usually the wrong trade anyway

Blocking removes you from the answer. It does not return the click. The answer still gets written, from whichever sources remain available. You have not created scarcity, you have created absence, and your competitor fills the space.

For a business that sells something other than content, the exchange is heavily in your favour. You give up traffic you were largely losing anyway on informational queries, and you gain the chance to be the named recommendation on commercial ones. That second thing is worth considerably more per instance.

The mistake I see most often

Not a deliberate decision. An accident.

A developer copied a robots.txt from a template, or a security plugin added agent blocks by default, or someone blocked bots during a scraping incident three years ago and never removed it. Nobody at the company chose this and nobody knows it is there.

Go and read yours right now at yourdomain.com/robots.txt. Look for these agent names:

  • GPTBot and OAI-SearchBot
  • ClaudeBot and Claude-User
  • PerplexityBot
  • Google-Extended
  • CCBot
  • Applebot-Extended

If any of them has a Disallow: / under it, you are invisible to that system by instruction rather than by competition. The free scan checks this automatically, because it is one of the highest-impact things on the list and one of the easiest to miss.

A note on Google-Extended specifically

Google-Extended is the control for whether your content can be used in Gemini and certain AI features. It is separate from normal Googlebot crawling, so blocking it does not remove you from Google Search. That separation confuses people in both directions. Some block it thinking it protects rankings. Others avoid blocking it thinking it would hurt rankings. Neither is correct.

The middle path

If you have genuinely valuable proprietary material, block the paths that contain it rather than the domain. Allow crawling of your marketing pages, your service pages and your public guides. Disallow the members area, the downloadable research, the client portal.

That gets you visibility where visibility is the goal and protection where protection is the goal, which is what you actually wanted.

What to do this week

  1. Read your robots.txt and note any AI agent blocks.
  2. Decide, deliberately, whether each one is intended.
  3. Remove the accidental ones.
  4. Add an explicit Allow: / for the agents you want, and point at your sitemap.
  5. Consider publishing an llms.txt while you are in there.

Want to know where your own site stands?

The free scan scores your top five pages and shows the gaps behind the number. About ten seconds, no email required.

Run the free scan Book a call

Reference

Related questions

Does blocking Google-Extended affect my Google rankings?
No. Google-Extended controls whether your content can be used in Gemini and certain AI features. It is separate from Googlebot, which handles Search crawling. Blocking one does not affect the other.
If I block AI crawlers, will they respect it?
The major named agents generally honour robots.txt. Smaller and less scrupulous scrapers may not, and robots.txt is a request rather than an enforcement mechanism. If you need actual enforcement you need server-level blocking, which is a different piece of work.
Can I allow some AI crawlers and block others?
Yes. robots.txt takes per-agent rules, so you can allow the ones whose answers you want to appear in and block the rest. You can also allow crawling of marketing pages while disallowing specific paths that hold proprietary material.
How do I check what my robots.txt currently says?
Visit yourdomain.com/robots.txt in a browser. Look for any AI agent name followed by Disallow: /. The free scan also checks this and flags it as a failed check, because it is one of the highest-impact issues on the list.

Read next

WhatsApp Robin