On October 15, 2026, Cloudflare and OpenAI shared early results from a research pilot that helped OpenAI find and index hundreds of millions of new web pages for ChatGPT search.
For a ChatGPT search law firm website plan, discovery is only half of it. A firm also has to let the right OpenAI crawler in. A plugin or CDN setting that blocks “AI bots” in one sweep may block ChatGPT search too.
ChatGPT search relies on OpenAI’s OAI-SearchBot crawler, which a site controls in robots.txt separately from GPTBot, the crawler used for model training.
A law firm can allow OAI-SearchBot so its pages can appear in ChatGPT search answers while still blocking GPTBot.
What the Cloudflare and OpenAI Pilot Found
Cloudflare and OpenAI started the pilot in July 2026 to test whether signals from Cloudflare’s network could help ChatGPT users find new content. The pilot used a cohort of about 1% of Cloudflare’s network, and it helped OpenAI discover and index hundreds of millions of new pages, according to the Cloudflare and OpenAI post on AI discovery.
Two details matter for law firms. OpenAI still evaluates and ranks those pages on its own, so the pilot is about finding pages, not ranking them. And the post says a signal about a page “doesn’t grant permission to access it.” The site owner still decides that.
The Three OpenAI Crawlers a Law Firm Should Know
OpenAI runs separate crawlers for separate jobs, and each one is set on its own, according to OpenAI’s overview of its crawlers.
OAI-SearchBot: ChatGPT search
OAI-SearchBot surfaces websites in ChatGPT’s search features. OpenAI says sites that opt out of it “will not be shown in ChatGPT search answers, though can still appear as navigational links.” For a firm that wants its pages cited in ChatGPT answers, this is the crawler that counts.
GPTBot: model training
GPTBot crawls content that may be used to train OpenAI’s generative AI foundation models. Blocking it tells OpenAI the site’s content should not be used for that training. Blocking GPTBot does not remove the site from ChatGPT search.
ChatGPT-User: actions a user asks for
ChatGPT-User visits a page when a person asks ChatGPT to do something that needs it. It does not crawl the web on its own. Because a user starts these visits, OpenAI says robots.txt rules “may not apply,” and it says ChatGPT-User is not used to decide whether content appears in search.
Three Robots.txt Setups a Law Firm Can Choose From
A firm’s robots.txt file sets this in a few lines. OpenAI says its search systems take about 24 hours to adjust after a change.
Allow ChatGPT search, block training. This fits most firms that want visibility but have doubts about training.
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /Allow both. Change GPTBot’s line to Allow: /. This fits firms comfortable with training use.
Block both. Change OAI-SearchBot’s line to Disallow: / as well. The firm’s pages will then not be shown in ChatGPT search answers.
How to Check What a Law Firm Site Blocks Today
A quick check takes a few minutes and should come before any change.
- Open the site’s robots.txt file at the root of the domain and look for OAI-SearchBot and GPTBot by name.
- Look for broad AI blocking rules added by a security plugin, a hosting dashboard or a CDN, which may not show up in robots.txt at all.
- Search the server logs for the OAI-SearchBot user agent to see whether it is reaching the site.
- Compare visiting addresses with the IP ranges OpenAI publishes for each crawler.
Cloudflare customers can also use Cloudflare’s AI Crawl Control to manage AI access.
Frequently Asked Questions
Blocking GPTBot does not keep a law firm out of ChatGPT search. OpenAI says GPTBot and OAI-SearchBot are set independently. GPTBot covers model training, while OAI-SearchBot decides whether a site can be shown in ChatGPT search answers, so a firm can block one and allow the other.
A robots.txt change takes about 24 hours to affect ChatGPT search, according to OpenAI. Its crawler documentation says it can take roughly that long after a site’s robots.txt update for OpenAI’s systems to adjust search results.
ChatGPT-User may not follow a law firm’s robots.txt. OpenAI says these visits are started by a user’s request in ChatGPT, so robots.txt rules may not apply. OpenAI also says ChatGPT-User is not used to decide whether content appears in search.
The Cloudflare and OpenAI pilot was about discovery, not ranking, based on what the two companies published. Their post says Cloudflare’s signals help OpenAI find pages, and OpenAI then independently evaluates and ranks how those pages appear.
- OAI-SearchBot controls whether a site can appear in ChatGPT search answers.
- GPTBot is the training crawler, and its setting is independent of OAI-SearchBot.
- Robots.txt changes take about 24 hours to reach ChatGPT search, per OpenAI.
- The pilot used about 1% of Cloudflare’s network.
Opening a ChatGPT Search Law Firm Website to the Right Crawler
A ChatGPT search law firm website plan starts with access. If OAI-SearchBot is blocked, no amount of good writing will get the firm’s pages into ChatGPT search answers. Allowing it, while making a separate choice about GPTBot, takes a few lines and a day to settle.
Access is only step one, though. Once ChatGPT can reach the pages, it still has to find a clear answer on them. Firms looking at how ChatGPT, Perplexity and Gemini treat law firm pages will see the same pattern: direct, well-organized answers get used.
Lexicon’s legal content experts build FAQ pages for law firms that answer client questions in plain, quotable language. Call 877-486-8123 to plan pages worth crawling.
