TL;DR
- ChatGPT shows up in your logs as two user agents. OAI-SearchBot crawls continuously to build ChatGPT's search index, while ChatGPT-User fetches pages live as an answer is written.
- In Discovered Labs' capture of ChatGPT traffic about its own site, ChatGPT-User fetched pages live on about half of turns. Across the broader prompt set in the same research, about a third of what the model read per turn came from live fetches.
- Blocking each agent does something different. Blocking OAI-SearchBot removes you from ChatGPT search answers apart from navigational links, while robots.txt rules may not apply to ChatGPT-User.
- No public per-site benchmark exists. Count requests per agent for a week, check which pages get fetched, then verify IPs against OpenAI's published ranges.
Organic search now runs on two surfaces, Google results and AI-generated answers. GA4 doesn't record AI crawler requests, so the clearest first-party record of ChatGPT retrieval sits in your server and CDN logs. This is the log-level detail behind how ChatGPT retrieves and cites: the two ChatGPT user agents that matter, what blocking each one changes, and a five-step checklist for tracking them yourself.
ChatGPT retrieval appears in your server and CDN logs as two user agents. OAI-SearchBot indexes pages ahead of time for ChatGPT search, while ChatGPT-User requests pages during an answer, seconds after a prompt triggers a search. Grep your access logs for both tokens, count requests per day and per URL, then check request IPs against OpenAI's published ranges before you trust the numbers. Treat your first week as the baseline.
Checking your own logs instead of vendor dashboards#
Your logs record ChatGPT requests that GA4 and vendor dashboards can't see.
Seeing what GA4 and dashboards miss#
Vendor dashboards sample prompts and estimate visibility. That's useful for trends, but it can't show you a single real request. GA4 can't fill the gap either. AI crawlers, including OpenAI's, skip JavaScript execution, so they never trigger the client-side tag. GA4 also automatically excludes known bots using Google research and the International Spiders and Bots List.
Server-side tagging won't close that gap, because a server container still receives its events from tags running in the visitor's browser. Only server or CDN logs record every request, whether or not JavaScript ran.
The same limit shapes what the crawlers see. If your key content is rendered client-side, OpenAI's crawlers may fetch an empty shell. Server-side rendering or static pre-rendering keeps that content visible to AI retrieval.
Getting access to your logs#
You need terminal access to your web server or access to your CDN's log export. On many Linux servers running Nginx or Apache, access logs sit in /var/log/nginx/ or /var/log/apache2/. On a managed host, ask for raw log access or install a logging plugin that captures user-agent strings.
Nginx's predefined combined format ends each line with the user agent in quotes, so one grep pulls every ChatGPT request:
grep -iE 'ChatGPT-User|OAI-SearchBot' /var/log/nginx/access.log
Each line gives you a timestamp, the requested URL, and a status code. Zero results means neither agent reached your server in the period that log covers.
If your site sits behind a CDN such as Cloudflare, check its logs as well. Requests the CDN serves from cache or blocks at the edge never reach your server, and your origin logs record the CDN's IPs instead of OpenAI's unless you restore visitor IPs.
Telling OAI-SearchBot and ChatGPT-User apart#
Each OpenAI agent sends a documented user-agent string, and the token in it tells you which job the request is doing.
Matching the user-agent strings#
OAI-SearchBot is OpenAI's search crawler, used to surface websites in search results in ChatGPT. OpenAI's documentation gives this user-agent string and notes the version number can change:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
ChatGPT-User handles certain user actions in ChatGPT and Custom GPTs, including fetching pages while an answer is written. Its user-agent string is:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
Mapping each agent's role#
Agent | User-agent token | Primary function | Request pattern |
|---|
ChatGPT-User | ChatGPT-User
| Live page fetches during answers | On-demand, in short bursts seconds after ChatGPT plans its search |
OAI-SearchBot | OAI-SearchBot
| Search indexing | Continuous crawling, volume varies by site |
GPTBot | GPTBot
| Training data collection | Varies by site |
OAI-SearchBot crawls continuously to build the index, so it governs whether your pages can appear in ChatGPT search answers. ChatGPT-User is event-driven. It fetches the pages the model is about to quote while the answer is being written.
GPTBot sits outside retrieval, but its crawling has grown fast. Cloudflare saw its raw requests rise 305% between May 2024 and May 2025.
Reading request patterns in your logs#
Live fetches show up as tight clusters of timestamps, while status codes tell you whether ChatGPT got the page.
Spotting live fetches#
Discovered Labs captured ChatGPT's traffic for prompts about its own site and compared it with the site's Cloudflare logs. On about half of turns, ChatGPT-User requested a few pages seconds after ChatGPT planned its search. Across the broader prompt set in the same research, about a third of what the model read per turn was fetched live.
Live fetches arrive in bursts. A single answer can trigger several ChatGPT-User fetches within a few seconds. To see them, list ChatGPT-User requests by timestamp:
grep 'ChatGPT-User' /var/log/nginx/access.log | awk '{print $4, $7}' | sort
Look for clusters in the timestamps. Several requests to different pages within a few seconds suggest a live answer rather than a one-off visit.
Note: The half-of-turns figure comes from a two-day capture of ChatGPT web app traffic for prompts about Discovered Labs' own site. The one-third figure comes from the broader prompt set in the same research. Treat them as patterns from this research and set your own baseline from your logs.
Reading status codes#
Status codes tell you whether ChatGPT got the page it asked for. This command counts them across both agents:
grep -iE 'ChatGPT-User|OAI-SearchBot' /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c
A 200 means the fetch succeeded. A 404 means the server couldn't find the requested page. A 403 means the server refused the request, and a high rate of them usually points to a firewall or bot-management rule blocking the agent. A robots.txt disallow won't show up as a 403, because a compliant crawler simply stops requesting the page.
Finding ChatGPT activity in CDN logs#
Cloudflare's edge logs carry the same user-agent, path, and status data as server logs. Other CDNs offer similar exports.
Exporting Cloudflare edge logs#
If you use Cloudflare, enable Logpush to send edge logs to a storage destination such as an S3 bucket. Then filter by the ClientRequestUserAgent field for ChatGPT-User or OAI-SearchBot.
Once the logs sit in a queryable store, a query like this lists the pages ChatGPT-User requests most. Replace cloudflare_http_requests with your own table name:
SELECT ClientRequestPath, COUNT(*) AS requests
FROM cloudflare_http_requests
WHERE ClientRequestUserAgent LIKE '%ChatGPT-User%'
GROUP BY ClientRequestPath
ORDER BY requests DESC
LIMIT 20;
The same dataset includes EdgeResponseStatus for status codes and ClientIP for IP verification.
Using other CDNs and managed hosts#
Fastly supports remote log streaming, and Akamai offers DataStream for streaming logs. Setup and exportable fields vary by provider, so confirm your plan can export the user-agent header.
On a managed WordPress host, request raw server logs or install a logging plugin that records user-agent strings. Some hosts expose logs by default, while others need a support request.
Deciding what to block in robots.txt#
Each OpenAI agent does a different job, so a robots.txt block changes something different for each one.
Blocking OAI-SearchBot#
To block OAI-SearchBot, add this to robots.txt:
User-agent: OAI-SearchBot
Disallow: /
Blocking OAI-SearchBot removes your site from ChatGPT search answers. OpenAI says opted-out sites will not be shown there, though they can still appear as navigational links. OpenAI also says its systems can take about 24 hours to adjust after a robots.txt change.
The block also covers new material. Fresh content, pricing changes, and new product pages can't surface in ChatGPT search answers once the block is in place.
Blocking ChatGPT-User and GPTBot#
OpenAI doesn't use ChatGPT-User to decide what appears in ChatGPT search. Because its actions are user-initiated, robots.txt rules may not apply, so don't count on robots.txt to stop ChatGPT-User fetches.
GPTBot is OpenAI's third crawler. Disallowing it tells OpenAI not to use your content for training foundation models. That makes it a separate decision from search: blocking GPTBot opts you out of training, while blocking OAI-SearchBot removes you from ChatGPT search answers.
Confirming a block is working#
After the change, count requests per day for the blocked agent:
grep 'OAI-SearchBot' /var/log/nginx/access.log | awk '{print $4}' | cut -d: -f1 | sort | uniq -c
Requests should fall away once OpenAI picks up the change. If they don't, check robots.txt for typos and look for conflicting firewall or CDN rules.
Connecting log activity to pipeline#
Server logs don't record the prompt that triggered a fetch. The Referer header is often absent or contains only the origin, so the query itself stays hidden. What you can do is line up log spikes with your own pipeline data.
Compare daily ChatGPT-User requests to key pages with demo requests, trial signups, and self-reported "how did you hear about us" answers that mention ChatGPT. A monthly note to leadership might read: "In [month], ChatGPT-User fetched our pricing page [N] times, and [X] demo requests self-reported ChatGPT."
Treat that as correlation. It's still first-party evidence you can stand behind. It's also clearer than reconciling GA4, HubSpot, and CRM data that each tell a different story.
Tracking ChatGPT retrieval with a five-step checklist#
Five weekly checks tell you whether ChatGPT reaches your site, what it reads, and whether the traffic is genuine.
1. Confirm both agents appear#
grep -c 'OAI-SearchBot' /var/log/nginx/access.log
grep -c 'ChatGPT-User' /var/log/nginx/access.log
A zero for either agent means none of its requests reached your server in the period your log covers. Check robots.txt, firewall rules, and your CDN's logs before assuming ChatGPT ignores your site.
2. Measure daily volume per agent#
grep 'OAI-SearchBot' /var/log/nginx/access.log | awk '{print $4}' | cut -d: -f1 | sort | uniq -c
Run it again with ChatGPT-User. Volumes vary widely from site to site, so treat your first week as the baseline. To keep a daily record, save each day's ChatGPT requests to a dated file:
grep -iE 'ChatGPT-User|OAI-SearchBot' /var/log/nginx/access.log | awk -v d="$(date '+%d/%b/%Y')" '$4 ~ d' > chatgpt-requests-$(date +%Y-%m-%d).log
3. Identify which pages get fetched#
grep 'OAI-SearchBot' /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
Swap in ChatGPT-User to see what the model fetches during answers. Check whether product, pricing, and comparison pages appear near the top. If an old announcement post leads the list, your content architecture may not match buyer intent. If your top-converting landing page has zero OAI-SearchBot hits in 30 days, it may be missing from ChatGPT's search index.
4. Verify the requests come from OpenAI#
Anyone can fake a user-agent string. List the IPs behind each agent's requests:
grep 'ChatGPT-User' /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -20
Run it again with OAI-SearchBot. Check them against OpenAI's published ChatGPT-User ranges and its OAI-SearchBot IP list. Requests from IPs outside those lists are likely someone else using the name. Behind a CDN, run this check on the CDN's ClientIP field instead.
5. Watch for shifts and dips#
Compare each week with the last. A drop to zero usually means a robots.txt change, a firewall rule, or a CDN setting is blocking the agent. A drop of 50% or more with no obvious cause is worth checking against recent site or content changes.
Turning your logs into a baseline#
Your server and CDN logs show what ChatGPT requested from your site. Separate OAI-SearchBot from ChatGPT-User, verify the IPs, then track both weekly. That gives you a first-party baseline for AI retrieval to set beside any dashboard estimate.
Log file analysis is part of our Outlier and custom engagements. Our team pairs an AI researcher who built production ML systems with full-time AI/ML engineers. Gladia grew sales-accepted leads 7x in 4 months, with 93% of AI-referred leads from LLM search, after a program that combined technical audits, AI-structured content, and Reddit work. If you want a second set of eyes on your logs, book a call and we'll tell you honestly whether we're a fit.
FAQs#
How often does ChatGPT-User visit my site?#
ChatGPT-User doesn't crawl on a schedule. OpenAI says it isn't used for automatic crawling. It fetches pages when someone asks ChatGPT or a Custom GPT a question that needs your content, so frequency depends on how often your pages come up in answers.
Can I see which queries triggered OAI-SearchBot or ChatGPT-User?#
Server logs don't show the prompt behind a request. You can correlate request spikes with your own pipeline data, such as demo requests and self-reported answers that mention ChatGPT.
Do both agents respect robots.txt?#
OAI-SearchBot follows robots.txt, and opted-out sites won't be shown in ChatGPT search answers. For ChatGPT-User, OpenAI says robots.txt rules may not apply because its actions are user-initiated.
What's a normal request volume for each agent?#
There's no published per-site benchmark for either agent. In Discovered Labs' two-day capture of its own site, ChatGPT-User fetched pages on about half of turns, but that's one site's pattern. Run the daily count for a week and use that as your baseline.
How do I verify a request came from OpenAI?#
Check the request IP against OpenAI's published IP ranges for each agent. A user-agent string alone can be faked, so an IP match is the check to rely on.
Key terms glossary#
ChatGPT-User: OpenAI's user agent for actions users take in ChatGPT and Custom GPTs, including fetching pages while an answer is written.
OAI-SearchBot: OpenAI's crawler for ChatGPT search. It crawls continuously to build the index ChatGPT search draws on.
GPTBot: OpenAI's crawler for training data. Disallowing it tells OpenAI not to use your content for foundation model training.
Live fetch vs indexed retrieval: Live fetch means ChatGPT-User requests a page during answer generation, so fresh content can appear immediately. Indexed retrieval means the answer draws on pages OAI-SearchBot already crawled, so changes may take longer to surface.
User-agent string: A text identifier sent in the HTTP request header that tells the server what software is making the request. ChatGPT agents identify themselves with tokens like ChatGPT-User or OAI-SearchBot.
CDN logs: Logs from a content delivery network such as Cloudflare, Fastly, or Akamai that record edge requests, including user-agent strings, timestamps, and URL paths.
