Agentic browsing tools: a practical guide to ChatGPT Atlas, ChatGPT Agent, Comet, Gemini, and Claude

TL;DR:

• Agentic browsing tools don't work like Googlebot. They use Chromium-based browsers with accessibility tree parsing, meaning JavaScript-heavy or poorly labeled sites get skipped.
• ChatGPT Atlas and Comet parse content using accessibility trees. Sites with weak ARIA labels and semantic HTML structure suppress their own citation rates.
• Operator was absorbed into ChatGPT Agent in August 2025. The underlying Computer-Using Agent model still powers agent interactions inside ChatGPT.
• Gemini grounds answers using Google Search infrastructure, not standalone browsing. It retrieves content from Google's index rather than conducting live browsing sessions.
• Priority actions: audit your robots.txt for retrieval bots, implement JSON-LD schema for Organization, FAQPage, and Product types, and verify successful agentic crawls in server logs. Initial citation signals appear within 1-2 weeks of technical fixes.

AI agents are being embedded in a growing share of enterprise applications. HUMAN Security's 2026 State of AI Traffic & Cyberthreat Benchmark Report recorded 7,851% year-over-year growth in traffic from AI agents and agentic browsers. Yet most B2B SaaS sites are still configured exclusively for Googlebot. For the strategic framework behind these requirements, see our agent readiness guide.

Traditional SEO focuses on document-level keyword ranking. Agentic retrieval focuses on passage-level extraction. This guide covers the technical requirements of major agentic browsing tools, including ChatGPT Atlas, ChatGPT Agent (formerly Operator), Comet, Gemini, and Claude, and provides a testing protocol to make sure your site is built for AI citation retrieval, not just keyword ranking.

Defining the mechanics of AI agent browsing

Agentic browsing is not passive indexing. An AI agent operates through a continuous Perception-Reasoning-Action (PRA) loop, perceiving the page via an accessibility tree, reasoning about which elements serve the query, and taking action by clicking, scrolling, or extracting specific passages in real time. The result is a fundamentally different retrieval model from Googlebot's document-level scan.

B2B commerce is increasingly influenced or automated by AI agents, and most site configurations aren't ready for them.

How agentic tools differ from standard crawlers

Traditional crawlers fetch a document and score it for keyword relevance and link authority. Agentic tools perceive page structure through an accessibility tree, reason about which elements match a query goal, and extract specific passages for synthesis. This is the shift from document-level indexing to passage-level retrieval, and it's the core reason that optimizing for Google rankings no longer guarantees AI citations.

Karpukhin et al. demonstrated that dense retrievers outperform keyword-based BM25 on passage retrieval tasks. The content structure that ranks on Google is not the same structure that gets cited in ChatGPT. Our CITABLE framework was built specifically around passage-retrieval mechanics for Answer Engine Optimization (AEO).

Protecting your brand in AI tools

Agentic browsing introduces security risks that standard SEO never created. Because agents interact with forms, buttons, and authenticated pages, misconfigured sites can be scraped at scale or expose internal data.

Enterprise deployment checklist

• Rate limiting and IP verification: Apply per-IP and per-user-agent rate limits, cross-referencing inbound agent traffic against published IP ranges from OpenAI, Google, and Perplexity where available
• Content licensing signals: Use structured data and meta tags to signal content licensing terms to agents that respect them
• WAF rules: Configure your Web Application Firewall to distinguish between retrieval bots (which you want) and training crawlers (which you may want to block)
• Access logging: Enable detailed server logging for all agent user-agents to track what was accessed and when

ChatGPT Atlas: technical requirements for citations

ChatGPT Atlas is OpenAI's browser-based agent that uses accessibility tree parsing to identify page elements. It appears to present as a standard Chrome browser instance, which means you cannot filter for it reliably via robots.txt or simple user-agent rules.

How Atlas parses page content

Atlas combines DOM processing and accessibility tree parsing to identify entities and relationships on a page. Pages with clean semantic HTML and proper ARIA roles parse successfully and contribute to citation outputs. JavaScript-heavy pages with lazy-loaded content may be abandoned before the accessibility tree is fully populated. Our AI Visibility Tracker tracks how often your brand appears in AI-generated answers across platforms.

As Tom Wentworth, CMO at incident.io, described the state most teams operate in before systematic optimization:

"Before Discovered Labs, we were using homegrown LLM prompts, without a clear strategy for what to optimize for or exactly how best to structure content." - incident.io case study

System requirements for Atlas browsing

For Atlas to extract content reliably, your pages need:

• Clean semantic HTML: Use <article>, <section>, and heading tags rather than div-heavy layouts
• ARIA labels: All interactive elements must have accessible names
• Server-side or hybrid rendering: Client-side-only rendering delays accessibility tree availability
• Fast response times: Page performance can affect content extraction
• JSON-LD schema: Organization, Product, and FAQPage schema types are recommended. Note that FAQPage no longer produces rich results in Google Search as of May 2026. Its value now is AI-citation extractability: structured FAQ blocks are among the most consistently retrieved passage types across ChatGPT, Claude, and Perplexity.

Technical audit for Atlas compatibility

1. Open Chrome DevTools' Accessibility panel and review the tree for key pages. Every content block should appear as a named node.
2. Test with a screen reader. If VoiceOver or NVDA can't read the page coherently, Atlas will struggle too.
3. Use Google's Rich Results Test or similar schema validation tools to confirm JSON-LD is valid and complete.
4. Review server logs for user-triggered ChatGPT browsing sessions. Our citation tracking workflow guide walks through the log analysis setup.

Operator (now ChatGPT Agent): technical requirements for AI citations

OpenAI shut down Operator as a standalone product on August 31, 2025, absorbing its capabilities into ChatGPT Agent. The underlying Computer-Using Agent (CUA) model still powers agent interactions inside ChatGPT.

How Operator parses your web pages

The CUA model combines screenshot analysis with accessibility tree parsing. It can execute sequences of clicks, scrolls, keystrokes, and form inputs. Technical analysis of similar agent architectures suggests these systems extract accessibility tree data and handle navigation programmatically.

Technical benchmarks for Operator crawling

The CUA model operates within practical session limits:

• Page size: Keep key content visible in the initial viewport and avoid requiring significant scroll to reach decision-relevant information
• Form interaction: Label all form fields with explicit <label> tags tied to input IDs

Optimizing site architecture for Operator

Site navigation should follow a flat, predictable structure. Three clicks or fewer from the homepage to any key decision page is the practical benchmark. Internal linking benefits from descriptive anchor text rather than generic labels like "learn more," because agent reasoning may use anchor text to predict destination content before clicking. Our Model Context Protocol (MCP) servers for AEO teams guide covers automating the internal linking audit for this requirement.

How Comet parses content for AI-driven answers

Comet is Perplexity's agentic browser. It prioritizes the DOM and accessibility tree over screenshot-based pixel analysis. Sites with weak accessibility implementation are harder for Comet to process correctly.

Mapping Comet's site interaction patterns

Comet uses the same action set as other CUA-style agents: clicks, scrolls, form inputs, and navigation. Its reliance on the accessibility tree is a deliberate design choice for handling complex DOMs.

Where Comet may struggle: custom interactive components that don't follow ARIA patterns, non-unique element IDs, and clickable elements that aren't marked as interactive in the accessibility tree.

Key Comet compatibility factors

Comet's compatibility requirements align with standard accessibility practice:

• Semantic HTML with proper element types
• ARIA roles for interactive elements
• Unique element IDs
• JSON-LD schema for relevant content types

Our analysis of what drives AI citations found no significant independent effect from schema markup alone. Schema likely contributes indirectly by reinforcing content structure rather than acting as a standalone citation signal, so treat it as a supporting technical requirement, not a primary lever. You can benchmark your current content using our free AEO content evaluator.

Decoding Gemini citation and retrieval logic

Gemini is not a standalone agentic browser. It grounds answers using Google's real-time search infrastructure (the google_search tool), retrieving and synthesizing content from Google's index rather than conducting a live browsing session. Its citation logic is tied directly to Google's indexing and Knowledge Graph.

How Gemini indexes and cites content

When a user prompt triggers a search need, Gemini generates one or more search queries, processes the results, synthesizes the information, and returns a grounded response with citations. The Gemini API documentation describes this as a fully automated flow.

If Google's Knowledge Graph doesn't have a clear, consistent entity record for your brand, Gemini's grounding step may pull from whatever Google surfaces, which could include outdated or competitor content.

Requirements for Gemini agentic browsing

Gemini's citations are shaped by Google's standard indexing requirements plus AEO-specific factors:

• Structured data: Organization, Product, and FAQPage schema implemented in JSON-LD. Note that both HowTo rich results (deprecated 2023) and FAQ rich results (fully removed from Google Search May 7, 2026) no longer produce SERP-level visual features. FAQPage schema still signals block structure to AI retrieval systems and supports Gemini's grounding step, so implement it for citation extractability, not for Google rich results.
• High-authority third-party mentions: Consistent claims about your brand across trade press, independent editorial publications, and review sites. Gemini appears to over-index on independent editorial sources and under-index social media citations compared to other AI systems. Our Reddit/ChatGPT citation research found Reddit occupied roughly 27% of ChatGPT's internal search slots during query processing, but Gemini shows a different citation pattern.
• robots.txt compliance: Google-Extended plus standard Google indexing directives apply
• Information consistency: Consistent claims across your site, third-party reviews, and comparison content

Assessing Gemini retrieval accuracy

1. Enable Google Search grounding in the Gemini API and run your highest-priority buyer queries.
2. Compare Gemini's citations against your actual pages. Mismatches may indicate entity disambiguation problems or schema gaps.
3. Cross-reference with Google Search Console impressions for the same queries. Our real citation rate benchmarks post covers why platform-reported numbers often understate actual citation performance.

Side-by-side comparison: which tool does what

The four tools use meaningfully different architectures. The tables below map the core technical differences and decision criteria for B2B SaaS sites.

Table 1: PRA loop and tool comparison

Table 2: Decision matrix for B2B SaaS use cases

How tools handle JavaScript execution

Atlas and ChatGPT Agent execute JavaScript via Chromium but read the accessibility tree before full JS hydration completes. Comet prioritizes the DOM and accessibility tree over screenshot-based rendering. Gemini accesses JavaScript-rendered content through Google's existing infrastructure rather than executing scripts in a live session, meaning pages need to be indexed with rendered content to appear in Gemini's grounding results. Client-side-rendered pages that depend on JavaScript to display their main content may face retrieval disadvantages across these tools. Our Claude Code for AEO workflows guide covers how to audit extractability systematically.

Handling paywalls and authentication

The four tools typically cannot handle hard paywalls or multi-step authentication flows. For content you want cited, keep it publicly accessible. If key product pages are behind a login, create publicly accessible equivalents such as comparison pages, feature overviews, or FAQ pages that agents can reach without authentication.

Configuring server logs for AI agent visits

Use these user-agent strings to filter server logs and identify agent visits by tool:

# ChatGPT Agent (user-triggered, reported)
User-agent: ChatGPT-User

# OAI retrieval bot (indexing, reported)
User-agent: OAI-SearchBot

# Perplexity retrieval bot (indexing, reported)
User-agent: PerplexityBot

# Perplexity user-triggered (reported)
User-agent: Perplexity-User

# Google Extended (training control)
User-agent: Google-Extended

# Claude retrieval and user-triggered (reported)
User-agent: Claude-SearchBot
User-agent: Claude-User

Note that ChatGPT Atlas presents as a standard Chrome browser instance, so Atlas-specific traffic may not be isolable via user-agent filtering alone. The ChatGPT-User string identifies user-triggered ChatGPT Agent sessions. Our AI visibility tracking guide explains how this connects to your broader citation rate measurement framework.

Technical testing protocols for agentic browsing

Testing agentic compatibility is best done in a dedicated sandbox environment rather than production to avoid triggering WAF rules and polluting analytics with test sessions.

Setting up and running compatibility tests

Set up a staging subdomain that mirrors production. Configure server logging to capture all requests, and verify that your staging robots.txt allows the agent user-agents you're testing.

To test ChatGPT browsing specifically:

1. Open a ChatGPT session with browsing enabled
2. Submit queries matching your core buyer questions, for example "best [category] software for [use case]" or "[your product] vs [competitor]"
3. Note which queries produce citations to your site versus competitors
4. Review staging server logs to confirm agent sessions actually fetched your pages
5. For pages that weren't cited, run the accessibility tree audit described in the Atlas section above

For the site-architecture side of this, including SSR, robots.txt, and checkout compatibility, see our agentic browsing readiness guide.

Resolving AI agent crawling blocks

The Web Bot Auth protocol is an emerging cryptographic solution to distinguish legitimate agent traffic from abuse. Implementing support now may position your site for the next generation of agent access control.

Measurement and attribution setup

Attribution tracking for AI-referred traffic has known limitations. GA4, HubSpot, and CRM data may give different answers for the same question. Here's a practical three-step setup to capture what you can:

1. UTM tagging: Append utm_source=chatgpt and utm_medium=ai_agent to any links you control in AI-cited content such as structured data, press releases, and syndicated content
2. "How did you hear about us" field: Add "AI Assistant (ChatGPT, Claude, Perplexity, Gemini)" as an explicit option on demo request and contact forms. Self-reported attribution is imperfect but directionally reliable at volume
3. CRM tagging: Flag leads who convert from sessions originating from AI assistant referrers with an AI attribution marker in HubSpot or Salesforce, separate from your standard organic source tagging

Our AI visibility platform buyer's guide and Profound vs Peec comparison cover how dedicated citation tracking tools integrate with these attribution workflows.

Addressing technical barriers to AI citation

A common reason B2B SaaS sites don't get cited in AI answers is configuration. Sites that block retrieval bots, serve unparseable JavaScript, or have inconsistent entity data across sources may suppress their own citation rates without realizing it.

Managing AI agent access via robots.txt

The distinction between retrieval bots, which you want, and training crawlers, which you may want to block, is the central configuration decision. Use these exact blocks to control agent access:

# Retrieval bots — ALLOW (these drive citations)
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# User-triggered agents — ALWAYS ALLOW
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Perplexity-User
Allow: /

# Training control tokens — OPTIONAL BLOCK
User-agent: Google-Extended
Disallow: /

# Training crawlers — OPTIONAL BLOCK
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

A practical approach is to block training, allow retrieval: keep your content visible inside ChatGPT, Claude, Gemini, and Perplexity answers while preventing AI vendors from using your content to train future foundation models without attribution. Note that Google-Extended controls both training and grounding for Gemini, so blocking it will also affect Gemini's ability to cite your content.

When to re-run agent compatibility tests

Re-test your agentic compatibility whenever:

• You complete a major site redesign or CMS migration
• Your citation rate drops materially month-over-month without a clear content explanation
• You implement new JavaScript frameworks or change rendering strategies

Resolving agentic browsing failures

Next steps for your agentic search strategy

Citation rate, mention rate, and AI-referred sessions now directly affect marketing-sourced pipeline and CAC payback. The sites that get cited are the ones agents can successfully parse and extract. The sites that don't get cited are often not failing on content quality. They're failing on technical configuration that's straightforward to fix once you know where to look.

The practical starting point is a full audit across all three surfaces: web search, citations, and training data. We run this as a Search Visibility Diagnostic, covering AI visibility across Google, Google AI Overviews, ChatGPT, Claude, Perplexity, and Gemini, with prioritized recommendations on what to fix on-site and where to build citations.

If you want a self-serve starting point first, our AEO content evaluator scores your existing content against the CITABLE framework at no cost. You can also see the full picture of what systematic AEO work produces in the incident.io case study, where optimization work was part of moving AI visibility from 38% to 64% and delivering a 22% lift in organic meetings booked. Book a call and we'll tell you honestly whether we're a fit.

FAQs

What user-agent strings should I allow in robots.txt for AI citation retrieval?

Allow OAI-SearchBot, Claude-SearchBot, and PerplexityBot for retrieval bots, and ChatGPT-User, Claude-User, and Perplexity-User for user-triggered agent fetches. These are the agents that drive citations, as distinct from training crawlers like GPTBot or ClaudeBot. Google-Extended is a separate case: it controls both training use and Gemini's grounding step, so blocking it stops AI vendors from training on your content but also removes Gemini's ability to cite it. Decide deliberately based on whether Gemini citation matters to your strategy, rather than defaulting to allow or block.

Does JavaScript rendering affect AI agent citation rates?

Yes. ChatGPT Atlas and ChatGPT Agent read the accessibility tree before full JS hydration completes, meaning client-side-rendered pages with delayed content may face retrieval disadvantages. Gemini accesses JavaScript-rendered content through Googlebot's infrastructure, so CSR pages that haven't been indexed with their rendered content may be structurally invisible to Gemini's grounding step.

What happened to Operator, and do I still need to optimize for it?

OpenAI shut down Operator as a standalone product on August 31, 2025, absorbing its capabilities into ChatGPT Agent. The underlying Computer-Using Agent model still powers agent interactions inside ChatGPT, so the technical requirements are expected to be similar: accessible interactive elements, clean semantic HTML, and server-side rendered content for key citation-target pages.

How much does a Search Visibility Diagnostic cost, and what does it cover?

The Search Visibility Diagnostic is available as an engagement. It maps your current visibility across Google, Google AI Overviews, ChatGPT, Claude, Perplexity, and Gemini, and delivers prioritized recommendations and optimized content using the CITABLE framework.

How long does it take to see a lift in citation rates after technical fixes?

Initial citations typically appear within 1 to 2 weeks of implementing schema, accessibility, and robots.txt fixes. A material citation rate lift typically takes several months of combined technical and content work across all three surfaces.

Key terms glossary

Agentic browsing: Real-time web interaction by an AI agent using a Perception-Reasoning-Action loop to navigate pages, extract content, and synthesize answers, as distinct from passive document indexing by traditional crawlers.

Citation rate: The percentage of target queries where an AI engine cites your brand as a source. This is a primary KPI for agentic retrieval optimization, distinct from impressions or click-based metrics.

Mention rate: The frequency with which your brand is named in AI-generated answers across a specific query set, whether or not your site is cited as a source. Mention rate may be high even when citation rate is low if brand associations exist in model training data.

AI-referred sessions: Website visits initiated by a user clicking a link within an AI engine's response. These sessions may appear as direct traffic in GA4, requiring UTM tagging and form field attribution to capture accurately.

Passage retrieval: The process by which LLMs extract specific sections of content for synthesis into an answer, as distinct from document-level keyword ranking. Passage retrieval plays a key role in whether your content gets cited, not whether your page ranks in traditional search results.