Answer Engine Optimization Without the Snake Oil

Answer engine optimization is being sold right now with a lot of new vocabulary and at least one product that doesn't do anything: the llms.txt file. Google's own guide says Google Search doesn't use it. People are still paying for it.

The honest version of this work is less exotic, and it starts somewhere most AI search agencies never look. Before anything else, I check whether ChatGPT, Perplexity and Bing's AI can even get into your site. A surprising number can't, because a firewall, a CDN setting or a security plugin is turning them away at the door.

After that it's three things: crawler access, a consistent entity (one clear, matching description of your business across the web), and pages that are worth quoting. If you'd like me to check where you stand, tell me about your site in the AI search questionnaire. I reply with a plan and a fixed quote within one business day.

Starting at $1,500 for an AI search audit, or $1,000/month for ongoing implementation. Both cover AI crawler access, entity consistency, citation-ready content, and setting up Bing's AI citation report.

What you get with BigRed's AI search optimization

• AI crawler access check. robots.txt, Cloudflare and firewall settings, security plugins, and your server logs, checked for the bots that feed AI answers: OAI-SearchBot, PerplexityBot, Bingbot and Googlebot.
• Entity consistency. Your business name, description, services, location and contact details lined up across your site, your Google Business Profile, LinkedIn and the directories AI tools read. When the web disagrees about who you are, AI answers tend to skip you.
• Citation-ready content. Your key pages reworked so each answers a real question in the first two sentences, states specific facts, and cites its sources. That's what gets a passage quoted.
• Bing AI citation report setup. Bing Webmaster Tools set up so you can see how often Copilot and Bing's AI answers cite your pages.
• A prioritized fix list (audit). Every finding ranked by impact, with the exact setting, file or page to change.
• Monthly implementation (if you choose it). I make the fixes, rework pages, chase listing corrections, and send a monthly note on what changed in citations and AI referral traffic.

How it works

1. You fill in the questionnaire. Your site, your main services, and the questions you'd like AI tools to answer with your name.
2. I reply with a plan and a fixed quote within one business day.
3. You share access. Search Console, Bing Webmaster Tools (or I'll set it up), GA4 read access, and Cloudflare or hosting read access if you have them.
4. The audit. Typically one to two weeks. I test the crawlers, read the logs, compare how your business is described across the web, and ask the AI tools the questions your customers ask.
5. Fixes. You can hand the list to your team, or I implement it month by month. Citations in AI answers typically build over months, and I'll show you the data rather than promise a date.

Honest AI search work: crawler access, entity, citations

What Google actually said

In May 2026, Google published a guide to optimizing for generative AI features in Search. It's short, and it contradicts a lot of what's being sold. Its key lines:

• Optimizing for generative AI search "is optimizing for the search experience, and thus still SEO."
• You don't need new machine-readable files, AI text files, markup or Markdown to appear in Google Search, and Google Search doesn't use llms.txt.
• Structured data isn't required for generative AI search, and there's no special schema.org markup for it.
• There's no requirement to break your content into tiny pieces.
• Seeking inauthentic "mentions" across the web isn't as helpful as it might seem.

So for Google, AI search work is good SEO done carefully. Anyone selling a separate "AI schema package" for Google is selling something Google says it doesn't need.

The llms.txt question

llms.txt is a proposed text file meant to give AI tools a summary of your site. Besides Google saying it doesn't use the file, SE Ranking studied nearly 300,000 domains and found no correlation between llms.txt and AI citations.

My position: it's free, it takes 15 minutes, and adding one does no harm. Never pay for it, and don't expect rankings or citations from it. I explain the history in why llms.txt doesn't move AI rankings.

Your firewall might be hiding you from ChatGPT

This is the part I care about most. OpenAI's documentation says sites that opt out of its search crawler will not be shown in ChatGPT search answers. It also says each of its bots is controlled separately. You can let OAI-SearchBot in for search and keep GPTBot out of model training. Perplexity runs the same split: PerplexityBot surfaces sites in its search results and isn't used for training.

The catch is that many sites block these bots without knowing it. Cloudflare started asking every new domain whether to allow AI crawlers in July 2025, with blocking as the starting point. Its settings changed again on September 15, 2026, and Cloudflare's docs now warn that crawlers mixing search and training can be caught by training blocks. Security plugins and host firewalls add their own rules on top. More on that in how Cloudflare can block ChatGPT.

Here's a robots.txt pattern that welcomes AI search and declines AI training:

# robots.txt: let AI search crawlers in, opt out of OpenAI model training.
# Could break: nothing in Google Search. It does NOT override a Cloudflare or firewall block.
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

On Google's side, the Google-Extended token controls Gemini training and grounding, and Google says it doesn't affect inclusion or ranking in Google Search. Blocking it can still cost you visibility in the Gemini app, so I don't block it by default. My AI crawler robots.txt templates cover more combinations.

robots.txt only says what you'd like. Your logs say what happened. This shows whether AI search bots are reaching you, and what status codes they get back:

# Read-only. Counts AI and search crawler requests in a standard access log.
grep -oiE "OAI-SearchBot|GPTBot|ChatGPT-User|PerplexityBot|bingbot|Googlebot" access.log | sort | uniq -c | sort -rn

# Status codes served to AI search bots. Lots of 403s means something is blocking them.
grep -iE "OAI-SearchBot|PerplexityBot" access.log | awk '{print $9}' | sort | uniq -c

Measure what you can, and admit what you can't

AI visibility is still hard to measure, and anyone selling a precise "AI rank" is guessing. Here's what's real, as of October 2026:

• Bing AI Performance. Since February 10, 2026, Bing Webmaster Tools has offered a public preview report of citations across Copilot, AI summaries in Bing and select partners, including cited pages and a sample of grounding queries (the searches the AI ran to find your page). It's free, and it's first-party data.
• Google Search Console. Google's guide points to a Generative AI performance report. Separately, Google says AI Overviews and AI Mode traffic is counted under the "Web" search type in the regular Performance report.
• GA4. Analytics now has an "AI Assistant" channel for visits from tools like ChatGPT. It excludes Google's own AI Overviews and AI Mode, and visits without a referrer still land in Direct.

And expect fewer clicks per answer. Pew found people clicked a regular result on 8% of visits with an AI summary, versus 15% without. That's why I judge this work by calls and form fills, not traffic alone. If your traffic has already slipped, see how to tell if AI Overviews caused your drop.

Who this is for (and who it isn't)

This is a good fit if:

• ChatGPT or Perplexity recommends your competitors and never you
• You use Cloudflare, a security plugin or a managed firewall and have never checked what it blocks
• Your business is described differently on your site, your profiles and directory listings
• You'd like real numbers on AI citations instead of screenshots of a chatbot

It isn't for you if your site has basic SEO problems: pages not indexed, broken redirects, thin content. AI search tools find their sources by crawling and searching the web, so a site that's hard to crawl or index is hard to cite. Fix those first. It also isn't for anyone who wants guaranteed ChatGPT mentions. Nobody controls those, and anyone promising them is selling hope. If you run a local business, start with local SEO, since your Google Business Profile feeds a lot of AI local answers.

AI search optimization pricing

For context, SE Ranking's survey of 260 agencies (December 2024) found an average of $937 per month for AI search optimization services. Mine, as of October 2026:

All prices are in USD, and Canadian clients can pay in CAD at the day's exchange rate. The full list sits on the pricing page.

Why hand it to me

I've spent 20+ years in search marketing, and AEO and GEO (answer and generative engine optimization) work is part of what I do now. The reason I start with access is that I've spent much of those years on servers: Cloudflare, cPanel/WHM, Plesk, Linux, and a lot of log files. Most AI search sellers can't read a WAF rule. I can, and I fix it.

I also set up Search Console, Bing Webmaster Tools and GA4 as part of the job, so the numbers you see come from the platforms themselves, not a third-party "AI visibility score."

You deal with one person, start to finish. The one who reads your logs is the one who writes your report.

Answer engine optimization FAQ

What is answer engine optimization?

Answer engine optimization (AEO) is the work of getting your business cited or recommended inside AI-generated answers, such as ChatGPT search, Perplexity, Copilot and Google's AI Overviews. In practice, it means AI crawlers can reach your site, your business is described consistently across the web, and your pages answer real questions clearly enough to be quoted. It builds on regular SEO rather than replacing it.

What's the difference between AEO, GEO and SEO?

SEO aims to rank pages in search results. AEO (answer engine optimization) and GEO (generative engine optimization) are two names for the same goal: being the source an AI answer cites. For Google, its own guide says optimizing for generative AI is still SEO. Other AI tools have their own crawlers and sources, which is why crawler access and consistent listings matter beyond Google.

Do I need an llms.txt file?

No. Google says Google Search doesn't use llms.txt, and a study of nearly 300,000 domains found no link between having the file and getting AI citations. If you want one, it's free and takes about 15 minutes, and it won't hurt. Just don't pay anyone to "optimize" it, and don't expect it to change your visibility in AI answers.

How do I get my business recommended by ChatGPT?

First, make sure OpenAI's search crawler, OAI-SearchBot, isn't blocked by your robots.txt, Cloudflare or firewall, because OpenAI says blocked sites aren't shown in ChatGPT search answers. Then make your business description consistent across your site and listings, earn genuine reviews and mentions, and write pages that answer customer questions directly. There's no switch to flip, and results build over months.

Can you guarantee my business will show up in AI answers?

No, and you should be wary of anyone who does. AI answers change from one query to the next and depend on sources nobody controls. What I can do is remove the blockers, fix the inconsistencies, improve the pages, and show you the citation data from Bing and Google so you can see whether it's working.

How much does answer engine optimization cost?

At BigRed, an AI search audit starts at $1,500 and ongoing implementation starts at $1,000 a month. One industry survey put the average for AI search optimization services at about $937 a month. Watch for packages built on llms.txt files, special "AI schema" or bulk mention campaigns, which Google's own guidance says aren't needed or helpful.

Next step

If AI tools keep recommending everyone but you, send me your site through the AI search questionnaire. I'll start by checking whether they can get in, and I'll come back with a plan and a fixed quote.