SEO and AI search for small businesses: what gets you found, and what got us dropped

Three doors, one floor: how people find a small business now

When someone looks for what you sell, they reach you in one of three ways. The first is the list of blue links Google has shown for twenty-five years. The second is Google's AI answers, AI Overviews and AI Mode, which write a summary at the top of the page and link to the sites they drew from. The third is an assistant such as ChatGPT, Perplexity or Copilot, which runs its own web search and cites a handful of sources in its reply.

It is tempting to treat these as three projects with three budgets. For a small business they are one project. Google says plainly that to appear as a supporting link in its AI features, a page only has to be indexed and eligible to show with a snippet in normal results, and that there is no special markup, file or optimisation to add. If your pages are not good enough for the blue links, they are not good enough for the AI box either.

The assistants outside Google work differently in one practical way: they send their own crawlers. ChatGPT's search feature, for example, reads the web with a crawler called OAI-SearchBot, which is separate from the GPTBot crawler OpenAI uses for model training. OpenAI's documentation says a site can allow OAI-SearchBot to appear in ChatGPT search while blocking GPTBot. So the one extra thing GEO asks of you is a decision about which crawlers you let in, which we cover below.

Everything else is the same floor: pages a machine can read, a site that is clearly about something, and content that someone would choose over the alternatives. The rest of this guide builds that floor in order, starting with the part that breaks most often and is cheapest to fix.

The foundation: can search engines read you at all?

Before anything about content, check that nothing is stopping a crawler. This takes twenty minutes with Google Search Console, which is free, and a browser. Google is clear that it does not guarantee it will crawl, index or serve any page, so the point of these checks is to remove every reason it might not.

• robots.txt. Open yoursite.com/robots.txt. A line reading Disallow: / under User-agent: * blocks everything. Many site builders and security plugins add rules you never chose. In September 2026 a change in Cloudflare's default crawler settings started blocking more bots than site owners expected, so check it even if you never touched the file.
• The robots meta tag. View the source of a page and search for noindex. A theme left in "discourage search engines" mode puts it on every page. The tag means do not show this page in search results, and it does exactly that.
• Canonical tags. Each page should declare itself, not another address, as the canonical version. A canonical pointing at your homepage tells Google to ignore the page you are on.
• The sitemap. Submit sitemap.xml in Search Console, and list only pages you want in search. Google uses the lastmod date in it only if it is consistently and verifiably accurate, meaning it changes when the content changes, not every time the site rebuilds.
• HTTPS and one address. http://, https:// and the www version should all land on one address with a permanent (301) redirect.

Then open Search Console's Pages report. It shows how many of your pages Google indexed and, for the rest, why not. "Crawled, currently not indexed" on a large share of your pages is a quality signal rather than a technical one: Google read them and decided they were not worth keeping. Hold on to that phrase, because it is where the next two sections lead.

We failed one of these checks ourselves while writing this guide. Our sitemap stamped the build date as the lastmod of seventy pages, so every deploy told Google they had all changed that day. It is a small thing, and exactly the kind of signal a search engine learns to ignore, along with the rest of your sitemap.

What Google rewards, and the policy small sites trip over

Google's guidance on content comes down to one question it asks you to put to every page: was it made for people, or made primarily to attract visits from search engines? The same document lists warning signs, and two of them describe what many small businesses now do with AI writing tools: using extensive automation to produce content on many topics, and producing large amounts of text with generative AI without real oversight.

AI itself is not the problem. Google's published position since 2023 is that it judges the quality of content, not how it was produced. What it acts against is the pattern its spam policies call scaled content abuse: many pages generated mainly to manipulate rankings rather than to help users. The policy does not care whether a person, a template or a model typed the words.

For a small business this has a practical reading. A page earns its place when it carries something only you can supply: your prices, your process, the questions your customers actually ask, the job that went wrong and why. A page that restates what a hundred other sites already say adds nothing, however well written. Ten of the first kind will do more for you than a hundred of the second, and a hundred of the second can pull the ten down with them.

That last point is the one most guides leave out, and it is the one we learned the hard way. What happened to us, below, only makes sense if Google judges a site as a whole as well as page by page: when most of what you publish is thin, off your subject, or written for search volume, the good pages on the same site stop being shown too. Which brings us to October.

Our own case: how this site lost Google in two days

BigLobster sells AI agents that run a small business's website and content. Since May 2026 our own agents had been writing this blog, in English and Spanish, at around 45 posts a month. Some of it was good. Much of it was about subjects our customers never come to us for: hiring, tax, industrial grants, workplace safety law, AI research news. The table shows what Google Search Console recorded, week by week, for web search.

The peak was partly an illusion. About 940 of the 2,440 impressions in those three months came from six posts about software price and product changes, news that stopped being searched soon after it happened, and 186 came from automated queries that no person typed. Even so, against the 100–140 a week we had before the peak, the drop was more than 90%.

We checked the plumbing first, and all of it passed: robots.txt open, every key page answering 200, self-referencing canonicals, no noindex, a valid sitemap. Then three tests that cost nothing. A site:biglobster.top search still listed our pages, including posts from the week of the drop, so we had not been removed. A Google search for the exact title of one of our posts, in quotes, did not show that post on the first page, and adding our brand name did not help. And a search for "biglobster", our own made-up name, no longer showed the site on page one; Search Console had it at an average position of 5 for that query before the drop. Finally, the Manual actions report was empty, which rules out a penalty applied by a Google reviewer.

Indexed, but not shown even for our own titles and name, with no manual action: that is an algorithmic judgement about the whole site. When we crossed every post with three months of Search Console data, the reason was not subtle. Of the 204 posts older than six weeks, 87 had not been shown in Google a single time. Our site looked like a general magazine about small business and AI, and a magazine needs a reputation a two-person company does not have.

What we changed, and what we would tell you to change

We did four things on 1 November 2026. None of them is clever; each one is something a small business can copy.

We took the dead weight out of Google, without deleting it. The 87 posts with zero impressions now carry a robots meta tag addressed to Google alone, <meta name="googlebot" content="noindex, follow">, which Google's documentation allows by naming googlebot in the tag. They stay published, linked and in our feeds, and they left the sitemap, which went from 342 to 255 addresses. We scoped the tag to Google because Google is where we had the problem and the data; other search engines and assistants were not part of the evidence, so we left them alone. Deleting was the other option. We chose noindex because it can be undone by removing one line.

We picked five subjects and stopped writing about anything else. They are the things we sell: getting found on Google and in AI answers, content that sells, turning visitors into enquiries, a website that works, and running AI agents in a small business. Each subject has a hub page, this guide among them, and every new post has to belong to one and link to it. The list lives in one file that every writing agent reads, so the rule is not left to each agent's judgement.

We cut the volume. From roughly 45 posts a month to at most one a week, and a Spanish version only once the English one has earned 20 Google impressions. A week with nothing worth saying is a week with no post.

We put first-hand material first. When two topics are equally good, the one built on our own data wins. This page is the first example.

We do not know yet whether it worked. Google re-evaluates a site over weeks or months, not days, and the first signal we are watching is simple: our own name back at the top for a search for "biglobster". We will update this section with what happens, including if nothing does.

If you run the same three tests on your site and get the same pattern, the remedy is the same, scaled to you. List what you sell. Pull every page that is not about one of those things and has earned nothing in three months out of Google. Publish less, and only on your subjects. Our Content Gap Hunter now works to exactly these rules, but the rules matter more than who follows them.

Getting cited by AI assistants without breaking anything

Once the floor is solid, being cited by AI assistants is mostly a matter of not locking them out and of writing in a way a machine can quote. Start with your crawler rules. In robots.txt, the useful split is between crawlers that search and crawlers that train. OpenAI documents both: OAI-SearchBot surfaces websites in ChatGPT's search features, while GPTBot gathers data to improve its models. Blocking the training crawler is a legitimate choice; blocking the search crawler removes you from the answers. Check each assistant's own documentation for its crawler names, and check what your hosting or CDN blocks on your behalf.

Then make each page easy to quote. Assistants lift short, self-contained passages, so the pages they cite tend to answer a question in the first lines and define their terms plainly. A practical habit: open each important page with two or three sentences that would still make sense if someone pasted them, alone, into a reply. The "Key concepts" box at the top of this guide is written that way on purpose.

Say clearly, and in the same words everywhere, what your business is. If your homepage, your About page and your social profiles describe you three different ways, a machine has three guesses to choose from. One plain sentence, repeated, is how an assistant learns what to cite you for.

Two things are less useful than they are made to sound. The first is llms.txt, a file that summarises your site for language models. It is a proposal for a standard, and the proposal itself does not claim that any search engine reads it. It costs little to add, so add it if you like, but do not expect it to move anything. The second is special AI markup: for Google's AI features there is no special structured data to add. Normal, accurate structured data for what you are, a local business, a product or an article, is enough.

A monthly routine for an owner with no marketing team

None of this needs a specialist, but it does need a habit. Once a month, set aside about an hour and work through the list below in Search Console and a private browser window. The order matters: each step only makes sense if the one before it passed.

The brand and exact-title searches are the two we wish we had been running every month. They take two minutes and they catch a site-wide problem before the traffic chart does. Our impressions looked healthy until the week they did not, and the title test would have shown the same thing from the first day.

Keep a one-line note of each month's results. Search engines move slowly, and the pattern over six months tells you more than any single check. If something does go wrong, that note is what lets you date it, which is the first thing you need to find the cause.

When to do nothing, and what to stop paying for

Some of the most useful SEO decisions for a small business are things not to do. If your site is small, accurate, about what you sell, and your customers find you through referrals or a local listing, a content programme may cost more than it returns. One good page per service, a correct Google Business Profile and the checks above can be the whole job.

Stop paying for volume. A plan that promises a number of posts a month, whoever or whatever writes them, is selling exactly what got our site demoted. Ask instead what each page will contain that is yours.

Do not chase the news. Posts about a price change or a product launch can bring a burst of visits, and that burst made our own chart look better than the site was. A month later those pages say something out of date, and they are still yours.

Do not delete pages in a panic. Removing a page that earns visits loses those visits; pulling pages that earn nothing out of Google with a noindex is reversible and enough. And do not block a crawler you have not identified: check what it is for before you shut the door, because some of them are the ones that send customers.

If you would rather hand the routine to someone, our SEO/GEO On-Site agent runs the technical checks on a schedule and fixes what it finds, and the Off-Site GEO Scout watches how your brand shows up outside your own website. Both work from the same principle this guide does: fewer, better, about one thing.

Frequently asked questions

Does Google penalise content written with AI?

Not for being AI-written. Google says it rewards quality regardless of how content is made. What it acts against is many pages produced mainly to rank rather than to help, which its spam policies call scaled content abuse, whether a person or a model wrote them.

Do I need special markup or a file to appear in Google's AI Overviews?

No. Google's documentation says a page needs to be indexed and eligible to show with a snippet, with no extra requirements, markup or AI-specific files. Normal SEO is the way in.

If I block GPTBot, will I disappear from ChatGPT?

Not from its search answers, as long as you allow OAI-SearchBot. OpenAI documents the two crawlers separately: OAI-SearchBot for search, GPTBot for model training.

How do I know if Google has penalised my site?

A penalty applied by a person shows in Search Console under Manual actions. An algorithmic drop shows nothing there: the signs are pages still indexed but not shown, even for their exact titles or your business name.

Should I delete old blog posts that get no traffic?

Pull them out of Google first, with a noindex, rather than deleting them. It is reversible, keeps the pages for anyone who follows a link, and removes them from Google's assessment of your site.