How I measured search visibility when the search engines pushed back

The problem

The program offered free advice to small businesses across a large region, and nobody knew how often it showed up when an owner searched for help. Paid SEO suites were out of budget, so I answered the question directly. I wrote hundreds of queries for the region and its industries, ran them on Google, Bing and DuckDuckGo with a Python script I wrote around Crawl4AI, an open-source crawler, and logged every result. The tables and charts went to the team with the page changes I recommended.

The method

1. I built the query bank deliberately. Each topic had several framings (the region’s name, a nearby town, “small business help”, a plain service phrase), so the result would show which framing won, not only which topic.
2. I added controls. Every batch included raw service queries the program should not rank for. If those scored above zero, the measurement was leaking.
3. I read three engines. Two through a self-hosted Crawl4AI instance driven by my query script, one through a real browser session.
4. I scored by expected clicks, not by “ranked or not” (below).
5. I reported gaps with their owners: which organisations held the spots the program was missing, and whether the gap was winnable or a different intent altogether.

When the engines push back

One engine served a block page as soon as the automated browser arrived. I assumed the IP address at first. It wasn’t. The tell was the browser announcing itself as automated. Running an ordinary browser session on the same connection, without the automation flag, returned normal results unattended, with no proxy and no CAPTCHA service.

Two smaller fixes mattered as much:

• One engine wrapped every result link in a redirect. The first parser read those as “no results” and reported false absences until I decoded the links.
• One engine rate-limited after about a dozen consecutive queries. I retried zero results after that point individually and slowly before they counted.

# Similar methodology, rewritten for this page:
# unwrap a redirect-style result link to its real destination.
from base64 import urlsafe_b64decode
from urllib.parse import urlparse, parse_qs, unquote

def real_url(href: str) -> str:
    q = parse_qs(urlparse(href).query)
    if "u" in q and q["u"][0].startswith("a1"):   # base64url after a prefix
        raw = q["u"][0][2:]
        return urlsafe_b64decode(raw + "=" * (-len(raw) % 4)).decode("utf-8", "replace")
    for key in ("uddg", "q", "url"):              # percent-encoded targets
        if key in q:
            return unquote(q[key][0])
    return href

The score

A first-place result gets far more clicks than a tenth-place one, so a simple “ranked or not” overstates weak placements. I weighted each rank by an expected click share, then took the mean for a query set.

What the synthetic run shows

The synthetic run shows the kind of pattern the method surfaces: a site that reaches page one only when a query carries one particular phrase, and is absent for the same topic phrased another way. A pattern like that turns a long list of gaps into one actionable change: lead titles and headings with the phrase people actually search.

What it proves

In an interview, I can walk through

• how I mined the query bank from a large keyword-research workbook without testing every row;
• why results can differ so much between engines, and what that means for a small organisation;
• the on-site changes I recommended, and the order I put them in.