Analysis: Google's `google.com/goto` Anti-Scraping Shift

Published: August 27, 2026 | Category: Google | Authors: Autom Team

Ready to scale your data collection? Stop fighting the bots and start scraping with Autom today. Discover the full potential of our API. [ $\text{Start for free} \rightarrow$ ]

🔍 What is Happening?

Google Search has implemented a significant change in how it handles organic search result links. Instead of providing the destination URL directly within the HTML source, Google is now rewriting these links to use a redirect wrapper: google.com/goto?url=...

When a user clicks a result, Google triggers a redirect to the actual destination. Essentially, the link in the HTML is now an opaque reference to a record within Google's own index.

Current Status

As of late August 2026, this behavior is consistently observed when:

The user is logged out.
The user is browsing in private/incognito mode.

While this may have started as an A/B test, it has evolved beyond a small subset of Search Engine Results Pages (SERPs) and is now widespread.

The Request Flow

The following diagram illustrates the new redirection logic:

⚖️ `google.com/url` vs. `google.com/goto`

It is important to note that this is not the same as the older google.com/url wrapper. The differences are fundamental to how scrapers must operate.

Feature	Old Format (`/url`)	New Format (`/goto`)
URL Structure	`google.com/url?q=[Encoded URL]`	`google.com/goto?url=[Opaque Blob]`
Offline Decoding	Possible via URL decoding	~~Impossible~~ (Requires server request)
Resolution Method	String manipulation	Reading the `Location` header
Visibility	Target link is readable in query	Target link is hidden

Note: Google still includes the domain, favicon, and attribution on the SERP to render the page correctly, so the URL exists in the HTML, but not as the primary href.

🎯 Why is Google Doing This?

This update is a strategic move to combat automated SERP harvesting. Google is specifically targeting AI crawlers and SEO tools that bulk-extract URLs to build competing indexes.

The "Cost" of Scraping

In the past, a scraper could parse thousands of destination links from a single HTML page without ever interacting with Google again. Now, the economics have shifted.

Using a simple mathematical representation of the request cost: $\text{Total Requests} = R_{page} + \sum_{i=1}^{n} R_{link_i}$ Where $R_{page}$ is the initial SERP request and $n$ is the number of links to be resolved.

The implications are clear:

Increased Latency: Scraping is now significantly slower.
Higher Visibility: Every link resolution creates a "ping" back to Google.
Easier Detection: Google can easily identify clients resolving hundreds of links in rapid succession.

This is part of a broader campaign to increase the "cost" of scraping, following the removal of the num=100 parameter and the strengthening of BotGuard and SearchGuard.

🛠️ The Autom Response

At Autom, we first noticed goto links appearing on a small fraction of results. Initially, we held off on a global fix to avoid disrupting stable responses. However, since the pattern is now dominant for private/logged-out sessions, we have deployed a permanent solution.

Our Technical Implementation

We have updated our Google Search pipeline to handle these redirects automatically. Our system now performs the following logic:

# Conceptual logic for resolving /goto links
if link.startswith("google.com/goto"):
    # Use a HEAD request to minimize bandwidth
    response = requests.head(link, allow_redirects=False)
    final_url = response.headers.get('Location')
    return final_url

What this means for you:

Automatic Resolution: We read the Location header and follow the redirect.
No Integration Changes: The final destination URL is returned in the same structured fields you already use.
Continuous Monitoring: We are tracking the rollout to adjust if the format changes again.

📚 Further Reading & Resources

If you want to dive deeper into the technical side of SERP changes, check out these articles:

google.com/goto: How to read Location with HEAD requests.
The death of the num=100 parameter.
Analysis of the Google vs. SerpAPI lawsuit and SearchGuard.
Comparative guide: Scraping SERPs across Google, Bing, and Brave.

Stop chasing the goalposts. Get reliable, live SERP data without the headache.

Trial: Get 1,000 free requests at Autom Pricing.
Access: Grab your API key at app.autom.dev/register.

Also from the Autom Blog:

Announcement: Our new website is live! (Jan 15, 2025)
Sales: Top 7 Google Sheets Extensions for Data Enrichment in 2026 (Feb 26, 2026)
Automation: MCP: Integrating Live SERP Data into Claude, Cursor, and Gemini (Apr 5, 2026)
Google: A viable alternative for Google Search Scraping (Jan 24, 2026)
SERP API: Why developers prefer Autom.

Analysis: Google's google.com/goto Anti-Scraping Shift