The Change (January 15th)
On January 15th, Google introduced a significant update to their website, requiring JavaScript execution to retrieve search results. Traditionally, the Search Engine Results Page (SERP) industry relied on HTTP requests to Google, such as curl requests, simulating browser behavior by including headers like User-Agent, Client-Hints, etc.
Previously, requests could even be made without cookies, effectively replicating an incognito browser search. With the new update, Google now serves a page containing a script (possibly accompanied by an anti-bot challenge) that redirects to the actual search results page with an additional dynamically generated query parameter, sei. However, directly accessing the URL with the sei parameter is not feasible due to the absence of proper cookies, leading to a block or CAPTCHA (the infamous "Sorry" page).
Solving the CAPTCHA is technically possible but highly inefficient and costly compared to traditional SERP requests. Moreover, even accessing the /search URL from the homepage results in the redirect page unless proper cookies are set. The only way to acquire these cookies is to execute a search, render the JavaScript, bypass the redirect, and reach the search results page. Notably, Google has added multiple new cookies to facilitate this process.
The Old Methods
Historically, Google search results were obtained using two approaches:
Both methods utilized the vx6 driver for efficient and lightweight interactions.
The Mitigation (January 16th–18th)
As the update rolled out, block rates began to rise, and the number of empty Google HTML responses increased significantly. To address this, we transitioned Google SERP requests to the vx8 driver. This involved rendering the JavaScript, waiting for the actual search results page to load, and retrieving the data.
While this ensured valid SERP requests, it came at a cost:
Response times increased from ~2–4 seconds to 10 seconds or more.
The system's ability to scale during traffic spikes was reduced, as creating and managing multiple browser instances required substantial resources.
The Solution (January 19th)
To maintain an efficient and competitive SERP API, we developed a custom cookie generation flow. This process involved performing a preliminary search for a random term, saving the resulting cookies, and using them for subsequent requests.
After deploying this solution, additional adjustments were made to address:
Following these fixes, the SERP API regained its original speed and accuracy.
Ads Optimization
The ads optimization feature, which previously relied on cookie-less curl requests, is no longer viable under the new changes. While further research is required to adapt to Google's updated ad policies, the feature has been transitioned to the vx8 driver. Although this approach is slower and more resource-intensive, it remains functional and appears to yield more ads in response.
Ongoing monitoring and adjustments will be necessary to optimize ad-related processes under the new system.
