How to scrape Google search results in Python (without running a scraper)

Scraping Google directly is a maintenance project, not a task. You need residential proxies, a headless browser that survives detection, HTML selectors that break whenever the results page changes, and a retry strategy for the days it all falls over anyway. For most teams the actual goal — get the results as data — never justifies that.

The alternative is to call a hosted search API: one HTTP request, JSON back, someone else's problem to keep working. This walks through doing that from Python with the Google Web Search API on JoJ API.

What you need

An API key from your workspace. The same key works across every API you subscribe to on the marketplace, and it goes in the X-JoJAPI-Key header.

The smallest thing that works

import requests

response = requests.get(
    "https://gsearch.jojapi.net/v1/",
    headers={"X-JoJAPI-Key": "YOUR_KEY"},
    params={"query": "best python http client", "limit": 20},
    timeout=30,
)
response.raise_for_status()
data = response.json()

for result in data["results"]:
    print(result["title"], "—", result["url"])

You can fire the same request from the playground on the endpoint page and inspect the response there before writing any code.

What comes back

Five top-level keys:

{
  "search_term": "Harry Potter",
  "next_cursor": "CBQ=",
  "results": [
    {
      "url": "https://www.harrypotter.com/",
      "title": "Harry Potter | Official home of Harry Potter, Hogwarts Sorting, and ...",
      "description": "Looking for Wizarding World? HarryPotter.com is the official home of...",
      "timestamp": null
    }
  ],
  "knowledge_panel": { "name": "Harry Potter", "label": "Buchreihe", "description": {}, "image": {}, "info": [] },
  "related_keywords": { "spelling_suggestion": null, "keywords": [] }
}

results is the organic listing, in rank order — results[0] is position one. Each entry has url, title, description and a timestamp that is null unless Google showed a date on the result.

The parameters

Parameter	Type	Required	What it does
`query`	string	yes	The search query, exactly as you'd type it into Google
`limit`	integer	no	How many results to return
`related_keywords`	boolean	no	Also return the related searches Google shows at the bottom
`cursor`	string	no	Pagination token, taken from the previous response

related_keywords is the one people miss. If you're doing keyword research or building a topic map, it hands you Google's own notion of what's adjacent to your query, which is considerably better than guessing. More on it below.

Paginating

Pagination is cursor-based: you pass back the token the previous response gave you rather than computing an offset, so the loop stays correct even when the underlying result set shifts between calls.

One thing to watch — the names don't match. You send cursor, but the response returns it as next_cursor. Reading payload["cursor"] gets you None and a loop that silently stops after one page.

import requests

API_URL = "https://gsearch.jojapi.net/v1/"
HEADERS = {"X-JoJAPI-Key": "YOUR_KEY"}


def search_all(query, pages=5, per_page=20):
    """Yield each page of results for a query."""
    cursor = None

    for _ in range(pages):
        params = {"query": query, "limit": per_page}
        if cursor:
            params["cursor"] = cursor

        response = requests.get(API_URL, headers=HEADERS, params=params, timeout=30)
        response.raise_for_status()
        payload = response.json()

        yield payload["results"]

        # Sent as `cursor`, returned as `next_cursor`. Absent on the last page.
        cursor = payload.get("next_cursor")
        if not cursor:
            break


for results in search_all("open source vector database"):
    for result in results:
        print(result["title"], "—", result["url"])

Cap the number of pages. An unbounded while cursor: loop is how you turn a five-dollar experiment into a surprise.

Handling the errors that actually happen

Three responses are worth branching on:

import time
import requests


def search(query, retries=3, **params):
    for attempt in range(retries):
        response = requests.get(
            API_URL,
            headers=HEADERS,
            params={"query": query, **params},
            timeout=30,
        )

        if response.status_code == 429:
            # Rate limited by the plan's request ceiling — back off and retry.
            time.sleep(2 ** attempt)
            continue

        if response.status_code == 402:
            # Out of balance, or the plan does not cover this request.
            raise RuntimeError("Top up your wallet or move to a larger plan")

        response.raise_for_status()
        return response.json()

    raise RuntimeError(f"Still rate limited after {retries} attempts")

429 is worth retrying with exponential backoff. 402 is not — no amount of retrying creates balance, and a retry loop against a payment error just burns time. 401 means the key is wrong or not subscribed to this API.

Knowing what it costs, per request

Every response through the gateway carries a usage header:

response = requests.get(API_URL, headers=HEADERS, params={"query": "example"})
print(response.headers.get("X-Jojapi-Results-Used"))

Log that alongside your requests and you have a per-query cost breakdown without waiting for an invoice — useful when you're deciding whether a nightly job over ten thousand keywords is worth running.

Writing results to CSV

Putting it together — a script that runs a list of queries and writes one row per result:

import csv
import requests

API_URL = "https://gsearch.jojapi.net/v1/"
HEADERS = {"X-JoJAPI-Key": "YOUR_KEY"}

QUERIES = [
    "best python http client",
    "python async http",
    "requests vs httpx",
]


def fetch(query, limit=20):
    response = requests.get(
        API_URL,
        headers=HEADERS,
        params={"query": query, "limit": limit, "related_keywords": True},
        timeout=30,
    )
    response.raise_for_status()
    return response.json()


with open("results.csv", "w", newline="", encoding="utf-8") as handle:
    writer = csv.writer(handle)
    writer.writerow(["query", "position", "title", "url", "description"])

    for query in QUERIES:
        payload = fetch(query)

        for position, result in enumerate(payload["results"], start=1):
            writer.writerow(
                [
                    query,
                    position,
                    result["title"],
                    result["url"],
                    result["description"],
                ]
            )

print("Wrote results.csv")

Related keywords, for research

With related_keywords=true you also get back what Google suggests around the query — and a spelling correction when it thinks you mistyped:

payload = fetch("harry potter")
related = payload["related_keywords"]

if related["spelling_suggestion"]:
    print("Did you mean:", related["spelling_suggestion"])

for item in related["keywords"]:
    print(item["position"], item["keyword"])

Each entry carries a position, the keyword itself, a keyword_html variant with Google's bold markup, and sometimes a knowledge block with a title, label and thumbnail. Run this across a seed list and you have the beginnings of a topic map without paying for a keyword tool.

The knowledge panel

When Google shows an entity panel, it comes back in knowledge_panel:

panel = payload.get("knowledge_panel")

if panel:
    print(panel["name"], "—", panel["label"])          # "Harry Potter — Novel series"
    print(panel["description"]["text"])
    print("source:", panel["description"]["site"], panel["description"]["url"])

info holds the attribute rows the panel displays — author, release date and so on — each as a title plus a list of labels. It's a cheap way to enrich an entity without a second lookup, and it's null for queries with no panel, so check before reading.

When you'd still write your own scraper

This approach is worse than a custom scraper in exactly one case: you need something the API doesn't expose, and you need it badly enough to own the maintenance. If you're extracting an unusual SERP feature, or scraping a site with no API equivalent at all, roll your own — and expect to keep fixing it.

For everything else, the hosted call is cheaper than the engineering time you'd spend keeping selectors alive.

Google Web Search API — full endpoint reference, plans and a live playground
Web scraping APIs — the rest of the scraping catalog
Search APIs — image search, SERP data and keyword tools