URL Inspection API limits: 2,000 a day, a Pacific midnight, and what that means at scale

Anyone who builds indexing monitoring on Google’s official API runs into the same wall within a week. The API is excellent: authoritative data, clean JSON, no scraping. It is also strictly rationed, and the ration is smaller than most people assume.

The numbers

2,000 inspections per day, per property.
600 inspections per minute, per property.

Both limits are per Search Console property — not per account, not per Cloud project, not per API key. A property is sc-domain:example.com or https://example.com/ as it exists in Search Console.

There is a separate per-project quota on the Search Console API as a whole, which matters when you serve many properties from one Google Cloud project, but it is not what constrains a single site. For one site, 2,000 a day is the ceiling and no amount of engineering moves it.

The consequence people miss

A 1,500-URL site can be inspected completely, every day, with room to spare.

A 20,000-URL site cannot. Not with two Cloud projects, not with ten, not with a queue that runs all night. The per-property limit is per property. Sharding across multiple Google Cloud projects raises how many different sites you can serve in parallel; it does nothing for one large site.

This is the single most common misconception we encounter, and it is worth stating plainly because it determines the architecture of everything built on top: at scale, indexing monitoring is a prioritisation problem, not a throughput problem.

The Pacific midnight

Google’s quotas reset at midnight Pacific Time. If you are anywhere in Europe, that is mid-morning — 09:00 or 10:00 depending on daylight saving.

The practical consequence catches almost everyone once. You schedule your nightly run for 03:00 local time, it consumes 3,800 inspections, and the next morning your dashboard reports one inspection used today. Nothing is broken: your night run landed at 17:00 or 18:00 Pacific, in the previous quota day.

Any accounting you build has to use Pacific dates, or your numbers will make no sense to anyone reading them at breakfast. We learned this the way everyone does — by staring at a counter reading 1 after a night that clearly did thousands.

Latency, and why batches are the wrong shape

A single inspection typically returns in three to seven seconds. The tail is long: a small share of requests take 40 to 75 seconds, and a few time out entirely.

If you process in fixed batches — fire 50, wait for all 50, fire the next 50 — every batch runs at the speed of its slowest member. With a tail that long, throughput collapses.

A sliding concurrency window fixes it: keep N requests in flight, and the moment one finishes, start the next. Same concurrency limit, dramatically better utilisation, and no single slow URL blocking anyone.

Reading errors correctly

Three status codes matter, and confusing them causes real damage.

403 is ambiguous. It means either “this principal has lost access to the property” or “you have exhausted quota”. A monitoring system that treats every 403 as lost access will send false alarms every time it hits a limit; one that treats every 403 as quota will stay silent for weeks after a client removes your access.

The way to tell them apart is at the batch level rather than the request level. If you reserve quota before dispatching — so you know you were within your own allowance — and every request in the batch comes back 403 with zero successes, that is access, not quota. A genuine quota exhaustion shows up as a partial batch: some successes, then refusals.

429 means back off. Not retry immediately, not retry the whole batch. Stop, return the unused reservation, and let the next scheduled run pick the URLs up.

Network failures are not verdicts. A timeout tells you nothing about the page. Leave the URL’s previous state untouched and put it first in the next queue. The alternative — recording “not indexed” because the request failed — turns your infrastructure hiccups into alerts about your customer’s site, which is the fastest way to make an alerting product worthless.

How we spend the 2,000

Since the ration is fixed, the only lever is deciding which URLs get today’s allowance. We sort every URL into one of four tiers and check them at different intervals:

• New — recently added, no verdict yet, or fewer than a handful of checks. Checked daily until the state stabilises.
• Hot — changed state recently, or currently not indexed. Checked daily; these are the URLs where something is actually happening.
• Stable — indexed, unchanged for a long stretch. Checked roughly weekly, because a page indexed for eight months is unlikely to change tonight, and if it does, catching it a few days later is an acceptable trade for covering everything else.
• Archive — deliberately deprioritised to a fortnightly cadence: dormant pages that still deserve an occasional look.

A slice of each property’s daily allowance is held back for manual re-checks, so that pressing “check now” after a fix returns an answer in seconds rather than failing because the night run consumed everything.

Two more rules keep the ration honest. URLs that return “not found” on several consecutive checks are removed from monitoring automatically — a deleted page should not consume a daily inspection forever. And when the day’s allowance genuinely runs out before the queue does, we say so on the project page rather than letting the “last checked” date quietly go stale, because an unexplained old timestamp looks exactly like a broken product.

Why the limits are a feature

It is tempting to see 2,000 a day as an obstacle to route around. It isn’t. It is the reason this data is trustworthy: the number comes from Google’s own index, for a property you have proven you control, without a scraper guessing from search results that are personalised, localised and deduplicated.

The constraint forces a genuinely better product — one that thinks about which pages matter today rather than checking everything at maximum frequency and calling that thoroughness.

Crawlert is built entirely inside those limits, and the scheduling described above is what runs every night.