A page that exists is not a page that is found. On most Winnipeg freight and manufacturing sites the pages that matter are outnumbered fifty to one by pages built for towns the company has no address in — and the crawler spends its time accordingly.
Ranking gets all the attention, but it is the last step of three. Before a page can rank it has to be indexed, and before it can be indexed it has to be discovered. Skip either of the first two and the content work behind the page was wasted.
This step is invisible from the outside. Nobody notices that a new service page has been sitting unindexed for eleven weeks, because nothing on the site looks broken. The page loads, the menu works, the form submits. It simply is not in the index, and no amount of rewriting will change that.
Discovery, indexing and ranking are not the same thing
Discovery is the crawler learning that a URL exists, usually from a link or a sitemap entry. Indexing is Google deciding the page is worth storing. Ranking is the contest that follows. Each stage drops pages, and the drop is silent.
Most owners assume publication equals availability. On a small site with clean internal linking that assumption often holds, and pages appear within days. On a site with several hundred URLs, most of them thin and near-identical, it stops holding — and it stops holding first for the newest pages, which are the ones you just paid for.
| Stage | What has to happen | What goes wrong |
|---|---|---|
| Discovery | A crawler finds the URL | No internal link, no sitemap entry |
| Crawl | The page is fetched | Budget spent on duplicates and parameters |
| Indexing | Google stores the page | Judged thin or duplicate, quietly dropped |
| Ranking | The page competes | Directories and load boards hold the results |
The Indexing Hub sits on the first two stages. It does not affect the third, and being clear about that boundary is the difference between using the tool well and expecting something it cannot deliver.
Location-page sprawl, and why it eats everything else
Here is the pattern, and it is close to universal on Manitoba trucking and 3PL sites built between 2016 and 2020. Somebody sold the company on local coverage. The result is a page for every town within four hundred kilometres — Steinbach, Selkirk, Morden, Winkler, Dauphin, Portage la Prairie — each one a copy of the last with the place name swapped in. Then came lane pages: one URL for every origin and destination pair the dispatch software could enumerate.
The arithmetic is brutal. Forty towns is forty pages. Forty origins against forty destinations is sixteen hundred lane URLs, generated automatically, differing by two proper nouns each. Meanwhile the company has perhaps twelve pages that describe something real: dry van, reefer, flatbed, LTL consolidation, cross-dock, bonded warehousing, customs paperwork, the equipment list, three case studies and a contact page.
Every one of those generated pages is a URL the crawler has to fetch, evaluate and mostly discard. They compete with each other for the same queries, so the pages view in Search Console shows Google picking a different one each month. And they claim addresses the company does not hold, which is a credibility problem as well as a technical one.
What to do instead is unglamorous and cheap. Consolidate the towns into one regional coverage page that lists them honestly as service areas rather than pretending to an address in each. Replace the lane matrix with a small number of corridor pages that describe something specific: the Winnipeg–Chicago run, northbound freight from the Emerson crossing, temperature-controlled service into Saskatchewan. Five real pages outperform sixteen hundred generated ones, and they cost less to maintain.
The town matrix
Near-identical pages for places with no address, no phone number and no staff. Redirect them to one coverage page.
- 301 to the regional page
- Drop them from the sitemap
The lane matrix
Origin-destination pairs generated in bulk. Keep the corridors you actually run and delete the rest.
- Keep five to ten corridors
- Write each one by hand
Equipment and service pages
Reefer, flatbed, LTL consolidation, bonded storage. These answer a specific question a shipper actually types.
Proof pages
Certifications, insurance, customs handling, case studies with a commodity and a route. Hard to generate, hard to copy.
Crawl budget, and the things that quietly spend it
Crawl budget is the attention a search engine gives your domain in a period. It is not published, it is not configurable, and on a small site it is rarely the binding constraint — until sprawl makes it one. What matters is the ratio: how much of the crawling lands on pages you care about.
The consumers are predictable. Generated location and lane pages come first. After them come URL parameters from filters and sorting, printer-friendly duplicates, session identifiers, staging copies left reachable, calendar and pagination chains that run forever, and old PDF rate sheets nobody has removed since 2018.
- Duplicate paths to the same page. With and without a trailing slash, http and https, www and bare domain, uppercase variants. Each doubles the work for no gain.
- Filter and sort parameters. A warehousing site with a searchable inventory can generate thousands of URL combinations from one page of stock.
- Redirect chains. Three hops to reach a page multiplies the cost of every visit. After a rebrand these accumulate quickly.
- Slow responses. A server that takes two seconds per page is crawled less than one that takes three hundred milliseconds. Hosting is a crawl question, not only a user one.
- Orphan pages. A page nothing links to is discovered only through a sitemap, if at all. Print shops accumulate these with every campaign microsite.
- Soft errors. Pages returning 200 with "no results found" look like content and are crawled like content.
None of these needs a developer retainer to fix. They need one afternoon with a crawl report and a list. The reason they persist on owner-run sites is not difficulty; it is that nothing visibly breaks, so nothing gets scheduled.
The sitemap is a discovery tool, and most are neither
A sitemap is usually treated as a file a plugin generates and nobody opens. Read properly it is a statement: these are the URLs I consider worth indexing. When it contains sixteen hundred lane pages, that statement is doing damage.
The Indexing Hub accepts a sitemap by file upload or by URL and parses it recursively to three levels — an index pointing to indexes pointing to sitemaps — with up to 1,000 sitemaps handled in a single job. That depth exists for genuinely large structures. Most sites here need two levels at most, and many need one.
Submission and sitemap parsing
The discovery half of the panel: what gets pushed, how fast, and what comes back.
- 1,000 URLs per day, per account. A hard daily ceiling on the URL tracker. It shapes every plan and cannot be negotiated upward by submitting harder.
- 10,000 URLs per bulk batch. The batch is the container; the daily budget is the rate at which it drains.
- Recursive sitemap parsing, three levels deep. Upload the file or point at the URL. Up to 1,000 sitemaps per job.
- Two concurrent jobs, twenty queued. Enough to run a full site and a section refresh at once without a scheduling system.
- IndexNow submission. Pushes to GoogleBot and BingBot rather than waiting to be found.
The two-concurrent limit is more useful than it sounds. A carrier consolidating its town pages can run the full sitemap in one job while a second job handles the fifteen corridor pages that replaced the lane matrix, and the queue absorbs everything else. Twenty queued jobs is more scheduling than an owner-run site will ever need.
One thousand a day, ten thousand a batch, and what IndexNow does
The two numbers work together and are frequently confused. The batch is how many URLs you can hand over at once: up to 10,000. The daily budget is how many are processed: 1,000 per account per day. A full 10,000-URL batch therefore takes ten days to drain, and no arrangement of the queue changes that.
This is the constraint that turns sprawl from an abstraction into a schedule. A site with 1,700 pages needs the better part of two days just to submit everything once, and it will spend most of that budget on pages you would delete if you looked at them. A site with 120 well-chosen pages submits its whole inventory in one morning and has budget left over for the next three months of changes.
IndexNow is the delivery mechanism. Instead of waiting for a crawler to arrive by its own schedule, the API notifies GoogleBot and BingBot that a URL is new or changed. It shortens the wait for a visit. It has no opinion about what happens after the visit.
| Site shape | URLs | Days to submit once | Sensible use of the budget |
|---|---|---|---|
| Sprawled carrier site | 1,700 | 2 | Consolidate first, then submit 150 |
| Consolidated carrier site | 150 | Under 1 | Full submission, then changes only |
| 3PL with inventory filters | 9,000 | 9 | Block parameters, submit the 200 real pages |
| Food processor after a rebuild | 400 | 1 | Whole site once, then weekly deltas |
| Print shop with microsites | 600 | 1 | Retire dead microsites before submitting |
The pattern in that table is consistent. In almost every case the right first move is not to submit more URLs. It is to have fewer of them, then submit those. The indexing tools in the panel make the second step quick; only you can do the first.
What the batch status actually tells you
Each URL carries its own submission record: the bot visit with a timestamp, the status, and error detail when something failed. Above that sit live counters for submitted, found and failed URLs. It is more feedback than most tools give, and it is still not a report on indexing.
Read the three counters in order. Submitted means the URL left your account — a fact about you, not about Google. Found means a bot arrived and fetched the page, which is the first genuinely external signal. Failed means the fetch broke, and the error detail usually names something you control: a 404 from a stale sitemap entry, a redirect loop, a timeout on a slow page, a robots rule blocking a directory somebody forgot about.
A fact about your account
The URL entered the queue. It says nothing about whether a crawler came or what it decided.
The bot arrived
A visit with a timestamp. Discovery worked. Indexing is a separate decision made later and reported elsewhere.
Usually your fault, usually fixable
404s, redirect loops, timeouts, blocked directories. The error detail is the most useful field in the whole log.
Found but never indexed
The common outcome for thin pages. No error, no explanation — the page was fetched and judged not worth storing.
That fourth case is the one worth internalising. A lane page can be submitted successfully, visited promptly, logged with a clean status, and still never appear in the index. Nothing in the batch log will flag it, because nothing went wrong technically. The page was simply not worth keeping, and the only place that shows up is the absence of impressions in Search Console weeks later.
Keeping it clean when nobody is watching the site
A consolidation is a one-off project. Keeping the URL inventory honest afterwards is not, and that is where owner-run companies lose the gains. The site gets a new page in March, a redirect breaks in June, and nobody notices until the following winter.
FullSEO
For a company with no internal marketing capacity that still needs the technical side maintained.
- A team rather than a tool. SEO specialists, developers and writers, which is what a consolidation of sixteen hundred URLs actually requires.
- Human review before changes go live. Nothing reaches the site until someone has read it — the standing condition when the owner signs off personally.
- Manual keyword selection with automatic fallback. Corridor and equipment terms chosen deliberately rather than harvested in bulk.
- Manual link placement with a Domain Authority target. Drawing on a partner network of more than 230,000 sites.
The lighter option is AutoSEO at 149 USD per month per domain, which runs keyword discovery, backlink building and on-site suggestions automatically and includes the same analytics. At recent rates the two tiers are roughly 200 and 680 Canadian dollars a month, approximate. Stream, the assistant in the My SEO area, accepts URL lists in bulk and keeps reports, placements and to-dos in one chronological project feed, which suits somebody who checks in every second week rather than daily.
Common questions
We deleted four hundred location pages. Will that hurt our traffic?
If they were near-identical and earning nothing, no. Redirect each to the regional coverage page that replaces it rather than letting them 404, remove them from the sitemap, and expect a few weeks of unstable reporting while Google reprocesses. Check the pages view first to confirm they really were earning nothing.
How long after submission should a page be indexed?
There is no guaranteed interval. Submission through IndexNow usually shortens the wait for a crawler visit to days rather than weeks, and the log will show the visit with a timestamp. Indexing is a separate decision with no timetable, and for a thin page it may never happen at all.
Should every page on the site be in the sitemap?
Only the ones you want indexed. Canonical URLs returning 200, no redirects, no parameter variants, no thank-you or login pages. A sitemap listing everything is a sitemap that recommends nothing, and it wastes budget on your behalf.
Our site has ninety pages. Does crawl budget matter to us at all?
Rarely as a limit, but the checks still pay. Redirect chains, duplicate paths and orphan pages slow discovery on any size of site, and ninety clean pages get crawled far more thoroughly than ninety pages hidden among four hundred parameter variants.
Can we run the full site and a new section at the same time?
Yes. Two sitemap jobs run concurrently and up to twenty wait in the queue, so a full inventory and a section refresh can go together. The daily ceiling of 1,000 URLs still applies across the account, so concurrency changes the ordering, not the throughput.
The closing calculation
Take the sprawled carrier site: 1,600 lane pages, 40 town pages, 60 genuine pages, 1,700 URLs in total. Submitting all of it consumes not quite two full days of the daily budget, and roughly ninety-six per cent of that goes to pages Google will fetch once and decline to store. Repeat it monthly and the pattern holds: the real pages get a fraction of the attention, permanently.
Now consolidate. Forty town pages become one coverage page. Sixteen hundred lane URLs become eight corridor pages written by hand. The inventory drops to about seventy URLs. The whole site submits in well under one day, with budget spare for every change made for the rest of the quarter — and the crawler now spends its visits on the pages that describe what the company sells.
The order matters more than the tooling. Delete and consolidate first, fix the duplicate paths and redirect chains second, submit third, and read the log fourth. Doing it in the reverse order is the common mistake, and it produces a very tidy record of a crawler repeatedly fetching pages nobody should have published. Related material sits on the blog, and the implementation side under our services.
If you want to see what your own inventory looks like before deciding what to cut, connect the domain in the Semalt dashboard, run one sitemap job and read the counters. For most sites here the useful finding arrives in the first hour: the number of URLs is four figures, and the number of pages anyone would defend is two.