SEO fundamentals: crawling, indexing and ranking explained
How search engines crawl, index and rank pages, where robots.txt, noindex, canonicals and sitemaps fit, and how to check indexing in Google Search Console.
SEO fundamentals are the three stages every search engine runs: crawling (finding and downloading pages), indexing (understanding and storing them) and ranking (choosing which stored pages to show for a query). A page that fails an earlier stage never reaches the next, so most SEO problems are crawling or indexing problems first.
Key takeaways
- Google's process has three stages, crawling, indexing and serving, and Google says it guarantees none of them
- Crawl budget matters only for very large or fast-changing sites
- robots.txt controls crawling, noindex controls indexing; never use both on one page
- Canonical tags and redirects are strong signals and sitemaps a weak one; Google still makes the final choice
- Use the Page indexing report for patterns and the URL Inspection tool for single pages
What are crawling, indexing and ranking?
Crawling, indexing and ranking are the three stages a search engine runs before your page can appear in results. Google's own guide, How Google Search works (updated 18 December 2025), names them crawling, indexing and serving search results:
- Crawling. Automated programs called crawlers download text, images and videos from pages they find. Google's crawler is Googlebot, and it also runs JavaScript.
- Indexing. Google analyses each page and stores what it learns in the Google index. Similar pages are grouped, and one, the canonical, is picked to represent the group.
- Ranking (serving). Google returns the indexed pages it judges most useful for the query, location, language and device.
Google "doesn't accept payment to crawl a site more frequently, or rank it higher", and "doesn't guarantee that it will crawl, index, or serve your page." Each stage is a filter. This article covers the basics of each; for how they connect to AI answers, see my guide to SEO, GEO, AEO and every other kind of engine optimisation.
Why do the fundamentals still matter?
Because every newer search surface sits on top of them. A page that is not crawled cannot be indexed, and a page that is not indexed cannot rank or be quoted in an AI answer. Fix the first two stages and every search surface benefits.
Most ranking problems are really crawling or indexing problems that nobody checked.
What is crawl budget, and is it a myth?
Crawl budget is the number of URLs a search engine can and wants to crawl on your site. It is real, but it only matters for a small number of sites. Google's crawl budget guide (updated 22 July 2026) says it is meant for sites with roughly 1 million or more unique pages that change about weekly, or 10,000 or more pages that change daily, or sites with many URLs stuck as "Discovered - currently not indexed". Google calls these "a rough estimate", not exact thresholds.
It has two parts: crawl capacity limit, how much load your server can take (it falls with slow responses or 5xx and 429 errors), and crawl demand, how much Google wants to crawl, driven by popularity and staleness.
Four common myths, checked against Google's own words:
- "Crawling more helps you rank." No. In a 2017 Search Central post, Gary Illyes wrote that "while crawling is necessary for being in the results, it's not a ranking signal."
- "My 300-page site has a crawl budget problem." Unlikely. The same post says a site with fewer than a few thousand URLs will, most of the time, be crawled efficiently.
- "Blocking pages in robots.txt frees budget for the rest." Only if Google is already hitting your capacity limit. Otherwise, the crawl budget guide says, the freed budget is not moved elsewhere.
- "Crawl-delay in robots.txt slows Googlebot." Googlebot does not process this non-standard rule, per the 2017 post.
How do robots.txt and noindex differ?
robots.txt controls crawling; noindex controls indexing. Mixing them up can keep a page in Google that you wanted out, or out of Google when you wanted it in.
Google's robots.txt introduction says the file manages which URLs crawlers can access, and "is not a mechanism for keeping a web page out of Google." A blocked URL that other sites link to can still be indexed and shown, just without a description.
noindex, a meta tag or HTTP header, makes Google drop the page from results whatever links point to it. The catch: the page must stay crawlable. If robots.txt blocks it, "the crawler will never see the noindex rule".
| You want to… | Use | Do not use |
|---|---|---|
| Keep a page out of search results | noindex (or a password) | robots.txt alone |
| Stop bots wasting time on endless filter or search URLs | robots.txt disallow | noindex (Google still has to fetch the page) |
| Remove a duplicate in favour of another version | rel="canonical" or a 301 redirect | robots.txt or noindex |
How does canonicalisation work?
Canonicalisation is choosing one preferred URL when several URLs show the same or very similar content, such as with and without "www" or with tracking parameters. If you do not choose, Google chooses for you.
Google's guide to consolidating duplicate URLs (updated 10 July 2026) ranks the signals you can send:
- 301 redirects: a strong signal. Use them when you retire a duplicate.
- rel="canonical": a strong signal, placed in the page's head or, for PDFs, in an HTTP header.
- Sitemap inclusion: a weak signal.
The signals stack, but they are not commands: Google still decides. The guide warns against using robots.txt, noindex or the URL removal tool to canonicalise.
Do you need an XML sitemap?
A sitemap helps search engines find your URLs, but it does not guarantee they are crawled or indexed. Google's sitemap overview says small, well-linked sites of about 500 pages or fewer may not need one, while large sites, new sites with few links, and sites heavy on video, images or news benefit most.
Google's build-a-sitemap guide
sets a maximum of 50,000 URLs or 50MB uncompressed per file, with absolute URLs. Google uses lastmod only when it is "consistently and verifiably" accurate, and ignores
priority and changefreq entirely.
IndexNow lets you tell engines the moment a URL changes. Its site lists Microsoft Bing, Naver, Seznam.cz, Yandex and Yep as supporters, not Google, so keep a sitemap regardless.
Which ranking systems does Google publicly document?
Google publishes a guide to its ranking systems (updated 10 December 2025). It names the systems that rank indexed pages, grouped here by what they do:
- Understanding language: BERT, MUM (used for specific applications, not general ranking), neural matching and RankBrain help Google match the meaning of a query to the meaning of a page, not just its words.
- Judging the page: the passage ranking system scores individual sections of a page; original content systems favour original reporting over pages that only cite it; the reviews system rewards expert reviews with original analysis.
- Trust and authority: link analysis systems and PageRank use links between pages; reliable information systems promote authoritative pages and demote low-quality ones.
- Keeping results useful: freshness, deduplication and site diversity systems (usually no more than two top results from one site), plus local news and crisis information systems.
- Fighting abuse: spam detection (including SpamBrain), removal-based demotion and the exact match domain system.
Four are listed as retired, folded into core ranking: Hummingbird, Panda, Penguin and the helpful content system (March 2024). Microsoft's page on how Bing delivers search results (March 2025) names six parameters "in general order of importance": relevance, quality and credibility, user engagement, freshness, location and language, and page load time.
How do you check indexing in Google Search Console?
Use two tools: the Page indexing report for patterns across the site, and the URL Inspection tool for one page. This is the routine I would run.
1. Read the Page indexing report
The Page indexing report shows indexed and not-indexed URLs and a "Why pages aren't indexed" table. Google says plainly, "Don't expect every URL on your site to be indexed", but "all of your important pages should be indexed". Learn the common reasons:
- Crawled - currently not indexed: Google fetched the page but has not indexed it.
- Discovered - currently not indexed: Google knows the URL but has not crawled it yet.
- Duplicate without user-selected canonical: Google picked another URL as canonical. Fine, unless it picked the wrong one.
- URL marked 'noindex' and URL blocked by robots.txt: check each one was deliberate.
2. Inspect single URLs
Paste a URL into the URL Inspection tool. It shows whether the URL is on Google, the Google-selected canonical and crawl details, based on the last indexed version, which is "not a live test".
3. Test live, then request indexing
After a fix, run the live test, then request indexing. Google says this usually takes about a day, is not guaranteed and has daily quotas. For many URLs, submit a sitemap instead.
4. Repeat on a schedule
Week one, fix anything important blocked or noindexed by mistake. Then check weekly, and after every launch or redesign. Do the same in Bing Webmaster Tools.
Where does this go wrong for Indian businesses?
It goes wrong most often at launch, when a staging site's robots.txt block or noindex goes live with the new site. Other repeat problems: city or language URL variants with no canonical, and key information published only as images or PDFs. Brands selling beyond the metros should make sure their Hindi or regional-language pages are separate, crawlable HTML pages, not translations hidden behind a script.
Who should own crawling and indexing?
The developer or agency that controls the site build should own robots.txt, sitemaps, redirects and canonicals. The marketer responsible for organic search should own the weekly Search Console check and raise problems. Writers and subject experts own the harder question: whether each page is worth indexing at all.
My view
My view is that SEO basics are boring, which is why they get skipped. I spent years managing performance budgets across Google, Meta and affiliate networks, where you can pay for a place on the page. Organic search has no such shortcut, so the only route is a site Google can read.
The trade-off is that this work produces no visible campaign, and the gain arrives weeks later. I would still do it first, because content written for a page Google cannot index is money spent on nothing.
Open Search Console today
List your ten most important pages and inspect each in the URL Inspection tool. If any is not on Google, find the reason in the Page indexing report before you write any new content.
Questions people ask
What is the difference between crawling and indexing?
Crawling is when a search engine's bot, such as Googlebot, finds and downloads a page. Indexing is when the engine analyses that page and stores it in its database. A page can be crawled and still not indexed; Google's Page indexing report calls this "Crawled - currently not indexed".
Should I use robots.txt or noindex to keep a page out of Google?
Use noindex. Google says robots.txt "is not a mechanism for keeping a web page out of Google", and a blocked URL can still be indexed if other pages link to it. A noindex tag only works if the page is crawlable, so do not block it in robots.txt at the same time.
Do small websites need to worry about crawl budget?
Usually not. Google's crawl budget guide targets sites with 1 million or more unique pages, or 10,000 or more that change daily. Smaller sites are, most of the time, crawled efficiently.
How do I check if a page is indexed by Google?
Paste the URL into the URL Inspection tool in Google Search Console. It shows whether the URL is on Google, the canonical Google chose and the last crawl. For site-wide patterns, use the Page indexing report.