Crawlability and Indexability: What They Are and How to Fix Common Problems
What crawlability and indexability mean, how they differ, the most common crawlability problems, when crawl budget matters and how to fix indexing issues.
Crawlability is whether search engines can find and reach a page; indexability is whether they can and will add it to their index. A page needs both before it can rank, which is why crawling and indexing problems are the first thing any technical audit checks.
This guide explains the difference, the most common crawlability problems, when crawl budget matters and how to fix pages that aren’t indexed. The fastest way to see the problems on a real site is Google Search Console’s Page indexing report, which a Search Console audit turns into a fix list, grouped by Google’s reason.
Crawlability vs indexability
| Crawlability | Indexability | |
|---|---|---|
| The question | Can Google reach the page? | Can and will Google store it? |
| Common blockers | robots.txt, broken links, orphan pages, server errors | noindex tags, canonicals to other pages, duplicates, thin content |
| Where to check | Crawl reports, server logs, Search Console crawl stats | Page indexing report, URL Inspection |
Common crawlability problems
Pages blocked by robots.txt
A disallow rule meant for one folder can block important sections. Check robots.txt after every site change, and remember that blocking a page in robots.txt stops crawling, not indexing. We explain the difference in noindex vs nofollow vs disallow.
Orphan pages
Pages with no internal links pointing to them are hard for Google to find, even if they’re in the sitemap. Link every important page from a relevant page, ideally within a few clicks of the homepage.
Broken links and redirect chains
Links to 404 pages waste crawling, and chains of redirects slow it down. Fix internal links to point directly to the final URL.
Server errors and slow responses
Frequent 5xx errors or very slow responses can make Google crawl less. Check Search Console’s crawl stats for spikes in errors.
Links Google can’t follow
Navigation built only with JavaScript click events, rather than real <a href> links, can hide pages. Use crawlable links for menus, pagination and filters you want indexed.
Common indexability problems
| Search Console reason | What it usually means | Typical fix |
|---|---|---|
| Excluded by ‘noindex’ tag | A noindex tag on the page | Remove it if the page should rank |
| Alternate page with proper canonical tag | Google is using another URL | Usually fine; check it’s the right one |
| Duplicate without user-selected canonical | Similar pages, no canonical set | Add canonicals or consolidate |
| Crawled – currently not indexed | Google saw it but chose not to index | Improve quality, add internal links |
| Discovered – currently not indexed | Google knows the URL but hasn’t crawled it | Improve internal links; check site health |
| Not found (404) | The page doesn’t exist | Redirect if it had value, or ignore |
“Crawled – currently not indexed” is the one clients ask about most. It usually means Google didn’t find enough value in the page compared with others. Better content and stronger internal links help more than resubmitting.
When crawl budget matters
Crawl budget is how many pages Google can and wants to crawl on a site. Google says it mainly matters for very large sites, such as sites with over a million pages, or tens of thousands of pages that change daily. For most business sites, crawl budget isn’t the problem; crawlability and quality are.
On large ecommerce sites, the usual crawl budget drains are faceted navigation, URL parameters and endless filter combinations. Control them with canonicals, robots.txt rules for useless parameters and internal links that point only to the versions you want indexed.
Crawlability and indexability checklist
| Check | Where |
|---|---|
| robots.txt doesn’t block important sections | robots.txt file, Search Console |
| No accidental noindex on key pages | Crawler, URL Inspection |
| XML sitemap lists only indexable, 200-status URLs | Sitemap, crawler |
| Every important page has internal links | Crawler (orphan pages) |
| No broken internal links or redirect chains | Crawler |
| Server errors are rare and short-lived | Search Console crawl stats |
| Navigation uses real links | Page source, crawler |
| Canonicals point to the right indexable URL | Crawler, URL Inspection |
| Faceted URLs and parameters are controlled | Crawler, Search Console |
How to fix crawling and indexing issues step by step
- Open the Page indexing report in Search Console and list each reason with its count.
- Decide which pages should be indexed. Not every excluded page is a problem.
- Fix blockers first: wrong noindex tags, robots.txt blocks and broken links.
- Fix duplicates with canonicals or by consolidating pages.
- Strengthen important pages with better content and internal links.
- Validate the fix in Search Console and check again in a few weeks.
These checks are the first section of any technical SEO checklist worth using, because nothing else matters until Google can crawl and index the pages that make money. Sites with several languages add their own indexing checks, covered in the international SEO checklist.