Noindex thin/duplicate pages, block faceted navigation, prune zero-traffic pages. Maximize quality pages vs total indexed pages.
Compare your indexed page count (site:yourdomain.com) against pages in your sitemap. If indexed pages are 2-3x your sitemap count, you have a crawl budget problem — Google is wasting time on junk pages.
Crawl budget only matters for large sites (10,000+ pages). For smaller sites, Google will crawl everything regardless. Don't over-optimize this if you have a few hundred pages.
Search "site:yourdomain.com" in Google and note the count. Compare to your sitemap page count. Go to Search Console > Pages to see indexed vs. excluded pages. Large gaps indicate crawl budget waste.
Add <meta name="robots" content="noindex, follow"> to pages that shouldn't appear in search. In Rank Math: edit the page > Advanced > Robots Meta > select "No Index". Common candidates: tag archives, author archives, internal search results, thin category pages.
E-commerce faceted navigation (filters for color, size, price) creates thousands of URL variations. Block these with robots.txt: "Disallow: /*?color=" or use canonical tags pointing filtered pages to the main category.
In GA4, go to Reports > Engagement > Pages. Sort by organic sessions. Pages with zero traffic over 6+ months are crawl budget waste. Either improve, consolidate into stronger pages, or noindex them.
Go to Settings > Crawl stats. Review total crawl requests, download size, and average response time. Look for crawl spikes on low-value pages. Check "Crawl requests breakdown" by response code and file type.
Help me optimize crawl budget for my website with [NUMBER] pages. My site has: - Product/service pages: [NUMBER] - Blog posts: [NUMBER] - Tag/category archives: [NUMBER] - Search result pages: [YES/NO they're indexable] - URL parameters: [LIST any filter/sort parameters] Provide: 1. Which page types should be noindexed 2. Robots.txt rules to block low-value crawl paths 3. Meta robots tag strategy for each page type 4. A faceted navigation handling approach 5. Criteria for identifying pages to prune or consolidate
Track your progress and get guided through every step.
Open Interactive Tool