Why a website is not indexed: a step-by-step check

Define what is actually missing

The question “why a website is not indexed” often describes three states. A search engine may not know the URL, may know it but exclude it, or may index it without showing it for the chosen phrase. A domain-wide site: check cannot distinguish them: the home page may appear while every product page carries a template-level noindex.

Choose one canonical URL from the affected template. Inspect it in Google Search Console and Yandex Webmaster: is it known, can it be crawled, which canonical did the system select, and what exclusion reason is reported? Open the same page without authentication on mobile and desktop. This turns an abstract complaint into a repeatable diagnosis.

Indexing does not equal ranking. If the tools confirm inclusion, ask whether the page satisfies the query, matches the target region, and provides standalone value. Repeating a keyword cannot repair a technical block and cannot make a weak page useful.

Crawling

A search crawler discovers a URL and downloads the server response.

Indexing

The engine processes a page and decides whether to store it as a possible result.

Ranking

Indexed documents are selected for a specific query and context.

Check the technical route in a fixed order

Begin with the server. A target page should consistently return 200, a moved page should use one permanent redirect, and a removed page should return 404 or 410. Timeouts, 5xx, certificate errors, and CDN protection can block crawlers even when an owner opens a cached session. Watch for soft 404s: a 200 response containing an empty page or unavailable-product message.

Then inspect access and consolidation. Robots.txt controls crawling, while meta robots and X-Robots-Tag control indexing of a response that can be loaded. If robots.txt blocks the URL, the crawler may never see its noindex. Canonicals, sitemaps, redirects, and internal links should identify the same preferred version.

For JavaScript websites, compare source and rendered HTML. The heading, body copy, links, and product state must be available after rendering without a click or scroll. Check blocked scripts, browser errors, and empty API containers. If this route is clean, investigate duplicates, page value, and search intent.

• HTTP status and redirects
• robots.txt and noindex
• canonical, sitemap, and links
• rendered primary content
• duplicates and query fit

Move from one URL to its template

Once the cause is confirmed, inspect neighbouring pages of the same type. A repeated noindex, canonical, or empty block is a template problem, not a list of CMS edits. If symptoms differ, separate them: a site-wide filter fault should not be bundled with one intentionally removed product.

KILENI uses follow-up crawls as implementation evidence. On eco-santeh.ru, 509/509 final URLs responded successfully, the checked export no longer showed missing primary tags, and Lighthouse moved from 36 → 57. On засорсервис.рф, 575/575 sitemap URLs opened and CLS moved from 0.519 to 0.0001 in two tests. These numbers do not forecast indexing.

Record the method and date. Keep the original response, corrected response, URL sample, and test limitation. For indexing, retain the status reported by webmaster tools because a successful private crawl does not prove inclusion by a search engine.

Fix the cause before requesting another crawl

Do not change robots.txt, canonicals, and the sitemap together as a guess. Correct the confirmed root cause, release it to a control group, and repeat the same test. If stable, apply the template change broadly. This release is easier to accept and reverse than unrelated SEO settings.

After deployment, request recrawling through official tools and monitor their reports. Submission does not guarantee immediate indexing. Engines must fetch and process the document again, and timing varies by website and change type. Technical readiness can be verified immediately; search results only after processing.

If the URL is indexed, return to intent. Compare the page with what the searcher expects: a service, guide, category, or product. Clarify region, availability, price, evidence, and next action. Do not add paragraphs to meet a length target; add facts that help someone choose or complete a task.