Technical SEO: crawl, index, render

Technical SEO decides whether content can be fetched, parsed, indexed, and trusted by Google, Bing, and AI answer engines (ChatGPT, Claude, Perplexity, Gemini). In 2026 the discipline split in two: Googlebot renders JavaScript and tolerates slow pages, AI crawlers do neither. Treat the raw HTML response as the product.

Work in this order: indexation, rendering, speed. A fast page that is not indexed earns nothing. An indexed page whose content only appears after client-side JavaScript runs is invisible to every AI engine except Google's. Every threshold below is labeled either official or measured (with a source URL) or field heuristic from 115+ agency audits.

Company knowledge first (Obsidian)

If the working environment contains an Obsidian vault or any local knowledge base (a folder of .md notes, often with a .obsidian directory), read the relevant notes before acting: brand and product facts, target keywords, competitors, and the SEO action log of what was already tried. Ground every recommendation in that context instead of asking the user for facts the vault already holds. At the end of the session, append the actions taken to the vault's SEO action log so the next session starts informed. Vault structure, read-first and write-back protocols: the obsidian-brain skill.

When to use this skill

Use it for:

• Full technical audits and pre-sales triage of any site
• "Why is my site (or page) not indexed" investigations
• robots.txt, meta robots, sitemap, canonical, and hreflang work
• Core Web Vitals (LCP, INP, CLS), PageSpeed, image weight
• JavaScript rendering decisions: SPA versus SSR versus SSG
• AI crawler access: this skill owns references/ai-crawlers.md, the canonical crawler reference for the whole skill set
• llms.txt questions (see the honest verdict section)
• Site migrations, redesigns, domain changes
• Hacked-site cleanup (injected spam links)

Hand off neighboring problems:

• Orphan pages and link architecture: use the seo-internal-linking skill
• JSON-LD and structured data: use the seo-schema-markup skill
• Writing content so AI engines quote it: use the geo-visibility skill
• Measuring AI citations and AI referral traffic: use the geo-tracking skill
• One-pass scored audit of a whole site: use the seo-geo-audit skill

Workflow

Step 0: Secure measurement access

Before any recommendation, confirm both:

1. Google Search Console is verified (domain property preferred) and you can actually open it.
2. GA4, or an equivalent analytics tool, is installed and collecting.

Field finding from 115+ agency audits: a large share of audited sites had neither installed, or had them installed with nobody able to access the accounts. Without GSC you cannot see coverage, queries, or manual actions, and every later step degrades into guessing. Set both up first, then add Bing Webmaster Tools: it imports GSC properties in a few clicks and matters in 2026 because Bing's index feeds ChatGPT search (https://yoast.com/chatgpt-search/).

For a fast automated first pass, run the bundled audit script from the seo-geo-audit skill (scripts/seo_audit.py), which checks robots.txt AI bot rules, sitemap, and on-page basics. Use its output to direct the manual work below.

Step 1: Read the indexation state

In GSC, open Indexing > Pages and compute the ratio of indexed to submitted pages.

• 75 percent or more indexed (for example 3,000 indexed of 4,000 submitted) is healthy on a mature site (field heuristic).
• Below 50 percent, stop and investigate before anything else: duplication, thin pages, crawl traps, leftover noindex, or rendering failures.

Then act on the gaps:

• Priority pages not indexed: inspect each with the URL Inspection tool, fix the reported cause, then click Request Indexing. The quota is roughly 10 to 12 requests per day per property, so spend it on money pages, not the archive.
• Bulk indexation, not page by page: ignore the "index 10 URLs per day" ritual some platforms (Wix and others) push. Manual one-by-one submission scales to nothing. Submit the full sitemap in GSC once and let Google discover every URL from it; reserve manual Request Indexing for a handful of priority pages (field heuristic from 115+ agency audits).
• Scaled and programmatic pages, the indexation condition: thousands of generated pages (one per city, per combination) only get indexed when each page carries genuinely unique content. The same block copied across all of them is the classic non-indexation cause: Google treats near-duplicate templates as one page and drops the rest. Uniqueness per page is the price of admission for scaled content, not an optimization (field heuristic from 115+ agency audits).
• 404s with history: any URL that previously earned traffic or backlinks and now returns 404 gets a 301 to the closest equivalent page. Never blanket-redirect everything to the homepage: Google treats irrelevant mass redirects as soft 404s.
• Orphan pages: target zero. Detection and fixes belong to the seo-internal-linking skill.

Measurement artifact (September 2025): Google removed the num=100 results parameter. Many properties saw desktop impressions collapse and average position improve overnight because rank-tracking bots stopped loading 100-result pages. Check clicks before declaring a loss: clicks typically stayed flat. Never present this artifact to a client as a traffic drop (https://searchengineland.com/google-num100-impact-data-462231).

Step 2: Crawl controls: robots.txt, sitemaps, meta robots

robots.txt:

• Must contain a Sitemap: line pointing at the sitemap index.
• Must not contain Disallow: / (shipped staging configs cause this more often than expected; check it first on any sudden deindexation).
• AI bot rules: decide with the GEO layer below and references/ai-crawlers.md.

Sitemap.xml:

• Segment by content type (sitemap-pages.xml, sitemap-posts.xml, sitemap-products.xml) under one index. Why: GSC reports coverage per sitemap, so segmentation reveals which template fails to get indexed.
• Keep lastmod honest: update it only when content meaningfully changes. Google states it ignores lastmod when it is uniformly fresh or inaccurate.
• Do not ping Google's sitemap endpoint: deprecated in June 2023, it returns 404. Submit through GSC and the robots.txt Sitemap: line (https://developers.google.com/search/blog/2023/06/sitemaps-lastmod-ping).
• Set up IndexNow for Bing: instant push of new and updated URLs, built into many CMS plugins and Cloudflare. Bing is a primary gateway into ChatGPT search, which makes Bing Webmaster Tools standard equipment in 2026 (https://yoast.com/chatgpt-search/).

Discovery for ultra-fresh topics: when the site publishes time-sensitive news, add a dedicated News sitemap plus an RSS or Atom feed so Google can pick up items within minutes; an RSS feed is also the fastest way to source breaking topics worth covering. This channel only fits genuinely fresh, newsworthy content, not the evergreen archive.

Meta robots versus robots.txt, the distinction behind most "why is this still indexed" tickets:

A disallowed page can appear in results as a URL-only listing. If something must disappear, allow the crawl, serve noindex (or 404/410), and use the Removals tool for urgent cases.

Crawl budget (sites above roughly 10,000 URLs):

• Faceted navigation, calendar archives, and internal search results generate near-infinite URL spaces. Disallow them in robots.txt and keep them out of sitemaps.
• Read GSC Settings > Crawl stats: spikes on parameter URLs mean budget burned on noise instead of money pages.
• Below a few thousand URLs, crawl budget is never the bottleneck; do not invoice work on it (field heuristic).

Step 3: JavaScript rendering, the number one technical check of 2026

No AI crawler executes JavaScript. The Vercel and MERJ study of 500M+ crawler fetches found that GPTBot, ClaudeBot, PerplexityBot, and Meta-ExternalAgent sometimes download JavaScript files (11.5 percent of ChatGPT fetches, 23.8 percent of Claude fetches) but execute none of it. Googlebot is the only major crawler that renders (https://vercel.com/blog/the-rise-of-the-ai-crawler and https://www.gsqi.com/marketing-blog/ai-search-javascript-rendering/).

Consequence map:

The test costs one minute. Fetch the raw HTML and search for a sentence that should be on the page:

curl -sL https://example.com/page/ | grep -c "exact sentence from the page"

Zero matches means AI engines see an empty shell. Cross-check with view-source in a browser (not DevTools Elements, which shows the rendered DOM) and with GSC URL Inspection for what Googlebot renders.

Decision rule: any page that should rank or be cited must serve its content in the initial HTML response. SSR or SSG is mandatory; client-side rendering is acceptable only for logged-in app surfaces.

Stack verdicts:

Inside a stable CMS, the theme still decides the technical floor. Pick a lean, SEO-oriented e-commerce theme (clean markup, no JavaScript bloat, content in raw HTML, fast Core Web Vitals out of the box) over a heavy multi-purpose one: on Shopify, for example, the gap between a performance-focused theme and a bloated one shows up directly in LCP. Choose the domain on the same logic: for a French local business, a .fr signals geography to Google and tends to earn a better local click-through than a .com (field heuristic from 115+ agency audits). Settle the domain before launch, since changing it later means a full migration (Step 7).

Step 4: Speed and Core Web Vitals

Judge mobile first: mobile carries 80 to 90 percent of traffic on typical lead-gen and e-commerce sites (field range). Measure with the free PageSpeed Insights web interface at https://pagespeed.web.dev: no API key, no account, and it returns both field data (CrUX, what real users experienced over 28 days) and lab data. Field data wins arguments; lab data locates causes.

Keep perspective: Core Web Vitals act as a tiebreaker, not a dominant ranking factor. Do not spend weeks chasing a score of 100 on a site with an indexation or rendering problem; sequence speed after Steps 1 to 3.

High-yield fixes, ordered by how often audits surface them:

1. Images: 200 KB or less each (field heuristic), WebP or AVIF, explicit width and height attributes to reserve layout space (prevents CLS), responsive srcset.
2. Never lazy-load above-the-fold images, least of all the LCP image; mark the LCP image fetchpriority="high".
3. Defer render-blocking third-party scripts; load chat widgets and trackers after interaction.
4. Text-to-HTML ratio under 10 percent signals a thin or bloated page (field heuristic): too little content or too much markup. Related: keep the DOM under roughly 1,500 nodes; Lighthouse flags excessive DOM size starting around 800 (https://developer.chrome.com/docs/lighthouse/performance/dom-size).

Step 5: Duplicates and international: canonicals and hreflang

Canonicals (the e-commerce deduplication tool):

• Faceted navigation, sort parameters, and session IDs multiply URLs. Point every variant at the clean URL with rel=canonical.
• A canonical is a hint, not a directive: Google ignores canonicals contradicted by stronger signals (internal links and sitemaps pointing at variants). Align all signals on the clean URL.
• Every page self-canonicalizes. It costs nothing and absorbs parameter pollution.
• Quick win, the homepage canonical: many sites set canonicals on every inner page but leave the homepage without one, even though it is the most linked URL and the one that collects the most parameter and tracking variants. Add a self-referencing canonical on the home page; it is a two-minute fix with outsized cleanup value (field heuristic from 115+ agency audits).

hreflang (international versions):

• Every language or region page lists all alternates including itself, plus one x-default pointing at the language selector or default market page.
• Reciprocity is mandatory: if the FR page references the EN page, the EN page must reference the FR page back, or Google discards the pair.
• Prefer hreflang in sitemaps on large sites: one file to audit instead of tags scattered across templates.
• hreflang routes users; it does not consolidate authority. Near-duplicate language variants (en-US versus en-GB) still need distinct value or consolidation.

Step 6: Security and domain integrity

• HTTPS everywhere: every HTTP URL 301s to HTTPS in one hop, no mixed content, valid certificate.
• Hack detection: search the raw HTML (hacks often cloak, so not just the rendered page) for injected casino, pharma, and replica anchors. Check GSC Security Issues and run a site: search for pages you do not recognize. On WordPress: clean the injection, update core and plugins, install Wordfence, rotate all credentials.
• Unintentional geo-blocking: a firewall rule or a country block (often shipped by a WAF, a CDN security preset, or a hosting default) can make the site unreachable from entire regions, so a slice of human visitors, search bots, and AI crawlers, and your own audit, see nothing. Symptoms: timeouts or 403s from one country but not another, missing impressions from a market that should convert. Test by fetching the site through a different-country exit (a VPN, or curl from a server in the target region) and compare the status code against your local result. Whitelist Googlebot, Bingbot, and the AI crawler ranges (references/ai-crawlers.md), and lift any country block that overlaps a real audience.
• Domain age is an asset: an old domain carries link history and trust. Never let a legacy domain expire; renew it and 301 it. Field rule: domains are cheap, history is not replaceable.

Step 7: Migrations (when applicable)

Migrations destroy more rankings than algorithm updates do (field observation across 115+ audits). Operating rule: it is easy to break rankings and very hard to build them.

1. Keep the same domain if humanly possible, and change one thing at a time: domain, URL structure, design, or content. Stacking all four into one launch makes any later diagnosis impossible.
2. Freeze a full URL inventory before launch: GSC performance export (16 months), GSC coverage, the sitemap, and a full crawl. The menu is not the site; only the export catches the long tail.
3. Map a 301 for 100 percent of known URLs to the closest equivalent. No chains, no mass redirect to the homepage.
4. Pages that rank but embarrass the new design: hide them from the menu instead of deleting them. They keep earning; deletion burns their history.
5. Keep titles, H1s, and content on top pages as close to identical as the redesign allows (light modifications only).
6. Deindex duplicate mirrors at launch: a Substack copy of the blog, a forgotten staging subdomain, a www and non-www split. One canonical home per piece of content.
7. After launch: recrawl, verify redirects, resubmit sitemaps, use GSC Change of Address for domain moves, then watch coverage daily for two weeks.

Spot-check commands

Fast verifications during any conversation, before reaching for tools. Each answers one question from the steps above:

Status code and redirect chain (flag chains longer than one hop)

curl -sIL https://example.com/page/ | grep -iE "^HTTP|^location"

Indexability: X-Robots-Tag header, then meta robots in raw HTML

curl -sIL https://example.com/page/ | grep -i "x-robots-tag" curl -sL https://example.com/page/ | grep -io '<meta name="robots"[^>]*>'

Canonical as served (must match the clean URL)

curl -sL https://example.com/page/ | grep -io '<link rel="canonical"[^>]*>'

robots.txt: sitemap declared, no accidental Disallow: /

curl -s https://example.com/robots.txt

Content present without JavaScript (the AI visibility test from Step 3)

curl -sL https://example.com/page/ | grep -c "a sentence from the page"

For anything beyond spot checks, run the bundled audit script from the seo-geo-audit skill (scripts/seo_audit.py).

Rules and thresholds

llms.txt: the honest verdict

What the data says:

• Adoption sits around 10 percent of the 300,000 domains SE Ranking studied in 2025.
• Server-log analyses find AI bots requesting the file in a tiny fraction of hits, commonly cited near 0.1 percent.
• No engine confirms reading it: not OpenAI, not Anthropic, not Perplexity, not Google. John Mueller publicly compared it to the meta keywords tag (https://www.seroundtable.com/google-does-not-endorse-llms-txt-40789.html).
• Google states sites do not need special AI text files for its AI features (https://developers.google.com/search/docs/appearance/ai-features).

Recommendation: generating one takes 10 minutes and does not hurt, so create it when asked or when it costs nothing. Never present it, prioritize it, or bill it as a visibility lever, and never let it displace work on rendering, indexation, or content. A vendor selling llms.txt as a GEO strategy is telling you something about the vendor.

GEO layer: AI crawler access

This skill is the canonical owner of AI crawler knowledge for the whole skill set. The full table (user agents, operators, robots.txt compliance, IP verification URLs) and paste-ready robots.txt blocks live in references/ai-crawlers.md. Read that file whenever a task touches robots.txt or bot access. Summary:

Three bot roles, three different blocking costs:

Default recommendation for brands, SaaS, lead-gen, e-commerce: allow everything, including training bots. Presence in training data means the model itself knows the brand and can recommend it even when the answer runs without web search. Defensive configuration for publishers whose content is the product: keep the search surfaces (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot, Googlebot), block training (GPTBot, ClaudeBot, CCBot, Google-Extended). Both configurations are ready to paste in references/ai-crawlers.md.

The Google-Extended trap: Google-Extended is a control token, not a crawler. Blocking it only opts the site out of Gemini training and grounding. It does NOT remove the site from AI Overviews or AI Mode, which are fed by regular Googlebot crawling. The only opt-outs from AI Overviews (nosnippet, max-snippet, noindex) also reduce classic search snippets. Decide with eyes open (https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers).

Two facts from earlier sections weigh double for AI bots: they read raw HTML only (Step 3), and Bing's index reaches ChatGPT (Step 2, IndexNow). Anti-spoofing: user-agent strings are trivially forged, so verify IPs against the published ranges listed in references/ai-crawlers.md before whitelisting anything or trusting log analyses.

For what to write so engines quote it, use the geo-visibility skill. For measuring citations and AI referral traffic, use the geo-tracking skill.

Output format

Deliver a prioritized action plan, never a data dump. Default structure:

Technical SEO audit: {domain} ({date})

Critical (blocks indexing or AI visibility)

Per finding: evidence (URL plus observed value), why it matters, exact fix, effort (S/M/L)

Important (costs rankings or citations)

Same fields

Maintenance (hygiene, batch later)

Same fields

Verified healthy

One line per check that passed, so the absence of findings is itself evidence

Severity rubric:

Every claim carries its evidence (the URL tested and the raw value observed), and every number is labeled official, measured, or field heuristic.

Common mistakes

Sources

• Sitemap ping deprecation: https://developers.google.com/search/blog/2023/06/sitemaps-lastmod-ping
• Bing index as ChatGPT gateway: https://yoast.com/chatgpt-search/
• num=100 removal impact: https://searchengineland.com/google-num100-impact-data-462231
• INP thresholds and rollout: https://web.dev/articles/inp
• AI crawlers and JavaScript, 500M+ fetches: https://vercel.com/blog/the-rise-of-the-ai-crawler
• AI search and JS rendering: https://www.gsqi.com/marketing-blog/ai-search-javascript-rendering/
• Google on AI features and text files: https://developers.google.com/search/docs/appearance/ai-features
• Mueller on llms.txt: https://www.seroundtable.com/google-does-not-endorse-llms-txt-40789.html
• Googlebot and Google-Extended documentation: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
• DOM size: https://developer.chrome.com/docs/lighthouse/performance/dom-size
• PageSpeed Insights, free web interface, no API key: https://pagespeed.web.dev
• Field heuristics: 115+ agency audits, labeled as such throughout