What is Technical SEO?

Four pillars: crawlability (robots.txt + sitemap + internal linking), renderability (JS rendering + Core Web Vitals + mobile-friendly layout), indexability (canonical tags + noindex directives + structured data), and performance (page speed + Largest Contentful Paint + Cumulative Layout Shift). Modern technical SEO also covers AI bot optimization — llms.txt, AI-friendly schema, server-side rendering for AI crawlers.

Technical SEO is about the infrastructure that lets engines access and understand your site at all — crawlability, rendering, indexing, speed. On-page SEO is about the content and metadata on a specific page — title tags, headings, keyword usage, internal linking within that content. A page can have perfect on-page SEO and still rank poorly if technical SEO issues (blocked by robots.txt, slow LCP, unrendered JS) prevent it from being properly crawled and indexed in the first place.

(1) Misconfigured canonical tags — pointing to the wrong URL or chains. (2) Robots.txt blocking critical paths. (3) JavaScript-rendered content invisible to AI bots. (4) Slow LCP from un-optimized hero images. (5) Missing or duplicate meta descriptions. (6) No structured data on pages that should have it. SearchChamp's Site Auditor catches all six on every audit.

Three measurable performance metrics: LCP (Largest Contentful Paint, target under 2.5s) — when the main content paints. INP (Interaction to Next Paint, target under 200ms) — how responsive interactions feel. CLS (Cumulative Layout Shift, target under 0.1) — how much the layout jumps after load. All three are confirmed Google ranking signals as of 2021.

Not inherently — but client-side-only rendering is risky for AI bots, which don't all execute JavaScript reliably. Server-side rendering (SSR) or static site generation (SSG) is safer. Modern frameworks (Next.js, Astro, SvelteKit) make SSR/SSG easy. SearchChamp uses Next.js with mostly server components for this reason.

Most AI bots (GPTBot, ClaudeBot, PerplexityBot) don't render JavaScript, so client-side-only content is invisible to them. They do parse structured data and meta tags reliably. Add llms.txt for explicit site summary, ship server-rendered HTML, use rich schema, and keep your robots.txt allowing AI bots. These four steps are the bulk of AI-bot technical SEO.

Fetch your robots.txt directly (yoursite.com/robots.txt) and look for a User-agent block matching GPTBot, ClaudeBot, PerplexityBot, or Google-Extended with a Disallow rule — or a wildcard User-agent: * block with a broad Disallow that would catch them too, since AI bots fall back to the wildcard rules when no bot-specific block exists. SearchChamp's Site Audit checks this automatically across all known AI bot User-Agents.

They're not interchangeable, and mixing them up is a classic trap. Disallow (in robots.txt) tells crawlers not to fetch a URL at all. noindex (a meta tag or HTTP header on the page itself) tells crawlers not to include an already-fetched page in search results. Disallowing a page also blocks the crawler from ever seeing its noindex tag — so a page you want removed from the index needs to stay crawlable (not Disallowed) with noindex in place until it's actually dropped, then can be Disallowed afterward if you also want to stop future crawl attempts.

Both work — the key requirement is that content is present in the initial HTML response rather than requiring JavaScript execution, since that's what most AI bots can't reliably do. SSG (static site generation) pre-builds pages at deploy time and is simplest for content that doesn't change per-request, like glossary or marketing pages. SSR (server-side rendering) renders on each request and suits pages with dynamic, per-user, or frequently-changing content. Client-side-only rendering is the one option to avoid for anything you want AI bots to read.

Start with crawl + index basics: robots.txt + sitemap + canonical tags. Then performance: LCP/INP/CLS via PageSpeed Insights. Then structured data: required schemas for your page types. Then AI-readiness: llms.txt, server-rendered content, AI-bot allow rules. SearchChamp's Site Auditor agent runs all four phases automatically and ranks fixes by impact.

Fetch the page with JavaScript disabled (curl, or your browser's dev tools with JS off) and check whether the content you care about is present in the raw HTML — if it only appears after JS executes, most AI bots won't see it. SearchChamp's Site Audit runs this class of check automatically as part of its AI-readiness scan, rather than needing a manual per-page fetch.

Continuously is ideal for a site that ships changes regularly, since a single deploy can silently introduce a blocking robots.txt rule or break server rendering. At minimum, audit after any significant site or CMS change, and on a monthly cadence otherwise. SearchChamp's Site Auditor is built to run repeatedly rather than as a one-time check, specifically because technical SEO regressions tend to be deploy-triggered, not gradual.

Both, in the same crawl — the 80+ checks span crawlability, render-readiness (detecting pages whose content only appears after JavaScript runs), indexing, performance, schema, and mobile-friendliness alongside AI-bot readiness (robots.txt AI User-Agent rules, server-rendered content, llms.txt presence), so you get one ranked list of fixes rather than needing a separate AI-specific audit.