Knowledge · Guidelines · architecture

Web Crawlability

Enforceable rules for robots.txt validity, sitemap.xml health, llms.txt format, and canonical URLs. Operationalises the seo-specialist role's crawlability mandate into pass/fail checks for every web product.

Andrei Solovev

Metadata

architecture

Description

Operational crawlability checklist. The seo-specialist role describes WHAT good crawlability is; this guideline makes it CHECKABLE so shredbx, BR, and every future product pass Google Search Console + Lighthouse SEO without rediscovery. Derived from real Search Console / Lighthouse findings (shredbx.com, 2026-07-01). Complements: seo-specialist.yml (role), web-assets.yml (favicon/og), web-performance.yml.