# SEO & AI Visibility Checklist > A comprehensive implementation contract for maximum visibility to AI agents, search engines, social media crawlers, and automated verification bots. --- ## Architecture - Build landing pages with a static site generator (e.g. Astro, Hugo, Next.js static export) - Pre-render all public pages to pure HTML at build time - No content should require JavaScript to be visible - JavaScript may be used for progressive enhancement only - Progressive enhancement scripts must live inside the document (no scripts after ``) --- ## 1. robots.txt Create a `robots.txt` at your site root with explicit allow blocks for all major crawlers: ``` User-agent: * Allow: / User-agent: Googlebot Allow: / # Controls whether content is used for Gemini AI training — not a traditional crawler User-agent: Google-Extended Allow: / User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / # Meta's AI training crawler — accounts for ~50% of all AI crawling traffic User-agent: Meta-ExternalAgent Allow: / # Controls whether content is used for Apple Intelligence AI training User-agent: Applebot-Extended Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Bytespider Allow: / User-agent: CCBot Allow: / User-agent: cohere-ai Allow: / Sitemap: https:///sitemap.xml ``` --- ## 2. llms.txt Serve an AI content discovery file at `/llms.txt` following the spec at [llmstxt.org](https://llmstxt.org/). Required: - H1 with your project or company name (this is the **only required element**) Optional (recommended): - Blockquote with a one-line summary - General content paragraphs or lists with background information - H2-delimited sections containing link lists in `[Name](URL): Description` format - A `## Optional` section for lower-priority resources (this is the only H2 name the spec assigns special meaning to) The spec does not prescribe other H2 section names — choose names that fit your site (e.g. `## Products`, `## Documentation`, `## API Reference`). Generate this from your product data source so it stays in sync automatically. --- ## 3. JSON-LD Structured Data Embed three schemas in ``: ### Organization ```json { "@context": "https://schema.org", "@type": "Organization", "@id": "https:///#organization", "name": "", "url": "https:///", "logo": { "@type": "ImageObject", "url": "https:///logo.png" }, "description": "", "sameAs": ["", ""], "contactPoint": { "@type": "ContactPoint", "email": "", "contactType": "Customer Support" }, "knowsAbout": ["", ""] } ``` ### WebSite ```json { "@context": "https://schema.org", "@type": "WebSite", "name": "", "url": "https:///", "publisher": { "@id": "https:///#organization" } } ``` ### ItemList (homepage only) ```json { "@context": "https://schema.org", "@type": "ItemList", "numberOfItems": , "itemListElement": [ { "@type": "ListItem", "position": 1, "item": { "@type": "SoftwareApplication", "name": "", "url": "", "applicationCategory": "", "description": "", "creator": { "@id": "https:///#organization" } } } ] } ``` Requirements: - Organization and WebSite on every public page - ItemList on homepage only - Cross-reference with `@id` - Keep `numberOfItems` data-driven - Add `alternateName` to Organization for name disambiguation (e.g. variations of your company name) - Add `legalName` to Organization - Validate at [validator.schema.org](https://validator.schema.org/) and [Google Rich Results Test](https://search.google.com/test/rich-results) ### BreadcrumbList Add to every page except the homepage: ```json { "@context": "https://schema.org", "@type": "BreadcrumbList", "itemListElement": [ { "@type": "ListItem", "position": 1, "name": "Home", "item": "https:///" }, { "@type": "ListItem", "position": 2, "name": "", "item": "https:///" } ] } ``` BreadcrumbList is one of Google's actively supported rich result types and generates visible breadcrumb trails in desktop search results. ### SiteNavigationElement ```json { "@context": "https://schema.org", "@type": "SiteNavigationElement", "name": ["Products", "About", "Contact"], "url": [ "https:///products", "https:///about", "https:///contact" ] } ``` This provides machine-readable navigation structure. Note: Google generates sitelinks algorithmically based on site structure, internal linking, and user behavior — this schema may provide a supplementary signal but is not guaranteed to influence sitelinks. ### FAQPage (homepage) ```json { "@context": "https://schema.org", "@type": "FAQPage", "mainEntity": [ { "@type": "Question", "name": "What is ?", "acceptedAnswer": { "@type": "Answer", "text": "" } } ] } ``` Requirements: - FAQ content must be **visible on the page** (not just in schema) - Use `
` / `` elements for expandable FAQ sections - Include questions about your company, products, and common queries - Add a question for each product with a link to try it > **Note:** Since August 2023, Google restricts FAQ rich results to well-known government and health authority websites. For other sites, FAQPage schema will not generate visible rich results in Google, but may still help AI systems and other search engines understand your content. ### SoftwareApplication (per-product pages) ```json { "@context": "https://schema.org", "@type": "SoftwareApplication", "name": "", "description": "", "applicationCategory": "Application", "url": "", "creator": { "@id": "https:///#organization" } } ``` Create individual product pages at `/products/` with per-product schema. --- ## 4. Product Pages Create a `/products` index page listing all products, and individual pages for each product: - `/products` — overview of all products grouped by category - `/products/` — dedicated page per product with: - Product name, description, status (live/in development) - Category and type - Link to the product's website - SoftwareApplication JSON-LD schema - Back links to `/products` and homepage Requirements: - Generate pages from your product data source (single source of truth) - Link to `/products` from your homepage footer - Include all product pages in `sitemap.xml` - Each product page must have unique ``, `<meta description>`, and canonical URL --- ## 5. Sitemap Create a `sitemap.xml` at your site root listing all indexable pages: ```xml <?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url> <loc>https://<your-domain>/</loc> <lastmod>YYYY-MM-DD</lastmod> </url> </urlset> ``` Requirements: - Maximum 50,000 URLs and 50MB per sitemap file - Only include canonical, indexable URLs (200 status, no `noindex`) - Use accurate `<lastmod>` dates (only update when content actually changes) - Google ignores `<priority>` and `<changefreq>` — do not include them - For large sites (50,000+ URLs), use a sitemap index file - Generate dynamically from your page/data source when possible - Reference your sitemap in `robots.txt` --- ## 6. Open Graph Meta Tags Required on every public page: ```html <meta property="og:type" content="website" /> <meta property="og:url" content="https://<your-domain>/<page>" /> <meta property="og:site_name" content="<Your Company>" /> <meta property="og:locale" content="en_US" /> <meta property="og:title" content="<Page Title>" /> <meta property="og:description" content="<Page Description>" /> <meta property="og:image" content="https://<your-domain>/og-image.png" /> <meta property="og:image:width" content="1200" /> <meta property="og:image:height" content="630" /> <meta property="og:image:type" content="image/png" /> <meta property="og:image:alt" content="<Descriptive alt text>" /> ``` Requirements: - `og:url`, `og:title`, `og:description` must be page-specific - `og:image` must be exactly 1200x630px - Use absolute HTTPS URLs for `og:image` - Specify explicit dimensions so platforms don't need to probe the image - For blog posts or articles, use `og:type` of `article` instead of `website` --- ## 7. Twitter/X Card Meta Tags Use `name=` attributes, NOT `property=`. Twitter's parser does fall back to Open Graph tags when Twitter-specific tags are missing. ```html <meta name="twitter:card" content="summary_large_image" /> <meta name="twitter:title" content="<Page Title>" /> <meta name="twitter:description" content="<Page Description>" /> <meta name="twitter:image" content="https://<your-domain>/og-image.png" /> <meta name="twitter:image:alt" content="<Descriptive alt text>" /> <meta name="twitter:site" content="@<your-handle>" /> <meta name="twitter:creator" content="@<your-handle>" /> ``` --- ## 8. Essential Head Tags Every public page must include: ```html <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> <link rel="canonical" href="https://<your-domain>/<page>" /> <link rel="icon" type="image/svg+xml" href="/favicon.svg" /> <link rel="apple-touch-icon" href="/apple-touch-icon.png" /> <meta name="description" content="<page description>" /> <meta name="author" content="<Your Company>" /> <meta name="theme-color" content="<#hex-color>" /> <title><Page Title> ``` Notes: - `` must be within the first 1024 bytes of the document - `` must use an absolute HTTPS URL matching the page's own URL - `` is optional — this is the default behavior - Use `` for pages that should not be indexed (utility pages, search results, staging environments) - Do not use `user-scalable=no` or `maximum-scale=1` in the viewport — users must be able to zoom - `` is ignored by Google (since 2009) and not used for ranking by Bing — omit it --- ## 9. Semantic HTML - Single `

` per page - `

` for sections, `

` for items - Proper landmarks: `