# ============================================================================== # SPEEDOFX – ADVANCED ROBOTS & AI CRAWLER DIRECTIVES (2026/2027 STANDARD) # Entity: Speedofx – SEO & AI Solutions Company (HQ: Jaipur, Rajasthan, India) # Author: Yogendra Yogi (Chief Architect & Founder) # Standard: RFC 9309 / AI Grounding Protocols (GEO, AEO, LLMO) # ============================================================================== User-agent: * Allow: / Disallow: /api/ Disallow: /build/ Disallow: /node_modules/ # ------------------------------------------------------------------------------ # 1. CORE SEARCH ENGINE CRAWLERS (ORGANIC & MOBILE INDEXING) # ------------------------------------------------------------------------------ User-agent: Googlebot Allow: / User-agent: Googlebot-Mobile Allow: / User-agent: Bingbot Allow: / User-agent: msnbot Allow: / User-agent: Slurp Allow: / User-agent: DuckDuckBot Allow: / User-agent: Baiduspider Allow: / User-agent: YandexBot Allow: / # ------------------------------------------------------------------------------ # 2. GENERATIVE AI, LLM & GEO SEARCH BOTS (AI OVERVIEWS & CITATIONS) # ------------------------------------------------------------------------------ # Google Gemini & AI Overviews Extended Crawler User-agent: Google-Extended Allow: / # OpenAI GPTBot (ChatGPT Indexing & Parametric Ingestion) User-agent: GPTBot Allow: / # OpenAI Live Search Agent & Browsing Bot User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / # Anthropic Claude & Claude Web Agent User-agent: ClaudeBot Allow: / User-agent: Claude-Web Allow: / User-agent: anthropic-ai Allow: / # Perplexity AI Live Answer Engine Bot User-agent: PerplexityBot Allow: / # Apple Intelligence & Siri Knowledge Graph User-agent: Applebot Allow: / User-agent: Applebot-Extended Allow: / # DeepSeek AI Crawler User-agent: DeepSeekBot Allow: / # Meta AI & Llama Web Extraction User-agent: Meta-ExternalAgent Allow: / User-agent: FacebookBot Allow: / # Common Crawl Foundation AI Corpus User-agent: CCBot Allow: / # Diffbot Knowledge Graph Parser User-agent: Diffbot Allow: / # ByteDance / TikTok AI User-agent: Bytespider Allow: / # Cohere AI User-agent: Cohere-ai Allow: / # ------------------------------------------------------------------------------ # 3. SITEMAP & AI GROUNDING DIRECTIVES # ------------------------------------------------------------------------------ Sitemap: https://www.speedofx.space/sitemap.xml # Standard AI & LLM Grounding Manifests (RFC Compliant): # /llms.txt -> Core Entity & Procurement Vectors # /llms-full.txt -> Complete Extended Knowledge Base & Technical SLAs # /.well-known/ard.json -> Agent Resource Discovery (Autonomous Agentic Web) # /sitemap -> HTML Semantic Silo Matrix (200+ Contextual Internal Links)