File size: 363 Bytes
fcbb9d8 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
User-agent: * Allow: / # Sitemap Sitemap: https://reubenfernandes.dev/sitemap.xml # Specific crawl instructions for major search engines User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / # Block unnecessary crawling of build files Disallow: /_next/ Disallow: /api/ # Crawl delay (optional - can be adjusted based on server capacity) Crawl-delay: 1 |