XenForo XenForo Robots.txt Guide – How to Use It

[CR LEAKS]

Step 1 :

Having a robots.txt is extremely important for your XenForo forum. It tells search engines which pages to crawl and which pages to skip. This is your chance to build out a proper structure for the robots.txt file so that it will exclude non-important pages

Step 2 :

When you install XenForo onto your hosting server, robots.txt is not be included in the package. However, creating one takes less than 30 seconds

Step 3 :

Here is our recommended robots.txt file for XenForo

User-agent: *
Disallow: /find-new/
Disallow: /account/
Disallow: /login/
Disallow: /admin.php
Disallow: /search/
Disallow: /whats-new/
Allow: /
Sitemap: https://domain.com/sitemap.xml