Excluding staging sites from search engines

It's important that when a website is crawled by a search engine it doesn't crawl a staging or example website


When developing a website for a client, it's important that the staging server doesn't ruin SEO rankings. The strategy below describes how you can exclude search engines from crawling staging sites.

  • .htaccess
<IfModule mod_rewrite.c>
    RewriteEngine on

    # Deny robots on staging server
    # Replace staging.example.com with your own domain
    RewriteCond %{HTTP_HOST} ^staging.example.com$ [NC]
    RewriteRule ^robots.txt$ robots.exclude.txt [L]

You now need to create two files which will be served depending on your domain. The file robots.txt is targeted for production environments, and robots.exclude.txt for testing/staging.

Once you've made the changes to your htaccess file, you can test the mod_rewrite redirects with the htaccess tester made by madewithlove.

  • robots.txt
User-agent: *
  • robots.exclude.txt
User-agent: *
Disallow: /