Here is my tip how to block competitors from spying on your links. If you build your private blog network or satellite sites linking to your money site it is very important to keep it as stealth as possible. What I use for my sites are two things:
1. robots.txt
2. .htaccess
I will not explain how to create such files as it is easy even for me and I'm not a programmer:)
Our main goal is to block robots such as Majestic seo, Ahrefs and Open site explorer. Very often those little bastards
don't respect comands in robots.txt this is why you need to block them with .htaccess file.
Here is how those files should look like:
in robots.txt ( which is basically notepad file called robots) paste this:
User-Agent: Googlebot
Allow: /
User-Agent: *
Disallow: /
This input will block access to your website to all bots apart of Google. In Theory. Many bots don't respect it so
it is good idea to block them through .htaccess file.
This is how my code looks like:
version for Wordpress:
and for any other site:
Do not simply change your .htaccess files for the second option because you might have some important commands in your .htaccess already.
However you would need to paste the code on top of this file. Version for Wordpress is copy- paste.
Like you can see above I block also other bots like Yandex, mainly because they are using a lot of transfer from the website and sometimes it might cost you additional fees at your hosting provider. You can also add other bots to this list if they're pissing you off:)
This rule: will send the bastards from your website to any site- simply put there whatever url. You can put here also website of your competitors :D
If you look at the ip's I'm blocking also countries such as China and Japan because I don't care about this traffic
and those bots consuming only transfer. You need to also remember it is the best to implement those files before linking campaign. If you put it later, there still will be record however new links won't appear in those backlink tools..
So this is my little input in the community and I hope it will help you to hide from grassers.
1. robots.txt
2. .htaccess
I will not explain how to create such files as it is easy even for me and I'm not a programmer:)
Our main goal is to block robots such as Majestic seo, Ahrefs and Open site explorer. Very often those little bastards
don't respect comands in robots.txt this is why you need to block them with .htaccess file.
Here is how those files should look like:
in robots.txt ( which is basically notepad file called robots) paste this:
User-Agent: Googlebot
Allow: /
User-Agent: *
Disallow: /
This input will block access to your website to all bots apart of Google. In Theory. Many bots don't respect it so
it is good idea to block them through .htaccess file.
This is how my code looks like:
version for Wordpress:
Code:
# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} .*Twice.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*Yand.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*Yahoo.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*Voil.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*libw.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*Java.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*Sogou.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*psbot.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*Exabot.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*boitho.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*ajSitemap.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*Rankivabot.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*DBLBot.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*MJ1.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*ask.*
RewriteCond %{HTTP_USER_AGENT} .*AhrefsBot.*
RewriteRule ^(.*)$ http://anysite.com/ [L,R=301]
Order Allow,Deny
Allow from all
Deny from 110.0.0.0/8
Deny from 111.0.0.0/8
Deny from 112.0.0.0/5
Deny from 120.0.0.0/6
Deny from 124.0.0.0/8
Deny from 125.0.0.0/8
Deny from 147.0.0.0/8
Deny from 169.208.0.0
Deny from 175.0.0.0/8
Deny from 180.0.0.0/8
Deny from 182.0.0.0/8
Deny from 183.0.0.0/8
Deny from 202.0.0.0/8
Deny from 203.0.0.0/8
Deny from 210.0.0.0/8
Deny from 211.0.0.0/8
Deny from 218.0.0.0/8
Deny from 219.0.0.0/8
Deny from 220.0.0.0/8
Deny from 221.0.0.0/8
Deny from 222.0.0.0/8
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPressCode:
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} .*Twice.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*Yand.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*Yahoo.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*Voil.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*libw.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*Java.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*Sogou.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*psbot.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*Exabot.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*boitho.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*ajSitemap.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*Rankivabot.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*DBLBot.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*MJ1.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*Rankivabot.* [OR]
RewriteCond %{HTTP_USER_AGENT} .*ask.*
RewriteCond %{HTTP_USER_AGENT} .*AhrefsBot.*
RewriteRule ^(.*)$ http://anysite.com/ [L,R=301]
Order Allow,Deny
Allow from all
Deny from 110.0.0.0/8
Deny from 111.0.0.0/8
Deny from 112.0.0.0/5
Deny from 120.0.0.0/6
Deny from 124.0.0.0/8
Deny from 125.0.0.0/8
Deny from 147.0.0.0/8
Deny from 169.208.0.0
Deny from 175.0.0.0/8
Deny from 180.0.0.0/8
Deny from 182.0.0.0/8
Deny from 183.0.0.0/8
Deny from 202.0.0.0/8
Deny from 203.0.0.0/8
Deny from 210.0.0.0/8
Deny from 211.0.0.0/8
Deny from 218.0.0.0/8
Deny from 219.0.0.0/8
Deny from 220.0.0.0/8
Deny from 221.0.0.0/8
Deny from 222.0.0.0/8However you would need to paste the code on top of this file. Version for Wordpress is copy- paste.
Like you can see above I block also other bots like Yandex, mainly because they are using a lot of transfer from the website and sometimes it might cost you additional fees at your hosting provider. You can also add other bots to this list if they're pissing you off:)
This rule: will send the bastards from your website to any site- simply put there whatever url. You can put here also website of your competitors :D
If you look at the ip's I'm blocking also countries such as China and Japan because I don't care about this traffic
and those bots consuming only transfer. You need to also remember it is the best to implement those files before linking campaign. If you put it later, there still will be record however new links won't appear in those backlink tools..
So this is my little input in the community and I hope it will help you to hide from grassers.