Robot.txt files - Post ID 939
Any help would be appreciated!!
Regarding robot.txt files, do a google search and you will find what they are, what they are used for etc.
And BTW, change your user name in your profile to something else than your mail address. We don't want to make it too easy for spammers to harvest mail addresses in this forum, do we?
This is mine for one of my sites.
# -- Harriers Online
# Robot Exclusion File -- robots.txt
# Author: Philip Lench
# Last Updated: 08/10/2006 11:00:00
So you can see from that that anything in those folders will be ignored and the spider moves on.
The forum is a contentious one because I want my forum to be found by Google etc but not all the individual threads.
Not easy to achieve so I decided to keep it out altogether.
Another benefit of blocking a forum is to cut down on bandwidth used by the spider wizzing through a thousand or so pages once a day and you seeing money leaking away.
If you want to go with a Robots.txt then just copy and alter the above into notepad, save as robots.txt and upload it to the root of your server.
Have something to add? We’d love to hear it!
You must have an account to participate. Please Sign In Here, then join the conversation.