Robot.txt files - Post ID 939

User 408675 Photo

Registered User
1 post

Although I have a programming background, I am brand new to website development. I have been reading about META tags and ROBOT.TXT. My understanding of the META tag is that I put keywords in to help the appropriate people find my sites. I think I've I figured out how to add the META tag line with my keywords; however, I'm still unsure about the ROBOT.TXT file ... exactly what it does and if I need it, how do I add it.

Any help would be appreciated!!

User 122279 Photo

Senior Advisor
14,450 posts

Yes, meta tags are put into the head section of an html document. In those tags you can specify keywords, contents, charset etc.

Regarding robot.txt files, do a google search and you will find what they are, what they are used for etc.

And BTW, change your user name in your profile to something else than your mail address. We don't want to make it too easy for spammers to harvest mail addresses in this forum, do we? ;)
Ha en riktig god dag!
Inger, Norway

My work in progress:
Components for Site Designer and the HTML Editor:

User 78330 Photo

Registered User
198 posts

Basically a Robots.txt is there to stop search engines spidering parts of your site that you don't want broadcast to the world.

This is mine for one of my sites.

# -- Harriers Online
# Robot Exclusion File -- robots.txt
# Author: Philip Lench
# Last Updated: 08/10/2006 11:00:00

User-agent: *
Disallow: /scripts
Disallow: /forum/
Disallow: /js
Disallow: /css
Disallow: /form_ima/
Disallow: /form_links/
Disallow: /form_contact/

So you can see from that that anything in those folders will be ignored and the spider moves on.

The forum is a contentious one because I want my forum to be found by Google etc but not all the individual threads.

Not easy to achieve so I decided to keep it out altogether.

Another benefit of blocking a forum is to cut down on bandwidth used by the spider wizzing through a thousand or so pages once a day and you seeing money leaking away.

If you want to go with a Robots.txt then just copy and alter the above into notepad, save as robots.txt and upload it to the root of your server.
User 447065 Photo

Trial User
4 posts

Just to add a useful resource for anyone new to robots.txt and would like to have a detail understanding and application of it :-

Have something to add? We’d love to hear it!
You must have an account to participate. Please Sign In Here, then join the conversation.