I'm looking for a way to prevent search engines from indexing a website that I am creating. I am only using VSD at the present time. I noticed that with HTML coding, a line of code such as -

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

will prevent most search engines from indexing a website. Is there any way to do this or something similar using only VSD?

Thanks in advance for any help.

Stew

Take this code and put it in a robots.txt file and place it in your main directory. Make sure its permissions are set so that visitors (such as search engines) can read it.

# robots.txt
User-agent: Googlebot
Disallow: /
User-agent: googlebot-image
Disallow: /
User-agent: googlebot-mobile
Disallow: /
User-agent: MSNBot
Disallow: /
User-agent: Slurp
Disallow: /
User-agent: Teoma
Disallow: /
User-agent: twiceler
Disallow: /
User-agent: Gigabot
Disallow: /
User-agent: Scrubby
Disallow: /
User-agent: Robozilla
Disallow: /
User-agent: Nutch
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: baiduspider
Disallow: /
User-agent: naverbot
Disallow: /
User-agent: yeti
Disallow: /
User-agent: yahoo-mmcrawler
Disallow: /
User-agent: psbot
Disallow: /
User-agent: asterias
Disallow: /
User-agent: yahoo-blogs/v3.9
Disallow: /
User-agent: *
Disallow: /
Disallow: /cgi-bin/

SirAGE, thank you for your reply to my question. I'm kind of new to using VSD, and was wondering how I would insert the code you provided above, into the main directory of my VSD website. Can I do this without using an HTML editor? If so, how?

Thanks very much for your help.

Stew

Use note pad, create robots.txt put my code in it. And then save it to your root directory by using a FTP program, I do not know much about how VSD works. But the robots.txt has nothing to do with it, I do not think you can load it through VSD but I may be wrong. I would just use a FTP program like Direct FTP.

And there is no coding that needs to be done, you are not saving a HTML file, you are saving the file as a .txt file. The robots.txt file is what robots look for when they craw your site, that look at it first and see what you say they can have access too which with the code I provided nothing.

Hope this helps.

SirAGE, thanks again! This definitely helps. Will give it a try.

Stew

Will the robots.txt file work at a sub-directory level? Meaning, I want the top level directory and many sub-directories to be indexed, all except one (which contains sub-sub-directories as well).

Just add these two lines to a robots.txt file.

User-agent: *
Disallow: /yourdirectorynamehere/

Also, this will allow the rest of your site to be crawled... But they will not enter the directory you mention in that disallow tag.