Robots.txt question

User 332337 Photo


Registered User
120 posts

Just revamped an old site with RSD, fewer pages needed - and looking at the server error pages I'm seeing the omission of a robots.txt file.

What is the general consensus on disallowing; the css, font and js folder?

User-agent: *
Disallow: /css/
Disallow: /fonts/
Disallow: /js/

Is this considered good practice?

Or should I just have a robots.txt file that allows all files to crawled?

Thanks...
User 53435 Photo


Registered User
79 posts
Online Now

Allow the pages you want indexed, which should have the keywords set at a meta tag. None of the ones listed above are going to be indexed so it is a waste of time to allow the crawler to read them.
Also remember to include the link to the sitemap XML file in the robots.txt file which is a big help to the crawler.

Have something to add? We’d love to hear it!
You must have an account to participate. Please Sign In Here, then join the conversation.