Robots.txt question - Post ID 276840

User 332337 Photo


Registered User
120 posts

Just revamped an old site with RSD, fewer pages needed - and looking at the server error pages I'm seeing the omission of a robots.txt file.

What is the general consensus on disallowing; the css, font and js folder?

User-agent: *
Disallow: /css/
Disallow: /fonts/
Disallow: /js/

Is this considered good practice?

Or should I just have a robots.txt file that allows all files to crawled?

Thanks...
User 53435 Photo


Registered User
79 posts

Allow the pages you want indexed, which should have the keywords set at a meta tag. None of the ones listed above are going to be indexed so it is a waste of time to allow the crawler to read them.
Also remember to include the link to the sitemap XML file in the robots.txt file which is a big help to the crawler.

Have something to add? We’d love to hear it!
You must have an account to participate. Please Sign In Here, then join the conversation.