Creating a site map for the first...

User 92156 Photo


Registered User
272 posts

Spinny wrote:
The way I figure it, Google's ambition is no less than making a map of the entire Internet ...

It looks that way.

Google's signature should actually be: :"Google Bwah ha ha ha"
User 126492 Photo


Ambassador
1,506 posts

Spinny wrote:
Actually, I put links to the sitemap on the handicapped-accessible nav section of every top-level page in my sites, and have never had any difficulties with the sites' indexing. If you're concerned about it, the easiest way to keep 'bots off the sitemap.html page is to have these lines in your robots.txt file:

User-agent: *
Disallow: /sitemap.html


That keeps you from having to make any mods to your actual site pages.


You don't want to be putting a 'Disallow' for sitemap.html in your robots.txt file.

What you should be adding to the robots.txt file is this line at the end, it is the path to the .xml file:-



The major search now recognise this line and is helpful if you have not submitted your map to Google Webmaster Tools or othe SE's/
Jim
---------------------------
User 597929 Photo


Registered User
1,332 posts

I actually have both. :cool:
"You can't be a real country unless you have a beer and an airline - it helps if you have some kind of a football team, or some nuclear weapons, but at the very least you need a beer." -- Frank Zappa

Visit Spinland Studios: http://www.spinland.biz
User 92156 Photo


Registered User
272 posts

Jim Cantrell wrote:
You don't want to be putting a 'Disallow' for sitemap.html in your robots.txt file.

Why is that?
Does having a path to the .xml file mean that the bot will sidestep the .html file?
User 148353 Photo


Ambassador
808 posts

Spinny wrote:
The way I figure it, Google's ambition is no less than making a map of the entire Internet.


Not just Google, but the Library of Congress. But the Library of Congress isn't stopping there, they want to archive every dadburn tweet as well!

Library of Congress Will Save Tweets
"Time heals everything. Know who said that? My Latin teacher at barber college!"
- Floyd Lawson

Widoktadwat - "Plays Well With Others"

User 1948478 Photo


Senior Advisor
1,850 posts

Lowell Ziegler wrote:
Not just Google, but the Library of Congress. But the Library of Congress isn't stopping there, they want to archive every dadburn tweet as well!

Library of Congress Will Save Tweets

Seems like the Library of Congress is run by a bunch of twits these days...

Have something to add? We’d love to hear it!
You must have an account to participate. Please Sign In Here, then join the conversation.