Yesterday I purchased Google Sitemapper and all of my sites that I've crawled have worked perfectly. However, I've run into an issue when crawling www.npug.org and I was hoping that someone here could help.

The NPUG web site features a vBulletin forum and when Google Sitemapper reaches the forums, it just crawls and crawls and crawls. For example, yesterday it crawled the forums for 10 hours until I finally cancelled the process and killed the application.

I'd like to include the forum in the sitemap, but it doesn't need to crawl every single page of the forums either. Anyone have any suggestions how how I can tweak the options for dealing with vB?

Thanks in advance!

Take a look at the Profile Options ( > Tools > Options ). You can have SiteMapper to ignore whole folders, single files and queries. For example forumdisplay.php is ok, but you choose to ignore showthread.php and members.php.

Ralph,

Thank you for the feedback. I took your suggestion to heart and added the paths in both the queries section and the files section. That helped somewhat, but I'm still getting an endless loop with the forumdisplay.php.

My solution has been to just stop the process, go back and delete the repeated elements and then save the file. That seems to work, but it's a bit tedious.

I would have thought that this would have been a KB item given that vB is so popular, but alas it's not.

Thanks again for the help.