SiteMapper spidering problem / Not...

User 103173 Photo


VP of Software Development
0 posts

Judit Kovacs wrote:
I have the same problem and it drives me nuts. Sitemapper finds only the index page in site www.kerolamps.com; I reviewed all and nothing I have done made a difference. It has a similar DHTML menu as my other site www.visitmajorscreek.com which it crawls without problems. Please help to restore my sanity!

Sitemapper scans your website in the same way as the spiders from popular search engines such as Google , Yahoo! , and Bing . This means that it does not follow links in JavaScript, DHTML, or Flash files. Here's why: Search engine spiders can only follow HTML.

Let's use Flash as an example. A Flash-based webpage is pretty and interactive when a human looks at it, but when a search engine spider tries to scan it, all it sees is an <embed> tag. It can't go into the Flash animation to find the links, so it skips past the page, and your site doesn't get a good ranking. Simple as that.

So what's the solution? Use HTML for all your links and navigation . If you want your site to rank well in search engine results, to work correctly on all browsers, and to be accessible to people with disabilities, you have to structure your site with semantically valid HTML and add CSS and JavaScript using the principles of progressive enhancement .



Learn the essentials with these quick tips for Responsive Site Designer, Responsive Email Designer, Foundation Framer, and the new Bootstrap Builder. You'll be making awesome, code-free responsive websites and newsletters like a boss.
User 2490574 Photo


Registered User
20 posts

Hmmm - I seem to have the 'non-seen' page problem on my site http://gblcg.com as well...

For example, I have seven pages under http://www.gblcg.com/goodman-bergh-values.html as referred to on that page in html. My sitemap only shows three...

When looking at http://www.gblcg.com/goodman-bergh-services.html, it references five pages which are all indexed correctly on the sitemap.

Bit stuck here - any idea what's going on and, more importantly, how to solve it?
User 187934 Photo


Senior Advisor
20,181 posts

I got the same result and didn't find anything that would be in common about the lost links. I would open a support topic and reference this thread.
I can't hear what I'm looking at.
It's easy to overlook something you're not looking for.

This is a site I built for my work.(RSD)
http://esmansgreenhouse.com
This is a site I built for use in my job.(HTML Editor)
https://pestlogbook.com
This is my personal site used for testing and as an easy way to share photos.(RLM imported to RSD)
https://ericrohloff.com
User 103173 Photo


VP of Software Development
0 posts

Raymond den Otter wrote:
Hmmm - I seem to have the 'non-seen' page problem on my site http://gblcg.com as well...

For example, I have seven pages under http://www.gblcg.com/goodman-bergh-values.html as referred to on that page in html. My sitemap only shows three...

When looking at http://www.gblcg.com/goodman-bergh-services.html, it references five pages which are all indexed correctly on the sitemap.

Bit stuck here - any idea what's going on and, more importantly, how to solve it?

This is because you are not using the same URL for all links. Some of them use http://www.gblcg.com and others use http://gblcg.com. That is why you are getting different results because those are 2 different URL's. You should either use relative path names (use a href="/goodman-bergh-values.html") or adjust them to all use the same URL http://www.gblcg.com

Always remember that WWW is a host name which could be a different location then domain.com (even though most are the same). It is similar to having shop.domain.com. For Sitemapper, all links have to be on the exact same domain as you are scanning.
Learn the essentials with these quick tips for Responsive Site Designer, Responsive Email Designer, Foundation Framer, and the new Bootstrap Builder. You'll be making awesome, code-free responsive websites and newsletters like a boss.
User 187934 Photo


Senior Advisor
20,181 posts

good catch Scott.;)
I can't hear what I'm looking at.
It's easy to overlook something you're not looking for.

This is a site I built for my work.(RSD)
http://esmansgreenhouse.com
This is a site I built for use in my job.(HTML Editor)
https://pestlogbook.com
This is my personal site used for testing and as an easy way to share photos.(RLM imported to RSD)
https://ericrohloff.com
User 122279 Photo


Senior Advisor
14,447 posts
Online Now

I learnt this from Scott some time ago in connection with WI. I hadn't actually thought as far as the Sitemapper, which I guess I should have. Thanks for reminding me - and others. ;)
Ha en riktig god dag!
Inger, Norway

My work in progress:
Components for Site Designer and the HTML Editor: https://mock-up.coffeecup.com



Have something to add? We’d love to hear it!
You must have an account to participate. Please Sign In Here, then join the conversation.