Query String Exclusions Not Working

User 515757 Photo


Registered User
9 posts

Hello:

Just downloaded the new(est) version of Site Mapper. Seems to run faster, but that could just be me...
However, when I add text strings to exclude URLs from the processing of my site - the final sitemap still includes these URLs. For example, I want to exclude all URLs that have text strings such as:

/offset
/reviews/
/search
/search/?tag=

These are very unique criteria, however in the final site map I can see almost all of the URLs that I DON'T want...Is anyone else excluding indexed URLs based on full or partial text strings?

Thanks for any help on this issue.

Mike P.

http://www.buffalospongecandy.com


User 187934 Photo


Senior Advisor
18,822 posts

Are you using the "Contains" Query?
I can't hear what I'm looking at.
It's easy to overlook something you're not looking for.

Here's my S-Drive site with
examples of what can be accomplished in VSD.
https://progrower.coffeecup.com/
Here's my CoffeeCup SCCP Shop with examples of what can be done.
https://progrower.coffeecup.com/shop/
This is a site I built for my work.(RSD)
http://esmansgreenhouse.com
This is a site I built for use in my job.(HTML Editor)
https://pestlogbook.com
This is my personal site used for testing and as an easy way to share photos.(RLM imported to RSD)
https://ericrohloff.com
User 515757 Photo


Registered User
9 posts

Eric:

I am using the "tools" / "options" to get to the pop-up window that prompts you to enter the URLs or queries to ignore...

I click "add" and get a prompt to enter the portion of the query to exclude.
I do not select the first radio button that prompts for the full URL to exclude.

I estimate that there are 100+ URLs that I would exclude so I don't want to enter a complete URL value one-buy-one...

Let me know if there is an error with my procedure...

Thanks for your time!

Mike P.
Buffalo, NY

ps: I tried to add a screen shot JPG file attachment but the upload doesn't work using the Chrome browser...
User 187934 Photo


Senior Advisor
18,822 posts

Right below that radio button for "Enter the portion" should be a arrow to select "Contains" ,"Starts with" or "Ends with".

See attached

I think your going to need the full url option to exclude most of the things you have listed.
Attachments:
I can't hear what I'm looking at.
It's easy to overlook something you're not looking for.

Here's my S-Drive site with
examples of what can be accomplished in VSD.
https://progrower.coffeecup.com/
Here's my CoffeeCup SCCP Shop with examples of what can be done.
https://progrower.coffeecup.com/shop/
This is a site I built for my work.(RSD)
http://esmansgreenhouse.com
This is a site I built for use in my job.(HTML Editor)
https://pestlogbook.com
This is my personal site used for testing and as an easy way to share photos.(RLM imported to RSD)
https://ericrohloff.com
User 126492 Photo


Ambassador
1,506 posts

Michael Pelone wrote:

/offset
/reviews/
/search
/search/?tag=


In the first three options above it looks to me as though you are trying to exclude three folders, in the fourth option it looks like you are again trying to exclude a folder plus a text string.

Have a look at the attached images and see if it helps how to distinguish the different options.

The sitemap1.png shows how a mixed set of exclusions would look like.

In the URL box, this tells sitemapper not to list any files within the /search folder, then if you added /search/?tag= into the Query option it would be ignored because of the full url you entered.

To just exclude the urls containing the text string ?tag= within the search folder you would enter it as in the query.png image, this will then tell Sitemapper to ignore any url containing ?tag= within it whichever folder it is found in.

A url is different to a text string query.

I hope this explains things a little better, if not, if you could give me the url of the site I might be able to explain it better after taking a look.
Attachments:
Jim
---------------------------
User 515757 Photo


Registered User
9 posts

Jim:

Yes, you are right...I was initially confused between the option to exclude a folder name versus excluding a text string. I should have spent more time reading the fine print...

I am now creating a new site map based on a series of partial text strings - looks like I have to add them one-by-one. It would be nice to be able to paste in a list of complete URLs into SiteMapper.

Recently, I have been experimenting with another free site map utility. it may not produce a pretty HTML file suitable for viewing by human visitors - but in my opinion, most of the views of my site maps are reported in my web log visitor files to be web spiders or Googlebot.

Check this link:
http://www.auditmypc.com/

Then scroll down to "sitemap generator"

Thanks for the comments...

Mike P.

www.buffalospongecandy.com
User 126492 Photo


Ambassador
1,506 posts

I don't know what you don't want Sitemapper to index but you should not need that many queries adding, using the query Contains should filter quite a lot out, remember the sitemap is not only for your visitors, it is a major part of how a search engine spider indexes your site.
Jim
---------------------------
User 440260 Photo


Registered User
26 posts

I have read this and other similar posts and even when I use the query Contains, the sitemapper still crawls the pages I want it to ignor.
User 126492 Photo


Ambassador
1,506 posts

It does take a little time to understand how the query string is fomatted so that it works in the way you want it too, but once you learn the basics it becomes much easier to use.

If you would like to post the url and an idea of which pages you don't want listing, then I will have a look and suggest some query strings for you.
Jim
---------------------------

Have something to add? We’d love to hear it!
You must have an account to participate. Please Sign In Here, then join the conversation.