Fine Tuning the Google Bot - Post ID...

User 458800 Photo


Registered User
78 posts

Oops, I initially posted this in the wrong section.

So here's a retry...



Greetings!

I am preparing a website for my home owner association/community.

On our old site, which I am preparing to replace with the new one done in VSD, the Google bot was reading our whole website and (I think) downloading our HOA documents. This was happening many times per hour, and caused us to go over our bandwidth agreement with our hosting company. I had to put in a ROBOTS.TXT file to prevent this from happening. After waiting for the new month to begin for our site to be re-activated, we monitored the traffic and saw that it had fallen WAY down.

That's kind of harsh, though. When I take the new VSD-created site live, I would like to have more granular control over the Google Bot. I don't mind if he looks at my site once a week or so, but I don't want him downloading a lot of data.

An alternative would be to just provide some keywords (like, for example, a couple of different spellings of the name of my community, or my city and the word "HOA") so that Google will show my site on a result. For this type of search engine result, I don't think it's necessary for a bot to actually read my whole site, right?

Any ideas, links to reference material, or general advice on this subject would be appreciated.

Thanks in advance!
Windows 11
Intel i9 (workstation) and i9 (laptop). Gobs of RAM and acres of SSD space on both machines.

Have something to add? We’d love to hear it!
You must have an account to participate. Please Sign In Here, then join the conversation.