Unusual traffic on one of my sites?

[ShaunMac]

Hi all,

Don't know if this is the right place to ask but can't see anywhere else on the forum where I could post this but..............

I've noticed a couple of things about the traffic on my site when analyzing my webstats.

First off, 13% of my bandwidth allocation was used up in 1 day at the start of this month!

When I looked at my stats I saw that this traffic could be pretty much traced back to one domain (www.covad.net) using a program called Xenu Link Sleuth. Any idea what this is all about and should I be concerned?

Secondly, I see that 50% or more of my traffic each month is down to bots crawling my site like Googlebot, Gigablast, MSNbot, etc. I know I've submitted my website to a number of search engines and was expecting to see my web site crawled periodically but some of these bots are crawling my site daily with Googlebot some days on my site almost the entire day. Is this sort of traffic typical or, again, should I be concerned? If I should be concerned then how can I take action against this?

Cheers

[Jacob]

Hi,

Xenu Link Sleuth is a broken link checker. If you do not bots to crawl part of your site, you can setup a robots.txt which will stop the vast majority of them. You can find more details at: http://www.robotstxt.org/

Regards,

Jacob

[ShaunMac]

Originally Posted by Jacob

Hi,

Xenu Link Sleuth is a broken link checker. If you do not bots to crawl part of your site, you can setup a robots.txt which will stop the vast majority of them. You can find more details at: http://www.robotstxt.org/

Regards,

Jacob

Cheers for that Jacob but should I be concerned over the amount of traffic? Xenu Link Sleuth alone used up 800Mb of my bandwidth. If I go beyond my allocation then I would need to pay for more.