Web search robots and robots.txt

[ShaunMac]

Hi all,

I've noticed - and think I've mentioned in the past here - that I seem to be getting a lot of traffic on my website from search engine bots and Googlebot in particular.

It's been suggested that I put a robots.txt file on my site and I'm seriously thinking of doing this now. I've used 1.3Gb of my bandwidth this month already with Goooglebot responsible for over 1Gb of that.

I think it's Googlebot crawling my forum but am not 100% sure. The only thing that's stopped me from putting a robots.txt file on my site is this:-

Would putting a robots.txt file on my site stop my site being visited by search engine bots altogether? I'd like my site to still be ranked by the major search engines and am concerned that a robots.txt file would result in my site not being ranked at all.

Secondly, is it possible to set up a robots.txt file to tell bots not to crawl my forum only? I'm sure that's the reason for all the search engine bot traffic and would like to stop that if I can.

Cheers.

[WhiskyFudge]

I suggest you do a bit of googling for more information about the robots.txt file.

You can customise it to (dis)allow robots and where they can have access to. So yes you will be able to stop them crawling your forum.

[ShaunMac]

Well, after a bit of Googling, I put the following robots.txt file on my site:-

User-agent: *
Disallow: /Forum/

that, hopefully, will stop the bots crawling my forum. I'll have to see how it affects the traffic on my site.

[ShaunMac]

Originally Posted by ShaunMac

Well, after a bit of Googling, I put the following robots.txt file on my site:-

User-agent: *
Disallow: /Forum/

that, hopefully, will stop the bots crawling my forum. I'll have to see how it affects the traffic on my site.

Well so much for the robots.txt file.

Just been on my site and found Googlebot crawling my forum despite the above text being in the file and the file being on my site.

Back to the drawing board. Anyone got any advice on how to stop these damn bots crawling my site and eating up all my bandwidth?

[ShaunMac]

Originally Posted by ShaunMac

Well so much for the robots.txt file.

Just been on my site and found Googlebot crawling my forum despite the above text being in the file and the file being on my site.

Back to the drawing board. Anyone got any advice on how to stop these damn bots crawling my site and eating up all my bandwidth?

Looks like I spoke too soon. I now seem to be getting virtually no bot traffic at all in my forum - at least, I've not seen any bots crawling my forum for quite a while.

I should have shown a bit more patience.