At RateItAll, we spend a lot of time battling rogue bots who scrape and crawl our site, sometimes hitting us so hard that they can bring our four servers to their knees.
After realizing he was spending up to eight hours per week battling bots, Brett Tabke over at WebmasterWorld got so frustrated that he blocked ALL bots. This was more a symbolic act than anything else - he soon had to soften his stance to allow Google, Yahoo, and MSN back in. Not many UGC sites can survive without organic search traffic.
As of yet, there is no mainstream tool that I'm aware of that can help webmasters manage this rogue traffic in an automated traffic. It's still a lot of manually blocking IP addresses, and writing scripts to detect non human behavior.
One company to watch in this area is CrawlWall, which is the labor of love of Incredibill - who is pretty well known in SEO circles.
