Comments on: War against the automated content scrapers

By: Venetsian

Venetsian — Wed, 13 May 2009 02:54:37 +0000

Yes I agree to a certain level, but it still blocks a large amount of spam bots. You can always add more user agents to the allowed list which solves your problem.
I think the best way is to have a centralized spam bot detection system but I don’t think anybody will go for it since you will have to establish trust with this type of organization in order to use such service which makes it quite complicated to arrange. If somebody does make it then it will solve the spam bot issue once and for all. If you do know which IPs are attacking you then you should edit your .HTACCESS file to deny their ip address ranges. If you don’t then .. well you are not protected.

By: San Luis Obispo Events

San Luis Obispo Events — Wed, 13 May 2009 02:15:14 +0000

This is not a good solution to block bots. It is common for spam bots to use the User Agent of a real browser to get past this filter. Also, although some of the mobile phone browsers use Opera or Safari, this script would block all that don’t use a standard browser.

By: anonymus

anonymus — Sun, 03 May 2009 19:25:44 +0000

Please note that lynx and other text-browsers will be banned
too by your php script

By: Venetsian

Venetsian — Fri, 10 Apr 2009 19:08:34 +0000

@okinawa Yes, you put that on top of the header so that it reads the user agent and returns the proper http header response code. If you print something before this code it won't work!! (should give you warning or error that header was already sent!). Venetsian

By: okinawa

okinawa — Sat, 04 Apr 2009 11:25:49 +0000

I would love to read more about your thoughts on Google’s duplicate content filter. I don’t know much about it. Do you place this in your website header?