Re: [squid-users] Mass Mailers & Spiders

>Does someone know how Mass Mailers and Spiders can be blocked via
>squid from functioning?

Mass mailers generally use SMTP over port 25, which Squid has nothing
to do with. Spiders, on the other hand, do use HTTP, and Squid can
be part of the solution there.

>I actually tried using HTB to restrict its bandwidth but it seemed
not to
>have any effect

The advanced routing queues can only restricy by IP Address and port
(and by packet flags), and are too blunt a tool to use in this case.
You need something (like Squid) that can read the traffic at the
application level.

However, more information would improve the quality of your answer.
Who is using the spider? What is it being used to do? Do you know
what program it is?

Your best bet would be to find some unique characteristic of the
spider (such as the User Agent string) and setup a delay pool to
slow it way down. You indicated you tried this before and it didn't
work - what was your setup like?

Outright blocking it will draw you into a cat and mouse game with
the spider's user - he/she will try to work around your blocking,
and you'll have to keep working to continue to block the user. Simply
slowing it way down may make the user think it's a connection problem
or a problem with the spider.

Hopefully you have a good acceptable use policy and can use it to
boot the user off your network - that will be the only sure solution.

Adam
Received on Sat Jul 19 2003 - 09:48:36 MDT