[squid-users] (Fwd) More flexible logging options?

This issue has reared its ugly head once again for us. This time, the 'spyware'
was Sun's Java autoupdater, which caused a single host to hit our proxy about
140 times per second most of yesterday, generating 1.3Gb of denials in our logs,
which are normally from 300-600Mb per day. The server handled the load just
fine, until it ran out of disk space trying to rotate the logs overnight.

Here's a sample log entry:
1143608899.081 3 10.2.120.18 TCP_DENIED/407 945 GET
http://java.sun.com/webapps/download/GetFile/1.5.0_03-b07/windows-
i586/jre1.5.0_03.msi - NONE/- text/html

I had a few replies in 2004 on how to deal with this problem, which I will re-visit,
but I'm curious how others are dealing with this issue, and if any new ideas have
come up since then.

We are running 2.5stable9 on Mandrake 9.2.

------- Forwarded message follows -------
From: Shawn Wright <swright@sls.bc.ca>
To: squid-users@squid-cache.org
Subject: More flexible logging options?
Send reply to: swright@sls.bc.ca
Date sent: Tue, 23 Nov 2004 14:43:52 -0800

We are finding squid's logging options quite limited, and are wondering if
there are any patches, or other ways to deal with some of the issues we
encounter. For example, in the past few weeks, we've had numerous
cases where a single client can generate 600Mb+ of log entries in a day,
all caused by spyware hitting a small group of URLs many times per
second. Of course, they are all denied, since we require authentication
for all except a few cases, and the spyware doesn't pass credentials to
the proxy.
During times when our proxy is being assaulted by spyware, it spends a
great deal of CPU time logging these denials. I would like to explore the
possibility of one or more of the following:

-handing off the logging to a separate process such as multilog
-finding some way to place log limits where multiple lines from a single
host would otherwise fill the logs. ie: maximum 5 denials logged per
second per host, with a burst of 20.
-limiting max # of connections allocated to a single IP per minute, since
delay pools won't help when all the connections are denials (I don't
think).

Thanks for any suggestions.

------- End of forwarded message -------
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Shawn Wright, I.T. Manager
Shawnigan Lake School
http://www.sls.bc.ca
swright@sls.bc.ca
Received on Wed Mar 29 2006 - 11:06:13 MST