RE: [squid-users] Memory Error when using large acl files

Yes that is what I was doing

Now I have converted to a dstdomain acl and everything is fine. Squid is
using 37MB memory on a PIII/1GHz with 512MB RAM.

When I load the current squid configuration from cache manager, squid
shoots up to 99% CPU load and then the browser crashes. I have to send a
shutdown signal to regain normality. I guess the browser just cannot
handle displaying 550,000 entries. It is a shame because I find the
config display very useful

Thanks for your help
Paul

-----Original Message-----
From: Henrik Nordstrom [mailto:henrik@henriknordstrom.net]
Sent: 28 February 2006 21:36
To: Paul Mattingly
Cc: squid-users@squid-cache.org
Subject: RE: [squid-users] Memory Error when using large acl files

tis 2006-02-28 klockan 15:54 +0000 skrev Paul Mattingly:

> Why does squid's memory usage increase by nearly 320MB when the file
is only 9MB?

I would guess because you are using regex acl, and each line gets
compiled into a compiled regex internally to speed up the processing.

A dstdomain ACL of 600K entries or 9MB uses 37MB of memory on 64-bit
platforms or 23MB of memory on 32-bit platforms in my tests.

Startup time for parsing this dstdomain acl was about 15-20 seconds.

> Which of the redirectors/plug-ins are best for managing large
blacklists if this way just won't work on this scale?

The Squid dstdomain ACL is about the fastest you can find at the moment.

The SquidGuard url ACL is the most flexible for more detailed matches
beyond only the hostname, but overhead of using a redirector is very
significant.

the regex type acls is bad performers in both. Not much which can be
done about that as regex have no structure.

SquidGuard has one nice feature in that it can use db files to avoid
building the complete index in memory on startup. And due to SquidGuard
being a redirector this also saves considerably amount of memory
compared to each copy of SquidGuard building it's own in-memory index..

Regards
Henrik
Received on Thu Mar 02 2006 - 09:06:55 MST