RE: [squid-users] Memory Error when using large acl files

Well i'm not so much in to the regex exept that the dstdomain_regex
worked with a much smaller file.

If I'm using a . infront of fx sex.com
so it'll be .sex.com in the dstdomain file I get the page error cannot
be displayed. The error page is "The server was not found, or a DNS problem
occurred) so it's like going to a page that does not exist.

and this happens for any page, it doesn't matter if it is located in the
dstdomain file or not.

If I get this to work I have a problem, since I'm not so familiar with all
the tools unix/linux has, I know there are commands to modify text files
so alittle help is needed for putting a . infront of all lines in a text
file. I know this is not the right place to ask this question, so I accept
that there are no responses to this, though I would be grateful if I get a
solution :-)

regards
Carsten Jensen

-----------
tis 2006-02-28 klockan 15:54 +0000 skrev Paul Mattingly:

> Why does squid's memory usage increase by nearly 320MB when the file is
only 9MB?

I would guess because you are using regex acl, and each line gets
compiled into a compiled regex internally to speed up the processing.

A dstdomain ACL of 600K entries or 9MB uses 37MB of memory on 64-bit
platforms or 23MB of memory on 32-bit platforms in my tests.

Startup time for parsing this dstdomain acl was about 15-20 seconds.

> Which of the redirectors/plug-ins are best for managing large blacklists
if this way just won't work on this scale?

The Squid dstdomain ACL is about the fastest you can find at the moment.

The SquidGuard url ACL is the most flexible for more detailed matches
beyond only the hostname, but overhead of using a redirector is very
significant.

the regex type acls is bad performers in both. Not much which can be
done about that as regex have no structure.

SquidGuard has one nice feature in that it can use db files to avoid
building the complete index in memory on startup. And due to SquidGuard
being a redirector this also saves considerably amount of memory
compared to each copy of SquidGuard building it's own in-memory index..

Regards
Henrik