Re: [squid-users] Squid network read()'s only 2k long?

On Mon, 1 Nov 2010 15:00:21 +0000, declanw_at_is.bbc.co.uk wrote:
> Greetings!
>
> I am poking a potential squid upgrade from squid 2 to 3.1.8 with a new
> config, but it's added around 40% more CPU load, and I'm looking for
> tune-ups.
>
> One thing I notice it doing when I truss is read()ing HTTP responses
with
> only a 2046 byte buffer, whereas squid2 used 24Kb. This is making for a
> lot of
> unnecessary system calls and go-arounds in the main polling loop.
>
> I went for a rummage in the code for the buffer size decisions, but got
> very
> very lost in the OO abstractions very quickly. Can anyone point me at
> anything I can tweak to fix this?

It's a global macro defined by auto-probing your operating systems TCP
receiving buffer when building.
Default is 16KB and max is 64KB. There may also be auto-probing done at
run time.

It is tunable at run-time with
http://www.squid-cache.org/Doc/config/tcp_recv_bufsize/

>
> Besides that, I have a laaarge url_regexp file to process, and I was
> wondering if there was any benefit to trying to break this regexp out to
a
> perl helper process (and if anyone has a precooked setup doing this I
can
> borrow)

The others have already covered the main points of this. ufdbGuard is
probably the way to go once you have restricted the size down by
elminiating all the entries which can be done with dstdomain and other
faster ACL types.

>
> Beyond that, I assume, to get the most out of a multi-cpu system I
should
> be running one squid per CPU, which means I need more IP's and that they
> can't share their memory or disk caches with each other directly, and I
> would need to switch on HTCP to try and re-merge them?

Possibly. You may want to test out 3.2 with SMP support. Reports have been
good so far (for a beta).

>
> While I'm musing here, is there any way to make an ACL construct that
makes
> a decision based on whether something is already cached? I have a lot of
> heavy ACL's, and if something is cached, in my case, it will prove that
it
> has previously passed all the ACL's and I can just return the cached
copy.

Not as such. The "acl source hier_code NONE" ACL could do this in the
http_reply_access tests where the source is known. The cache index lookup
is one of the heavier operations and depends on all request adaptation
being done first. It may be possible to make a cache_status ACL or such
which works in adapted_http_access and later tests.

All ACL with external data sources (DNS, external_acl_type, ident, etc)
cache their results for faster re-use. Tuning the TTL for each of these can
reduce their impact.

>
> Build: Sun Solaris 9
> PATH=~/sunstudio12.0/bin:$PATH ./configure CC=cc CXX=CC CFLAGS="-fast
> -xtarget=ultra3i -m64 -xipo" CXXFLAGS="-fast -xtarget=ultra3i -m64
-xipo"
> --enable-cache-digests --enable-removal-policies=lru,heap
> --enable-storeio=aufs,ufs --enable-devpoll

Ah. You will definitely be wanting 3.1.9. /dev/poll support is included
and several ACL problems specific to the S9 are fixed.

>
> Tuney bits of Config:
> htcp_port 0
> icp_port 0
> digest_generation off
> quick_abort_min 0 KB
> quick_abort_max 0 KB
> read_ahead_gap 64 KB
> store_avg_object_size 16 KB
> read_timeout 5 minutes
> request_timeout 30 seconds
> persistent_request_timeout 30 seconds
> pconn_timeout 3 seconds

NOTE: pconn_timeout tuning can no longer be done based on info from older
versions. There have been a LOT of fixes that make 3.1.8+ pconn support
HTTP compliant, used more often and less resources hungry than older
versions.

> cache_mem 512 MB
> maximum_object_size_in_memory 64 KB

NP: It's worth noting that 3.x has fixed the large file in memory problems
which 2.x suffers from. 3.x will handle them in linear time instead of with
exponential CPU load.

> memory_replacement_policy heap GDSF
> ignore_expect_100 on

If this is actually a problem you may benefit extra from 3.2 beta here as
well.

Amos
Received on Mon Nov 01 2010 - 21:36:57 MDT