Re: [squid-users] investigate squid eating 100% CPU

On 27/03/2013 6:09 a.m., Youssef Ghorbal wrote:
>>> the cachemanager can be usefull to see the actual activity of your squid :
>>>
>>> squidclient localhost mgr:5min
>>>
>>> gives you the last 5 min stats. (see if the n° of req/s is coherent
>>> with what you expect )
>>
>> Here after the output of the mgr:5min
>> It show that we are around 168 req/s
>> for a cpu usage around 99%
>> I don't think that 160 req/s is such a big number that can explain full CPU time.
> Forgot to mention that it somtimes drops to ~30% CPU for some periods.
> For example right now we are :
> client_http.requests = 160.370710/sec
> cpu_usage = 39.305030%
>
> Which makes me think of some kind of requests getting it to go crazy.
>
> [complete output here]
>
> sample_start_time = 1364317190.222942 (Tue, 26 Mar 2013 16:59:50 GMT)
> sample_end_time = 1364317490.271499 (Tue, 26 Mar 2013 17:04:50 GMT)
> client_http.requests = 160.370710/sec
> client_http.hits = 17.510499/sec
> client_http.errors = 0.016664/sec
> client_http.kbytes_in = 278.614904/sec
> client_http.kbytes_out = 3589.275718/sec

client<->squid : 31Mbps.

> client_http.all_median_svc_time = 0.092188 seconds
> client_http.miss_median_svc_time = 0.097357 seconds
> client_http.nm_median_svc_time = 0.000911 seconds
> client_http.nh_median_svc_time = 0.016481 seconds
> client_http.hit_median_svc_time = 0.000911 seconds
> server.all.requests = 145.123178/sec
> server.all.errors = 0.000000/sec
> server.all.kbytes_in = 3294.903365/sec
> server.all.kbytes_out = 269.543039/sec

Squid<->server : 28Mbps

Total: 59Mbps.

Which is slightly higher than the known "good" performance limit for
Squid-3.1. Which is up to ~50Mbps, tuning both in Squid and the system
can reach around 100Mbps IIRC. But that sort of numbers you are looking
at specific traffic types as well.

> select_loops = 2316.561716/sec
<snip>
> syscalls.sock.reads = 1316.883520/sec
> syscalls.sock.writes = 1770.096831/sec

It looks like sending and receiving data in ~2KB chunks, just over one
packet per cycle.

> syscalls.sock.recvfroms = 58.203913/sec
> syscalls.sock.sendtos = 30.974986/sec
> cpu_time = 117.934176 seconds
> wall_time = 300.048557 seconds
> cpu_usage = 39.305030%

CPU usage at 40%.

Squid-3.1 does use around 30% CPU to push data at line rate, which is
what this set of counters seems to be showing. We have done a bunch of
work aimed at improving things all over Squid in the more recent
releases, if you want to assist the developers tracking this down please
upgrade to the latest release before delving too deeply into the
analysis as all fixes need to go in through 3.HEAD anyway and you maybe
repeating already done work (only might there is still CPU issues hiding
in the latest development code).

What do things look like when its at or close to 100% CPU?

Also any sign of the OS swapping virtual memory when the CPU climbs?
that is a known performance killer for a number of reasons we can't fix
easily.

Amos
Received on Wed Mar 27 2013 - 10:19:59 MDT