Re: [squid-users] Squid memory usage

On 31/05/2013 3:00 p.m., Nathan Hoad wrote:
> Fantastic explanation! Thanks heaps Amos. Would it make sense for this
> to go onto wiki.squid-cache.org somewhere?

Yes. Francesco and I are trying to document all of the manager reports
in there.
http://wiki.squid-cache.org/Features/CacheManager#Available_Reports
But, it is slow going with just the two of us documenting in spare time.
(hint, hint, anyone?)

Amos

> --
> Nathan Hoad
> Software Developer
> www.getoffmalawn.com
>
>
> On Wed, May 29, 2013 at 7:00 PM, Amos Jeffries <squid3_at_treenet.co.nz> wrote:
>> On 29/05/2013 2:19 p.m., Nathan Hoad wrote:
>>> On Tue, May 28, 2013 at 3:23 PM, Amos Jeffries <squid3_at_treenet.co.nz>
>>> wrote:
>>>> On 28/05/2013 3:59 p.m., Nathan Hoad wrote:
>>>>
>>>> I take it you are referring to the 2.0g resident size?
>>> That is what I'm referring to, yes - the resident size has increased
>>> to 2.5g since my previous mail, virtual to 2.6g.
>>>
>>>> 1GB is within the reasonable use limit for a fully loaded Squid under
>>>> peak
>>>> traffic. The resident size reported is usually the biggest "ever" size of
>>>> memory usage by the process.
>>>>
>>>> FWIW: The memory report shows about 324MB being tracked by Squid as
>>>> currently in use for other things than cache_mem with 550 active clients
>>>> doing 117 transactions at present. The client transaction related pools
>>>> show
>>>> that the current values are 1/3 of peak traffic, so 3x 360MB ==> ~1GB
>>>> under
>>>> peak traffic appears entirely possible for your Squid.
>>> Out of interest, how did you come to the 324MB? I'd be interested in
>>> learning how to read the output a bit better :)
>>
>> Okay. (for anyone reading the report is a TSV format [open in Libre Calc or
>> Excel as Tab-separated columns]).
>>
>> The final row of the big table is Totals of all rows above. I took the
>> 1360314 from Allocated section "(KB)" column [~1360 MB] and subtracted 1GB /
>> 1024MB. That is the current Total memory usage Squid is aware of either in
>> active use or waiting re-allocation, minus what you said cache_mem was
>> configured to.
>>
>> NP: Before subtracting I did a quick check of the mem_node (ie cache_mem
>> memory 'pages') there is ~1025 MB. Enough for the full 1024 MB cache_mem and
>> some extra MB of items in-transit right now that are not cacheable - which
>> use mem_nodes as well.
>>
>>
>> Using a bit of developer inside-knowledge I identify in the Allocated
>> section "(#)" column the main state objects which are allocated one for each
>> client connection:
>> * "cbdata ConnStateData" shows 2749 - at 1 per currently open client TCP
>> connection.
>> * "cbdata ClientHttpRequest " shows 550 - at 1 each per client HTTP request.
>> ++ sorry I got that wrong earlier myself.
>> * "cbdata ClientReplyContext" also shows 550 - at 1 each per currently
>> underway client HTTP responses.
>>
>> These objects also give me the details to estimate current versus peak
>> traffic memory requirements. For example:
>> cbdata ClientReplyContext shows 550 current allocated, using 35269 KB, with
>> a highest-ever allocation of 100933 KB - roughy 3x the current memory usage.
>>
>>
>> The difference between Allocated "(KB)" column and "high (KB)" columns shows
>> us how much is allocated now versus the highest ever allocated. A leak
>> usually shows up as both those columns being nearly identical values,
>> although it is possible that 1:10 objects leaks or something weird like that
>> which can hide it.
>>
>> I may have to change my reading a bit. Thinking about that column meaning
>> now I see the Total value for allocations "high (KB)" only shows 1462831 KB
>> - its not very accurate, but does show that if all objects were reaching
>> their max at the same time it would still be ~500MB short of that 2GB the
>> system reports. Another option that usually adds fuzz to these numbers is
>> spawning of helpers - the fork() used for that allocates the child process a
>> whole duplicate of the current memory space usage of the parent process.
>> Effectively doubling the OS-reported memory values from whatever the reality
>> is.
>>
>>
>> Like Alex said, leaks show up as an ever-increasing value in these numbers
>> somewhere. Regular snapshots of that report and the system values taken
>> across a week or two should be able to show if there is anything constantly
>> growing at a regular rate.
>>
>> Amos
Received on Fri May 31 2013 - 06:46:29 MDT