Re: [squid-users] top reports twice memory as much as Total in mgr:mem

Amos,

The cache size is configured around 128GB, as reported by mgr:config:
cache_dir aufs /export/squid/cache 131072 32 256 IOEngine=DiskThreads
cache_swap_low 90
cache_swap_high 95
cache_mem -1073741824 bytes
memory_replacement_policy lru
cache_replacement_policy lru
memory_pools on
memory_pools_limit 5242880 bytes

In a VIRT=11.5g RES=11g instance, mgr:info reports something like:
        Process Data Segment Size via sbrk(): 1020120 KB
        Maximum Resident Size: 47922032 KB
        Storage Swap size: 34233780 KB
        Storage Swap capacity: 25.5% used, 74.5% free
        Storage Mem size: 3375668 KB
        Storage Mem capacity: 46.4% used, 53.6% free
        memPoolAlloc calls: 21345922346
        memPoolFree calls: 21401173704
    Internal Data Structures:
        451308 StoreEntries
         48956 StoreEntries with MemObjects
         48927 Hot Object Cache Items
        451254 on-disk objects

I guess it is not a mem_pools_limit issue, since memory_pools_limit is
the default 5MB, which means the malloc library will handle memory
above 5MB. Not sure whether the malloc library returns freed memory to
OS or not (guess not).

So the problem might be a rush of requests bring the host to its
knees, and it won't have a chance to stand on its feet again before
the squid restarts? Lots of swap in/out and few memory for filesystem
cache when squid holds above 90 percent of system. The memory
starvation simply exacerbates system load.

Any solution?

Thanks,
kc

2010/10/19 Amos Jeffries <squid3_at_treenet.co.nz>:
> On 19/10/10 22:19, Kaiwang Chen wrote:
>>
>> Hello,
>>
>> In a 3.1.6 squid installation on a CentOS 5.5 x86 host with customized
>> kernel, top reports
>>
>> VERT 19.8g, and RES 14g
>>
>> while mgr:mem reports
>> Total Allocated 7.25g (96.4% of which is mem_node) high 7.27g,  In Use
>> 7.25g %alloc 99.957
>>
>> In squid.conf, the following is explicitly configured(mem_pool is
>> default):
>> cache_mem 7168 MB
>> maximum_object_size 51200 KB
>> maximum_object_size_in_memory 1024 KB
>>
>> Probably memory leak.  However, it is not reliably reproducible.
>> Anyway, a restart will return memory to OS.
>
> see question 4 of the memory FAQ:
> http://wiki.squid-cache.org/SquidFaq/SquidMemory
>
>>
>> Additionally, mgr reports negative values:
>>
>
> Bit of 32-bit signed wrap in the display. Annoying but nothing to worry
> overly much about. It's been fixed in later releases.
>
> Amos
> --
> Please be using
>  Current Stable Squid 2.7.STABLE9 or 3.1.8
>  Beta testers wanted for 3.2.0.2
>
Received on Thu Oct 21 2010 - 15:07:44 MDT