Again, a performance problem on Digital Unix (OSF1)

                                
After a long time of satisfying work, our cache has now started to stop
again.

<h1>The symptoms</h1>

We currently see 30 to 60 seconds of response time for retrieving a small
HTML page already in the cache (measured by echoping or mconnect and
logged in access.log). TCP connections to the cache manager are also
incredibly slow.

This occurs only when the system is loaded. During the week-end,
everything is fine.

The system worked fine a few days ago. We added three new cache children
and performance suddenly dropped.

<h1>The configuration</h1>

An AlphaServer 1000A 4/266 with a StorageWorks disk controller, driving
14 gigabytes of Raid 0 disks (USF, not ADVfs). It has 320 megabytes of
memory. It runs Digital Unix 4.0.

The software is Squid 1.1.8, linked with GNU-malloc, compiled with gcc.
The machine does only Web caching.

<h1>The details</h1>

Nothing special is logged in cache.log. Here are the outputs of the cache
manager and the configuration file :

info:

{Squid Object Cache: Version 1.1.8}
{Start Time: Fri, 07 Mar 1997 17:22:51 GMT}
{Current Time: Mon, 10 Mar 1997 13:08:21 GMT}
{Connection information for squid:}
{ Number of TCP connections: 830436}
{ Number of UDP connections: 920782}
{ Connections per hour: 25845.1}
{Cache information for squid:}
{ Storage Swap size: 12951 MB}
{ Storage Mem size: 155053 KB}
{ Storage LRU Expiration Age: 0.49 days}
{Resource usage for squid:}
{ CPU Time: 19390 seconds (7021 user 12369 sys)}
{ CPU Usage: 8%}
{ Maximum Resident Size: 239096 KB}
{ Page faults with physical i/o: 1112867}
{File descriptor usage for squid:}
{ Max number of file desc available: 4096}
{ Largest file desc currently in use: 2348}
{ Available number of file descriptors: 1873}
{ Reserved number of file descriptors: 100}
{Internal Data Structures:}
{ 772454 StoreEntries}
{ 1433 StoreEntries with MemObjects}
{ 1433 StoreEntries with MemObject Data}
{ 263 Hot Object Cache Items}
{Accounted Memory Usage:}
{ StoreEntry 772454 x 72 bytes = 54313 KB}
{ URL strings = 37141 KB}
{ IPCacheEntry 920 x 64 bytes = 57 KB}
{ FQDNCacheEntry 0 x 104 bytes = 0 KB}
{ Hash link 263 x 24 bytes = 6 KB}
{ Pool MemObject structures 1433 x 144 bytes = 201 KB (
0 free)}
{ Pool for Request structur 1334 x 4424 bytes = 5763 KB (
0 free)}
{ Pool for in-memory object 39323 x 4096 bytes = 157292 KB (
0 free)}
{ Pool for disk I/O 494 x 8192 bytes = 3952 KB (
0 free)}
{ NetDB Address Entries 0 x 88 bytes = 0 KB}
{ NetDB Host Entries 0 x 16 bytes = 0 KB}
{ ClientDB Entries 43 x 304 bytes = 12 KB}
{ Miscellaneous = 8899 KB}
{ Total Accounted = 267639 KB}
}

stats/dns :

{dnsserver requests: 59776}
{dnsserver replies: 59757}
{number of dnsservers: 32}
{dnsservers use histogram:}
{ dnsserver #1: 39524}
{ dnsserver #2: 10752}
{ dnsserver #3: 3796}
{ dnsserver #4: 1701}
{ dnsserver #5: 1061}
{ dnsserver #6: 728}
{ dnsserver #7: 515}
{ dnsserver #8: 389}
{ dnsserver #9: 288}
{ dnsserver #10: 230}
{ dnsserver #11: 186}
{ dnsserver #12: 141}
{ dnsserver #13: 110}
{ dnsserver #14: 87}
{ dnsserver #15: 65}
{ dnsserver #16: 51}
{ dnsserver #17: 36}
{ dnsserver #18: 30}
{ dnsserver #19: 23}
{ dnsserver #20: 17}
{ dnsserver #21: 11}
{ dnsserver #22: 9}
{ dnsserver #23: 8}
{ dnsserver #24: 6}
{ dnsserver #25: 4}
{ dnsserver #26: 3}
{ dnsserver #27: 2}
{ dnsserver #28: 2}
{ dnsserver #29: 72}
{ dnsserver #30: 71}
{ dnsserver #31: 68}
{ dnsserver #32: 68}
}

}

parameter :

{VM-Max 40 "# Maximum hot-vm cache (MB)"}
{VM-High 90 "# High water mark hot-vm cache (%)"}
{VM-Low 75 "# Low water mark hot-vm cache (%)"}
{Swap-Max 14000 "# Maximum disk cache (MB)"}
{Swap-High 95 "# High water mark disk cache (%)"}
{Swap-Low 90 "# Low water mark disk cache (%)"}
{Neg-TTL 300 "# TTL for negative cache (s)"}
{ReadTimeout 1800 "# Maximum idle connection (s)"}
{ClientLifetime 24000 "# Lifetime for incoming HTTP requests"}
{CleanRate 31536000 "# Rate for periodic object expiring"}
{HttpAccelMode 0 "# Is operating as an HTTP accelerator"}
}

filedescriptors:

2300 file descriptors.

Here is the running Squid. Memory leak is obvious since it never went
over 250 megabytes before and it is still increasing:

cache 7732 3.8 65.9 368M 210M ?? R Mar 07 05:24:10
/usr/local/squid/bin/squid

Extract from squid.conf. Details available of course:

cache_mem 40
cache_swap 14000
cache_swap_low 90
cache_swap_high 95
cache_mem_low 75
cache_mem_high 90
dns_children 32
request_size 100
swap_level1_dirs 32
swap_level2_dirs 256

Values edited in /etc/sysconfigtab:

vm:
        ubc-maxpercent = 80

vfs:
        name-cache-size = 3499

inet:
        somaxconn = 32767
        sominconn = 32767
        tcbhashsize = 1024

proc:
        max-proc-per-user = 256
Received on Mon Mar 10 1997 - 06:01:56 MST