RE: [squid-users] squid crashes rapidly with glibc errors

Hi Henrik!

Two months ago you suggested me to run squid through the valgrind (sorry for
not replying). Could you please have a look at the result and advice the
direction for further investigation? The valgrind output is below.

Thank you,
Anton Golubev
ENGECON

[root@ ~]# valgrind -v --show-reachable=yes squid
==16329== Memcheck, a memory error detector.
==16329== Copyright (C) 2002-2005, and GNU GPL'd, by Julian Seward et al.
==16329== Using LibVEX rev 1575, a library for dynamic binary translation.
==16329== Copyright (C) 2004-2005, and GNU GPL'd, by OpenWorks LLP.
==16329== Using valgrind-3.1.1, a dynamic binary instrumentation framework.
==16329== Copyright (C) 2000-2005, and GNU GPL'd, by Julian Seward et al.
==16329==
--16329-- Command line
--16329-- squid
--16329-- Startup, with flags:
--16329-- -v
--16329-- --show-reachable=yes
--16329-- Contents of /proc/version:
--16329-- Linux version 2.6.9-42.EL (buildcentos@build-i386) (gcc version
3.4.6 20060404 (Red Hat 3.4.6-3)) #1 Sat Aug 12 09:17:58 CDT 2006
--16329-- Arch and subarch: X86, x86-sse2
--16329-- Valgrind library directory: /usr/lib/valgrind
--16329-- Reading syms from /lib/ld-2.3.4.so (0x4000000)
--16329-- Reading syms from /usr/local/sbin/squid (0x8048000)
--16329-- Reading syms from /usr/lib/valgrind/x86-linux/memcheck
(0xB0000000)
--16329-- object doesn't have a dynamic symbol table
--16329-- Reading suppressions file: /usr/lib/valgrind/default.supp
--16329-- REDIR: 0x4012450 (index) redirected to 0xB001BC86
(vgPlain_x86_linux_REDIR_FOR_index)
--16329-- Reading syms from /usr/lib/valgrind/x86-linux/vgpreload_core.so
(0x4018000)
--16329-- Reading syms from
/usr/lib/valgrind/x86-linux/vgpreload_memcheck.so
(0x401B000)
--16329-- REDIR: 0x40125F0 (strlen) redirected to 0x401DE90 (strlen)
--16329-- Reading syms from /lib/libcrypt-2.3.4.so (0x402A000)
--16329-- Reading syms from /lib/tls/libpthread-2.3.4.so (0x4058000)
--16329-- Reading syms from /lib/tls/libm-2.3.4.so (0x406A000)
--16329-- Reading syms from /lib/libnsl-2.3.4.so (0x408D000)
--16329-- Reading syms from /lib/tls/libc-2.3.4.so (0x40A3000)
--16329-- REDIR: 0x40007A0 (_dl_sysinfo_int80) redirected to 0xB001BC83
(???)
--16329-- REDIR: 0x410D870 (memset) redirected to 0x401E588 (memset)
--16329-- REDIR: 0x410DD90 (memcpy) redirected to 0x401E1A8 (memcpy)
--16329-- REDIR: 0x410CA10 (rindex) redirected to 0x401DB9C (rindex)
--16329-- REDIR: 0x4108910 (mallopt) redirected to 0x401D8AA (mallopt)
--16329-- REDIR: 0x410E5F0 (rawmemchr) redirected to 0x401E618 (rawmemchr)
--16329-- REDIR: 0x410C630 (strlen) redirected to 0x401DE74 (strlen)
--16329-- REDIR: 0x4107660 (malloc) redirected to 0x401C386 (malloc)
--16329-- REDIR: 0x4107380 (calloc) redirected to 0x401D639 (calloc)
--16329-- REDIR: 0x4107CA0 (realloc) redirected to 0x401D6E4 (realloc)
--16329-- REDIR: 0x410C0D0 (strcmp) redirected to 0x401E114 (strcmp)
--16329-- REDIR: 0x410D370 (memchr) redirected to 0x401E184 (memchr)
--16329-- REDIR: 0x410BF60 (index) redirected to 0x401DC4C (index)
--16329-- REDIR: 0x410C140 (strcpy) redirected to 0x401DEC8 (strcpy)
--16329-- REDIR: 0x41055B0 (free) redirected to 0x401CE7D (free)
--16329-- REDIR: 0x410C830 (strncmp) redirected to 0x401E0C4 (strncmp)
--16329-- REDIR: 0x410E6C0 (strchrnul) redirected to 0x401E5FC (strchrnul)
--16329-- REDIR: 0x410DA60 (stpcpy) redirected to 0x401E348 (stpcpy)
--16329-- Reading syms from /lib/libnss_files-2.3.4.so (0x42DB000)
--16329-- Reading syms from /lib/libnss_dns-2.3.4.so (0x4021000)
--16329-- Reading syms from /lib/libresolv-2.3.4.so (0x42E6000)
--16329-- REDIR: 0x410C950 (strncpy) redirected to 0x401DF84 (strncpy)
--16329-- REDIR: 0x410C6E0 (strnlen) redirected to 0x401DE50 (strnlen)
--16329-- discard syms at 0x42DB000-0x42E6000 in /lib/libnss_files-2.3.4.so
due to munmap()
--16329-- discard syms at 0x4021000-0x4027000 in /lib/libnss_dns-2.3.4.so
due
to munmap()
--16329-- discard syms at 0x42E6000-0x42F9000 in /lib/libresolv-2.3.4.so due

to munmap()
==16329==
==16329== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 27 from 2)
--16329--
--16329-- supp: 21 Ubuntu-stripped-ld.so
--16329-- supp: 6 dl_relocate_object
==16329== malloc/free: in use at exit: 717,860 bytes in 1,271 blocks.
==16329== malloc/free: 1,681 allocs, 410 frees, 748,566 bytes allocated.
==16329==
==16329== searching for pointers to 1,271 not-freed blocks.
==16329== checked 1,890,128 bytes.
==16329==
==16329== LEAK SUMMARY:
==16329== definitely lost: 0 bytes in 0 blocks.
==16329== possibly lost: 3,192 bytes in 95 blocks.
==16329== still reachable: 714,668 bytes in 1,176 blocks.
==16329== suppressed: 0 bytes in 0 blocks.
--16329-- memcheck: sanity checks: 18 cheap, 1 expensive
--16329-- memcheck: auxmaps: 0 auxmap entries (0k, 0M) in use
--16329-- memcheck: auxmaps: 0 searches, 0 comparisons
--16329-- memcheck: secondaries: 30 issued (1920k, 1M)
--16329-- memcheck: secondaries: 54 accessible and distinguished (3456k,
3M)
--16329-- tt/tc: 20,739 tt lookups requiring 21,882 probes
--16329-- tt/tc: 20,739 fast-cache updates, 6 flushes
--16329-- translate: new 7,725 (157,693 -> 2,835,558; ratio 179:10)
[0
scs]
--16329-- translate: dumped 0 (0 -> ??)
--16329-- translate: discarded 607 (12,936 -> ??)
--16329-- scheduler: 932,092 jumps (bb entries).
--16329-- scheduler: 18/15,346 major/minor sched events.
--16329-- sanity: 19 cheap, 1 expensive checks.
--16329-- exectx: 30,011 lists, 428 contexts (avg 0 per list)
--16329-- exectx: 2,118 searches, 1,692 full compares (798 per 1000)
--16329-- exectx: 0 cmp2, 80 cmp4, 0 cmpAll
--16331-- discard syms at 0x42DB000-0x42E6000 in /lib/libnss_files-2.3.4.so
due to munmap()
--16331-- discard syms at 0x4021000-0x4027000 in /lib/libnss_dns-2.3.4.so
due
to munmap()
--16331-- discard syms at 0x42E6000-0x42F9000 in /lib/libresolv-2.3.4.so due

to munmap()
==16331==
==16331== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 27 from 2)
--16331--
--16331-- supp: 21 Ubuntu-stripped-ld.so
--16331-- supp: 6 dl_relocate_object
==16331== malloc/free: in use at exit: 717,860 bytes in 1,271 blocks.
==16331== malloc/free: 1,681 allocs, 410 frees, 748,566 bytes allocated.
==16331==
==16331== searching for pointers to 1,271 not-freed blocks.
==16331== checked 1,890,124 bytes.
==16331==
==16331== LEAK SUMMARY:
==16331== definitely lost: 0 bytes in 0 blocks.
==16331== possibly lost: 3,192 bytes in 95 blocks.
==16331== still reachable: 714,668 bytes in 1,176 blocks.
==16331== suppressed: 0 bytes in 0 blocks.
--16331-- memcheck: sanity checks: 18 cheap, 1 expensive
--16331-- memcheck: auxmaps: 0 auxmap entries (0k, 0M) in use
--16331-- memcheck: auxmaps: 0 searches, 0 comparisons
--16331-- memcheck: secondaries: 30 issued (1920k, 1M)
--16331-- memcheck: secondaries: 54 accessible and distinguished (3456k,
3M)
--16331-- tt/tc: 20,709 tt lookups requiring 21,859 probes
--16331-- tt/tc: 20,709 fast-cache updates, 6 flushes
--16331-- translate: new 7,669 (156,655 -> 2,821,259; ratio 180:10)
[0
scs]
--16331-- translate: dumped 0 (0 -> ??)
--16331-- translate: discarded 607 (12,936 -> ??)
--16331-- scheduler: 931,529 jumps (bb entries).
--16331-- scheduler: 18/15,381 major/minor sched events.
--16331-- sanity: 19 cheap, 1 expensive checks.
--16331-- exectx: 30,011 lists, 428 contexts (avg 0 per list)
--16331-- exectx: 2,118 searches, 1,692 full compares (798 per 1000)
--16331-- exectx: 0 cmp2, 80 cmp4, 0 cmpAll
--16342-- discard syms at 0x42DB000-0x42E6000 in /lib/libnss_files-2.3.4.so
due to munmap()
--16342-- discard syms at 0x4021000-0x4027000 in /lib/libnss_dns-2.3.4.so
due
to munmap()
--16342-- discard syms at 0x42E6000-0x42F9000 in /lib/libresolv-2.3.4.so due

to munmap()
==16342==
==16342== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 27 from 2)
--16342--
--16342-- supp: 21 Ubuntu-stripped-ld.so
--16342-- supp: 6 dl_relocate_object
==16342== malloc/free: in use at exit: 717,860 bytes in 1,271 blocks.
==16342== malloc/free: 1,703 allocs, 432 frees, 767,174 bytes allocated.
==16342==
==16342== searching for pointers to 1,271 not-freed blocks.
==16342== checked 1,890,140 bytes.
==16342==
==16342== LEAK SUMMARY:
==16342== definitely lost: 0 bytes in 0 blocks.
==16342== possibly lost: 3,192 bytes in 95 blocks.
==16342== still reachable: 714,668 bytes in 1,176 blocks.
==16342== suppressed: 0 bytes in 0 blocks.
--16342-- memcheck: sanity checks: 18 cheap, 1 expensive
--16342-- memcheck: auxmaps: 0 auxmap entries (0k, 0M) in use
--16342-- memcheck: auxmaps: 0 searches, 0 comparisons
--16342-- memcheck: secondaries: 31 issued (1984k, 1M)
--16342-- memcheck: secondaries: 54 accessible and distinguished (3456k,
3M)
--16342-- tt/tc: 21,587 tt lookups requiring 22,915 probes
--16342-- tt/tc: 21,587 fast-cache updates, 6 flushes
--16342-- translate: new 8,097 (166,847 -> 2,981,031; ratio 178:10)
[0
scs]
--16342-- translate: dumped 0 (0 -> ??)
--16342-- translate: discarded 607 (12,936 -> ??)
--16342-- scheduler: 938,637 jumps (bb entries).
--16342-- scheduler: 18/15,905 major/minor sched events.
--16342-- sanity: 19 cheap, 1 expensive checks.
--16342-- exectx: 30,011 lists, 457 contexts (avg 0 per list)
--16342-- exectx: 2,160 searches, 1,705 full compares (789 per 1000)
--16342-- exectx: 0 cmp2, 80 cmp4, 0 cmpAll
--16354-- discard syms at 0x42DB000-0x42E6000 in /lib/libnss_files-2.3.4.so
due to munmap()
--16354-- discard syms at 0x4021000-0x4027000 in /lib/libnss_dns-2.3.4.so
due
to munmap()
--16354-- discard syms at 0x42E6000-0x42F9000 in /lib/libresolv-2.3.4.so due

to munmap()
==16354==
==16354== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 27 from 2)
--16354--
--16354-- supp: 21 Ubuntu-stripped-ld.so
--16354-- supp: 6 dl_relocate_object
==16354== malloc/free: in use at exit: 717,860 bytes in 1,271 blocks.
==16354== malloc/free: 1,715 allocs, 444 frees, 784,553 bytes allocated.
==16354==
==16354== searching for pointers to 1,271 not-freed blocks.
==16354== checked 1,890,140 bytes.
==16354==
==16354== LEAK SUMMARY:
==16354== definitely lost: 0 bytes in 0 blocks.
==16354== possibly lost: 3,192 bytes in 95 blocks.
==16354== still reachable: 714,668 bytes in 1,176 blocks.
==16354== suppressed: 0 bytes in 0 blocks.
--16354-- memcheck: sanity checks: 18 cheap, 1 expensive
--16354-- memcheck: auxmaps: 0 auxmap entries (0k, 0M) in use
--16354-- memcheck: auxmaps: 0 searches, 0 comparisons
--16354-- memcheck: secondaries: 31 issued (1984k, 1M)
--16354-- memcheck: secondaries: 54 accessible and distinguished (3456k,
3M)
--16354-- tt/tc: 21,605 tt lookups requiring 22,937 probes
--16354-- tt/tc: 21,605 fast-cache updates, 6 flushes
--16354-- translate: new 8,098 (166,855 -> 2,981,484; ratio 178:10)
[0
scs]
--16354-- translate: dumped 0 (0 -> ??)
--16354-- translate: discarded 607 (12,936 -> ??)
--16354-- scheduler: 942,342 jumps (bb entries).
--16354-- scheduler: 18/15,966 major/minor sched events.
--16354-- sanity: 19 cheap, 1 expensive checks.
--16354-- exectx: 30,011 lists, 457 contexts (avg 0 per list)
--16354-- exectx: 2,182 searches, 1,727 full compares (791 per 1000)
--16354-- exectx: 0 cmp2, 80 cmp4, 0 cmpAll
--16363-- discard syms at 0x42DB000-0x42E6000 in /lib/libnss_files-2.3.4.so
due to munmap()
--16363-- discard syms at 0x4021000-0x4027000 in /lib/libnss_dns-2.3.4.so
due
to munmap()
--16363-- discard syms at 0x42E6000-0x42F9000 in /lib/libresolv-2.3.4.so due

to munmap()
==16363==
==16363== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 27 from 2)
--16363--
--16363-- supp: 21 Ubuntu-stripped-ld.so
--16363-- supp: 6 dl_relocate_object
==16363== malloc/free: in use at exit: 717,860 bytes in 1,271 blocks.
==16363== malloc/free: 1,727 allocs, 456 frees, 801,932 bytes allocated.
==16363==
==16363== searching for pointers to 1,271 not-freed blocks.
==16363== checked 1,890,140 bytes.
==16363==
==16363== LEAK SUMMARY:
==16363== definitely lost: 0 bytes in 0 blocks.
==16363== possibly lost: 3,192 bytes in 95 blocks.
==16363== still reachable: 714,668 bytes in 1,176 blocks.
==16363== suppressed: 0 bytes in 0 blocks.
--16363-- memcheck: sanity checks: 18 cheap, 1 expensive
--16363-- memcheck: auxmaps: 0 auxmap entries (0k, 0M) in use
--16363-- memcheck: auxmaps: 0 searches, 0 comparisons
--16363-- memcheck: secondaries: 31 issued (1984k, 1M)
--16363-- memcheck: secondaries: 54 accessible and distinguished (3456k,
3M)
--16363-- tt/tc: 21,636 tt lookups requiring 22,971 probes
--16363-- tt/tc: 21,636 fast-cache updates, 6 flushes
--16363-- translate: new 8,105 (167,087 -> 2,985,137; ratio 178:10)
[0
scs]
--16363-- translate: dumped 0 (0 -> ??)
--16363-- translate: discarded 607 (12,936 -> ??)
--16363-- scheduler: 946,056 jumps (bb entries).
--16363-- scheduler: 18/16,034 major/minor sched events.
--16363-- sanity: 19 cheap, 1 expensive checks.
--16363-- exectx: 30,011 lists, 457 contexts (avg 0 per list)
--16363-- exectx: 2,204 searches, 1,749 full compares (793 per 1000)
--16363-- exectx: 0 cmp2, 80 cmp4, 0 cmpAll
--16372-- discard syms at 0x42DB000-0x42E6000 in /lib/libnss_files-2.3.4.so
due to munmap()
--16372-- discard syms at 0x4021000-0x4027000 in /lib/libnss_dns-2.3.4.so
due
to munmap()
--16372-- discard syms at 0x42E6000-0x42F9000 in /lib/libresolv-2.3.4.so due

to munmap()
==16372==
==16372== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 27 from 2)
--16372--
--16372-- supp: 21 Ubuntu-stripped-ld.so
--16372-- supp: 6 dl_relocate_object
==16372== malloc/free: in use at exit: 717,860 bytes in 1,271 blocks.
==16372== malloc/free: 1,739 allocs, 468 frees, 819,311 bytes allocated.
==16372==
==16372== searching for pointers to 1,271 not-freed blocks.
==16372== checked 1,890,140 bytes.
==16372==
==16372== LEAK SUMMARY:
==16372== definitely lost: 0 bytes in 0 blocks.
==16372== possibly lost: 3,192 bytes in 95 blocks.
==16372== still reachable: 714,668 bytes in 1,176 blocks.
==16372== suppressed: 0 bytes in 0 blocks.
--16372-- memcheck: sanity checks: 18 cheap, 1 expensive
--16372-- memcheck: auxmaps: 0 auxmap entries (0k, 0M) in use
--16372-- memcheck: auxmaps: 0 searches, 0 comparisons
--16372-- memcheck: secondaries: 31 issued (1984k, 1M)
--16372-- memcheck: secondaries: 54 accessible and distinguished (3456k,
3M)
--16372-- tt/tc: 21,656 tt lookups requiring 22,995 probes
--16372-- tt/tc: 21,656 fast-cache updates, 6 flushes
--16372-- translate: new 8,107 (167,117 -> 2,985,525; ratio 178:10)
[0
scs]
--16372-- translate: dumped 0 (0 -> ??)
--16372-- translate: discarded 607 (12,936 -> ??)
--16372-- scheduler: 949,762 jumps (bb entries).
--16372-- scheduler: 18/16,096 major/minor sched events.
--16372-- sanity: 19 cheap, 1 expensive checks.
--16372-- exectx: 30,011 lists, 457 contexts (avg 0 per list)
--16372-- exectx: 2,226 searches, 1,771 full compares (795 per 1000)
--16372-- exectx: 0 cmp2, 80 cmp4, 0 cmpAll
--16381-- discard syms at 0x42DB000-0x42E6000 in /lib/libnss_files-2.3.4.so
due to munmap()
--16381-- discard syms at 0x4021000-0x4027000 in /lib/libnss_dns-2.3.4.so
due
to munmap()
--16381-- discard syms at 0x42E6000-0x42F9000 in /lib/libresolv-2.3.4.so due

to munmap()
==16381==
==16381== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 27 from 2)
--16381--
--16381-- supp: 21 Ubuntu-stripped-ld.so
--16381-- supp: 6 dl_relocate_object
==16381== malloc/free: in use at exit: 717,860 bytes in 1,271 blocks.
==16381== malloc/free: 1,751 allocs, 480 frees, 836,690 bytes allocated.
==16381==
==16381== searching for pointers to 1,271 not-freed blocks.
==16381== checked 1,890,140 bytes.
==16381==
==16381== LEAK SUMMARY:
==16381== definitely lost: 0 bytes in 0 blocks.
==16381== possibly lost: 3,192 bytes in 95 blocks.
==16381== still reachable: 714,668 bytes in 1,176 blocks.
==16381== suppressed: 0 bytes in 0 blocks.
--16381-- memcheck: sanity checks: 18 cheap, 1 expensive
--16381-- memcheck: auxmaps: 0 auxmap entries (0k, 0M) in use
--16381-- memcheck: auxmaps: 0 searches, 0 comparisons
--16381-- memcheck: secondaries: 32 issued (2048k, 2M)
--16381-- memcheck: secondaries: 54 accessible and distinguished (3456k,
3M)
--16381-- tt/tc: 21,672 tt lookups requiring 23,013 probes
--16381-- tt/tc: 21,672 fast-cache updates, 6 flushes
--16381-- translate: new 8,107 (167,117 -> 2,985,525; ratio 178:10)
[0
scs]
--16381-- translate: dumped 0 (0 -> ??)
--16381-- translate: discarded 607 (12,936 -> ??)
--16381-- scheduler: 953,468 jumps (bb entries).
--16381-- scheduler: 18/16,156 major/minor sched events.
--16381-- sanity: 19 cheap, 1 expensive checks.
--16381-- exectx: 30,011 lists, 457 contexts (avg 0 per list)
--16381-- exectx: 2,248 searches, 1,793 full compares (797 per 1000)
--16381-- exectx: 0 cmp2, 80 cmp4, 0 cmpAll
--16390-- discard syms at 0x42DB000-0x42E6000 in /lib/libnss_files-2.3.4.so
due to munmap()
--16390-- discard syms at 0x4021000-0x4027000 in /lib/libnss_dns-2.3.4.so
due
to munmap()
--16390-- discard syms at 0x42E6000-0x42F9000 in /lib/libresolv-2.3.4.so due

to munmap()
==16390==
==16390== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 27 from 2)
--16390--
--16390-- supp: 21 Ubuntu-stripped-ld.so
--16390-- supp: 6 dl_relocate_object
==16390== malloc/free: in use at exit: 717,860 bytes in 1,271 blocks.
==16390== malloc/free: 1,763 allocs, 492 frees, 854,069 bytes allocated.
==16390==
==16390== searching for pointers to 1,271 not-freed blocks.
==16390== checked 1,890,140 bytes.
==16390==
==16390== LEAK SUMMARY:
==16390== definitely lost: 0 bytes in 0 blocks.
==16390== possibly lost: 3,192 bytes in 95 blocks.
==16390== still reachable: 714,668 bytes in 1,176 blocks.
==16390== suppressed: 0 bytes in 0 blocks.
--16390-- memcheck: sanity checks: 19 cheap, 1 expensive
--16390-- memcheck: auxmaps: 0 auxmap entries (0k, 0M) in use
--16390-- memcheck: auxmaps: 0 searches, 0 comparisons
--16390-- memcheck: secondaries: 32 issued (2048k, 2M)
--16390-- memcheck: secondaries: 54 accessible and distinguished (3456k,
3M)
--16390-- tt/tc: 21,688 tt lookups requiring 23,031 probes
--16390-- tt/tc: 21,688 fast-cache updates, 6 flushes
--16390-- translate: new 8,107 (167,117 -> 2,985,525; ratio 178:10)
[0
scs]
--16390-- translate: dumped 0 (0 -> ??)
--16390-- translate: discarded 607 (12,936 -> ??)
--16390-- scheduler: 957,174 jumps (bb entries).
--16390-- scheduler: 19/16,217 major/minor sched events.
--16390-- sanity: 20 cheap, 1 expensive checks.
--16390-- exectx: 30,011 lists, 457 contexts (avg 0 per list)
--16390-- exectx: 2,270 searches, 1,815 full compares (799 per 1000)
--16390-- exectx: 0 cmp2, 80 cmp4, 0 cmpAll
--16399-- discard syms at 0x42DB000-0x42E6000 in /lib/libnss_files-2.3.4.so
due to munmap()
--16399-- discard syms at 0x4021000-0x4027000 in /lib/libnss_dns-2.3.4.so
due
to munmap()
--16399-- discard syms at 0x42E6000-0x42F9000 in /lib/libresolv-2.3.4.so due

to munmap()
==16399==
==16399== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 27 from 2)
--16399--
--16399-- supp: 21 Ubuntu-stripped-ld.so
--16399-- supp: 6 dl_relocate_object
==16399== malloc/free: in use at exit: 717,860 bytes in 1,271 blocks.
==16399== malloc/free: 1,775 allocs, 504 frees, 871,448 bytes allocated.
==16399==
==16399== searching for pointers to 1,271 not-freed blocks.
==16399== checked 1,890,140 bytes.
==16399==
==16399== LEAK SUMMARY:
==16399== definitely lost: 0 bytes in 0 blocks.
==16399== possibly lost: 3,192 bytes in 95 blocks.
==16399== still reachable: 714,668 bytes in 1,176 blocks.
==16399== suppressed: 0 bytes in 0 blocks.
--16399-- memcheck: sanity checks: 19 cheap, 1 expensive
--16399-- memcheck: auxmaps: 0 auxmap entries (0k, 0M) in use
--16399-- memcheck: auxmaps: 0 searches, 0 comparisons
--16399-- memcheck: secondaries: 32 issued (2048k, 2M)
--16399-- memcheck: secondaries: 54 accessible and distinguished (3456k,
3M)
--16399-- tt/tc: 21,704 tt lookups requiring 23,049 probes
--16399-- tt/tc: 21,704 fast-cache updates, 6 flushes
--16399-- translate: new 8,107 (167,117 -> 2,985,525; ratio 178:10)
[0
scs]
--16399-- translate: dumped 0 (0 -> ??)
--16399-- translate: discarded 607 (12,936 -> ??)
--16399-- scheduler: 960,879 jumps (bb entries).
--16399-- scheduler: 19/16,277 major/minor sched events.
--16399-- sanity: 20 cheap, 1 expensive checks.
--16399-- exectx: 30,011 lists, 457 contexts (avg 0 per list)
--16399-- exectx: 2,292 searches, 1,837 full compares (801 per 1000)
--16399-- exectx: 0 cmp2, 80 cmp4, 0 cmpAll
--16408-- discard syms at 0x42DB000-0x42E6000 in /lib/libnss_files-2.3.4.so
due to munmap()
--16408-- discard syms at 0x4021000-0x4027000 in /lib/libnss_dns-2.3.4.so
due
to munmap()
--16408-- discard syms at 0x42E6000-0x42F9000 in /lib/libresolv-2.3.4.so due

to munmap()
==16408==
==16408== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 27 from 2)
--16408--
--16408-- supp: 21 Ubuntu-stripped-ld.so
--16408-- supp: 6 dl_relocate_object
==16408== malloc/free: in use at exit: 717,860 bytes in 1,271 blocks.
==16408== malloc/free: 1,787 allocs, 516 frees, 888,827 bytes allocated.
==16408==
==16408== searching for pointers to 1,271 not-freed blocks.
==16408== checked 1,890,140 bytes.
==16408==
==16408== LEAK SUMMARY:
==16408== definitely lost: 0 bytes in 0 blocks.
==16408== possibly lost: 3,192 bytes in 95 blocks.
==16408== still reachable: 714,668 bytes in 1,176 blocks.
==16408== suppressed: 0 bytes in 0 blocks.
--16408-- memcheck: sanity checks: 19 cheap, 1 expensive
--16408-- memcheck: auxmaps: 0 auxmap entries (0k, 0M) in use
--16408-- memcheck: auxmaps: 0 searches, 0 comparisons
--16408-- memcheck: secondaries: 32 issued (2048k, 2M)
--16408-- memcheck: secondaries: 54 accessible and distinguished (3456k,
3M)
--16408-- tt/tc: 21,720 tt lookups requiring 23,067 probes
--16408-- tt/tc: 21,720 fast-cache updates, 6 flushes
--16408-- translate: new 8,107 (167,117 -> 2,985,525; ratio 178:10)
[0
scs]
--16408-- translate: dumped 0 (0 -> ??)
--16408-- translate: discarded 607 (12,936 -> ??)
--16408-- scheduler: 964,585 jumps (bb entries).
--16408-- scheduler: 19/16,337 major/minor sched events.
--16408-- sanity: 20 cheap, 1 expensive checks.
--16408-- exectx: 30,011 lists, 457 contexts (avg 0 per list)
--16408-- exectx: 2,314 searches, 1,859 full compares (803 per 1000)
--16408-- exectx: 0 cmp2, 80 cmp4, 0 cmpAll
--16417-- discard syms at 0x42DB000-0x42E6000 in /lib/libnss_files-2.3.4.so
due to munmap()
--16417-- discard syms at 0x4021000-0x4027000 in /lib/libnss_dns-2.3.4.so
due
to munmap()
--16417-- discard syms at 0x42E6000-0x42F9000 in /lib/libresolv-2.3.4.so due

to munmap()
==16417==
==16417== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 27 from 2)
--16417--
--16417-- supp: 21 Ubuntu-stripped-ld.so
--16417-- supp: 6 dl_relocate_object
==16417== malloc/free: in use at exit: 717,860 bytes in 1,271 blocks.
==16417== malloc/free: 1,799 allocs, 528 frees, 906,206 bytes allocated.
==16417==
==16417== searching for pointers to 1,271 not-freed blocks.
==16417== checked 1,890,140 bytes.
==16417==
==16417== LEAK SUMMARY:
==16417== definitely lost: 0 bytes in 0 blocks.
==16417== possibly lost: 3,192 bytes in 95 blocks.
==16417== still reachable: 714,668 bytes in 1,176 blocks.
==16417== suppressed: 0 bytes in 0 blocks.
--16417-- memcheck: sanity checks: 19 cheap, 1 expensive
--16417-- memcheck: auxmaps: 0 auxmap entries (0k, 0M) in use
--16417-- memcheck: auxmaps: 0 searches, 0 comparisons
--16417-- memcheck: secondaries: 33 issued (2112k, 2M)
--16417-- memcheck: secondaries: 54 accessible and distinguished (3456k,
3M)
--16417-- tt/tc: 21,740 tt lookups requiring 23,091 probes
--16417-- tt/tc: 21,740 fast-cache updates, 6 flushes
--16417-- translate: new 8,107 (167,117 -> 2,985,525; ratio 178:10)
[0
scs]
--16417-- translate: dumped 0 (0 -> ??)
--16417-- translate: discarded 607 (12,936 -> ??)
--16417-- scheduler: 968,309 jumps (bb entries).
--16417-- scheduler: 19/16,401 major/minor sched events.
--16417-- sanity: 20 cheap, 1 expensive checks.
--16417-- exectx: 30,011 lists, 457 contexts (avg 0 per list)
--16417-- exectx: 2,336 searches, 1,881 full compares (805 per 1000)
--16417-- exectx: 0 cmp2, 80 cmp4, 0 cmpAll
--16426-- discard syms at 0x42DB000-0x42E6000 in /lib/libnss_files-2.3.4.so
due to munmap()
--16426-- discard syms at 0x4021000-0x4027000 in /lib/libnss_dns-2.3.4.so
due
to munmap()
--16426-- discard syms at 0x42E6000-0x42F9000 in /lib/libresolv-2.3.4.so due

to munmap()
==16426==
==16426== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 27 from 2)
--16426--
--16426-- supp: 21 Ubuntu-stripped-ld.so
--16426-- supp: 6 dl_relocate_object
==16426== malloc/free: in use at exit: 717,860 bytes in 1,271 blocks.
==16426== malloc/free: 1,811 allocs, 540 frees, 923,585 bytes allocated.
==16426==
==16426== searching for pointers to 1,271 not-freed blocks.
==16426== checked 1,890,140 bytes.
==16426==
==16426== LEAK SUMMARY:
==16426== definitely lost: 0 bytes in 0 blocks.
==16426== possibly lost: 3,192 bytes in 95 blocks.
==16426== still reachable: 714,668 bytes in 1,176 blocks.
==16426== suppressed: 0 bytes in 0 blocks.
--16426-- memcheck: sanity checks: 19 cheap, 1 expensive
--16426-- memcheck: auxmaps: 0 auxmap entries (0k, 0M) in use
--16426-- memcheck: auxmaps: 0 searches, 0 comparisons
--16426-- memcheck: secondaries: 33 issued (2112k, 2M)
--16426-- memcheck: secondaries: 54 accessible and distinguished (3456k,
3M)
--16426-- tt/tc: 21,758 tt lookups requiring 23,112 probes
--16426-- tt/tc: 21,758 fast-cache updates, 6 flushes
--16426-- translate: new 8,107 (167,117 -> 2,985,525; ratio 178:10)
[0
scs]
--16426-- translate: dumped 0 (0 -> ??)
--16426-- translate: discarded 607 (12,936 -> ??)
--16426-- scheduler: 972,024 jumps (bb entries).
--16426-- scheduler: 19/16,463 major/minor sched events.
--16426-- sanity: 20 cheap, 1 expensive checks.
--16426-- exectx: 30,011 lists, 457 contexts (avg 0 per list)
--16426-- exectx: 2,358 searches, 1,903 full compares (807 per 1000)
--16426-- exectx: 0 cmp2, 80 cmp4, 0 cmpAll
--16435-- discard syms at 0x42DB000-0x42E6000 in /lib/libnss_files-2.3.4.so
due to munmap()
--16435-- discard syms at 0x4021000-0x4027000 in /lib/libnss_dns-2.3.4.so
due
to munmap()
--16435-- discard syms at 0x42E6000-0x42F9000 in /lib/libresolv-2.3.4.so due

to munmap()
==16435==
==16435== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 27 from 2)
--16435--
--16435-- supp: 21 Ubuntu-stripped-ld.so
--16435-- supp: 6 dl_relocate_object
==16435== malloc/free: in use at exit: 717,860 bytes in 1,271 blocks.
==16435== malloc/free: 1,823 allocs, 552 frees, 940,964 bytes allocated.
==16435==
==16435== searching for pointers to 1,271 not-freed blocks.
==16435== checked 1,890,140 bytes.
==16435==
==16435== LEAK SUMMARY:
==16435== definitely lost: 0 bytes in 0 blocks.
==16435== possibly lost: 3,192 bytes in 95 blocks.
==16435== still reachable: 714,668 bytes in 1,176 blocks.
==16435== suppressed: 0 bytes in 0 blocks.
--16435-- memcheck: sanity checks: 19 cheap, 1 expensive
--16435-- memcheck: auxmaps: 0 auxmap entries (0k, 0M) in use
--16435-- memcheck: auxmaps: 0 searches, 0 comparisons
--16435-- memcheck: secondaries: 33 issued (2112k, 2M)
--16435-- memcheck: secondaries: 54 accessible and distinguished (3456k,
3M)
--16435-- tt/tc: 21,774 tt lookups requiring 23,130 probes
--16435-- tt/tc: 21,774 fast-cache updates, 6 flushes
--16435-- translate: new 8,107 (167,117 -> 2,985,525; ratio 178:10)
[0
scs]
--16435-- translate: dumped 0 (0 -> ??)
--16435-- translate: discarded 607 (12,936 -> ??)
--16435-- scheduler: 975,730 jumps (bb entries).
--16435-- scheduler: 19/16,523 major/minor sched events.
--16435-- sanity: 20 cheap, 1 expensive checks.
--16435-- exectx: 30,011 lists, 457 contexts (avg 0 per list)
--16435-- exectx: 2,380 searches, 1,925 full compares (808 per 1000)
--16435-- exectx: 0 cmp2, 80 cmp4, 0 cmpAll
--16330-- discard syms at 0x42DB000-0x42E6000 in /lib/libnss_files-2.3.4.so
due to munmap()
--16330-- discard syms at 0x4021000-0x4027000 in /lib/libnss_dns-2.3.4.so
due
to munmap()
--16330-- discard syms at 0x42E6000-0x42F9000 in /lib/libresolv-2.3.4.so due

to munmap()
==16330==
==16330== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 27 from 2)
--16330--
--16330-- supp: 21 Ubuntu-stripped-ld.so
--16330-- supp: 6 dl_relocate_object
==16330== malloc/free: in use at exit: 717,860 bytes in 1,271 blocks.
==16330== malloc/free: 1,841 allocs, 570 frees, 967,026 bytes allocated.
==16330==
==16330== searching for pointers to 1,271 not-freed blocks.
==16330== checked 1,890,160 bytes.
==16330==
==16330== LEAK SUMMARY:
==16330== definitely lost: 0 bytes in 0 blocks.
==16330== possibly lost: 3,192 bytes in 95 blocks.
==16330== still reachable: 714,668 bytes in 1,176 blocks.
==16330== suppressed: 0 bytes in 0 blocks.
--16330-- memcheck: sanity checks: 19 cheap, 1 expensive
--16330-- memcheck: auxmaps: 0 auxmap entries (0k, 0M) in use
--16330-- memcheck: auxmaps: 0 searches, 0 comparisons
--16330-- memcheck: secondaries: 33 issued (2112k, 2M)
--16330-- memcheck: secondaries: 54 accessible and distinguished (3456k,
3M)
--16330-- tt/tc: 21,966 tt lookups requiring 23,344 probes
--16330-- tt/tc: 21,966 fast-cache updates, 6 flushes
--16330-- translate: new 8,234 (169,487 -> 3,024,912; ratio 178:10)
[0
scs]
--16330-- translate: dumped 0 (0 -> ??)
--16330-- translate: discarded 607 (12,936 -> ??)
--16330-- scheduler: 982,674 jumps (bb entries).
--16330-- scheduler: 19/16,642 major/minor sched events.
--16330-- sanity: 20 cheap, 1 expensive checks.
--16330-- exectx: 30,011 lists, 468 contexts (avg 0 per list)
--16330-- exectx: 2,413 searches, 1,947 full compares (806 per 1000)
--16330-- exectx: 0 cmp2, 80 cmp4, 0 cmpAll

-----Original Message-----
From: Henrik Nordstrom [mailto:henrik@henriknordstrom.net]
Sent: Friday, April 27, 2007 5:37 PM
To: Anton Golubev
Cc: squid-users@squid-cache.org
Subject: Re: [squid-users] squid crashes rapidly with glibc errors

fre 2007-04-27 klockan 15:46 +0400 skrev Anton Golubev:
> Hello squid users,
>
> I'm trying to figure out, why usually quite stable squid started to
> crash rapidly on newly installed server with CentOS 4.4. Time to crash
> is up to several seconds, then moderate requests load is applied.
> Relevant log records are following:
>
> 2007/04/25 17:42:54| Validated 10907 Entries
> 2007/04/25 17:42:54| store_swap_size = 107484k
> 2007/04/25 17:42:54| storeLateRelease: released 0 objects
> *** glibc detected *** corrupted double-linked list: 0x0a378818 ***

Ugh.. something stomped on anothers memory.

Can you try if this problem can be trapped in more detail using valgrind?

Install valgrind and valgrind-devel

Rebuild Squid with the --with-valgrind option

Start Squid using valgrind and try making it crash (or at least get ugly
warnings from valgrind). Squid will run a bit slower, so some patience may
be needed.

Regards
Henrik