RE: [squid-users] Oom-killer and Squid

Denys,
Thanks for the reply, the system is 64 bit. Right now, "ps axv" shows
squid as the largest memory user at almost 600MB and growing at about
2MB/min (makes sense with the 3 week crash time). Memory leak maybe?
Dave

-----Original Message-----
From: Denys [mailto:nuclearcat@nuclearcat.com]
Sent: Tuesday, March 06, 2007 3:32 PM
To: squid-users@squid-cache.org
Subject: Re: [squid-users] Oom-killer and Squid

Is it 32-bit system?
If yes, i guess with PAE process cannot take more than 2GB of memory.
But i
think i am wrong (it is 64-bit?).
try to do sometimes "ps axv", and see how much ram eated by processes

On Tue, 6 Mar 2007 14:47:02 -0500, Dave Rhodes wrote
> Hello All:
>
> I am running Squid 2.6 Stable5 on an HP DL390 w/6GB RAM, 60GB cache,
> 2GB swap w/SuSE 10.1. As a rule, thanks to some help from Henrik,
> everything runs well. Twice now though, I've had oom-killer jump in
> and kill Squid and only Squid. I am running a very small test group
> of about 30 users so it takes awhile (about 3 weeks) to run out of
memory.
>
> After the last crash, I put in memory monitors to see what the status
> was just before and just after the crash. I also added Webmin
> monitors to let me know when the process stopped. Just before this
> particular crash, the free memory was only about 15MB but that is not
> unusual with Linux and I had watched it on several occasions jump from

> that number to almost 100MB. Apparently, it ran out of buffers and
> oom- killer shut the Squid process down. After restarting the Squid
> process a check of the free memory showed 5.4GB.
>
> If anyone can tell me why this happens, I would certainly appreciate
> it.
>
> Below is the system log output detailing the oom-killer sequence.
> The access.log and cache.log show no problems at all. Thanks, Dave
>
> System Log:
>
> Mar 6 11:01:10 Squid1 kernel: oom-killer: gfp_mask=0x201d2, order=0
> Mar 6 11:01:10 Squid1 kernel:
> Mar 6 11:01:10 Squid1 kernel: Call Trace:
> <ffffffff8015af7b>{out_of_memory+93}
> <ffffffff8015cde9>{__alloc_pages+552}
> Mar 6 11:01:10 Squid1 kernel:
> <ffffffff8015e617>{__do_page_cache_readahe
> ad+149} <ffffffff802cfa4a>{__wait_on_bit_lock+91}
> Mar 6 11:01:10 Squid1 kernel: <ffffffff80157f94>{__lock_page+114}
> <fffff
> fff8015a59b>{filemap_nopage+323}
> Mar 6 11:01:10 Squid1 kernel:
> <ffffffff80164cc1>{__handle_mm_fault+911}
> <ffffffff802d235e>{do_page_fault+965}
> Mar 6 11:01:10 Squid1 kernel:
> <ffffffff801291aa>{default_wake_function+0
> } <ffffffff8010de94>{do_gettimeofday+80}
>
> Mar 6 11:01:10 Squid1 kernel: <ffffffff8010b451>{error_exit+0}
> Mar 6 11:01:10 Squid1 kernel: Mem-info:
> Mar 6 11:01:10 Squid1 kernel: Node 0 DMA per-cpu:
> Mar 6 11:01:10 Squid1 kernel: cpu 0 hot: high 0, batch 1 used:0 Mar
> 6
> 11:01:10 Squid1 kernel: cpu 0 cold: high 0, batch 1 used:0 Mar 6
> 11:01:10 Squid1 kernel: cpu 1 hot: high 0, batch 1 used:0
> Mar 6 11:01:11 Squid1 squid[5079]: Squid Parent: child process 5081
> exited due to signal 9
> Mar 6 11:01:12 Squid1 kernel: cpu 1 cold: high 0, batch 1 used:0
> Mar 6 11:01:15 Squid1 kernel: Node 0 DMA32 per-cpu:
> Mar 6 11:01:16 Squid1 kernel: cpu 0 hot: high 186, batch 31 used:161
> Mar 6 11:01:17 Squid1 kernel: cpu 0 cold: high 62, batch 15 used:14
> Mar 6 11:01:17 Squid1 kernel: cpu 1 hot: high 186, batch 31 used:158
> Mar 6 11:01:17 Squid1 kernel: cpu 1 cold: high 62, batch 15 used:61
> Mar 6 11:01:17 Squid1 kernel: Node 0 Normal per-cpu:
> Mar 6 11:01:17 Squid1 kernel: cpu 0 hot: high 186, batch 31 used:177
> Mar 6 11:01:17 Squid1 kernel: cpu 0 cold: high 62, batch 15 used:15
> Mar 6 11:01:17 Squid1 kernel: cpu 1 hot: high 186, batch 31 used:156
> Mar 6 11:01:17 Squid1 kernel: cpu 1 cold: high 62, batch 15 used:11
> Mar 6 11:01:17 Squid1 kernel: Node 0 HighMem per-cpu: empty
> Mar 6 11:01:17 Squid1 kernel: Free pages: 32040kB (0kB HighMem)
> Mar 6 11:01:17 Squid1 kernel: Active:1046725 inactive:437196
> dirty:0 writeback: 0
unstable:0
> free:8010 slab:5467 mapped:1480421 pagetables:5102
>
> Mar 6 11:01:17 Squid1 kernel: Node 0 DMA free:12388kB min:16kB
low:20kB
> high:24 kB active:0kB
> inactive:0kB present:12032kB pages_scanned:2868 all_unreclaimable? yes
> Mar 6 11:01:17 Squid1 kernel: lowmem_reserve[]: 0 3512 5974 5974
> Mar 6 11:01:17 Squid1 kernel: Node 0 DMA32 free:15616kB min:5808kB
> low:7260kB h igh:8712kB
> active:2228388kB inactive:1308212kB present:3596460kB pages_scanned:5
> 390781 all_unreclaimable? yes
> Mar 6 11:01:17 Squid1 kernel: lowmem_reserve[]: 0 0 2461 2461
> Mar 6 11:01:17 Squid1 kernel: Node 0 Normal free:4036kB min:4068kB
> low:5084kB h igh:6100kB
> active:1958512kB inactive:440572kB present:2520960kB pages_scanned:98
> 21648 all_unreclaimable? yes
> Mar 6 11:01:17 Squid1 kernel: lowmem_reserve[]: 0 0 0 0
> Mar 6 11:01:17 Squid1 kernel: Node 0 HighMem free:0kB min:128kB
> low:128kB high: 128kB
> active:0kB inactive:0kB present:0kB pages_scanned:0
> all_unreclaimable? no
> Mar 6 11:01:17 Squid1 kernel: lowmem_reserve[]: 0 0 0 0
> Mar 6 11:01:17 Squid1 kernel: Node 0 DMA: 7*4kB 5*8kB 4*16kB 5*32kB
> 3*64kB 3*12 8kB 1*256kB
> 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12388kB
> Mar 6 11:01:17 Squid1 kernel: Node 0 DMA32: 2*4kB 1*8kB 1*16kB
> 3*32kB 0*64kB 1* 128kB
0*256kB
> 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15616kB
>
> Mar 6 11:01:17 Squid1 kernel: Node 0 Normal: 1*4kB 0*8kB 0*16kB
2*32kB
> 0*64kB 1 *128kB 1*256kB
> 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4036kB
> Mar 6 11:01:17 Squid1 kernel: Node 0 HighMem: empty
>
> Mar 6 11:01:17 Squid1 kernel: Swap cache: add 1031992, delete
> 1031737, find 761
> 567/818419, race 0+14
> Mar 6 11:01:17 Squid1 kernel: Free swap = 0kB
> Mar 6 11:01:17 Squid1 kernel: Total swap = 2104476kB
> Mar 6 11:01:17 Squid1 kernel: Free swap: 0kB
> Mar 6 11:01:17 Squid1 kernel: 1687551 pages of RAM
> Mar 6 11:01:17 Squid1 kernel: 174888 reserved pages
> Mar 6 11:01:17 Squid1 kernel: 10669 pages shared
> Mar 6 11:01:17 Squid1 kernel: 255 pages swap cached
> Mar 6 11:01:17 Squid1 kernel: oom-killer: gfp_mask=0x201d2, order=0
> Mar 6 11:01:17 Squid1 kernel:
> Mar 6 11:01:17 Squid1 kernel: Call Trace:
<ffffffff8015af7b>{out_of_memory+93}
> <ffffffff8015cde9>{__alloc_pages+552}
>
> Mar 6 11:01:17 Squid1 kernel:
> <ffffffff8012a186>{__wake_up+56} <ffffffff
8015e617>{__do_page_cache_readahead+149}
> Mar 6 11:01:17 Squid1 kernel: <ffffffff801e97bf>{find_next_bit+85}
> <ffff
> ffff80128243>{find_busiest_group+356}
> Mar 6 11:01:17 Squid1 kernel: <ffffffff8015a59b>{filemap_nopage+323}
> <ff
> ffffff80164cc1>{__handle_mm_fault+911}
>
> Mar 6 11:01:17 Squid1 kernel:
<ffffffff80143d9d>{finish_wait+50}
> <ffffff
> ff802d235e>{do_page_fault+965}
> Mar 6 11:01:17 Squid1 kernel:
> <ffffffff80143d1a>{autoremove_wake_functio
> n+0} <ffffffff8010b451>{error_exit+0}
> Mar 6 11:01:17 Squid1 kernel: Mem-info:
> Mar 6 11:01:17 Squid1 kernel: Node 0 DMA per-cpu:
> Mar 6 11:01:17 Squid1 kernel: cpu 0 hot: high 0, batch 1 used:0
> Mar 6 11:01:17 Squid1 kernel: cpu 0 cold: high 0, batch 1 used:0
> Mar 6 11:01:17 Squid1 kernel: cpu 1 hot: high 0, batch 1 used:0
> Mar 6 11:01:17 Squid1 kernel: cpu 1 cold: high 0, batch 1 used:0
> Mar 6 11:01:18 Squid1 kernel: Node 0 DMA32 per-cpu:
>
> Mar 6 11:01:18 Squid1 kernel: cpu 0 hot: high 186, batch 31 used:161
> Mar 6 11:01:18 Squid1 kernel: cpu 0 cold: high 62, batch 15 used:14
> Mar 6 11:01:18 Squid1 kernel: cpu 1 hot: high 186, batch 31 used:158
> Mar 6 11:01:18 Squid1 kernel: cpu 1 cold: high 62, batch 15 used:61
> Mar 6 11:01:18 Squid1 kernel: Node 0 Normal per-cpu:
> Mar 6 11:01:18 Squid1 kernel: cpu 0 hot: high 186, batch 31 used:177
> Mar 6 11:01:18 Squid1 kernel: cpu 0 cold: high 62, batch 15 used:15
> Mar 6 11:01:18 Squid1 kernel: cpu 1 hot: high 186, batch 31 used:156
> Mar 6 11:01:18 Squid1 kernel: cpu 1 cold: high 62, batch 15 used:11
> Mar 6 11:01:18 Squid1 kernel: Node 0 HighMem per-cpu: empty
> Mar 6 11:01:18 Squid1 kernel: Free pages: 32040kB (0kB HighMem)
> Mar 6 11:01:18 Squid1 kernel: Active:1046467 inactive:437482
> dirty:0 writeback: 0
unstable:0
> free:8010 slab:5467 mapped:1480421 pagetables:5102
>
> Mar 6 11:01:18 Squid1 kernel: Node 0 DMA free:12388kB min:16kB
low:20kB
> high:24 kB active:0kB
> inactive:0kB present:12032kB pages_scanned:2868 all_unreclaimable? yes
> Mar 6 11:01:18 Squid1 kernel: lowmem_reserve[]: 0 3512 5974 5974
> Mar 6 11:01:18 Squid1 kernel: Node 0 DMA32 free:15616kB min:5808kB
> low:7260kB h igh:8712kB
> active:2228060kB inactive:1308540kB present:3596460kB pages_scanned:5
> 440499 all_unreclaimable? yes
> Mar 6 11:01:18 Squid1 kernel: lowmem_reserve[]: 0 0 2461 2461
> Mar 6 11:01:18 Squid1 kernel: Node 0 Normal free:4036kB min:4068kB
> low:5084kB h igh:6100kB
> active:1957808kB inactive:441276kB present:2520960kB pages_scanned:98
> 21962 all_unreclaimable? yes
> Mar 6 11:01:18 Squid1 kernel: lowmem_reserve[]: 0 0 0 0
> Mar 6 11:01:18 Squid1 kernel: Node 0 HighMem free:0kB min:128kB
> low:128kB high: 128kB
> active:0kB inactive:0kB present:0kB pages_scanned:0
> all_unreclaimable? no
> Mar 6 11:01:18 Squid1 kernel: lowmem_reserve[]: 0 0 0 0
> Mar 6 11:01:18 Squid1 kernel: Node 0 DMA: 7*4kB 5*8kB 4*16kB 5*32kB
> 3*64kB 3*12 8kB 1*256kB
> 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12388kB
> Mar 6 11:01:18 Squid1 kernel: Node 0 DMA32: 2*4kB 1*8kB 1*16kB
> 3*32kB 0*64kB 1* 128kB
0*256kB
> 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15616kB
>
> Mar 6 11:01:18 Squid1 kernel: Node 0 Normal: 1*4kB 0*8kB 0*16kB
2*32kB
> 0*64kB 1 *128kB 1*256kB
> 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4036kB
> Mar 6 11:01:18 Squid1 kernel: Node 0 HighMem: empty
>
> Mar 6 11:01:18 Squid1 kernel: Swap cache: add 1031992, delete
> 1031737, find 761
> 567/818419, race 0+14
> Mar 6 11:01:18 Squid1 kernel: Free swap = 0kB
> Mar 6 11:01:18 Squid1 kernel: Total swap = 2104476kB
> Mar 6 11:01:18 Squid1 kernel: Free swap: 0kB
> Mar 6 11:01:18 Squid1 kernel: oom-killer: gfp_mask=0x201d2, order=0
> Mar 6 11:01:18 Squid1 kernel:
> Mar 6 11:01:18 Squid1 kernel: Call Trace:
<ffffffff8015af7b>{out_of_memory+93}
> <ffffffff8015cde9>{__alloc_pages+552}
> Mar 6 11:01:18 Squid1 kernel:
> <ffffffff8015e617>{__do_page_cache_readahe
> ad+149} <ffffffff802ceec0>{thread_return+0}
> Mar 6 11:01:18 Squid1 kernel: <ffffffff8015a59b>{filemap_nopage+323}
> <ff
> ffffff80164cc1>{__handle_mm_fault+911}
> Mar 6 11:01:18 Squid1 kernel: <ffffffff802d235e>{do_page_fault+965}
> <fff
> fffff8018cb56>{__pollwait+0}
>
> Mar 6 11:01:18 Squid1 kernel:
<ffffffff8018cb23>{sys_select+795}
> <ffffff
> ff8010b451>{error_exit+0}
> Mar 6 11:01:18 Squid1 kernel: Free swap = 0kB
> Mar 6 11:01:18 Squid1 kernel: Total swap = 2104476kB
> Mar 6 11:01:18 Squid1 kernel: Free swap: 0kB
> Mar 6 11:01:18 Squid1 kernel: 1687551 pages of RAM
> Mar 6 11:01:18 Squid1 kernel: 174888 reserved pages
> Mar 6 11:01:18 Squid1 kernel: 10675 pages shared
> Mar 6 11:01:18 Squid1 kernel: 255 pages swap cached
>
> Mar 6 11:01:18 Squid1 kernel: Out of Memory: Kill process 5081
> (squid) score 29 271 and
> children.
> Mar 6 11:01:18 Squid1 kernel: Out of memory: Killed process 5082
> (unlinkd).

--
Virtual ISP S.A.L.