Re: [squid-users] Squid slows under load

Hi Julian,

The one stat that I can't see here is disk access. I know you said that
you have SSD's, but what is the disk stats for your logging volume and
the squid volume ? If you totally bypass the proxy, does it improve ?
(could be that the squid server is getting shaped ?)

Cheers,

Pieter

On 4/03/2011 06:46, Julian Pilfold-Bagwell wrote:
> Hi All,
>
> I've been having some problems with Squid and Dansguardian for a while
> now and despite lots of time on Google, haven't found a solution.
>
> The problem started a week or so back when I noticed that squid was
> slowing. A quick look through the logs showed it was running out of
> file descriptors so I upped the level to take account. The server was
> ancient so I bought in an HP Proliant DL120 (dual Pentium 2.80Ghz
> G6950 CPU & 4GB of RAM). At the same time, I bought in 2 x 60GB SSD
> drives to use as cache space with the system on a RAID 1 array with
> 160GB SATA II disks.
>
> On this, I installed Ubuntu server 10.04.2 LTS with Squid 2.7 (from
> apt) and Dansguardian 2.10.1.1. The kernel version is 2.6.32-24-server
> and the server authenticates via a Samba PDC (v 3.5.6) using
> OpenLDAP/Winbind. The Samba version on the proxy machine is v 3.4.7
> as supplied from the Ubuntu repo.
>
> This however also seems to run out of steam. My first thought was
> that it may have been running out of RAM so I ran htop. Both CPUs
> were topping out at 20% and out of the 4GB of RAM, 1.3GB was used.
> Next I checked the load on the NIC and found that it was running on
> average 400kB/s, with the odd burst at 5MB/s. As the load increased,
> web pages were taking up to 30-45 seconds to load. I bypassed
> Dansguardian and went in on 3128 with no change in performance.
>
> Following the recommendations on other sites discovered via Google, I
> tuned and tweaked settings with no real benefit and I can't see that I
> changed anything to cause it to happen. The log files look fine, I
> have 10000 file descriptors available and cachemgr shows plenty of
> spares. There are 50% more NTLM authenticators than are in use at any
> given time.
>
> The config file for Squid is shown below. I have had the number of
> authenticators set to 400 as I have 350 users but the number in use
> still peaked at around 50. If I've been a numpty and done something
> glaringly obvious, I'd be grateful if someone could point it out. If
> not, ask for info and I'll provide it.
>
> Thanks,
>
> Jools
>
>
> ## Squid.conf
> ## Start with authentication for clients
>
> auth_param ntlm program /usr/bin/ntlm_auth
> --helper-protocol=squid-2.5-ntlmssp
> auth_param ntlm_param children 100
> auth_param ntlm keep_alive on
>
> auth_param basic program /usr/bin/ntlm_auth
> --helper-protocol=squid-2.5-basic
> auth_param basic children 100
> auth_param basic realm Squid proxy-caching web server
> auth_param basic credentialsttl 2 hours
>
> ## Access Control Lists for filter bypass ##
> acl realtek dstdomain .realtek.com.tw
> acl tes dstdomain .tes.co.uk
> acl glogster dstdomain .glogster.com
> acl adobe-installer dstdomain .adobe.com # allow installs from adobe
> download manager
> acl actihealth dstdomain .actihealth.com .actihealth.net # Allow
> direct access for PE dept activity monitors
> acl spybotupdates dstdomain .safer-networking.org .spybotupdates.com #
> Allow updates for Spybot S&D
> acl sims-update dstdomain .kcn.org.uk .capitaes.co.uk
> .capitasolus.co.uk .sims.co.uk # Allow SIMS to update itself directly
> acl kcc dstdomain .kenttrustweb.org.uk # Fix problem with county
> acl frenchconference dstdomain flashmeeting.e2bn.net
> acl emsonline dstdomain .emsonline.kent.gov.uk
> acl clamav dstdomain .db.gb.clamav.net
> acl ubuntu dstdomain .ubuntu.com .warwick.ac.uk
> acl windowsupdate dstdomain windowsupdate.microsoft.com
> acl windowsupdate dstdomain .update.microsoft.com
> acl windowsupdate dstdomain download.windowsupdate.com
> acl windowsupdate dstdomain redir.metaservices.microsoft.com
> acl windowsupdate dstdomain images.metaservices.microsoft.com
> acl windowsupdate dstdomain c.microsoft.com
> acl windowsupdate dstdomain www.download.windowsupdate.com
> acl windowsupdate dstdomain wustat.windows.com
> acl windowsupdate dstdomain crl.microsoft.com
> acl windowsupdate dstdomain sls.microsoft.com
> acl windowsupdate dstdomain productactivation.one.microsoft.com
> acl windowsupdate dstdomain ntservicepack.microsoft.com
> acl windowsupdate dstdomain download.adobe.com
> acl comodo dstdomain download.comodo.com
> acl simsb2b dstdomain emsonline.kent.gov.uk
> acl powerman dstdomain pmstats.org
> acl ability dstdomain ability.com
> acl fulston dstdomain fulstonmanor.kent.sch.uk
> acl httpsproxy dstdomain .retiredsanta.com .atunnel.com .btunnel.com
> .ctunnel.com .dtunnel.com .ztunnel.com .partyaccount.com
>
> ## Access Control for filtered users ##
> acl all src 0.0.0.0/0.0.0.0
> acl manager proto cache_object
> acl localhost src 127.0.0.1/255.255.255.255
> acl to_localhost dst 127.0.0.0/8
> acl SSL_ports port 443
> acl ntlm_users proxy_auth REQUIRED
>
> acl SSL_ports port 443 # https
> acl SSL_ports port 563 # snews
> acl SSL_ports port 873 # rsync
> acl Safe_ports port 80 # http
> acl Safe_ports port 21 # ftp
> acl Safe_ports port 443 # https
> acl Safe_ports port 70 # gopher
> acl Safe_ports port 210 # wais
> acl Safe_ports port 1025-65535 # unregistered ports
> acl Safe_ports port 280 # http-mgmt
> acl Safe_ports port 488 # gss-http
> acl Safe_ports port 591 # filemaker
> acl Safe_ports port 777 # multiling http
> acl Safe_ports port 631 # cups
> acl Safe_ports port 873 # rsync
> acl Safe_ports port 901 # SWAT
> acl purge method PURGE
> acl CONNECT method CONNECT
>
> ## Allow/Deny Lists ##
> http_access allow manager localhost
> http_access deny manager
> http_access allow purge localhost
> http_access deny purge
> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
>
> http_access allow emsonline
> http_access allow clamav
> http_access allow realtek
> http_access allow ubuntu
> http_access allow tes
> http_access allow glogster
> http_access allow kcc
> http_access allow fulston
> http_access allow comodo
> http_access allow ability
> http_access allow powerman
> http_access allow windowsupdate
> http_access allow simsb2b
> http_access allow adobe-installer
> http_access allow actihealth
> http_access allow spybotupdates
> http_access allow sims-update
> http_access allow frenchconference
> http_access allow ntlm_users
> http_access deny httpsproxy
> http_access allow localhost
> http_access deny all
> icp_access deny all
>
> ## Cache Settings ##
> log_fqdn off
> half_closed_clients off
> maximum_object_size 1024 KB
> cache_access_log none
> cache_store_log none
> http_port 3128
> redirect_children 750
> hierarchy_stoplist cgi-bin ?
> cache_mem 128 MB
> memory_replacement_policy lru
> cache_replacement_policy lru
> cache_dir ufs /fastcache1 15000 16 256
> cache_dir ufs /fastcache2 15000 16 256
> refresh_pattern ^ftp: 1440 20% 10080
> refresh_pattern ^gopher: 1440 0% 1440
> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
> refresh_pattern (Release|Package(.gz)*)$ 0 20% 2880
> refresh_pattern . 0 20% 4320
> acl shoutcast rep_header X-HTTP09-First-Line ^ICY.[0-9]
> upgrade_http0.9 deny shoutcast
> acl apache rep_header Server ^Apache
> broken_vary_encoding allow apache
> extension_methods REPORT MERGE MKACTIVITY CHECKOUT
> cache_effective_user proxy
> ## Hash out effective group as it stops access to winbind privileged
> pipe and breaks authentication - jpb
> # cache_effective_group proxy
> max_filedescriptors 10000
> dns_nameservers 172.20.0.253 172.31.49.46 172.31.81.46
> hosts_file /etc/hosts
> coredump_dir /var/spool/squid
>
Received on Thu Mar 03 2011 - 17:59:24 MST