[squid-users] Windows Updates, YouTube and WoW

Hi all.

This is currently a test environment so making changes isn't an issue.

Initially I had issues with hosts updating <any flavour of Microsoft
Windows> but solved that with the included squid.conf. I'm even
getting real cache hits on some of the Windows XP and Windows 7
updates in my test lab, so the amount of effort I've put in so far is
pretty well justified. Since the target audience won't have access to
a local WSUS, I can pretty well count it as a win, even if the rest of
this email becomes moot.

Then came the big issue - World of Warcraft installation via the
downloaded client. Things pretty well fell apart. It would install up
to 20% and crash. Then it would install up to 25% and crash. Then 30%
and crash. It did that, crashing further in the process each time,
until it finally installed the base game (roughly 15 crashes). Due to
clamping down on P2P I disabled that update mechanism and told the
downloader to use only direct download. I'm averaging 0.00KB/s with
bursts from 2KB/s to 64 KB/s. If I take squid out of the line I get
speeds between 1 and 3 MB/s+ and things just work - but that sort of
defeats the purpose in having a device that will cache
non-authenticated user content. Having one user download a new 1 GB
patch, and it being available locally for the other couple of hundred,
would be ideal. Still, it isn't a deal breaker.

I understand that it could be related to the partial content reply for
the request and I understand that it could also be related to the
<URL>/<foo>? style request. Is the best approach to just automatically
pass anything for blizzard.com/worldofwarcraft.com straight through
and not attempt to cache the updates? I've seen some comments where
using

acl QUERY urlpath_regex cgi-bin \?
cache deny QUERY

will cause those requests to not be cached (and I understand why that
is) but I'm wondering if I should just ignore them altogether,
especially given the third item - YouTube.

The target population for this cache is rather large. Typically,
youtube is a huge culprit for bandwidth usage and a lot of the times
it's hundreds of people hitting the same videos. I've been looking at
how to cache those and it seems like it's required to either not use
the above ACL or it's to setup another ACL that specifically allows
youtube.

All of those comments and workarounds have been regarding the 2.x set
of squid, though. I'm curious if there is a cleaner way to go about
caching youtube (or, perhaps I should say, video.google.com) in 3.1.x,
or if it's possible to cache things like the WoW updates now? We're
looking to experiment with some proprietary devices that claim to be
able to cache Windows Updates, YouTube/Google Video, etc., but I'm
wondering if my woes are just because of my inexperience with squid or
if they're just that far ahead in terms of functionality?

Any hints, tips or suggestions would be more than welcome!

Relevant version information and configuration files:

fergie# squid -v
Squid Cache: Version 3.1.9
configure options: '--with-default-user=squid'
'--bindir=/usr/local/sbin' '--sbindir=/usr/local/sbin'
'--datadir=/usr/local/etc/squid'
'--libexecdir=/usr/local/libexec/squid' '--localstatedir=/var/squid'
'--sysconfdir=/usr/local/etc/squid' '--with-logdir=/var/log/squid'
'--with-pidfile=/var/run/squid/squid.pid'
'--enable-removal-policies=lru heap' '--disable-linux-netfilter'
'--disable-linux-tproxy' '--disable-epoll' '--disable-translation'
'--enable-auth=basic digest negotiate ntlm'
'--enable-basic-auth-helpers=DB NCSA PAM MSNT SMB squid_radius_auth'
'--enable-digest-auth-helpers=password'
'--enable-external-acl-helpers=ip_user session unix_group
wbinfo_group' '--enable-ntlm-auth-helpers=smb_lm' '--without-pthreads'
'--enable-storeio=ufs diskd' '--enable-disk-io=AIO Blocking
DiskDaemon' '--disable-ipv6' '--disable-snmp' '--disable-htcp'
'--disable-wccp' '--enable-pf-transparent' '--disable-ecap'
'--disable-loadable-modules' '--enable-kqueue' '--with-large-files'
'--prefix=/usr/local' '--mandir=/usr/local/man'
'--infodir=/usr/local/info/' '--build=amd64-portbld-freebsd8.1'
'build_alias=amd64-portbld-freebsd8.1' 'CC=cc' 'CFLAGS=-O2 -pipe
-fno-strict-aliasing' 'LDFLAGS=' 'CPPFLAGS=' 'CXX=c++' 'CXXFLAGS=-O2
-pipe -fno-strict-aliasing' 'CPP=cpp'
--with-squid=/usr/ports/www/squid31/work/squid-3.1.9
--enable-ltdl-convenience

It's running in transparent mode on

fergie# uname -m -r -s -v
FreeBSD 8.1-RELEASE FreeBSD 8.1-RELEASE #0: Mon Jul 19 02:36:49 UTC
2010 root_at_mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC
amd64

which is basically a vanilla FreeBSD 8.1 install with squid installed
from ports.

My squid.conf:

###################################################
#
# Recommended minimum configuration:
#

acl manager proto cache_object
acl localhost src 127.0.0.1/32 ::1
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1

# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly
plugged) machines

acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT

#
# Recommended minimum Access Permission configuration:
#
# Only allow cachemgr access from localhost
http_access allow manager localhost
http_access deny manager

# Deny requests to certain unsafe ports
http_access deny !Safe_ports

# Deny CONNECT to other than secure SSL ports
http_access deny CONNECT !SSL_ports

# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
#http_access deny to_localhost

#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#

# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
http_access allow localnet
http_access allow localhost

acl windowsupdate dstdomain windowsupdate.microsoft.com
acl windowsupdate dstdomain .update.microsoft.com
acl windowsupdate dstdomain download.windowsupdate.com
acl windowsupdate dstdomain redir.metaservices.microsoft.com
acl windowsupdate dstdomain images.metaservices.microsoft.com
acl windowsupdate dstdomain c.microsoft.com
acl windowsupdate dstdomain www.download.windowsupdate.com
acl windowsupdate dstdomain wustat.windows .com
acl windowsupdate dstdomain crl.microsoft.com

acl wuCONNECT dstdomain www.update.microsoft.com

#no_cache deny windowsupdate
#no_cache deny wuCONNECT

http_access allow CONNECT wuCONNECT localnet
http_access allow windowsupdate localnet

# And finally deny all other access to this proxy
http_access deny all

# Squid normally listens to port 3128
http_port 3128 transparent

# We recommend you to use at least the following line.
hierarchy_stoplist cgi-bin ?

# Uncomment and adjust the following to add a disk cache directory.
cache_dir ufs /var/squid/cache 175000 16 256

# Cache Mem - ideal amount of RAM to use
cache_mem 2048 MB

# Maximum object size - default is 4MB, not nearly enough to be useful
maximum_object_size 1024 MB

# Maximum object size in memory - we have 4GB, we can handle larger objects
maximum_object_size_in_memory 512 MB

# Read Timeout - BASE can take ages to read data, even more than the
default 15 minutes
read_timeout 30 minutes

# always pull the entire file from the start when a range is requested
range_offset_limit -1

# allow full object to download when client disconnects
quick_abort_min -1 KB

# Leave coredumps in the first cache dir
coredump_dir /var/squid/cache

#refresh_pattern -i ak\.worldofwarcraft\.com\.edgesuite\.net(.*)/(.*)
1440 90% 100800 reload-into-ims override-expire
#refresh_pattern -i blizzard\.vo\.llnwd\.net/(.*) 1440 90% 100800
reload-into-ims override-expire
#refresh_pattern -i attdist\.blizzard\.com/(.*) 1440 90% 100800
reload-into-ims override-expire

# Add any of your own refresh_pattern entries above these.
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320

###################################################

My /etc/pf.conf:

fergie# pfctl -f /etc/pf.conf
No ALTQ support in kernel
ALTQ related functions disabled
fergie# cat /etc/pf.conf

int_if=bge1
ext_if=bge0

rdr pass log(all) on $int_if proto tcp from any to any port 80 ->
127.0.0.1 port 3128

nat pass log(all) on $ext_if from $int_if:network to any -> ($ext_if)

pass log(all) on $ext_if
pass log(all) on $int_if
Received on Mon Nov 08 2010 - 23:32:58 MST