Re: [squid-users] possible bug on 2.7S9

On 07/11/10 13:21, Leonardo Rodrigues wrote:
>
> Hi,
>
> i'll try to describe with the most details i can what i think is
> something like a forwarding-loop-detection bug on 2.7S9
>
> i have squid 2.7S9 running on a CentOS 5.5 x64 box whici has 4 NICs. 3
> NICs are for internal networks (192.168.x) and 1 NIC is for internet
> (189.73.x.x). It was built with:
>
> [root_at_firewall squid]# squid -v
> Squid Cache: Version 2.7.STABLE9
> configure options: '--prefix=/usr' '--exec-prefix=/usr/bin'
> '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--libexecdir=/usr/bin'
> '--sysconfdir=/etc/squid' '--datadir=/var/squid' '--localstatedir=/var'
> '--enable-removal-policies=heap,lru' '--enable-storeio=ufs,aufs,null'
> '--enable-delay-pools' '--enable-http-violations' '--with-maxfd=8192'
> '--enable-async-io=8' '--enable-err-languages=Portuguese English'
> '--enable-default-err-language=Portuguese' '--enable-snmp'
> '--disable-ident-lookups' '--enable-linux-netfilter'
> '--enable-auth=basic digest ntlm negotiate'
> '--enable-basic-auth-helpers=DB LDAP NCSA SMB'
> '--enable-digest-auth-helpers=password ldap'
> '--enable-external-acl-helpers=ip_user ldap_group session wbinfo_group'
> '--enable-negotiate-auth-helpers=squid_kerb_auth'
> '--enable-ntlm-auth-helpers=fakeauth no_check' '--enable-useragent-log'
> '--enable-referer-log' '--disable-wccp' '--disable-wccpv2'
> '--enable-arp-acl' '--with-large-files' '--enable-large-cache-files'
> '--enable-ssl' '--enable-icmp'
>
>
> i've setup squid with something like:
>
> acl localhost src 127.0.0.1/255.255.255.255
> acl localhost_to dst 127.0.0.1/255.255.255.255
>
> acl network1 src 192.168.1.0/255.255.255.0
> acl network1_to dst 192.168.1.0/255.255.255.0
>
> acl network2 src 192.168.2.0/255.255.255.0
> acl network2_to dst 192.168.2.0/255.255.255.0
>
> acl network3 src 192.168.3.0/255.255.255.0
> acl network3_to dst 192.168.3.0/255.255.255.0
>
> http_port 8080 transparent
> http_port 3128 transparent
>
> tcp_outgoing_address 127.0.0.1 localhost_to
> tcp_outgoing_address 192.168.1.1 network1_to
> tcp_outgoing_address 192.168.2.1 network2_to
> tcp_outgoing_address 192.168.3.1 network3_to
> tcp_outgoing_address 189.73.x.x all
>
>
>
> config is OK, it runs just fine.

Obviously not. Or you would not be reporting this problem.

>
> problem is, on a given day, squid stop responding new connections and i
> have to stop it (service squid stop). After searching logs, i have found
> some interesting requests:
>
>
> 1288136326.944 48437 192.168.2.15 TCP_MISS/000 0 GET
> http://localhost:8080/sync/sis/index.php - DIRECT/127.0.0.1 -

DoS attack by 192.168.2.15.

> 1288136326.944 48426 127.0.0.1 TCP_MISS/000 0 GET
> http://localhost:8080/sync/sis/index.php - DIRECT/127.0.0.1 -
> (and this second line repeated about 13000 times)

This is why the default Squid ruleset includes:
  "http_access deny to_localhost"

>
> and during these, i got also on cache.log:
>
> 2010/10/26 21:37:59| WARNING! Your cache is running out of filedescriptors
> 2010/10/26 21:38:15| WARNING! Your cache is running out of filedescriptors
> 2010/10/26 21:38:31| WARNING! Your cache is running out of filedescriptors
> 2010/10/26 21:38:48| WARNING! Your cache is running out of filedescriptors
> 2010/10/26 21:39:04| WARNING! Your cache is running out of filedescriptors
> 2010/10/26 21:39:20| WARNING! Your cache is running out of filedescriptors
>
> i'm running with 8192 filedescriptors on a 150 clients network, that's
> more than enough filedescriptors for normal usage.
>
> (from cache.log)
> 2010/10/31 12:27:50| Starting Squid Cache version 2.7.STABLE9 for
> x86_64-unknown-linux-gnu...
> 2010/10/31 12:27:50| Process ID 16093
> 2010/10/31 12:27:50| With 8192 file descriptors available
>
>
> Well ..... after found that, i tried to reproduce it doing some request
> to localhost:8080 on 8080 squid port and i could successfully reproduce
> it, all the times, with the above squid.conf configuration.
>
> after some tryings, i have found that:
>
> 1) removing the:
> tcp_outgoing_address 127.0.0.1 localhost_to
>
> would avoid the problem and make the forwarding-loop-detection works fine
>

Indicating that your NAT rules are incorrect.

The above line is simply forcing Squid to send from 127.0.0.1. It would
only have any effect if your NAT intercept rules were forcing all
localhost traffic back into Squid.

Removing the above line may mean that you are simply shifting the
problem from your Squid to some web server elsewhere. Your Squid will be
passing it requests for "http://localhost:8080/...". The upside is that
at least it will not be a DoS flood when it arrives there.

> 2) removing the transparent from
> http_port 8080 transparent
>
> would avoid the problem too, even with the tcp_outgoing_address
> 127.0.0.1 active

Yes. Not doing NAT is good protection against this whole class of IP
address problems.

>
> question is ..... squid NOT detecting this forwarding-loop should be
> expected with this transparent and tcp_outgoing_address combination ?

Do you have both "via on" and "forwarded_for on" set in your squid.conf?
They are both needed.

> Are we talking of a bug or are we talking of some expected behavior ? Is
> there any other information that i could provide to help tracking this ?
>

The forwarding detection is not perfect. The to_localhost DoS protection
exists to prevent this.

In other news ... why do you have 2 interception ports? they are only
relevant between Squid and the NAT logic on the local box.

Also using two well-known ports such as 8080 and 3128 is just asking to
be kicked by CVE-2009-0801 and this particular DoS attack you have seen.

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE9 or 3.1.9
   Beta testers wanted for 3.2.0.2