Build failed in Jenkins: 3.2-matrix » rio.treenet #116 from noc_at_squid-cache.org on 2011-04-03 (squid-dev)

From: <noc_at_squid-cache.org>
Date: Sun, 3 Apr 2011 16:28:47 +0200 (CEST)

See <http://build.squid-cache.org/job/3.2-matrix/./label=rio.treenet/116/changes>

Changes:

[Amos Jeffries] Simulate DIRECT tunnel to origin peers on CONNECT

Within reason. Check that at least the port matches. That gives us some
small measure of reason to believe its the same protocol inside or the
same app being CONNECTed to.

[Amos Jeffries] Language: fr

[Amos Jeffries] Language: hy

[Amos Jeffries] Optimization (performance regression fix): Use bigger buffer for server reads.

Change the server read buffer limits to 16KB minimum and 256KB maximum.
Used to be: 2KB and 2GB. And before r9766: 4KB and SQUID_TCP_SO_RCVBUF.

Trunk r9766 (Remove limit on HTTP headers read) made the default HTTP
server read buffer size 2KB instead of 4KB, visibly slowing down Squid
when kernel network buffers are full and can sustain larger Squid reads.
Doing up to twice as many network reads is expensive (and probably not
just because of the extra system call overheads).

We never grow that buffer size if the _parser_ does not need a bigger
buffer: Even if the HTTP client is slower than the server, the buffer
stays small because it gives all the data to Store and Store eventually
just stalls reading via delayAwareRead() and read_ahead_gap. The
situation may be different with RESPMOD, but if the adaptation service
is fast, the buffer would still not grow.

This change does not reset the minimum buffer size to the old 4KB
default because memory is much cheaper compared to the days where that
default was set. 8KB may have worked too, but with 12KB median typical
response size a larger buffer may be a good idea for a busy Squid. More
performance work is needed to find the optimal value (which could depend
on the environment).

This change does not set the maximum buffer size to the current 2GB
limit because we have not tested how header and chunking parsers would
cope with malicious messages trying to run Squid out of RAM; and also
because no good parser should need that much lookahead space. Is 256KB
enough for all legitimate real-world response headers? We do not know.

It is tempting to use Config.tcpRcvBufsz or SQUID_TCP_SO_RCVBUF to find
the right minimum or maximum buffer size, but those parameters deal with
low-level TCP buffering aspects while this buffer deals with HTTP
parsing.

[Amos Jeffries] Log all transactions including those with uncertain status or no sent response.

Excluding those transactions from access log hides valuable information.

For example, it hides certain aborted client transactions which sometimes
indicate Squid bugs or configuration problems that the administrator can fix.
It also leads to wrong statistics reporting if the reporting tools are based
on access logs information.

The change just removes the logging guard and adds debugging. The logging
code itself is unchanged except for indenting.

[Amos Jeffries] Optimization: Do not create adaptation history to check whether it is empty.

[Amos Jeffries] If a worker process crashes during shutdown, dump core and prevent restarts.

Before the change, if a worker process crashes during shutdown, death()
handler would exit with code 1, and master process would restart the
worker. Now workers send SIGUSR1 to master when shutting down. When
master process gets the SIGUSR1 signal, it stops restarting workers.

SIGUSR1 is already used for log rotation, but it is fine to use SIGUSR1
for master process shutdown notifications because master is never
responsible for both log rotation and kid restarts.

Terminate with abort(3) instead of exit(3) to leave a core dump if Squid
worker crashes during shutdown.

Also the patch fixes potential infinite loop in master process. Master
used to finish only when all kids exited with success, or all kids are
hopeless, or all kids were killed by a signal, but when some kids are
hopeless and others were killed, the master process would not exit.
After the change, master exits when there are no running kids and no
kids should be restarted.

Add syslog notice if kid becomes hopeless.

[Amos Jeffries] Better reporting of REQMOD failures during body processing.

When REQMOD body processing fails, the server-side needs to abort the
in-progress transaction. Use HTTP 500 (Internal Server Error) instead of 502
(Bad Gateway) status code and a new custom error detail for this case.

There is no perfect status code for ICAP errors because some view ICAP
processing as an integral part of the proxy (closer to the internal
proxy code) and some treat it as an external entity (closer to an
"upstream" web server).

Nevertheless, 502 (Bad Gateway) feels like a worse fit because from web
user and traffic flow point of views ICAP is less of a "gateway" or
"upstream" server than it is an "internally" used auxiliary service.
When both status codes were used, we have received reasonable complaints
about 502 responses but not (IIRC) about 500 responses.

Using custom error detail is better than errno in this context because
errno is often not set for ICAP errors and because by the time we
generate an error, the errno value is likely to be from a different
system call error anyway.

[Amos Jeffries] Instead of exiting, disable optional eCAP services that fail initialization.

Report all service initialization failures but mark optional services
down instead of killing Squid by propagating the failure to the Squid
core. An initialization failure of an optional (bypass=1) service
should not lead to Squid quitting because, according to squid.conf, such
a service may be replaced by other services (using adaptation sets) or
simply ignored.

Initialization failures of essential services still lead to fatal
errors, but they are now reported better.

Ecap services should indicate failures by throwing an exception. Squid
reports exception details using std::exception::what() method if
possible.

[Amos Jeffries] Support dynamic adaptation plans that cover multiple vectoring points.

The dynamic adaptation plan is specified using X-Next-Services ICAP
header or eCAP meta-info, as usual. A REQMOD adaptation service may
construct an adaptation plan that starts with REQMOD and ends with
RESPMOD. Multiple adaptations may be planned at each point.

The natural transaction handling order must be preserved: the plan
cannot go from RESPMOD back to REQMOD.

Adaptation::History object is used to keep future plan steps when
crossing vectoring points.

[Amos Jeffries] Cleanup: Use prefix increment in place of postfix where appropriate in mem.cc

[Amos Jeffries] Cleanup: Include iosfwd instead of ostream in Mem.h

[Amos Jeffries] Cleanup: make clientParseRequest() a member of ConnStateData

This allows the HttpParser to also become a member field and persistent
across all requests on the connection instead of newely allocated on
the stack for every read cycle.
That in turn allows the parser to retain state for efficient 'trickle'
parsing across multiple read cycles.

For now the old behaviour of reset on every read is retained in order to
prevent this shuffling from causing behaviour changes. That negates most
of the actual performance gains (for now).

[Amos Jeffries] Bug 3007: CONNECT to cache_peer returns 000 status code

Send 200 on successful tunnel relay setup. 500 on peer connect failures.

This archive was generated by hypermail 2.2.0 : Mon Apr 04 2011 - 12:00:03 MDT