Re: [squid-users] Tuning for very expensive bandwidth links

On 01/04/11 12:09, Ed W wrote:
> Hi
>
>>> My thought was to investigate having the internet side proxy add etag
>>> headers to all content based on some quality hash function. Then have
>>> the (expensive) remote side proxy rewrite the request headers to always
>>> use If-None-Match? The idea is that the bandwidth is cheap on internet
>>> connected side, so it can refresh it's cache of the whole page, generate
>>> a new hash, but still return a "not modified" response if the end result
>>> is the same string of bytes. How much of that can I implement in Squid
>>> 3.x today..?
>>
>> 3.1.10+ will validate If-None-Match and ETag, but will not add them to
>> requests itself.
>
> Thanks - can you expand on what it means to "validate" in this case?
>
> I think you mean that if the content is cached with a given eTag then
> requests for that content will be returned from cache if the request has
> an appropriate If-None-Match - is this the case?

I mean Squid will produce 412 or 304 replies using those headers in the
HTTP/1.1 if-modified-since and if-none-match checking algorithms to
reduce bandwidth.

So you will not have to alter the receiving Squid to meet your needs.
Only the sending one. With modern browsers adding those headers on their
own you may not even have to add them.

>
>>> Note, I realise this could lead to some side effects where the action of
>>> visiting the web page itself causes some other side effect, however, I
>>> think this is a manageable problem for this requirement?
>>>
>>> Thanks for any pointers to ideas or other products that might help?
>>
>> ICAP or eCAP would be the way to go here for quick results. Making a
>> plugin to do the ETag generation and alterations before sending off.
>
> Understood.
>
> So the remote (client) side proxy would need an eCAP plugin that would
> modify the initial request to include an ETag. This would require some
> ability to interrogate what we have in cache and generate/request the
> ETag associated with what we have already - do you have a pointer to any
> API/code that I would need to look at to do this?

I'm unsure sorry. Alex at The Measurement Factory has better info on
specific details of what the eCAP API can do.

>
> Then on the internet side proxy we would do whatever we need to retrieve
> the content, say fetch the asset. Then our eCap on that side would
> generate a consistent ETag using our favourite hash function?

Yes. I'd start with that side of the link and see if the modern browsers
ETag support plays well and eliminates the need for the client-side eCAP
trouble.

>
> The part I'm unsure how to implement would be examining what's in
> squid's cache in order to generate an ETag based on what we have got (ie
> for remote side)?
>

Me too.

>
>> You could also look at cutting bodies off 304 replies at the Internet
>> side to avoid the bandwidth expensive TCP_REFRESH_UNMODIFIED responses.
>
> Hmm, yes that would be very sensible. Apart from via eCAP are there
> other ways I might do that?
>

Not currently.

>
>> NP: if you want to go ahead and alter Squid code adding If-None-Match on
>> outbound requests is an open bug. As is proper ETag variant caching
>> support.
>
> I don't know if I have the time/ability to hack on squid code? Is there
> someone who might be interested on working on this for an affordable fee?

IIRC we have Dimitry with The Measurement Factory assisting with HTTP
compliance fixes. I'm sure sponsorship towards a specific fix will be
welcomed.

>
> Thanks for the very helpful feedback. Note if there are any existing
> ecap/icap modules I should look at then please educate me? (I'm
> currently using "Ziproxy" and looking at moving the interesting bits to
> a Squid ecap module. I have also used "Rabbit" proxy which is somewhat
> similar)

The one public eCAP adapter we have bee notified about happens to be for
doing gzip. http://code.google.com/p/squid-ecap-gzip/

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE9 or 3.1.11
   Beta testers wanted for 3.2.0.5