Re: [squid-users] caching behavior during COSS rebuild

Just tested this - -F appears to work for aufs rebuilds but not COSS
rebuilds.

To reproduce:

- Config squid with an aufs and a coss store, like so:

cache_dir aufs /usr/squidcache 5000 16 256 min-size=500000
cache_dir coss /usr/squidcache/coss1.dat 500 block-size=4096 max-
size=500000 membufs=100
cache_swap_log /usr/squidcache/%s.swap

- Start squid, send it a bunch of queries (I use a script that grabs
random URLs from recent access.log files)

- Stop squid, delete /usr/squidcache/usr.squidcache.swap

- Start sending squid requests (again, the random URL script above)

- Restart squid with -F

What I'm seeing when I do the above is that as soon as the AUFS stores
finish rebuilding, squid starts answering queries - but the COSS isn't
rebuilt yet, and until that completes, all objects < 500K are cache
misses (SO_FAIL in store.log).

Will get a bugzilla entry in place for this.

-C

On Apr 22, 2009, at 10:00 AM, Chris Woodfield wrote:

> ...and sure enough, it's right there in -h output...
>
> cache$ /usr/local/squid/sbin/squid -h
> ...
> -F Don't serve any requests until store is rebuilt.
> ...
>
> /me goes to write "I will RTFM Before Posting To squid-users" 100
> times on the whiteboard... :)
>
> -C
>
> On Apr 22, 2009, at 9:56 AM, Adrian Chadd wrote:
>
>> Well, I killed the swaplog writing entirely in Lusca - the COSS
>> rebuild code doesn't read from it (it was broken for various reasons
>> revolving mostly around code bitrot IIRC.)
>>
>> There's a flag you can pass Squid to not handle requests until the
>> store is rebuilt - its the "-F" flag.
>>
>> I'm fixing the store rebuild times in Lusca-HEAD at the moment and
>> this includes writing some new COSS rebuild-from-index, rebuild-
>> from-log
>> and rebuild-from-rawdevice tools.
>>
>>
>>
>> Adrian
>>
>>
>> On Wed, Apr 22, 2009, Chris Woodfield wrote:
>>>
>>> On Apr 22, 2009, at 4:56 AM, Amos Jeffries wrote:
>>>
>>>> Chris Woodfield wrote:
>>>>> So I'm running with COSS under 2.7STABLE6, we've noticed (as I can
>>>>> see others have, teh Googles tell me so) that the COSS rebuild a.
>>>>> happens every time squid is restarted, and b. takes quite a while
>>>>> if the COSS stripes are large. However, I've noticed that while
>>>>> the
>>>>> stripes are being rebuilt, squid still listens for and handles
>>>>> requests - it just SO_FAILs on every object that would normally
>>>>> get
>>>>> saved to a COSS stripe. So much for that hit ratio.
>>>>> SO - the questions are:
>>>>> 1. Is there *any* way to prevent the COSS rebuild if squid is
>>>>> terminated normally?
>>>>
>>>> The indexes are stored in swap.state. Check that it is being done
>>>> properly by your Squid.
>>>>
>>>
>>> This could be the issue - when I exit squid, I notice that my
>>> $coss_file.dat and $coss_file.dat.last-clean files all have zero
>>> size.
>>> Any idea why this might be happening?
>>>
>>> The relevant section of our squid.conf reads as follows:
>>>
>>> cache_dir aufs /usr/squidcache.0/cache/ 750000 16 256 min-
>>> size=1000000
>>> cache_dir coss /usr/squidcache.0/cache/coss1.dat 30000 block-
>>> size=4096
>>> max-size=1000000 membufs=100
>>> cache_dir coss /usr/squidcache.0/cache/coss2.dat 30000 block-
>>> size=4096
>>> max-size=1000000 membufs=100
>>> cache_dir coss /usr/squidcache.0/cache/coss3.dat 30000 block-
>>> size=4096
>>> max-size=1000000 membufs=100
>>>
>>> cache_swap_log /usr/squidcache.0/cache/%s
>>>
>>> Thanks,
>>>
>>> -C
>>>
>>>>> 2. Is there a way to prevent squid from handling requests until
>>>>> the
>>>>> COSS stripe is fully rebuilt (this is obviously not good if you
>>>>> don't have redundant squids, but that's not a problem for us) ?
>>>>
>>>> I believe its possible. If its not a local failure to find
>>>> swap.state for the COSS dir then it will be a code fix.
>>>> Unfortunately we developers are no longer actively working on
>>>> Squid-2 without a paid support contract. Also Adrian our storage
>>>> expert who would be the best to ask has retired from active
>>>> alterations.
>>>>
>>>> Amos
>>>> --
>>>> Please be using
>>>> Current Stable Squid 2.7.STABLE6 or 3.0.STABLE14
>>>> Current Beta Squid 3.1.0.7
>>>>
>>
>> --
>> - Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial
>> Squid Support -
>> - $25/pm entry-level VPSes w/ capped bandwidth charges available in
>> WA -
>>
>
Received on Wed Apr 22 2009 - 15:42:47 MDT