Re: [squid-users] How can I clean out old objects - refresh patt

On 19-Sep-07 My Secret NSA Wiretap Overheard Adrian Chadd Saying :
> Files aren't deleted when they expire.
>
> Files are deleted when:
>
> * A request occurs and squid checks the file for freshness, or
> * Squid issues a validation requests and determines the local copy is stale,
> or
> * Squid needs to make space (as the disk store is full) and starts running
> the object replacement policy to purge objects - but then, it doesn't
> maintain
> a list of "stale" objects to purge; it just deletes the 'oldest' objects.
>
> Adrian

 Thanks for the clarification, but Eeek!

 So then, I guess this raises the question: If you have plenty of disk, there
really is nothing from keeping ancient files hanging around, using up space
and enlarging your swap.state file?

 I thought it was an either not enough space Or older than expire time would
 delete objects.

 So it seems like, I either have to manually purge old files every so often, or
set my disk space artificially to prevent too many objects based on my
servers's memory or increase my memory?

 Is that what people generally do?

  Nicole

 
> On Tue, Sep 18, 2007, Nicole wrote:
>> Hate to respond to myself, but I wanted to add more info..
>>
>> In a well duh moment I ran find and found objects going back to July.
>> find /cache -type f -mtime +30 -exec ls {} \;
>>
>> If my headers from my web servers are set to expire in 2 weeks:
>> Cache-Control: max-age=1728000
>> Connection: close
>> Date: Wed, 19 Sep 2007 00:38:51 GMT
>> Accept-Ranges: bytes
>> ETag: "1155193587"
>> Server: lighttpd
>> Content-Length: 68424
>> Content-Type: image/jpeg
>> Expires: Tue, 09 Oct 2007 00:38:51 GMT
>> Last-Modified: Mon, 17 Sep 2007 20:03:00 GMT
>> Client-Date: Wed, 19 Sep 2007 00:38:51 GMT
>> Client-Response-Num: 1
>>
>> How did my expires make it keep files for so long and why did my new ones
>> not
>> start a purge?
>>
>>
>> Thanks
>>
>>
>> Nicole
>>
>>
>> > Hello all
>> >
>> > I have a few squid servers that seem to have gotten a bit out of control.
>> >
>> > They are using up all the systems memory and starting to serve items
>> > slowly.
>> >
>> > As near as I can tell, it seems to just want more memory than I have to
>> > serve
>> > and manage all the objects in the cache.
>> >
>> > Internal Data Structures:
>> > 20466526 StoreEntries
>> > 24888 StoreEntries with MemObjects
>> > 24870 Hot Object Cache Items
>> > 20466434 on-disk objects
>> >
>> >
>> > I have tried reducing my refresh pattern from:
>> > refresh_pattern -i \.jpg 10080 150% 40320 ignore-reload
>> > to:
>> > refresh_pattern -i \.jpg 5040 100% 4320 ignore-reload
>> >
>> > and doing a reload.
>> >
>> > However, I have not noticed it expiring out old objects and freeing up
>> > disk
>> > space like I thought it would.
>> >
>> > Do objects get stored based on their original refresh pattern? So even if
>> > I
>> > change it, they won't expire until they expire based on the pattern they
>> > were
>> > stored with?
>> >
>> > Is there any way to tell the age of the objects eating up my cache
>> > storage
>> > space? Any reccomendations on how to reduce my object count besides
>> > reducing
>> > disk space? This is for a reverse proxy cache and we have the cache
>> > header set to expire objects in 2 weeks. I really can't believe that I
>> > have
>> > 20Million 2 week old objects.
>> >
>> >
>> >
>> > Thanks!
>> >
>> >
>> >
>> > Nicole
>> >
>> >
>> > --
>> > |\ __ /| (`\
>> > | o_o |__ ) )
>> > // \\
>> > - nmh@daemontech.com - Powered by FreeBSD -
>> > ------------------------------------------------------
>> > "The term "daemons" is a Judeo-Christian pejorative.
>> > Such processes will now be known as "spiritual guides"
>> > - Politicaly Correct UNIX Page
>>
>>
>> --
>> |\ __ /| (`\
>> | o_o |__ ) )
>> // \\
>> - nmh@daemontech.com - Powered by FreeBSD -
>> ------------------------------------------------------
>> "The term "daemons" is a Judeo-Christian pejorative.
>> Such processes will now be known as "spiritual guides"
>> - Politicaly Correct UNIX Page
>>
>>
>
> --
> - Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support
> -
> - $25/pm entry-level bandwidth-capped VPSes available in WA -

--
                     |\ __ /|   (`\            
                     | o_o  |__  ) )           
                    //      \\                 
  -  nmh@daemontech.com  -  Powered by FreeBSD  -
------------------------------------------------------
 "The term "daemons" is a Judeo-Christian pejorative.
 Such processes will now be known as "spiritual guides"
  - Politicaly Correct UNIX Page