Re: [squid-users] squid not storing objects to disk andgettingRELEASED on the fly

On 29/10/10 05:28, Rajkumar Seenivasan wrote:
> switching from LFUDA to GDSF didn't make much of a difference.
>
> I assume the following is happening...
> I pre-cache around 2 to 3GB of data everyday and get 40 to 50% HITS everyday.

Check that those 40-50% hits are from actual visitors, not from your
pre-cache scan. Ideally your pre-cache scan will be entirely MISSes for
content which is cacheable through until at least the next scan.

> Once the cache_dir size reaches the cache_swap_low threshold, squid is
> not aggressive enough in removing the old objects. Infact, I think
> squid is not doing anything to remove the old objects.

Correct. Aggressive removal starts when cache_dir stores more than
*_high threshold % its max data and continues until *_low is reached by
the erasures.
  Slow removal occurs with every visitor request as Squid becomes aware
of things being stale.

>
> So the pre-caching requests are not getting into the store and the HIT
> rate goes down big time.

huh? pre-cache requests for stuff which is cacheable will always get
them into cache. Things requested earlier may get dropped, even if it
was from earlier in the pre-cache scan.

HIT ratio dropping means only that items not in the cache are wanted
more than those already there. When working correctly the things in
cache are dropped and the now-wanted ones are added.

This is *exactly* how the pre-caching run works. It scans URLs through
Squid pretending to be a regular client which does not care about the
loading lag. By making Squid aware of and update any "pre-cache" items
which are stale they get reduced lag for later visitors. If they become
stale between pre-cache runs the next visitor to fetch them will face
the same slow-down.

NP: all of this pre-caching stuff is a different problem to the issue of
how much RAM squid is using.

> When this happens, if I increase the store size, I can see better HIT rates.

Yes. More stuff gets cached, thus more can be served from the cache.

>
> What can be done to resolve this issue? Is there a equivalent of
> "reference_age" for squid V3.1.8?

heap LFUDA and lru should provide it in the usual place. They are the
only replacement policies which base things on age.
heap GDSF bases removal on object size, so avg. object size will reduce
when it does removal.

> cache mgr always reports swap.files_cleaned = 0.
> my understanding is that this counter will show the # of objects
> removed from the disk store based on the replacement policy.

My understanding matches yours.

>
> I changed the cache_replacement_policy from "heap GDSF" to "lru"
> yesterday to see if it makes a difference.
> Removal policy: lru
> LRU reference age: 1.15 days
>
>
> issues with memory usage:
> both squids are running with 100% mem usage (15GB). Nothing else is
> running on these 2 servers.
> Stopping and starting the squid doesn't bring down the memory usage.
>
> The only way to release memory is to stop squid, move the cache dir to
> something_old, recreate and start squid with empty cache
> AND DELETE the old cache dirs.
> If I don't delete the old cache dir, memory is not getting released.

I saw earlier that you had Squid mempools turned off. This means that
memory usage is directly under control of the operating system default
allocator. They tend to like to keep memory alocations around, at least
as virtual memory in case its needed again.

>
> squid runs in accel mode and serves only sqllite and xml files. nothing else.

Can you provide me with some example URLs which I can run some tests on?
And if possible, access to make "manager" requests to your proxies. (I
visit from 58.28.153.233 or 2002:3a1c:99e9:0:206:5bff:fe7c:b8a if allowed)

>
> Squid Cache: Version 3.1.8
> configure options: '--enable-icmp'
> '--enable-removal-policies=heap,lru' '--enable-useragent-log'
> '--enable-referer-log' '--enable-follow-x-forwarded-for'
> '--enable-default-hostsfile=/etc/hosts' '--enable-x-accelerator-vary'
> '--disable-ipv6' '--enable-htcp' '--enable-icp'
> '--enable-storeio=diskd,aufs' '--with-large-files'
> '--enable-http-violations' '--disable-translation'
> '--disable-auto-locale' '--enable-async-io'
> --with-squid=/root/downlaods/squid/squid-3.1.8
> --enable-ltdl-convenience
>

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE9 or 3.1.9
   Beta testers wanted for 3.2.0.2