Re: memory mapped store entries

On Thu, Aug 27, 1998 at 12:41:29PM +0300, Andres Kroonmaa wrote:

> oh, Solaris, for eg. guarantees that dirty page is flushed in 30
> secs at worst (by default)

Do you _know_ this for sure - have you tested this?

I'd offer to test it here, but I'm afraid if I go killing Solaris
boxes in the middle of doing stuff, bad things will happen.

> yeah, it will as old as from last page flush, not more. It will
> just lack recent changes, but not corrupt. Same goes for current
> squid. And unless you use syncronous disk io, you'd have FS a mess
> anyway, so what's the point in arguing about mmap or any other
> delayed write.

In inconsistency between the swap and the store is a corruption IMO.
If the swap looses track of 5% of the store, bad things will happen,
file will never get purged, disk space will run low or worse....

One solution I can think of to this, is to have a maintenance thread
running at ultra-low priority, perhaps sleeping most of the time,
that when the disks are idle, or more idle than busy, to walk the
store and check the consistency of the swap.

A careful balancing act would be required here though if you weren't
to trash your buffer-cache, although I'm not sure what good 50-100M
of buffer cache is with 30GB of disk blocks.... (yes, I know its
helps a lot with metadata and directories, but a store scan probably
won't affect this in a multi-level buffer cache because the block
involved in the sweeping won't get promoted sufficiently quickly and
hence will be purged before truly more frequently used blocks).

> > Yes, I agree. mmap is a win - but its not 100% foolproof against
> > corruption because my test show the kernel does not flush dirty
> > buffers at all of very frequently, not my kernel anyhow.
>
> then your kernel is broken, or timed page flushes explicitly
> disabled.

Indeed - the kernel may be broken, but its out there, and its
probably used by more squids that any other kernel.

I'll see what can be done about flushing dirty mmap pages under linux
anyhow.

> Anyway, we have arrived at understanding, that we must have ifdef
> that would do msync on some systems regularly...

Why an #ifdef - if solaris/whatever does it on a regular basis
anyhow, then calling msync every now and then won't do anything,
except perhaps flush a few pages dirtied since the last kernel-flush,
and its not going to cost that much....

-cw
Received on Tue Jul 29 2003 - 13:15:53 MDT