Re: Deferred reads

On Wed, May 31, 2000, Duane Wessels wrote:
>
>
> On Wed, 31 May 2000, Adrian Chadd wrote:
>
> >
> > The next step in my nefarious plan with the comm code is to make
> > squid work well with event driven systems such as FreeBSD's kqueue
> > system. In this light, I'd like to kill deferred reads. The
>
> I'll argue that deferred reads are (however ugly) a very important
> part of squid's operation. As much as you may dislike them, they
> work.
>

[snip]

> Do you have any data from real caches? Is it ever the case
> that "you have lots of deferred reads?"

I have been doing my testing with a datacomm-1 polygraph load. I will
dig up the figures tonight for you in the kqueue case. The trouble with
things like kqueue is that if you defer the read, you end up just
getting the read back the next time you get the list of completed
events. The only way around this is to ditch caring about the fd until
after the fd is deferred, but currently you can only tell the state of
a deferred fd when you call commReadDefer(), not at event register time.

I'll throw some profile info at the list tomorrow sometime. I have to
throw some debugging statements back into the sourcecode.

> > There are five functions passed to commSetDefer:
> >
> > httpAcceptDefer - defers accepting HTTP connections if we are out of fd's .
> > clientReadDefer - haven't figured this ne out yet
>
> clientReadDefer is used for half-closed connections, and also
> for pipelined requests.

I have since figured this one out.

> > sslDeferServerRead - used only if DELAY_POOLS is enabled
> > pumpReadDefer - used so PUMP_MAXBUFFER isn't overflowed
> > fwdCheckDeferRead - don't read too far ahead from server if clients are lagged
> >
> > Now, from what I can see these can be replaced with either periodic events
> > or properly scheduled reads.
>
> I don't like the sound of "periodic events." How do you choose the
> period? If the period is too long then the request is delayed
> unnecessarily. If its too short, and you still don't want to read,
> then you have almost the same problem you're trying to fix -- a lot of
> cycles spent not reading.

> Also, can you explain what a properly scheduled read is?
> How is that supposed to work?

For example, only scheduling a read when you only want data, and not
when its deferred. I'll pull up some example code to demonstrate what
I'm thinking.

> > Since this is part of the commloops development, I'm going to
> > start removing these and replacing them with suitable interfaces.
> > Can anyone think of a reason to keep deferred reads?
>
> To me, deferring reads are tightly coupled with the use of select and
> poll. I'm far from convinced that you can/should "kill them" because
> kqueue works differently. Admittedly, I don't know enough about the
> kqueue implementation, so maybe you can educate me further. If a
> kqueue event (or whatever) says that data is read, but you want to
> ignore it (or, the application isn't ready to accept the data), how do
> you handle that?

The trouble is that in poll/select, you can throw away a returned
fd rather easily because you should have all the other fds that are
ready to service, and you only sacrifice a little CPU. With an event
driven system like kqueue, if you throw away the read because you are
not ready to deal with it yet (a la deferred reads), you schedule
another one and it will return immediately. If you end up with
a few deferred reads, then you end up getting mostly these read ready
fds. kqueue doesn't have any way of dealing with this. You can't
tell whether a fd is going to be deferred at event add time, and you
don't get an update when the read defer is active and inactive.

> Whatever you do, please do not commit any of these changes until you
> have run the code for a while on live caches.

I'm not going to commit this code until its been thoroughly profiled
and tested.

I'm open to more suggestions. :)

Adrian

-- 
Adrian Chadd			Build a man a fire, and he's warm for the
<adrian@creative.net.au>	rest of the evening. Set a man on fire and
				he's warm for the rest of his life.