Major pending change in 2.5 for multi-gigabyte file support

As some of you may have know the support in Squid for handling
multi-gigabyte files has long been non-existing at the best, both not at
all being able to handle files above 1.9 GB, bad interactions with the
cache (both on-disk and memory) in how largeish objects is processed and
the inability to write large log files on 32-bit platforms.

The lfs branch "corrects" this by ensuring file sizes, counters etc is 64
bits on all platforms supporting 64 bit integers and a bit more (swapout
logics changed slightly to support abandoning writing of an object no
longer cacheable).

Due to the nature of the patch it is fairly intrusive in terms of number
of lines changed, but it has undergone fairly intensive testing during
development.

The majority of these is just type changes relating to file size/offsets
arguments or removal of incorrect casts of the same to int. The other
changes are:

   - new squid_off_t data type for storing file sizes/offsets. Similar to
off_t, but may be larger if the OS and compiler supports long long
but doesn't support large files.

   - New header type to support large integers (content-length)

   - --enable-large-files configure option to enable OS-level LFS support

   - --enable-large-cache-files configure option to enable OS-level LFS
support and also support for caching such large files. This is a separate
option as it changes the cache format.

   - new config parsing helper function to parse large integers

   - swapout changes to stop swapping out objects when they become
uncacheable in the middle of a request. This covers both objects larger
than maximum_object_size and objects being replaced by a newer version
while in transit.

   - forward deferred read change to support the above

   - trivial store client changes to support the above two items

   - FTP directory listings changes to display very large file sizes in a
meaningful manner.

   - and perhaps some more I have forgotten to mention.

The branch is now considered finished and I welcome independent tests.
Areas I have not been able to run a full cover test on due to limited disk
space (my HDD is only 6GB in total, and quite full already..) is caching
of very large files > 2GB or range processing on the same.

Unless someone objects I plan to incorporate these changes into Squid-2.5
shortly (14 days or so).

Regards
Henrik
Received on Fri Mar 11 2005 - 18:53:49 MST