SMP Caching, part 1: Core changes and addition of the shared memory cache. * Added MemObject::expectedReplySize() and used it instead of object_sz. When deciding whether an object with a known content length can be swapped out, do not wait until the object is completely received and its size (mem_obj->object_sz) becomes known (while asking the store to recheck in vain with every incoming chunk). Instead, use the known content length, if any, to make the decision. This optimizes the common case where the complete object is eventually received and swapped out, preventing accumulating potentially large objects in RAM while waiting for the end of the response. Should not affect objects with unknown content length. Side-effect1: probably fixes several cases of unknowingly using negative (unknown) mem_obj->object_sz in calculations. I added a few assertions to double check some of the remaining object_sz/objectLen() uses. Side-effect2: When expectedReplySize() is stored on disk as StoreEntry metadata, it may help to detect truncated entries when the writer process dies before completing the swapout. * Removed mem->swapout.memnode in favor of mem->swapout.queue_offset. The code used swapout.memnode pointer to keep track of the last page that was swapped out. The code was semi-buggy because it could reset the pointer to NULL if no new data came in before the call to doPages(). Perhaps the code relied on the assumption that the caller will never doPages if there is no new data, but I am not sure that assumption was correct in all cases (it could be that I broke the calling code, of course). Moreover, the page pointer was kept without any protection from page disappearing during asynchronous swapout. There were "Evil hack time" comments discussing how the page might disappear. Fortunately, we already have mem->swapout.queue_offset that can be fed to getBlockContainingLocation to find the page that needs to be swapped out. There is no need to keep the page pointer around. The queue_offset-based math is the same so we are not adding any overheads by using that offset (in fact, we are removing some minor computations). * Added "close how?" parameter to storeClose() and friends. The old code would follow the same path when closing swapout activity for an aborted entry and when completing a perfectly healthy swapout. In non-shared case, that could have been OK because the abort code would then release the entry, removing any half-written entry from the index and the disk (but I am not sure that release happened fast enough in 100% of cases). When the index and disk storage is shared among workers, such "temporary" inconsistencies result in truncated responses being delivered by other workers to the user because once the swapout activity is closed, other workers can start using the entry. By adding the "close how?" parameter to closing methods we allow the core and SwapDir-specific code to handle aborted swapouts appropriately. Since swapin code is "read only", we do not currently distinguish between aborted and fully satisfied readers: The readerGone enum value applies to both cases. If needed, the SwapDir reading code can make that distinction by analyzing how much was actually swapped in. * Moved "can you store this entry?" code to virtual SwapDir::canStore(). The old code had some of the tests in SwapDir-specific canStore() methods and some in storeDirSelect*() methods. This resulted in inconsistencies, code duplication, and extra calculation overheads. Making this call virtual allows individual cache_dir types to do custom access controls. The same method is used for cache_dir load reporting (if it returns true). Load management needs more work, but the current code is no worse than the old one in this aspect, and further improvements are outside this change scope. * Minimized from-disk StoreEntry loading/unpacking code duplication. Moved common (and often rather complex!) code from store modules into storeRebuildLoadEntry, storeRebuildParseEntry, and storeRebuildKeepEntry. * Do not set object_sz when the entry is aborted because the true object size (HTTP reply headers + body) is not known in this case. Setting object_sz may fool client-side code into believing that the object is complete. This addresses an old RBC's complaint. * When swapout initiation fails, mark swapout decision as MemObject::SwapOut::swImpossible. This prevents the caller code from trying to swap out again and again because swap_status becomes SWAPOUT_NONE. TODO: Consider add SWAPOUT_ERROR, STORE_ERROR, and similar states. It may solve several problems where the code sees _NONE or _OK and thinks everything is peachy when in fact there was an error. * Call haveParsedReplyHeaders() before entry->replaceHttpReply(). HaveParsedReplyHeaders() sets the entry public key and various flags (at least). ReplaceHttpReply() packs reply headers, starting swapout process. It feels natural to adjust the entry _before_ we pack/swap it, but I may be missing some side-effects here. The change was necessary because we started calling checkCachable() from swapoutPossible(). If haveParsedReplyHeaders() is not called before we swap out checks, the entry will still have the private key and will be declared impossible to cache. * Extracted the write-to-store step from StoreEntry::replaceHttpReply(). This allows the caller to set the reply for the entry and then update the entry and the reply before writing them to store. For example, the server-side haveParsedReplyHeaders() code needs to set the entry timestamps and make the entry key public before the entry starts swapping out, but the same code also needs access to entry->getReply() and such for timestampsSet() and similar code to work correctly. TODO: Calls to StoreEntry::replaceHttpReply() do not have to be modified because replaceHttpReply() does write by default. However, it is likely that callers other than ServerStateData::setFinalReply() should take advantage of the new split interface because they call timestampsSet() and such after replaceHttpReply(). * Moved SwapDir::cur_size and n_disk_objects to specific SwapDirs. Removed updateSize(). Some cache_dirs maintain their own maps and size statistics, making the one-size-fits-all SwapDir members inappropriate. * A new SwapDir public method swappedOut() added. It is called from storeSwapOutFileClosed() to notify SwapDir that an object was swapped out. * Change SwapDir::max_size to bytes, make it protected, use maxSize() instead. Change SwapDir::cur_size to bytes, make it private, use currentSize() instead. Store Config.Store.avgObjectSize in bytes to avoid repeated and error-prone KB<->bytes conversions. * Change Config.cacheSwap.swapDirs and StoreEntry::store() type to SwapDir. This allows using SwapDir API without dynamic_cast. * Always call StoreEntry::abort() instead of setting ENTRY_ABORTED manually. * Rely on entry->abort() side-effects if ENTRY_ABORTED was set. * Added or updated comments to better document current code. * Added operator << for dumping StoreEntry summary into the debugging log. Needs more work to report more info (and not report yet-unknown info). * Added initial shared memory cache implementation (MemStore). The shared memory cache keeps its own compact index of cached entries using extended Ipc::StoreMap class (MemStoreMap). The cache also strives to keep its Root.get() results out of the store_table except during transit. Eventually, the non-shared/local memory cache should also be implemented using a MemStore-like class, I think. This will allow to clearly isolate local from shared memory cache code. * Fixed blocking reads that were sometimes reading from random file offsets. Core "disk file" reading code assumed that if the globally stored disk.offset matches the desired offset, there is no reason to seek. This was probably done to reduce seek overhead between consecutive reads. Unfortunately, the disk writing code did not know about that optimization and left F->disk.offset unchanged after writing. This may have worked OK for UFS if it never writes to the file it reads from, but it does not work for store modules that do both kinds of I/O at different offsets of the same disk file. Eventually, implement this optimization correctly or remove disk.offset. * Added RunnersRegistry, an API to register and, later, run a group of actions. Useful for keeping general initialization/cleanup management code (e.g., main.cc) independent from specific initialization/cleanup code (e.g., see ThePagePool in src/ipc/mem/Pages.cc) during staged initialization and cleaning. TODO: Portability. Add mem_cache shared=on|off parameter. Eventually, support shared memory caching of multi-page entries. === modified file 'include/Array.h' --- include/Array.h 2009-07-13 01:20:26 +0000 +++ include/Array.h 2011-01-27 21:14:56 +0000 @@ -97,6 +97,8 @@ Vector &operator += (E item) {push_back(item); return *this;}; void insert (E); + const E &front() const; + E &front(); E &back(); E pop_back(); E shift(); // aka pop_front @@ -248,6 +250,22 @@ } template +const E & +Vector::front() const +{ + assert (size()); + return items[0]; +} + +template +E & +Vector::front() +{ + assert (size()); + return items[0]; +} + +template void Vector::prune(E item) { === modified file 'src/DiskIO/Blocking/BlockingFile.cc' --- src/DiskIO/Blocking/BlockingFile.cc 2010-11-20 11:31:38 +0000 +++ src/DiskIO/Blocking/BlockingFile.cc 2011-01-27 21:14:56 +0000 @@ -146,6 +146,7 @@ assert (fd > -1); assert (ioRequestor.getRaw()); readRequest = aRequest; + debugs(79, 3, HERE << aRequest->len << " for FD " << fd << " at " << aRequest->offset); file_read(fd, aRequest->buf, aRequest->len, aRequest->offset, ReadDone, this); } @@ -160,7 +161,7 @@ void BlockingFile::write(WriteRequest *aRequest) { - debugs(79, 3, "storeUfsWrite: FD " << fd); + debugs(79, 3, HERE << aRequest->len << " for FD " << fd << " at " << aRequest->offset); writeRequest = aRequest; file_write(fd, aRequest->offset, @@ -216,7 +217,7 @@ BlockingFile::writeDone(int rvfd, int errflag, size_t len) { assert (rvfd == fd); - debugs(79, 3, "storeUfsWriteDone: FD " << fd << ", len " << len); + debugs(79,3, HERE << "FD " << fd << ", len " << len); WriteRequest::Pointer result = writeRequest; writeRequest = NULL; === modified file 'src/Makefile.am' --- src/Makefile.am 2011-04-11 06:05:36 +0000 +++ src/Makefile.am 2011-04-25 15:14:10 +0000 @@ -464,8 +466,8 @@ Server.h \ structs.h \ swap_log_op.h \ - SwapDir.cc \ - SwapDir.h \ + SwapDir.cc MemStore.cc \ + SwapDir.h MemStore.h \ time.cc \ tools.cc \ tunnel.cc \ @@ -543,6 +545,7 @@ eui/libeui.la \ acl/libstate.la \ $(AUTH_LIBS) \ + $(DISK_LIBS) \ acl/libapi.la \ base/libbase.la \ libsquid.la \ @@ -558,7 +561,6 @@ $(XTRA_OBJS) \ $(DISK_LINKOBJS) \ $(REPL_OBJS) \ - $(DISK_LIBS) \ $(DISK_OS_LIBS) \ $(CRYPTLIB) \ $(REGEXLIB) \ @@ -1350,7 +1368,7 @@ StoreSwapLogData.cc \ tools.cc \ tunnel.cc \ - SwapDir.cc \ + SwapDir.cc MemStore.cc \ url.cc \ URLScheme.cc \ urn.cc \ @@ -1657,6 +1675,7 @@ time.cc \ tools.cc \ tunnel.cc \ + SwapDir.cc MemStore.cc \ url.cc \ URLScheme.cc \ urn.cc \ @@ -1831,6 +1850,7 @@ time.cc \ tools.cc \ tunnel.cc \ + SwapDir.cc MemStore.cc \ url.cc \ URLScheme.cc \ urn.cc \ @@ -2172,7 +2192,7 @@ event.cc \ tools.cc \ tunnel.cc \ - SwapDir.cc \ + SwapDir.cc MemStore.cc \ url.cc \ URLScheme.cc \ urn.cc \ @@ -2878,7 +2931,7 @@ event.cc \ tools.cc \ tunnel.cc \ - SwapDir.cc \ + SwapDir.cc MemStore.cc \ urn.cc \ wccp2.cc \ whois.cc \ === modified file 'src/MemObject.cc' --- src/MemObject.cc 2011-01-27 12:23:25 +0000 +++ src/MemObject.cc 2011-04-28 22:45:55 +0000 @@ -71,6 +71,15 @@ return Pool().inUseCount(); } +void +MemObject::resetUrls(char const *aUrl, char const *aLog_url) +{ + safe_free(url); + safe_free(log_url); /* XXX account log_url */ + log_url = xstrdup(aLog_url); + url = xstrdup(aUrl); +} + MemObject::MemObject(char const *aUrl, char const *aLog_url) { debugs(20, 3, HERE << "new MemObject " << this); @@ -90,6 +99,8 @@ object_sz = -1; /* XXX account log_url */ + + swapout.decision = SwapOut::swNeedsCheck; } MemObject::~MemObject() @@ -233,6 +244,15 @@ return data_hdr.endOffset(); } +void +MemObject::markEndOfReplyHeaders() +{ + const int hdr_sz = endOffset(); + assert(hdr_sz >= 0); + assert(_reply); + _reply->hdr_sz = hdr_sz; +} + int64_t MemObject::size() const { @@ -242,6 +262,22 @@ return object_sz; } +int64_t +MemObject::expectedReplySize() const { + debugs(20, 7, HERE << "object_sz: " << object_sz); + if (object_sz >= 0) // complete() has been called; we know the exact answer + return object_sz; + + if (_reply) { + const int64_t clen = _reply->bodySize(method); + debugs(20, 7, HERE << "clen: " << clen); + if (clen >= 0 && _reply->hdr_sz > 0) // yuck: HttpMsg sets hdr_sz to 0 + return clen + _reply->hdr_sz; + } + + return -1; // not enough information to predict +} + void MemObject::reset() { @@ -341,7 +377,7 @@ * there will be a chunk of the data which is not in memory * and is not yet on disk. * The -1 makes sure the page isn't freed until storeSwapOut has - * walked to the next page. (mem->swapout.memnode) + * walked to the next page. */ int64_t on_disk; === modified file 'src/MemObject.h' --- src/MemObject.h 2010-11-27 06:44:33 +0000 +++ src/MemObject.h 2011-04-28 22:45:55 +0000 @@ -58,12 +58,19 @@ MemObject(char const *, char const *); ~MemObject(); + /// replaces construction-time URLs with correct ones; see hidden_mem_obj + void resetUrls(char const *aUrl, char const *aLog_url); + void write(StoreIOBuffer, STMCB *, void *); void unlinkRequest(); HttpReply const *getReply() const; void replaceHttpReply(HttpReply *newrep); void stat (MemBuf * mb) const; int64_t endOffset () const; + void markEndOfReplyHeaders(); ///< sets _reply->hdr_sz to endOffset() + /// negative if unknown; otherwise, expected object_sz, expected endOffset + /// maximum, and stored reply headers+body size (all three are the same) + int64_t expectedReplySize() const; int64_t size() const; void reset(); int64_t lowestMemReaderOffset() const; @@ -106,9 +113,12 @@ { public: - int64_t queue_offset; /* relative to in-mem data */ - mem_node *memnode; /* which node we're currently paging out */ + int64_t queue_offset; ///< number of bytes sent to SwapDir for writing StoreIOState::Pointer sio; + + /// Decision states for StoreEntry::swapoutPossible() and related code. + typedef enum { swNeedsCheck = 0, swImpossible = -1, swPossible = +1 } Decision; + Decision decision; ///< current decision state }; SwapOut swapout; === added file 'src/MemStore.cc' --- src/MemStore.cc 1970-01-01 00:00:00 +0000 +++ src/MemStore.cc 2011-05-12 03:58:16 +0000 @@ -0,0 +1,377 @@ +/* + * $Id$ + * + * DEBUG: section 20 Memory Cache + * + */ + +#include "config.h" +#include "base/RunnersRegistry.h" +#include "ipc/mem/Page.h" +#include "ipc/mem/Pages.h" +#include "MemObject.h" +#include "MemStore.h" +#include "HttpReply.h" + +/// shared memory segment path to use for MemStore maps +static const char *ShmLabel = "cache_mem"; + +// XXX: support storage using more than one page per entry + +MemStore::MemStore(): map(NULL), theCurrentSize(0) +{ +} + +MemStore::~MemStore() +{ + delete map; +} + +void +MemStore::init() { + const int64_t entryLimit = EntryLimit(); + if (entryLimit <= 0) + return; // no memory cache configured or a misconfiguration + + const int64_t diskMaxSize = Store::Root().maxObjectSize(); + const int64_t memMaxSize = maxObjectSize(); + if (diskMaxSize == -1) { + debugs(20, DBG_IMPORTANT, "WARNING: disk-cache maximum object size " + "is unlimited but mem-cache maximum object size is " << + memMaxSize / 1024.0 << " KB"); + } else if (diskMaxSize > memMaxSize) { + debugs(20, DBG_IMPORTANT, "WARNING: disk-cache maximum object size " + "is too large for mem-cache: " << + diskMaxSize / 1024.0 << " KB > " << + memMaxSize / 1024.0 << " KB"); + } + + map = new MemStoreMap(ShmLabel); + map->cleaner = this; +} + +void +MemStore::stat(StoreEntry &e) const +{ + storeAppendPrintf(&e, "\n\nShared Memory Cache\n"); + + storeAppendPrintf(&e, "Maximum Size: %.0f KB\n", Config.memMaxSize/1024.0); + + if (map) { + const int limit = map->entryLimit(); + storeAppendPrintf(&e, "Maximum entries: %9d\n", limit); + if (limit > 0) { + storeAppendPrintf(&e, "Current entries: %"PRId64" %.2f%%\n", + currentCount(), (100.0 * currentCount() / limit)); + + if (limit < 100) { // XXX: otherwise too expensive to count + Ipc::ReadWriteLockStats stats; + map->updateStats(stats); + stats.dump(e); + } + } + } +} + +void +MemStore::maintain() +{ +} + +uint64_t +MemStore::minSize() const +{ + return 0; // XXX: irrelevant, but Store parent forces us to implement this +} + +uint64_t +MemStore::maxSize() const +{ + return 0; // XXX: make configurable +} + +uint64_t +MemStore::currentSize() const +{ + return theCurrentSize; +} + +uint64_t +MemStore::currentCount() const +{ + return map ? map->entryCount() : 0; +} + +int64_t +MemStore::maxObjectSize() const +{ + return Ipc::Mem::PageSize(); +} + +void +MemStore::reference(StoreEntry &) +{ +} + +bool +MemStore::dereference(StoreEntry &) +{ + // no need to keep e in the global store_table for us; we have our own map + return false; +} + +int +MemStore::callback() +{ + return 0; +} + +StoreSearch * +MemStore::search(String const, HttpRequest *) +{ + fatal("not implemented"); + return NULL; +} + +StoreEntry * +MemStore::get(const cache_key *key) +{ + if (!map) + return NULL; + + // XXX: replace sfileno with a bigger word (sfileno is only for cache_dirs) + sfileno index; + const Ipc::StoreMapSlot *const slot = map->openForReading(key, index); + if (!slot) + return NULL; + + const Ipc::StoreMapSlot::Basics &basics = slot->basics; + const MemStoreMap::Extras &extras = map->extras(index); + + // create a brand new store entry and initialize it with stored info + StoreEntry *e = new StoreEntry(); + e->lock_count = 0; + + e->swap_file_sz = basics.swap_file_sz; + e->lastref = basics.lastref; + e->timestamp = basics.timestamp; + e->expires = basics.expires; + e->lastmod = basics.lastmod; + e->refcount = basics.refcount; + e->flags = basics.flags; + + e->store_status = STORE_OK; + e->mem_status = IN_MEMORY; // setMemStatus(IN_MEMORY) requires mem_obj + //e->swap_status = set in StoreEntry constructor to SWAPOUT_NONE; + e->ping_status = PING_NONE; + + EBIT_SET(e->flags, ENTRY_CACHABLE); + EBIT_CLR(e->flags, RELEASE_REQUEST); + EBIT_CLR(e->flags, KEY_PRIVATE); + EBIT_SET(e->flags, ENTRY_VALIDATED); + + const bool copied = copyFromShm(*e, extras); + + // we copied everything we could to local memory; no more need to lock + map->closeForReading(index); + + if (copied) { + e->hashInsert(key); + return e; + } + + debugs(20, 3, HERE << "mem-loading failed; freeing " << index); + map->free(index); // do not let others into the same trap + return NULL; +} + +void +MemStore::get(String const key, STOREGETCLIENT aCallback, void *aCallbackData) +{ + // XXX: not needed but Store parent forces us to implement this + fatal("MemStore::get(key,callback,data) should not be called"); +} + +bool +MemStore::copyFromShm(StoreEntry &e, const MemStoreMap::Extras &extras) +{ + const Ipc::Mem::PageId &page = extras.page; + + StoreIOBuffer sourceBuf(extras.storedSize, 0, + static_cast(PagePointer(page))); + + // XXX: We do not know the URLs yet, only the key, but we need to parse and + // store the response for the Root().get() callers to be happy because they + // expect IN_MEMORY entries to already have the response headers and body. + // At least one caller calls createMemObject() if there is not one, so + // we hide the true object until that happens (to avoid leaking TBD URLs). + e.createMemObject("TBD", "TBD"); + + // emulate the usual Store code but w/o inapplicable checks and callbacks: + + // from store_client::readBody(): + HttpReply *rep = (HttpReply *)e.getReply(); + const ssize_t end = headersEnd(sourceBuf.data, sourceBuf.length); + if (!rep->parseCharBuf(sourceBuf.data, end)) { + debugs(20, DBG_IMPORTANT, "Could not parse mem-cached headers: " << e); + return false; + } + // local memory stores both headers and body + e.mem_obj->object_sz = sourceBuf.length; // from StoreEntry::complete() + + storeGetMemSpace(sourceBuf.length); // from StoreEntry::write() + + assert(e.mem_obj->data_hdr.write(sourceBuf)); // from MemObject::write() + const int64_t written = e.mem_obj->endOffset(); + // we should write all because StoreEntry::write() never fails + assert(written >= 0 && + static_cast(written) == sourceBuf.length); + // would be nice to call validLength() here, but it needs e.key + + debugs(20, 7, HERE << "mem-loaded all " << written << " bytes of " << e << + " from " << page); + + e.hideMemObject(); + + return true; +} + +void +MemStore::considerKeeping(StoreEntry &e) +{ + if (!e.memoryCachable()) { + debugs(20, 7, HERE << "Not memory cachable: " << e); + return; // cannot keep due to entry state or properties + } + + assert(e.mem_obj); + if (!willFit(e.mem_obj->endOffset())) { + debugs(20, 5, HERE << "No mem-cache space for " << e); + return; // failed to free enough space + } + + keep(e); // may still fail +} + +bool +MemStore::willFit(int64_t need) +{ + // TODO: obey configured maximum entry size (with page-based rounding) + return need <= static_cast(Ipc::Mem::PageSize()); +} + +/// allocates map slot and calls copyToShm to store the entry in shared memory +void +MemStore::keep(StoreEntry &e) +{ + if (!map) { + debugs(20, 5, HERE << "No map to mem-cache " << e); + return; + } + + sfileno index = 0; + Ipc::StoreMapSlot *slot = map->openForWriting(reinterpret_cast(e.key), index); + if (!slot) { + debugs(20, 5, HERE << "No room in mem-cache map to index " << e); + return; + } + + MemStoreMap::Extras &extras = map->extras(index); + if (copyToShm(e, extras)) { + slot->set(e); + map->closeForWriting(index, false); + } else { + map->abortIo(index); + } +} + +/// uses mem_hdr::copy() to copy local data to shared memory +bool +MemStore::copyToShm(StoreEntry &e, MemStoreMap::Extras &extras) +{ + Ipc::Mem::PageId page; + if (!Ipc::Mem::GetPage(page)) { + debugs(20, 5, HERE << "No mem-cache page for " << e); + return false; // GetPage is responsible for any cleanup on failures + } + + const int64_t bufSize = Ipc::Mem::PageSize(); + const int64_t eSize = e.mem_obj->endOffset(); + + StoreIOBuffer sharedSpace(bufSize, 0, + static_cast(PagePointer(page))); + + // check that we kept everything or purge incomplete/sparse cached entry + const ssize_t copied = e.mem_obj->data_hdr.copy(sharedSpace); + if (eSize != copied) { + debugs(20, 2, HERE << "Failed to mem-cache " << e << ": " << + eSize << "!=" << copied); + // cleanup + PutPage(page); + return false; + } + + debugs(20, 7, HERE << "mem-cached all " << eSize << " bytes of " << e << + " in " << page); + + theCurrentSize += Ipc::Mem::PageSize(); + // remember storage location and size + extras.page = page; + extras.storedSize = copied; + return true; +} + +void +MemStore::cleanReadable(const sfileno fileno) +{ + Ipc::Mem::PutPage(map->extras(fileno).page); + theCurrentSize -= Ipc::Mem::PageSize(); +} + +/// calculates maximum number of entries we need to store and map +int64_t +MemStore::EntryLimit() +{ + if (!Config.memMaxSize) + return 0; // no memory cache configured + + const int64_t entrySize = Ipc::Mem::PageSize(); // for now + const int64_t entryLimit = Config.memMaxSize / entrySize; + return entryLimit; +} + + +/// initializes shared memory segments used by MemStore +class MemStoreRr: public RegisteredRunner +{ +public: + /* RegisteredRunner API */ + MemStoreRr(): owner(NULL) {} + virtual void run(const RunnerRegistry &); + virtual ~MemStoreRr(); + +private: + MemStoreMap::Owner *owner; +}; + +RunnerRegistrationEntry(rrAfterConfig, MemStoreRr); + + +void MemStoreRr::run(const RunnerRegistry &) +{ + if (!UsingSmp()) + return; + + if (IamMasterProcess()) { + Must(!owner); + const int64_t entryLimit = MemStore::EntryLimit(); + if (entryLimit <= 0) + return; // no memory cache configured or a misconfiguration + owner = MemStoreMap::Init(ShmLabel, entryLimit); + } +} + +MemStoreRr::~MemStoreRr() +{ + delete owner; +} === added file 'src/MemStore.h' --- src/MemStore.h 1970-01-01 00:00:00 +0000 +++ src/MemStore.h 2011-05-12 03:58:16 +0000 @@ -0,0 +1,68 @@ +/* + * $Id$ + */ + +#ifndef SQUID_MEMSTORE_H +#define SQUID_MEMSTORE_H + +#include "ipc/mem/Page.h" +#include "ipc/StoreMap.h" +#include "Store.h" + +// StoreEntry restoration info not already stored by Ipc::StoreMap +struct MemStoreMapExtras { + Ipc::Mem::PageId page; ///< shared memory page with the entry content + int64_t storedSize; ///< total size of the stored entry content +}; +typedef Ipc::StoreMapWithExtras MemStoreMap; + +/// Stores HTTP entities in RAM. Current implementation uses shared memory. +/// Unlike a disk store (SwapDir), operations are synchronous (and fast). +class MemStore: public Store, public Ipc::StoreMapCleaner { +public: + MemStore(); + virtual ~MemStore(); + + /// cache the entry or forget about it until the next considerKeeping call + void considerKeeping(StoreEntry &e); + + /* Store API */ + virtual int callback(); + virtual StoreEntry * get(const cache_key *); + virtual void get(String const key , STOREGETCLIENT callback, void *cbdata); + virtual void init(); + virtual uint64_t maxSize() const; + virtual uint64_t minSize() const; + virtual uint64_t currentSize() const; + virtual uint64_t currentCount() const; + virtual int64_t maxObjectSize() const; + virtual void stat(StoreEntry &) const; + virtual StoreSearch *search(String const url, HttpRequest *); + virtual void reference(StoreEntry &); + virtual bool dereference(StoreEntry &); + virtual void maintain(); + + static int64_t EntryLimit(); + +protected: + bool willFit(int64_t needed); + void keep(StoreEntry &e); + + bool copyToShm(StoreEntry &e, MemStoreMap::Extras &extras); + bool copyFromShm(StoreEntry &e, const MemStoreMap::Extras &extras); + + // Ipc::StoreMapCleaner API + virtual void cleanReadable(const sfileno fileno); + +private: + MemStoreMap *map; ///< index of mem-cached entries + uint64_t theCurrentSize; ///< currently used space in the storage area +}; + +// Why use Store as a base? MemStore and SwapDir are both "caches". + +// Why not just use a SwapDir API? That would not help much because Store has +// to check/update memory cache separately from the disk cache. And same API +// would hurt because we can support synchronous get/put, unlike the disks. + +#endif /* SQUID_MEMSTORE_H */ === modified file 'src/Server.cc' --- src/Server.cc 2011-03-11 23:02:23 +0000 +++ src/Server.cc 2011-04-28 22:45:55 +0000 @@ -162,8 +162,10 @@ assert(rep); theFinalReply = HTTPMSGLOCK(rep); - entry->replaceHttpReply(theFinalReply); - haveParsedReplyHeaders(); + // give entry the reply because haveParsedReplyHeaders() expects it there + entry->replaceHttpReply(theFinalReply, false); // but do not write yet + haveParsedReplyHeaders(); // update the entry/reply (e.g., set timestamps) + entry->startWriting(); // write the updated entry to store return theFinalReply; } @@ -381,10 +383,10 @@ } if (io.flag) { - debugs(11, 1, "sentRequestBody error: FD " << io.fd << ": " << xstrerr(errno)); + debugs(11, 1, "sentRequestBody error: FD " << io.fd << ": " << xstrerr(io.xerrno)); ErrorState *err; err = errorCon(ERR_WRITE_ERROR, HTTP_BAD_GATEWAY, fwd->request); - err->xerrno = errno; + err->xerrno = io.xerrno; fwd->fail(err); abortTransaction("I/O error while sending request body"); return; @@ -526,7 +528,7 @@ purgeEntriesByHeader(request, reqUrl, theFinalReply, HDR_CONTENT_LOCATION); } -// called (usually by kids) when we have final (possibly adapted) reply headers +/// called when we have final (possibly adapted) reply headers; kids extend void ServerStateData::haveParsedReplyHeaders() { === modified file 'src/Store.h' --- src/Store.h 2010-12-04 13:10:19 +0000 +++ src/Store.h 2011-05-12 03:58:16 +0000 @@ -57,8 +57,8 @@ class AsyncCall; class StoreClient; class MemObject; -class Store; class StoreSearch; +class SwapDir; typedef struct { @@ -85,7 +85,7 @@ virtual const char *getMD5Text() const; StoreEntry(); StoreEntry(const char *url, const char *log_url); - virtual ~StoreEntry() {} + virtual ~StoreEntry(); virtual HttpReply const *getReply() const; virtual void write (StoreIOBuffer); @@ -95,7 +95,8 @@ virtual void complete(); virtual store_client_t storeClientType() const; virtual char const *getSerialisedMetaData(); - virtual void replaceHttpReply(HttpReply *); + void replaceHttpReply(HttpReply *, bool andStartWriting = true); + void startWriting(); ///< pack and write reply headers and, maybe, body virtual bool swapoutPossible(); virtual void trimMemory(); void abort(); @@ -110,16 +111,18 @@ void cacheNegatively(); /** \todo argh, why both? */ void invokeHandlers(); void purgeMem(); + void cacheInMemory(); ///< start or continue storing in memory cache void swapOut(); bool swapOutAble() const; - void swapOutFileClose(); + void swapOutFileClose(int how); const char *url() const; int checkCachable(); int checkNegativeHit() const; int locked() const; int validToSend() const; - int keepInMemory() const; + bool memoryCachable() const; ///< may be cached in memory void createMemObject(const char *, const char *); + void hideMemObject(); ///< no mem_obj for callers until createMemObject void dump(int debug_lvl) const; void hashDelete(); void hashInsert(const cache_key *); @@ -141,9 +144,10 @@ bool hasIfNoneMatchEtag(const HttpRequest &request) const; /** What store does this entry belong too ? */ - virtual RefCount store() const; + virtual RefCount store() const; MemObject *mem_obj; + MemObject *hidden_mem_obj; ///< mem_obj created before URLs were known RemovalPolicyNode repl; /* START OF ON-DISK STORE_META_STD TLV field */ time_t timestamp; @@ -208,6 +212,8 @@ bool hasOneOfEtags(const String &reqETags, const bool allowWeakMatch) const; }; +std::ostream &operator <<(std::ostream &os, const StoreEntry &e); + /// \ingroup StoreAPI class NullStoreEntry:public StoreEntry { @@ -295,6 +301,15 @@ /** The minimum size the store will shrink to via normal housekeeping */ virtual uint64_t minSize() const = 0; + /** current store size */ + virtual uint64_t currentSize() const = 0; + + /** the total number of objects stored */ + virtual uint64_t currentCount() const = 0; + + /** the maximum object size that can be stored, -1 if unlimited */ + virtual int64_t maxObjectSize() const = 0; + /** * Output stats to the provided store entry. \todo make these calls asynchronous @@ -313,12 +328,15 @@ /* pulled up from SwapDir for migration.... probably do not belong here */ virtual void reference(StoreEntry &) = 0; /* Reference this object */ - virtual void dereference(StoreEntry &) = 0; /* Unreference this object */ + /// Undo reference(), returning false iff idle e should be destroyed + virtual bool dereference(StoreEntry &e) = 0; virtual void maintain() = 0; /* perform regular maintenance should be private and self registered ... */ - /* These should really be private */ - virtual void updateSize(int64_t size, int sign) = 0; + // XXX: This method belongs to Store::Root/StoreController, but it is here + // because test cases use non-StoreController derivatives as Root + /// called when the entry is no longer needed by any transaction + virtual void handleIdleEntry(StoreEntry &e) {} private: static RefCount CurrentRoot; === modified file 'src/StoreHashIndex.h' --- src/StoreHashIndex.h 2010-12-04 01:41:43 +0000 +++ src/StoreHashIndex.h 2011-05-12 03:58:16 +0000 @@ -67,21 +67,26 @@ virtual uint64_t minSize() const; + virtual uint64_t currentSize() const; + + virtual uint64_t currentCount() const; + + virtual int64_t maxObjectSize() const; + virtual void stat(StoreEntry&) const; virtual void reference(StoreEntry&); - virtual void dereference(StoreEntry&); + virtual bool dereference(StoreEntry&); virtual void maintain(); - virtual void updateSize(int64_t, int); - virtual StoreSearch *search(String const url, HttpRequest *); private: /* migration logic */ StorePointer store(int const x) const; + SwapDir &dir(int const idx) const; }; class StoreHashIndexEntry : public StoreEntry === modified file 'src/StoreIOState.h' --- src/StoreIOState.h 2009-01-21 03:47:47 +0000 +++ src/StoreIOState.h 2011-02-15 04:02:28 +0000 @@ -84,13 +84,19 @@ virtual void read_(char *buf, size_t size, off_t offset, STRCB * callback, void *callback_data) = 0; virtual void write(char const *buf, size_t size, off_t offset, FREE * free_func) = 0; - virtual void close() = 0; + + typedef enum { + wroteAll, ///< success: caller supplied all data it wanted to swap out + writerGone, ///< failure: caller left before swapping out everything + readerDone ///< success or failure: either way, stop swapping in + } CloseHow; + virtual void close(int how) = 0; ///< finish or abort swapping per CloseHow sdirno swap_dirn; sfileno swap_filen; StoreEntry *e; /* Need this so the FS layers can play god */ mode_t mode; - off_t offset_; /* current on-disk offset pointer */ + off_t offset_; ///< number of bytes written or read for this entry so far STFNCB *file_callback; /* called on delayed sfileno assignments */ STIOCB *callback; void *callback_data; @@ -107,7 +113,7 @@ StoreIOState::Pointer storeCreate(StoreEntry *, StoreIOState::STFNCB *, StoreIOState::STIOCB *, void *); StoreIOState::Pointer storeOpen(StoreEntry *, StoreIOState::STFNCB *, StoreIOState::STIOCB *, void *); -SQUIDCEXTERN void storeClose(StoreIOState::Pointer); +SQUIDCEXTERN void storeClose(StoreIOState::Pointer, int how); SQUIDCEXTERN void storeRead(StoreIOState::Pointer, char *, size_t, off_t, StoreIOState::STRCB *, void *); SQUIDCEXTERN void storeIOWrite(StoreIOState::Pointer, char const *, size_t, off_t, FREE *); === modified file 'src/StoreMetaUnpacker.cc' --- src/StoreMetaUnpacker.cc 2010-12-13 11:31:14 +0000 +++ src/StoreMetaUnpacker.cc 2011-01-27 21:14:56 +0000 @@ -39,6 +39,24 @@ int const StoreMetaUnpacker::MinimumBufferLength = sizeof(char) + sizeof(int); +/// useful for meta stored in pre-initialized (with zeros) db files +bool +StoreMetaUnpacker::isBufferZero() +{ + // We could memcmp the entire buffer, but it is probably safe enough + // to test a few bytes because if we do not detect a corrupted entry + // it is not a big deal. Empty entries are not isBufferSane anyway. + const int depth = 10; + if (buflen < depth) + return false; // cannot be sure enough + + for (int i = 0; i < depth; ++i) { + if (buf[i]) + return false; + } + return true; +} + bool StoreMetaUnpacker::isBufferSane() { === modified file 'src/StoreMetaUnpacker.h' --- src/StoreMetaUnpacker.h 2010-11-21 04:40:05 +0000 +++ src/StoreMetaUnpacker.h 2011-01-27 21:14:56 +0000 @@ -41,6 +41,7 @@ public: StoreMetaUnpacker (const char *buf, ssize_t bufferLength, int *hdrlen); StoreMeta *createStoreMeta(); + bool isBufferZero(); ///< all-zeros buffer, implies !isBufferSane bool isBufferSane(); private: === modified file 'src/SwapDir.cc' --- src/SwapDir.cc 2010-12-14 14:01:14 +0000 +++ src/SwapDir.cc 2011-05-12 03:58:16 +0000 @@ -38,8 +38,18 @@ #include "StoreFileSystem.h" #include "ConfigOption.h" +SwapDir::SwapDir(char const *aType): theType(aType), + max_size(0), + path(NULL), index(-1), min_objsize(0), max_objsize (-1), + repl(NULL), removals(0), scanned(0), + cleanLog(NULL) +{ + fs.blksize = 1024; +} + SwapDir::~SwapDir() { + // TODO: should we delete repl? xfree(path); } @@ -61,6 +71,9 @@ void SwapDir::stat(StoreEntry &output) const { + if (!doReportStat()) + return; + storeAppendPrintf(&output, "Store Directory #%d (%s): %s\n", index, type(), path); storeAppendPrintf(&output, "FS Block Size %d Bytes\n", @@ -90,8 +103,11 @@ void SwapDir::reference(StoreEntry &) {} -void -SwapDir::dereference(StoreEntry &) {} +bool +SwapDir::dereference(StoreEntry &) +{ + return true; // keep in global store_table +} int SwapDir::callback() @@ -99,6 +115,30 @@ return 0; } +bool +SwapDir::canStore(const StoreEntry &e, int64_t diskSpaceNeeded, int &load) const +{ + debugs(47,8, HERE << "cache_dir[" << index << "]: needs " << + diskSpaceNeeded << " maxSize()) + return false; // already overflowing + + /* Return 999 (99.9%) constant load; TODO: add a named constant for this */ + load = 999; + return true; // kids may provide more tests and should report true load +} + + void SwapDir::sync() {} @@ -151,6 +191,25 @@ return theType; } +bool +SwapDir::active() const +{ + if (IamWorkerProcess()) + return true; + + // we are inside a disker dedicated to this disk + if (IamDiskProcess() && index == (KidIdentifier-1 - Config.workers)) + return true; + + return false; // Coordinator, wrong disker, etc. +} + +bool +SwapDir::needsDiskStrand() const +{ + return false; +} + /* NOT performance critical. Really. Don't bother optimising for speed * - RBC 20030718 */ @@ -269,13 +328,11 @@ storeAppendPrintf(e, " max-size=%"PRId64, max_objsize); } -/* Swapdirs do not have an index of their own - thus they ask their parent.. - * but the parent child relationship isn't implemented yet - */ +// some SwapDirs may maintain their indexes and be able to lookup an entry key StoreEntry * SwapDir::get(const cache_key *key) { - return Store::Root().get(key); + return NULL; } void === modified file 'src/SwapDir.h' --- src/SwapDir.h 2010-12-07 09:14:33 +0000 +++ src/SwapDir.h 2011-05-12 03:58:16 +0000 @@ -37,14 +37,14 @@ /* forward decls */ class RemovalPolicy; +class MemStore; /* Store dir configuration routines */ /* SwapDir *sd, char *path ( + char *opt later when the strtok mess is gone) */ class ConfigOption; -/* New class that replaces the static SwapDir methods as part of the Store overhaul */ - +/// hides memory/disk cache distinction from callers class StoreController : public Store { @@ -58,6 +58,9 @@ virtual void get(String const, STOREGETCLIENT, void * cbdata); + /* Store parent API */ + virtual void handleIdleEntry(StoreEntry &e); + virtual void init(); virtual void maintain(); /* perform regular maintenance should be private and self registered ... */ @@ -66,6 +69,12 @@ virtual uint64_t minSize() const; + virtual uint64_t currentSize() const; + + virtual uint64_t currentCount() const; + + virtual int64_t maxObjectSize() const; + virtual void stat(StoreEntry &) const; virtual void sync(); /* Sync the store prior to shutdown */ @@ -74,9 +83,7 @@ virtual void reference(StoreEntry &); /* Reference this object */ - virtual void dereference(StoreEntry &); /* Unreference this object */ - - virtual void updateSize(int64_t size, int sign); + virtual bool dereference(StoreEntry &); /* Unreference this object */ /* the number of store dirs being rebuilt. */ static int store_dirs_rebuilding; @@ -84,7 +91,8 @@ private: void createOneStore(Store &aStore); - StorePointer swapDir; + StorePointer swapDir; ///< summary view of all disk caches + MemStore *memStore; ///< memory cache }; /* migrating from the Config based list of swapdirs */ @@ -108,20 +116,23 @@ SQUIDCEXTERN int storeDirGetBlkSize(const char *path, int *blksize); SQUIDCEXTERN int storeDirGetUFSStats(const char *, int *, int *, int *, int *); - +/// manages a single cache_dir class SwapDir : public Store { public: - SwapDir(char const *aType) : theType (aType), cur_size(0), max_size(0), min_objsize(0), max_objsize(-1), cleanLog(NULL) { - fs.blksize = 1024; - path = NULL; - } + typedef RefCount Pointer; + SwapDir(char const *aType); virtual ~SwapDir(); virtual void reconfigure(int, char *) = 0; char const *type() const; + virtual bool needsDiskStrand() const; ///< needs a dedicated kid process + virtual bool active() const; ///< may be used in this strand + /// whether stat should be reported by this SwapDir + virtual bool doReportStat() const { return active(); } + /* official Store interface functions */ virtual void diskFull(); @@ -132,19 +143,28 @@ virtual uint64_t maxSize() const { return max_size;} virtual uint64_t minSize() const; + + virtual int64_t maxObjectSize() const { return max_objsize; } + virtual void stat (StoreEntry &anEntry) const; virtual StoreSearch *search(String const url, HttpRequest *) = 0; - virtual void updateSize(int64_t size, int sign); - /* migrated from store_dir.cc */ bool objectSizeIsAcceptable(int64_t objsize) const; + /// called when the entry is about to forget its association with cache_dir + virtual void disconnect(StoreEntry &) {} + + /// called when entry swap out is complete + virtual void swappedOut(const StoreEntry &e) = 0; + protected: void parseOptions(int reconfiguring); void dumpOptions(StoreEntry * e) const; virtual ConfigOption *getOptionTree() const; + int64_t sizeInBlocks(const int64_t size) const { return (size + fs.blksize - 1) / fs.blksize; } + private: bool optionReadOnlyParse(char const *option, const char *value, int reconfiguring); void optionReadOnlyDump(StoreEntry * e) const; @@ -152,9 +172,10 @@ void optionObjectSizeDump(StoreEntry * e) const; char const *theType; -public: - uint64_t cur_size; ///< currently used space in the storage area +protected: uint64_t max_size; ///< maximum allocatable size of the storage area + +public: char *path; int index; /* This entry's index into the swapDirs array */ int64_t min_objsize; @@ -174,11 +195,11 @@ virtual bool doubleCheck(StoreEntry &); /* Double check the obj integrity */ virtual void statfs(StoreEntry &) const; /* Dump fs statistics */ virtual void maintain(); /* Replacement maintainence */ - /* <0 == error. > 1000 == error */ - virtual int canStore(StoreEntry const &)const = 0; /* Check if the fs will store an object */ + /// check whether we can store the entry; if we can, report current load + virtual bool canStore(const StoreEntry &e, int64_t diskSpaceNeeded, int &load) const = 0; /* These two are notifications */ virtual void reference(StoreEntry &); /* Reference this object */ - virtual void dereference(StoreEntry &); /* Unreference this object */ + virtual bool dereference(StoreEntry &); /* Unreference this object */ virtual int callback(); /* Handle pending callbacks */ virtual void sync(); /* Sync the store prior to shutdown */ virtual StoreIOState::Pointer createStoreIO(StoreEntry &, StoreIOState::STFNCB *, StoreIOState::STIOCB *, void *) = 0; === modified file 'src/adaptation/icap/Elements.cc' --- src/adaptation/icap/Elements.cc 2010-09-12 22:35:35 +0000 +++ src/adaptation/icap/Elements.cc 2011-01-27 21:16:49 +0000 @@ -8,6 +8,7 @@ { const XactOutcome xoUnknown = "ICAP_ERR_UNKNOWN"; +const XactOutcome xoGone = "ICAP_ERR_GONE"; const XactOutcome xoRace = "ICAP_ERR_RACE"; const XactOutcome xoError = "ICAP_ERR_OTHER"; const XactOutcome xoOpt = "ICAP_OPT"; === modified file 'src/adaptation/icap/Elements.h' --- src/adaptation/icap/Elements.h 2010-09-12 22:35:35 +0000 +++ src/adaptation/icap/Elements.h 2011-01-27 21:16:49 +0000 @@ -64,6 +64,7 @@ typedef const char *XactOutcome; ///< transaction result for logging extern const XactOutcome xoUnknown; ///< initial value: outcome was not set +extern const XactOutcome xoGone; ///< initiator gone, will not continue extern const XactOutcome xoRace; ///< ICAP server closed pconn when we started extern const XactOutcome xoError; ///< all kinds of transaction errors extern const XactOutcome xoOpt; ///< OPTION transaction === modified file 'src/adaptation/icap/ServiceRep.h' --- src/adaptation/icap/ServiceRep.h 2011-03-08 23:56:22 +0000 +++ src/adaptation/icap/ServiceRep.h 2011-04-14 16:58:28 +0000 @@ -92,6 +92,8 @@ virtual void finalize(); + void invalidate(); // call when the service is no longer needed or valid + virtual bool probed() const; // see comments above virtual bool up() const; // see comments above === modified file 'src/adaptation/icap/Xaction.cc' --- src/adaptation/icap/Xaction.cc 2011-04-07 12:42:02 +0000 +++ src/adaptation/icap/Xaction.cc 2011-04-14 16:58:28 +0000 @@ -196,10 +196,14 @@ theService->cfg().port, NULL, client_addr); disableRetries(); } else { - //status() adds leading spaces. - debugs(93,3, HERE << "closing pconn" << status()); + const bool reset = al.icap.outcome == xoGone || al.icap.outcome == xoError; + debugs(93,3, HERE << (reset ? "RST" : "FIN") << "-closing" << + status()); // comm_close will clear timeout - comm_close(connection); + if (reset) + comm_reset_close(connection); + else + comm_close(connection); } writer = NULL; @@ -443,8 +447,10 @@ { if (theInitiator.set()) { + debugs(93,4, HERE << "Initiator gone before ICAP transaction ended"); clearInitiator(); detailError(ERR_DETAIL_ICAP_INIT_GONE); + setOutcome(xoGone); mustStop("initiator aborted"); } === modified file 'src/base/Makefile.am' --- src/base/Makefile.am 2010-11-18 08:01:53 +0000 +++ src/base/Makefile.am 2011-04-14 04:22:25 +0000 @@ -15,6 +15,8 @@ TidyPointer.h \ CbcPointer.h \ InstanceId.h \ + RunnersRegistry.cc \ + RunnersRegistry.h \ Subscription.h \ TextException.cc \ TextException.h === added file 'src/base/RunnersRegistry.cc' --- src/base/RunnersRegistry.cc 1970-01-01 00:00:00 +0000 +++ src/base/RunnersRegistry.cc 2011-04-14 04:22:25 +0000 @@ -0,0 +1,58 @@ +#include "config.h" +#include "base/RunnersRegistry.h" +#include +#include + +typedef std::list Runners; +typedef std::map Registries; + +/// all known registries +static Registries *TheRegistries = NULL; + +/// returns the requested runners list, initializing structures as needed +static Runners & +GetRunners(const RunnerRegistry ®istryId) +{ + if (!TheRegistries) + TheRegistries = new Registries; + + if (TheRegistries->find(registryId) == TheRegistries->end()) + (*TheRegistries)[registryId] = new Runners; + + return *(*TheRegistries)[registryId]; +} + +int +RegisterRunner(const RunnerRegistry ®istryId, RegisteredRunner *rr) +{ + Runners &runners = GetRunners(registryId); + runners.push_back(rr); + return runners.size(); +} + +int +ActivateRegistered(const RunnerRegistry ®istryId) +{ + Runners &runners = GetRunners(registryId); + typedef Runners::iterator RRI; + for (RRI i = runners.begin(); i != runners.end(); ++i) + (*i)->run(registryId); + return runners.size(); +} + +void +DeactivateRegistered(const RunnerRegistry ®istryId) +{ + Runners &runners = GetRunners(registryId); + typedef Runners::iterator RRI; + while (!runners.empty()) { + delete runners.back(); + runners.pop_back(); + } +} + +bool +UseThisStatic(const void *) +{ + return true; +} === added file 'src/base/RunnersRegistry.h' --- src/base/RunnersRegistry.h 1970-01-01 00:00:00 +0000 +++ src/base/RunnersRegistry.h 2011-04-14 04:22:25 +0000 @@ -0,0 +1,49 @@ +#ifndef SQUID_BASE_RUNNERSREGISTRY_H +#define SQUID_BASE_RUNNERSREGISTRY_H + +/** + * This API allows different modules to register with a well-known registry, + * be activated by some central processor at some registry-specific time, and + * be deactiveated by some central processor at some registry-specific time. + * + * For example, main.cc may activate registered I/O modules after parsing + * squid.conf and deactivate them before exiting. + * + */ + +/// well-known registries (currently, deactivation is not performed for these) +typedef enum { + rrAfterConfig, ///< activated by main.cc after parsing squid.conf + rrEnd ///< not a real registry, just a label to mark the end of enum +} RunnerRegistry; + +/// a runnable registrant API +class RegisteredRunner { +public: + // called when this runner's registry is deactivated + virtual ~RegisteredRunner() {} + + // called when this runner's registry is activated + virtual void run(const RunnerRegistry &r) = 0; +}; + + +/// registers a given runner with the given registry and returns registry count +int RegisterRunner(const RunnerRegistry ®istry, RegisteredRunner *rr); + +/// calls run() methods of all runners in the given registry +int ActivateRegistered(const RunnerRegistry ®istry); +/// deletes all runners in the given registry +void DeactivateRegistered(const RunnerRegistry ®istry); + + +/// convenience function to "use" an otherwise unreferenced static variable +bool UseThisStatic(const void *); + +/// convenience macro: register one RegisteredRunner kid as early as possible +#define RunnerRegistrationEntry(Registry, Who) \ + static const bool Who ## _RegisteredWith_ ## Registry = \ + RegisterRunner(Registry, new Who) > 0 && \ + UseThisStatic(& Who ## _RegisteredWith_ ## Registry); + +#endif /* SQUID_BASE_RUNNERSREGISTRY_H */ === modified file 'src/cache_cf.cc' --- src/cache_cf.cc 2011-04-11 03:25:32 +0000 +++ src/cache_cf.cc 2011-04-27 23:34:13 +0000 @@ -608,7 +608,7 @@ if (0 == Store::Root().maxSize()) /* people might want a zero-sized cache on purpose */ (void) 0; - else if (Store::Root().maxSize() < (Config.memMaxSize >> 10)) + else if (Store::Root().maxSize() < Config.memMaxSize) /* This is bogus. folk with NULL caches will want this */ debugs(3, 0, "WARNING cache_mem is larger than total disk cache space!"); @@ -1908,18 +1908,6 @@ static void parse_cachedir(SquidConfig::_cacheSwap * swap) { - // The workers option must preceed cache_dir for the IamWorkerProcess check - // below to work. TODO: Redo IamWorkerProcess to work w/o Config and remove - if (KidIdentifier > 1 && Config.workers == 1) { - debugs(3, DBG_CRITICAL, - "FATAL: cache_dir found before the workers option. Reorder."); - self_destruct(); - } - - // Among all processes, only workers may need and can handle cache_dir. - if (!IamWorkerProcess()) - return; - char *type_str; char *path_str; RefCount sd; @@ -1986,6 +1974,9 @@ ++swap->n_configured; + if (sd->needsDiskStrand()) + ++swap->n_strands; + /* Update the max object size */ update_maxobjsize(); } === modified file 'src/cf.data.pre' --- src/cf.data.pre 2011-04-05 20:57:57 +0000 +++ src/cf.data.pre 2011-05-11 22:38:35 +0000 @@ -2702,6 +2702,16 @@ higher hit ratio at the expense of an increase in response time. + The rock store type: + + cache_dir rock Directory-Name Mbytes + + The Rock Store type is a database-style storage. All cached + entries are stored in a "database" file, using fixed-size slots, + one entry per slot. The database size is specified in MB. The + slot size is specified in bytes using the max-size option. See + below for more info on the max-size option. + The coss store type: NP: COSS filesystem in Squid-3 has been deemed too unstable for @@ -2852,8 +2862,9 @@ ' output as-is - left aligned - width field width. If starting with 0 the - output is zero padded + width minimum and/or maximum field width: [min][.max] + When minimum starts with 0, the field is zero-padded. + String values exceeding maximum width are truncated. {arg} argument such as header name etc Format codes: @@ -4019,8 +4030,8 @@ DOC_END NAME: store_avg_object_size -COMMENT: (kbytes) -TYPE: kb_int64_t +COMMENT: (bytes) +TYPE: b_int64_t DEFAULT: 13 KB LOC: Config.Store.avgObjectSize DOC_START @@ -6516,11 +6527,12 @@ returning a chain of services to be used next. The services are specified using the X-Next-Services ICAP response header value, formatted as a comma-separated list of service names. - Each named service should be configured in squid.conf and - should have the same method and vectoring point as the current - ICAP transaction. Services violating these rules are ignored. - An empty X-Next-Services value results in an empty plan which - ends the current adaptation. + Each named service should be configured in squid.conf. Other + services are ignored. An empty X-Next-Services value results + in an empty plan which ends the current adaptation. + + Dynamic adaptation plan may cross or cover multiple supported + vectoring points in their natural processing order. Routing is not allowed by default: the ICAP X-Next-Services response header is ignored. === modified file 'src/client_side.cc' --- src/client_side.cc 2011-04-09 04:01:00 +0000 +++ src/client_side.cc 2011-04-14 16:58:28 +0000 @@ -1813,12 +1813,12 @@ void ClientSocketContext::writeComplete(int aFileDescriptor, char *bufnotused, size_t size, comm_err_t errflag) { - StoreEntry *entry = http->storeEntry(); + const StoreEntry *entry = http->storeEntry(); http->out.size += size; assert(aFileDescriptor > -1); debugs(33, 5, "clientWriteComplete: FD " << aFileDescriptor << ", sz " << size << ", err " << errflag << ", off " << http->out.size << ", len " << - entry ? entry->objectLen() : 0); + (entry ? entry->objectLen() : 0)); clientUpdateSocketStats(http->logType, size); assert (this->fd() == aFileDescriptor); @@ -2277,8 +2277,10 @@ int ConnStateData::getAvailableBufferLength() const { - int result = in.allocatedSize - in.notYetUsed - 1; - assert (result >= 0); + assert (in.allocatedSize > in.notYetUsed); // allocated more than used + const size_t result = in.allocatedSize - in.notYetUsed - 1; + // huge request_header_max_size may lead to more than INT_MAX unused space + assert (static_cast(result) <= INT_MAX); return result; } === modified file 'src/client_side_reply.cc' --- src/client_side_reply.cc 2011-03-02 07:27:24 +0000 +++ src/client_side_reply.cc 2011-04-14 16:58:28 +0000 @@ -1043,6 +1043,8 @@ int clientReplyContext::storeOKTransferDone() const { + assert(http->storeEntry()->objectLen() >= 0); + assert(http->storeEntry()->objectLen() >= headers_sz); if (http->out.offset >= http->storeEntry()->objectLen() - headers_sz) { debugs(88,3,HERE << "storeOKTransferDone " << " out.offset=" << http->out.offset << === modified file 'src/comm.cc' --- src/comm.cc 2011-01-28 07:58:53 +0000 +++ src/comm.cc 2011-04-21 15:19:31 +0000 @@ -144,7 +144,7 @@ bool isOpen(const int fd) { - return fd >= 0 && fd_table[fd].flags.open != 0; + return fd >= 0 && fd_table && fd_table[fd].flags.open != 0; } /** === modified file 'src/disk.cc' --- src/disk.cc 2011-01-28 07:58:53 +0000 +++ src/disk.cc 2011-04-14 16:58:28 +0000 @@ -231,12 +231,14 @@ assert(fdd->write_q->len > fdd->write_q->buf_offset); - debugs(6, 3, "diskHandleWrite: FD " << fd << " writing " << (fdd->write_q->len - fdd->write_q->buf_offset) << " bytes"); + debugs(6, 3, "diskHandleWrite: FD " << fd << " writing " << + (fdd->write_q->len - fdd->write_q->buf_offset) << " bytes at " << + fdd->write_q->file_offset); errno = 0; if (fdd->write_q->file_offset != -1) - lseek(fd, fdd->write_q->file_offset, SEEK_SET); + lseek(fd, fdd->write_q->file_offset, SEEK_SET); /* XXX ignore return? */ len = FD_WRITE_METHOD(fd, fdd->write_q->buf + fdd->write_q->buf_offset, @@ -433,7 +435,11 @@ PROF_start(diskHandleRead); +#if WRITES_MAINTAIN_DISK_OFFSET if (F->disk.offset != ctrl_dat->offset) { +#else + { +#endif debugs(6, 3, "diskHandleRead: FD " << fd << " seeking to offset " << ctrl_dat->offset); lseek(fd, ctrl_dat->offset, SEEK_SET); /* XXX ignore return? */ statCounter.syscalls.disk.seeks++; === modified file 'src/fs/coss/CossSwapDir.h' --- src/fs/coss/CossSwapDir.h 2008-11-11 10:38:40 +0000 +++ src/fs/coss/CossSwapDir.h 2011-04-28 12:23:55 +0000 @@ -37,7 +37,7 @@ virtual StoreSearch *search(String const url, HttpRequest *); virtual void unlink (StoreEntry &); virtual void statfs (StoreEntry &)const; - virtual int canStore(StoreEntry const &)const; + virtual bool canStore(const StoreEntry &e, int64_t diskSpaceNeeded, int &load) const; virtual int callback(); virtual void sync(); virtual StoreIOState::Pointer createStoreIO(StoreEntry &, StoreIOState::STFNCB *, StoreIOState::STIOCB *, void *); @@ -49,6 +49,9 @@ virtual void logEntry(const StoreEntry & e, int op) const; virtual void parse (int index, char *path); virtual void reconfigure (int, char *); + virtual void swappedOut(const StoreEntry &e); + virtual uint64_t currentSize() const { return cur_size; } + virtual uint64_t currentCount() const { return n_disk_objects; } /* internals */ virtual off_t storeCossFilenoToDiskOffset(sfileno); virtual sfileno storeCossDiskOffsetToFileno(off_t); @@ -76,6 +79,16 @@ CossMemBuf *createMemBuf(off_t start, sfileno curfn, int *collision); sfileno allocate(const StoreEntry * e, int which); void startMembuf(); + StoreEntry *addDiskRestore(const cache_key *const key, + int file_number, + uint64_t swap_file_sz, + time_t expires, + time_t timestamp, + time_t lastref, + time_t lastmod, + uint32_t refcount, + uint16_t flags, + int clean); private: void changeIO(DiskIOModule *module); @@ -88,6 +101,8 @@ const char *ioModule; mutable ConfigOptionVector *currentIOOptions; const char *stripe_path; + uint64_t cur_size; ///< currently used space in the storage area + uint64_t n_disk_objects; ///< total number of objects stored }; /// \ingroup COSS === modified file 'src/fs/coss/store_coss.h' --- src/fs/coss/store_coss.h 2009-03-31 12:39:30 +0000 +++ src/fs/coss/store_coss.h 2011-04-17 10:27:49 +0000 @@ -79,7 +79,7 @@ off_t st_size; void read_(char *buf, size_t size, off_t offset, STRCB * callback, void *callback_data); void write(char const *buf, size_t size, off_t offset, FREE * free_func); - void close(); + virtual void close(int); void doCallback(int errflag); void lockMemBuf(); === modified file 'src/fs/coss/store_dir_coss.cc' --- src/fs/coss/store_dir_coss.cc 2011-01-28 07:58:53 +0000 +++ src/fs/coss/store_dir_coss.cc 2011-04-28 12:23:55 +0000 @@ -49,7 +49,7 @@ #include "StoreFScoss.h" #include "Parsing.h" #include "swap_log_op.h" -//#include "SquidMath.h" +#include "SquidMath.h" #define STORE_META_BUFSZ 4096 @@ -74,16 +74,6 @@ static char *storeCossDirSwapLogFile(SwapDir *, const char *); static EVH storeCossRebuildFromSwapLog; -static StoreEntry *storeCossAddDiskRestore(CossSwapDir * SD, const cache_key * key, - int file_number, - uint64_t swap_file_sz, - time_t expires, - time_t timestamp, - time_t lastref, - time_t lastmod, - uint32_t refcount, - uint16_t flags, - int clean); static void storeCossDirRebuild(CossSwapDir * sd); static void storeCossDirCloseTmpSwapLog(CossSwapDir * sd); static FILE *storeCossDirOpenTmpSwapLog(CossSwapDir *, int *, int *); @@ -358,7 +348,7 @@ CossIndexNode *coss_node = (CossIndexNode *)e->repl.data; e->repl.data = NULL; dlinkDelete(&coss_node->node, &sd->cossindex); - coss_index_pool->free(coss_node); + coss_index_pool->freeOne(coss_node); sd->count -= 1; } @@ -427,9 +417,8 @@ /* * Make sure we don't unlink the file, it might be * in use by a subsequent entry. Also note that - * we don't have to subtract from store_swap_size - * because adding to store_swap_size happens in - * the cleanup procedure. + * we don't have to subtract from cur_size because + * adding to cur_size happens in the cleanup procedure. */ e->expireNow(); e->releaseRequest(); @@ -481,10 +470,9 @@ continue; } - /* update store_swap_size */ rb->counts.objcount++; - e = storeCossAddDiskRestore(rb->sd, s.key, + e = rb->sd->addDiskRestore(s.key, s.swap_filen, s.swap_file_sz, s.expires, @@ -503,8 +491,8 @@ /* Add a new object to the cache with empty memory copy and pointer to disk * use to rebuild store from disk. */ -static StoreEntry * -storeCossAddDiskRestore(CossSwapDir * SD, const cache_key * key, +StoreEntry * +CossSwapDir::addDiskRestore(const cache_key *const key, int file_number, uint64_t swap_file_sz, time_t expires, @@ -524,7 +512,7 @@ * already in use! */ e = new StoreEntry(); e->store_status = STORE_OK; - e->swap_dirn = SD->index; + e->swap_dirn = index; e->setMemStatus(NOT_IN_MEMORY); e->swap_status = SWAPOUT_DONE; e->swap_filen = file_number; @@ -541,8 +529,10 @@ EBIT_CLR(e->flags, KEY_PRIVATE); e->ping_status = PING_NONE; EBIT_CLR(e->flags, ENTRY_VALIDATED); + cur_size += fs.blksize * sizeInBlocks(e->swap_file_sz); + ++n_disk_objects; e->hashInsert(key); /* do it after we clear KEY_PRIVATE */ - storeCossAdd(SD, e); + storeCossAdd(this, e); assert(e->swap_filen >= 0); return e; } @@ -925,13 +915,12 @@ swap = open(stripePath(), O_WRONLY | O_CREAT | O_TRUNC | O_BINARY, 0600); /* TODO just set the file size */ - /* swap size is in K */ - char *block[1024]; - - memset(&block, '\0', 1024); - - for (off_t offset = 0; offset < max_size; ++offset) { - if (write (swap, block, 1024) < 1024) { + char block[1024]; + Must(maxSize() % sizeof(block) == 0); + memset(block, '\0', sizeof(block)); + + for (uint64_t offset = 0; offset < maxSize(); offset += sizeof(block)) { + if (write (swap, block, sizeof(block)) != sizeof(block)) { debugs (47, 0, "Failed to create COSS swap space in " << path); } } @@ -959,26 +948,14 @@ safe_free(stripe_path); } -/* - * storeCossDirCheckObj - * - * This routine is called by storeDirSelectSwapDir to see if the given - * object is able to be stored on this filesystem. COSS filesystems will - * not store everything. We don't check for maxobjsize here since its - * done by the upper layers. - */ -int -CossSwapDir::canStore(StoreEntry const &e)const +bool +CossSwapDir::canStore(const StoreEntry &e, int64_t diskSpaceNeeded, int &load) const { - - /* Check if the object is a special object, we can't cache these */ - - if (EBIT_TEST(e.flags, ENTRY_SPECIAL)) - return -1; - - /* Otherwise, we're ok */ - /* Return load, cs->aq.aq_numpending out of MAX_ASYNCOP */ - return io->load(); + if (!SwapDir::canStore(e, diskSpaceNeeded, load)) + return false; + + load = io->load(); + return true; } /* @@ -996,10 +973,10 @@ CossSwapDir::statfs(StoreEntry & sentry) const { storeAppendPrintf(&sentry, "\n"); - storeAppendPrintf(&sentry, "Maximum Size: %lu KB\n", max_size); - storeAppendPrintf(&sentry, "Current Size: %lu KB\n", cur_size); + storeAppendPrintf(&sentry, "Maximum Size: %"PRIu64" KB\n", maxSize() >> 10); + storeAppendPrintf(&sentry, "Current Size: %.2f KB\n", currentSize() / 1024.0); storeAppendPrintf(&sentry, "Percent Used: %0.2f%%\n", - (100.0 * (double)cur_size / (double)max_size) ); + Math::doublePercent(currentSize(), maxSize()) ); storeAppendPrintf(&sentry, "Number of object collisions: %d\n", (int) numcollisions); #if 0 /* is this applicable? I Hope not .. */ @@ -1023,21 +1000,15 @@ void CossSwapDir::parse(int anIndex, char *aPath) { - unsigned int i; - unsigned int size; - off_t max_offset; - - i = GetInteger(); - size = i << 10; /* Mbytes to Kbytes */ - - if (size <= 0) + const int i = GetInteger(); + if (i <= 0) fatal("storeCossDirParse: invalid size value"); index = anIndex; path = xstrdup(aPath); - max_size = size; + max_size = i << 20; // MBytes to Bytes parseOptions(0); @@ -1057,12 +1028,12 @@ * largest possible sfileno, assuming sfileno is a 25-bit * signed integer, as defined in structs.h. */ - max_offset = (off_t) 0xFFFFFF << blksz_bits; + const uint64_t max_offset = (uint64_t) 0xFFFFFF << blksz_bits; - if ((off_t)max_size > (max_offset>>10)) { + if (maxSize() > max_offset) { debugs(47, 0, "COSS block-size = " << (1<> 10)); + storeAppendPrintf(&entry, " %"PRIu64, maxSize() >> 20); dumpOptions(&entry); } -CossSwapDir::CossSwapDir() : SwapDir ("coss"), swaplog_fd(-1), count(0), current_membuf (NULL), current_offset(0), numcollisions(0), blksz_bits(0), io (NULL), ioModule(NULL), currentIOOptions(new ConfigOptionVector()), stripe_path(NULL) +CossSwapDir::CossSwapDir() : SwapDir ("coss"), swaplog_fd(-1), count(0), current_membuf (NULL), current_offset(0), numcollisions(0), blksz_bits(0), io (NULL), ioModule(NULL), currentIOOptions(new ConfigOptionVector()), stripe_path(NULL), cur_size(0), n_disk_objects(0) { membufs.head = NULL; membufs.tail = NULL; === modified file 'src/fs/coss/store_io_coss.cc' --- src/fs/coss/store_io_coss.cc 2010-12-13 11:31:14 +0000 +++ src/fs/coss/store_io_coss.cc 2011-04-28 12:23:55 +0000 @@ -83,8 +83,7 @@ allocsize = e->objectLen() + e->mem_obj->swap_hdr_sz; /* Check if we have overflowed the disk .. */ - /* max_size is int, so cast to (off_t) *before* bit-shifting */ - if ((current_offset + allocsize) > ((off_t)max_size << 10)) { + if (current_offset + allocsize > static_cast(maxSize())) { /* * tried to allocate past the end of the disk, so wrap * back to the beginning @@ -134,6 +133,10 @@ CossSwapDir::unlink(StoreEntry & e) { debugs(79, 3, "storeCossUnlink: offset " << e.swap_filen); + if (e.swap_status == SWAPOUT_DONE && EBIT_TEST(e.flags, ENTRY_VALIDATED)) { + cur_size -= fs.blksize * sizeInBlocks(e.swap_file_sz); + --n_disk_objects; + } StoreFScoss::GetInstance().stats.unlink.ops++; StoreFScoss::GetInstance().stats.unlink.success++; storeCossRemove(this, &e); @@ -275,8 +278,9 @@ return sio; } +/// COSS does not distinguish different closure types void -CossState::close() +CossState::close(int) { debugs(79, 3, "storeCossClose: offset " << swap_filen); === modified file 'src/fs/ufs/store_dir_ufs.cc' --- src/fs/ufs/store_dir_ufs.cc 2011-04-07 11:58:46 +0000 +++ src/fs/ufs/store_dir_ufs.cc 2011-05-12 03:58:16 +0000 @@ -57,13 +57,17 @@ * object is able to be stored on this filesystem. UFS filesystems will * happily store anything as long as the LRU time isn't too small. */ -int -UFSSwapDir::canStore(StoreEntry const &e)const +bool +UFSSwapDir::canStore(const StoreEntry &e, int64_t diskSpaceNeeded, int &load) const { + if (!SwapDir::canStore(e, diskSpaceNeeded, load)) + return false; + if (IO->shedLoad()) - return -1; + return false; - return IO->load(); + load = IO->load(); + return true; } @@ -76,14 +80,14 @@ if (i <= 0) fatal("UFSSwapDir::parseSizeL1L2: invalid size value"); - size_t size = i << 10; /* Mbytes to kbytes */ + const uint64_t size = i << 20; // MBytes to Bytes /* just reconfigure it */ if (reconfiguring) { - if (size == max_size) - debugs(3, 2, "Cache dir '" << path << "' size remains unchanged at " << size << " KB"); + if (size == maxSize()) + debugs(3, 2, "Cache dir '" << path << "' size remains unchanged at " << i << " MB"); else - debugs(3, 1, "Cache dir '" << path << "' size changed to " << size << " KB"); + debugs(3, 1, "Cache dir '" << path << "' size changed to " << i << " MB"); } max_size = size; @@ -242,7 +246,7 @@ createSwapSubDirs(); } -UFSSwapDir::UFSSwapDir(char const *aType, const char *anIOType) : SwapDir(aType), IO(NULL), map(file_map_create()), suggest(0), swaplog_fd (-1), currentIOOptions(new ConfigOptionVector()), ioType(xstrdup(anIOType)) +UFSSwapDir::UFSSwapDir(char const *aType, const char *anIOType) : SwapDir(aType), IO(NULL), map(file_map_create()), suggest(0), swaplog_fd (-1), currentIOOptions(new ConfigOptionVector()), ioType(xstrdup(anIOType)), cur_size(0), n_disk_objects(0) { /* modulename is only set to disk modules that are built, by configure, * so the Find call should never return NULL here. @@ -312,10 +316,10 @@ int x; storeAppendPrintf(&sentry, "First level subdirectories: %d\n", l1); storeAppendPrintf(&sentry, "Second level subdirectories: %d\n", l2); - storeAppendPrintf(&sentry, "Maximum Size: %"PRIu64" KB\n", max_size); - storeAppendPrintf(&sentry, "Current Size: %"PRIu64" KB\n", cur_size); + storeAppendPrintf(&sentry, "Maximum Size: %"PRIu64" KB\n", maxSize() >> 10); + storeAppendPrintf(&sentry, "Current Size: %.2f KB\n", currentSize() / 1024.0); storeAppendPrintf(&sentry, "Percent Used: %0.2f%%\n", - (double)(100.0 * cur_size) / (double)max_size); + Math::doublePercent(currentSize(), maxSize())); storeAppendPrintf(&sentry, "Filemap bits in use: %d of %d (%d%%)\n", map->n_files_in_map, map->max_n_files, Math::intPercent(map->n_files_in_map, map->max_n_files)); @@ -361,7 +365,7 @@ RemovalPurgeWalker *walker; - double f = (double) (cur_size - minSize()) / (max_size - minSize()); + double f = (double) (currentSize() - minSize()) / (maxSize() - minSize()); f = f < 0.0 ? 0.0 : f > 1.0 ? 1.0 : f; @@ -378,7 +382,7 @@ walker = repl->PurgeInit(repl, max_scan); while (1) { - if (cur_size < minSize()) + if (currentSize() < minSize()) break; if (removed >= max_remove) @@ -420,13 +424,15 @@ * This routine is called whenever the last reference to an object is * removed, to maintain replacement information within the storage fs. */ -void +bool UFSSwapDir::dereference(StoreEntry & e) { debugs(47, 3, "UFSSwapDir::dereference: referencing " << &e << " " << e.swap_dirn << "/" << e.swap_filen); if (repl->Dereferenced) repl->Dereferenced(repl, &e, &e.repl); + + return true; // keep e in the global store_table } StoreIOState::Pointer @@ -713,6 +719,8 @@ e->ping_status = PING_NONE; EBIT_CLR(e->flags, ENTRY_VALIDATED); mapBitSet(e->swap_filen); + cur_size += fs.blksize * sizeInBlocks(e->swap_file_sz); + ++n_disk_objects; e->hashInsert(key); /* do it after we clear KEY_PRIVATE */ replacementAdd (e); return e; @@ -1284,6 +1292,10 @@ { debugs(79, 3, "storeUfsUnlink: dirno " << index << ", fileno "<< std::setfill('0') << std::hex << std::uppercase << std::setw(8) << e.swap_filen); + if (e.swap_status == SWAPOUT_DONE && EBIT_TEST(e.flags, ENTRY_VALIDATED)) { + cur_size -= fs.blksize * sizeInBlocks(e.swap_file_sz); + --n_disk_objects; + } replacementRemove(&e); mapBitReset(e.swap_filen); UFSSwapDir::unlinkFile(e.swap_filen); @@ -1322,7 +1334,7 @@ void UFSSwapDir::dump(StoreEntry & entry) const { - storeAppendPrintf(&entry, " %"PRIu64" %d %d", (max_size >> 10), l1, l2); + storeAppendPrintf(&entry, " %"PRIu64" %d %d", maxSize() >> 20, l1, l2); dumpOptions(&entry); } @@ -1359,6 +1371,13 @@ IO->sync(); } +void +UFSSwapDir::swappedOut(const StoreEntry &e) +{ + cur_size += fs.blksize * sizeInBlocks(e.swap_file_sz); + ++n_disk_objects; +} + StoreSearch * UFSSwapDir::search(String const url, HttpRequest *request) { === modified file 'src/fs/ufs/store_io_ufs.cc' --- src/fs/ufs/store_io_ufs.cc 2010-08-29 00:12:52 +0000 +++ src/fs/ufs/store_io_ufs.cc 2011-02-15 04:02:28 +0000 @@ -190,11 +190,11 @@ * when it is safe to actually signal the lower layer for closing. */ void -UFSStoreState::close() +UFSStoreState::close(int) { debugs(79, 3, "UFSStoreState::close: dirno " << swap_dirn << ", fileno "<< std::setfill('0') << std::hex << std::uppercase << std::setw(8) << swap_filen); - tryClosing(); + tryClosing(); // UFS does not distinguish different closure types } void === modified file 'src/fs/ufs/ufscommon.cc' --- src/fs/ufs/ufscommon.cc 2010-12-13 11:31:14 +0000 +++ src/fs/ufs/ufscommon.cc 2011-04-18 10:28:07 +0000 @@ -362,63 +362,14 @@ rebuildFromDirectory(); } -struct InitStoreEntry : public unary_function { - InitStoreEntry(StoreEntry *anEntry, cache_key *aKey):what(anEntry),index(aKey) {} - - void operator()(StoreMeta const &x) { - switch (x.getType()) { - - case STORE_META_KEY: - assert(x.length == SQUID_MD5_DIGEST_LENGTH); - memcpy(index, x.value, SQUID_MD5_DIGEST_LENGTH); - break; - - case STORE_META_STD: - struct old_metahdr { - time_t timestamp; - time_t lastref; - time_t expires; - time_t lastmod; - size_t swap_file_sz; - u_short refcount; - u_short flags; - } *tmp; - tmp = (struct old_metahdr *)x.value; - assert(x.length == STORE_HDR_METASIZE_OLD); - what->timestamp = tmp->timestamp; - what->lastref = tmp->lastref; - what->expires = tmp->expires; - what->lastmod = tmp->lastmod; - what->swap_file_sz = tmp->swap_file_sz; - what->refcount = tmp->refcount; - what->flags = tmp->flags; - break; - - case STORE_META_STD_LFS: - assert(x.length == STORE_HDR_METASIZE); - memcpy(&what->timestamp, x.value, STORE_HDR_METASIZE); - break; - - default: - break; - } - } - - StoreEntry *what; - cache_key *index; -}; - void RebuildState::rebuildFromDirectory() { - LOCAL_ARRAY(char, hdr_buf, SM_PAGE_SIZE); currentEntry(NULL); cache_key key[SQUID_MD5_DIGEST_LENGTH]; struct stat sb; - int swap_hdr_len; int fd = -1; - StoreMeta *tlv_list; assert(this != NULL); debugs(47, 3, "commonUfsDirRebuildFromDirectory: DIR #" << sd->index); @@ -447,114 +398,27 @@ continue; } - if ((++counts.scancount & 0xFFFF) == 0) - debugs(47, 3, " " << sd->path << " " << std::setw(7) << counts.scancount << " files opened so far."); - debugs(47, 9, "file_in: fd=" << fd << " "<< std::setfill('0') << std::hex << std::uppercase << std::setw(8) << filn); - - - statCounter.syscalls.disk.reads++; - - int len; - - if ((len = FD_READ_METHOD(fd, hdr_buf, SM_PAGE_SIZE)) < 0) { - debugs(47, 1, "commonUfsDirRebuildFromDirectory: read(FD " << fd << "): " << xstrerror()); - file_close(fd); - store_open_disk_fd--; - fd = -1; - continue; - } + MemBuf buf; + buf.init(SM_PAGE_SIZE, SM_PAGE_SIZE); + if (!storeRebuildLoadEntry(fd, sd->index, buf, counts)) + return; + + StoreEntry tmpe; + const bool loaded = storeRebuildParseEntry(buf, tmpe, key, counts, + (int64_t)sb.st_size); file_close(fd); store_open_disk_fd--; fd = -1; - swap_hdr_len = 0; - - StoreMetaUnpacker aBuilder(hdr_buf, len, &swap_hdr_len); - - if (!aBuilder.isBufferSane()) { - debugs(47, 1, "commonUfsDirRebuildFromDirectory: Swap data buffer length is not sane."); - /* XXX shouldn't this be a call to commonUfsUnlink ? */ - sd->unlinkFile ( filn); - continue; - } - - tlv_list = aBuilder.createStoreMeta (); - - if (tlv_list == NULL) { - debugs(47, 1, "commonUfsDirRebuildFromDirectory: failed to get meta data"); - /* XXX shouldn't this be a call to commonUfsUnlink ? */ - sd->unlinkFile (filn); - continue; - } - - debugs(47, 3, "commonUfsDirRebuildFromDirectory: successful swap meta unpacking"); - memset(key, '\0', SQUID_MD5_DIGEST_LENGTH); - - StoreEntry tmpe; - InitStoreEntry visitor(&tmpe, key); - for_each(*tlv_list, visitor); - storeSwapTLVFree(tlv_list); - tlv_list = NULL; - - if (storeKeyNull(key)) { - debugs(47, 1, "commonUfsDirRebuildFromDirectory: NULL key"); - sd->unlinkFile(filn); - continue; - } - - tmpe.key = key; - /* check sizes */ - - if (tmpe.swap_file_sz == 0) { - tmpe.swap_file_sz = (uint64_t) sb.st_size; - } else if (tmpe.swap_file_sz == (uint64_t)(sb.st_size - swap_hdr_len)) { - tmpe.swap_file_sz = (uint64_t) sb.st_size; - } else if (tmpe.swap_file_sz != (uint64_t)sb.st_size) { - debugs(47, 1, "commonUfsDirRebuildFromDirectory: SIZE MISMATCH " << - tmpe.swap_file_sz << "!=" << - sb.st_size); - - sd->unlinkFile(filn); - continue; - } - - if (EBIT_TEST(tmpe.flags, KEY_PRIVATE)) { - sd->unlinkFile(filn); - counts.badflags++; - continue; - } - - /* this needs to become - * 1) unpack url - * 2) make synthetic request with headers ?? or otherwise search - * for a matching object in the store - * TODO FIXME change to new async api - * TODO FIXME I think there is a race condition here with the - * async api : - * store A reads in object foo, searchs for it, and finds nothing. - * store B reads in object foo, searchs for it, finds nothing. - * store A gets called back with nothing, so registers the object - * store B gets called back with nothing, so registers the object, - * which will conflict when the in core index gets around to scanning - * store B. - * - * this suggests that rather than searching for duplicates, the - * index rebuild should just assume its the most recent accurate - * store entry and whoever indexes the stores handles duplicates. - */ - e = Store::Root().get(key); - - if (e && e->lastref >= tmpe.lastref) { - /* key already exists, current entry is newer */ - /* keep old, ignore new */ - counts.dupcount++; - continue; - } else if (NULL != e) { - /* URL already exists, this swapfile not being used */ - /* junk old, load new */ - e->release(); /* release old entry */ - counts.dupcount++; - } + + if (!loaded) { + // XXX: shouldn't this be a call to commonUfsUnlink? + sd->unlinkFile(filn); // should we unlink in all failure cases? + continue; + } + + if (!storeRebuildKeepEntry(tmpe, key, counts)) + continue; counts.objcount++; // tmpe.dump(5); @@ -644,9 +508,8 @@ /* * Make sure we don't unlink the file, it might be * in use by a subsequent entry. Also note that - * we don't have to subtract from store_swap_size - * because adding to store_swap_size happens in - * the cleanup procedure. + * we don't have to subtract from cur_size because + * adding to cur_size happens in the cleanup procedure. */ currentEntry()->expireNow(); currentEntry()->releaseRequest(); @@ -783,7 +646,6 @@ (void) 0; } - /* update store_swap_size */ counts.objcount++; currentEntry(sd->addDiskRestore(swapData.key, === modified file 'src/fs/ufs/ufscommon.h' --- src/fs/ufs/ufscommon.h 2011-04-07 11:58:46 +0000 +++ src/fs/ufs/ufscommon.h 2011-05-12 03:58:16 +0000 @@ -62,9 +62,9 @@ virtual void unlink(StoreEntry &); virtual void statfs(StoreEntry &)const; virtual void maintain(); - virtual int canStore(StoreEntry const &)const; + virtual bool canStore(const StoreEntry &e, int64_t diskSpaceNeeded, int &load) const; virtual void reference(StoreEntry &); - virtual void dereference(StoreEntry &); + virtual bool dereference(StoreEntry &); virtual StoreIOState::Pointer createStoreIO(StoreEntry &, StoreIOState::STFNCB *, StoreIOState::STIOCB *, void *); virtual StoreIOState::Pointer openStoreIO(StoreEntry &, StoreIOState::STFNCB *, StoreIOState::STIOCB *, void *); virtual void openLog(); @@ -76,6 +76,9 @@ virtual void reconfigure(int, char *); virtual int callback(); virtual void sync(); + virtual void swappedOut(const StoreEntry &e); + virtual uint64_t currentSize() const { return cur_size; } + virtual uint64_t currentCount() const { return n_disk_objects; } void unlinkFile(sfileno f); // move down when unlink is a virtual method @@ -135,7 +138,8 @@ void optionIODump(StoreEntry * e) const; mutable ConfigOptionVector *currentIOOptions; char const *ioType; - + uint64_t cur_size; ///< currently used space in the storage area + uint64_t n_disk_objects; ///< total number of objects stored }; #include "RefCount.h" @@ -206,7 +210,7 @@ void operator delete (void *); UFSStoreState(SwapDir * SD, StoreEntry * anEntry, STIOCB * callback_, void *callback_data_); ~UFSStoreState(); - virtual void close(); + virtual void close(int how); virtual void closeCompleted(); // protected: virtual void ioCompletedNotification(); === modified file 'src/globals.h' --- src/globals.h 2011-01-28 07:58:53 +0000 +++ src/globals.h 2011-04-18 10:28:07 +0000 @@ -108,7 +108,6 @@ extern char *snmp_agentinfo; #endif - extern int n_disk_objects; /* 0 */ extern iostats IOStats; extern struct acl_deny_info_list *DenyInfoList; /* NULL */ @@ -117,7 +116,6 @@ extern int starting_up; /* 1 */ extern int shutting_down; /* 0 */ extern int reconfiguring; /* 0 */ - extern unsigned long store_swap_size; /* 0 */ extern time_t hit_only_mode_until; /* 0 */ extern StatCounters statCounter; extern double request_failure_ratio; /* 0.0 */ === modified file 'src/ipc/Coordinator.cc' --- src/ipc/Coordinator.cc 2011-02-03 07:39:31 +0000 +++ src/ipc/Coordinator.cc 2011-04-14 16:58:28 +0000 @@ -47,10 +47,27 @@ void Ipc::Coordinator::registerStrand(const StrandCoord& strand) { - if (StrandCoord* found = findStrand(strand.kidId)) + debugs(54, 3, HERE << "registering kid" << strand.kidId << + ' ' << strand.tag); + if (StrandCoord* found = findStrand(strand.kidId)) { + const String oldTag = found->tag; *found = strand; - else + if (oldTag.size() && !strand.tag.size()) + found->tag = oldTag; // keep more detailed info (XXX?) + } else { strands_.push_back(strand); + } + + // notify searchers waiting for this new strand, if any + typedef Searchers::iterator SRI; + for (SRI i = searchers.begin(); i != searchers.end();) { + if (i->tag == strand.tag) { + notifySearcher(*i, strand); + i = searchers.erase(i); + } else { + ++i; + } + } } void Ipc::Coordinator::receive(const TypedMsgHdr& message) @@ -58,8 +75,16 @@ switch (message.type()) { case mtRegistration: debugs(54, 6, HERE << "Registration request"); - handleRegistrationRequest(StrandCoord(message)); - break; + handleRegistrationRequest(HereIamMessage(message)); + break; + + case mtStrandSearchRequest: { + const StrandSearchRequest sr(message); + debugs(54, 6, HERE << "Strand search request: " << sr.requestorId << + " tag: " << sr.tag); + handleSearchRequest(sr); + break; + } case mtSharedListenRequest: debugs(54, 6, HERE << "Shared listen request"); @@ -102,14 +127,14 @@ } } -void Ipc::Coordinator::handleRegistrationRequest(const StrandCoord& strand) +void Ipc::Coordinator::handleRegistrationRequest(const HereIamMessage& msg) { - registerStrand(strand); + registerStrand(msg.strand); // send back an acknowledgement; TODO: remove as not needed? TypedMsgHdr message; - strand.pack(message); - SendMessage(MakeAddr(strandAddrPfx, strand.kidId), message); + msg.pack(message); + SendMessage(MakeAddr(strandAddrPfx, msg.strand.kidId), message); } void @@ -154,6 +179,39 @@ Mgr::Inquirer::HandleRemoteAck(response); } +void +Ipc::Coordinator::handleSearchRequest(const Ipc::StrandSearchRequest &request) +{ + // do we know of a strand with the given search tag? + const StrandCoord *strand = NULL; + typedef StrandCoords::const_iterator SCCI; + for (SCCI i = strands_.begin(); !strand && i != strands_.end(); ++i) { + if (i->tag == request.tag) + strand = &(*i); + } + + if (strand) { + notifySearcher(request, *strand); + return; + } + + searchers.push_back(request); + debugs(54, 3, HERE << "cannot yet tell kid" << request.requestorId << + " who " << request.tag << " is"); +} + +void +Ipc::Coordinator::notifySearcher(const Ipc::StrandSearchRequest &request, + const StrandCoord& strand) +{ + debugs(54, 3, HERE << "tell kid" << request.requestorId << " that " << + request.tag << " is kid" << strand.kidId); + const StrandSearchResponse response(strand); + TypedMsgHdr message; + response.pack(message); + SendMessage(MakeAddr(strandAddrPfx, request.requestorId), message); +} + #if SQUID_SNMP void Ipc::Coordinator::handleSnmpRequest(const Snmp::Request& request) === modified file 'src/ipc/Coordinator.h' --- src/ipc/Coordinator.h 2011-01-31 14:04:00 +0000 +++ src/ipc/Coordinator.h 2011-04-14 16:58:28 +0000 @@ -14,12 +14,16 @@ #include "ipc/Port.h" #include "ipc/SharedListen.h" #include "ipc/StrandCoords.h" +#include "ipc/StrandSearch.h" #include "mgr/forward.h" #if SQUID_SNMP #include "snmp/forward.h" #endif + +#include #include + namespace Ipc { @@ -42,7 +46,12 @@ StrandCoord* findStrand(int kidId); ///< registered strand or NULL void registerStrand(const StrandCoord &); ///< adds or updates existing - void handleRegistrationRequest(const StrandCoord &); ///< register,ACK + void handleRegistrationRequest(const HereIamMessage &); ///< register,ACK + + /// answer the waiting search request + void notifySearcher(const StrandSearchRequest &request, const StrandCoord&); + /// answers or queues the request if the answer is not yet known + void handleSearchRequest(const StrandSearchRequest &request); /// returns cached socket or calls openListenSocket() void handleSharedListenRequest(const SharedListenRequest& request); @@ -58,6 +67,9 @@ private: StrandCoords strands_; ///< registered processes and threads + typedef std::list Searchers; ///< search requests + Searchers searchers; ///< yet unanswered search requests in arrival order + typedef std::map Listeners; ///< params:fd map Listeners listeners; ///< cached comm_open_listener() results === modified file 'src/ipc/Forwarder.cc' --- src/ipc/Forwarder.cc 2011-02-03 08:02:28 +0000 +++ src/ipc/Forwarder.cc 2011-04-14 16:58:28 +0000 @@ -89,6 +89,8 @@ { debugs(54, 3, HERE); request->requestId = 0; + // Do not clear ENTRY_FWD_HDR_WAIT or do entry->complete() because + // it will trigger our client side processing. Let job cleanup close. } /// Ipc::Forwarder::requestTimedOut wrapper === modified file 'src/ipc/Kid.cc' --- src/ipc/Kid.cc 2011-03-30 19:39:14 +0000 +++ src/ipc/Kid.cc 2011-04-14 16:58:28 +0000 @@ -13,6 +13,8 @@ #include #endif +int TheProcessKind = pkOther; + Kid::Kid(): badFailures(0), pid(-1), === modified file 'src/ipc/Kid.h' --- src/ipc/Kid.h 2011-03-30 19:39:14 +0000 +++ src/ipc/Kid.h 2011-04-14 16:58:28 +0000 @@ -85,4 +85,19 @@ status_type status; ///< exit status of a stopped kid }; + +// TODO: processes may not be kids; is there a better place to put this? + +/// process kinds +typedef enum { + pkOther = 0, ///< we do not know or do not care + pkCoordinator = 1, ///< manages all other kids + pkWorker = 2, ///< general-purpose worker bee + pkDisker = 4, ///< cache_dir manager +} ProcessKind; + +/// ProcessKind for the current process +extern int TheProcessKind; + + #endif /* SQUID_IPC_KID_H */ === modified file 'src/ipc/Kids.cc' --- src/ipc/Kids.cc 2011-03-30 19:39:14 +0000 +++ src/ipc/Kids.cc 2011-04-14 16:58:28 +0000 @@ -7,6 +7,7 @@ #include "config.h" #include "ipc/Kids.h" +#include "protos.h" Kids TheKids; KidName TheKidName; @@ -16,26 +17,31 @@ } /// maintain n kids -void Kids::init(size_t n) +void Kids::init() { - assert(n > 0); - if (storage.size() > 0) storage.clean(); - storage.reserve(n); + storage.reserve(NumberOfKids()); char kid_name[32]; - // add Kid records for all n main strands - for (size_t i = 1; i <= n; ++i) { - snprintf(kid_name, sizeof(kid_name), "(squid-%d)", (int)i); + // add Kid records for all workers + for (int i = 0; i < Config.workers; ++i) { + snprintf(kid_name, sizeof(kid_name), "(squid-%d)", (int)(storage.size()+1)); + storage.push_back(Kid(kid_name)); + } + + // add Kid records for all disk processes + // (XXX: some cache_dirs do not need this) + for (int i = 0; i < Config.cacheSwap.n_configured; ++i) { + snprintf(kid_name, sizeof(kid_name), "(squid-disk-%d)", (int)(storage.size()+1)); storage.push_back(Kid(kid_name)); } // if coordination is needed, add a Kid record for Coordinator - if (n > 1) { - snprintf(kid_name, sizeof(kid_name), "(squid-coord-%d)", (int)(n + 1)); + if (storage.size() > 1) { + snprintf(kid_name, sizeof(kid_name), "(squid-coord-%d)", (int)(storage.size()+1)); storage.push_back(Kid(kid_name)); } } === modified file 'src/ipc/Kids.h' --- src/ipc/Kids.h 2011-03-30 19:39:14 +0000 +++ src/ipc/Kids.h 2011-04-14 16:58:28 +0000 @@ -21,8 +21,8 @@ Kids& operator= (const Kids&); ///< not implemented public: - /// maintain n kids - void init(size_t n); + /// initialize all kid records based on Config + void init(); /// returns kid by pid Kid* find(pid_t pid); === modified file 'src/ipc/Messages.h' --- src/ipc/Messages.h 2011-02-06 19:50:52 +0000 +++ src/ipc/Messages.h 2011-04-14 16:58:28 +0000 @@ -15,7 +15,9 @@ /// message class identifier typedef enum { mtNone = 0, mtRegistration, + mtStrandSearchRequest, mtStrandSearchResponse, mtSharedListenRequest, mtSharedListenResponse, + mtIpcIoNotification, mtCacheMgrRequest, mtCacheMgrResponse #if SQUID_SNMP , === modified file 'src/ipc/Strand.cc' --- src/ipc/Strand.cc 2011-02-03 07:39:31 +0000 +++ src/ipc/Strand.cc 2011-04-14 16:58:28 +0000 @@ -11,10 +11,13 @@ #include "ipc/StrandCoord.h" #include "ipc/Messages.h" #include "ipc/SharedListen.h" +#include "ipc/StrandSearch.h" #include "ipc/Kids.h" #include "mgr/Request.h" #include "mgr/Response.h" #include "mgr/Forwarder.h" +#include "DiskIO/IpcIo/IpcIoFile.h" /* XXX: scope boundary violation */ +#include "SwapDir.h" /* XXX: scope boundary violation */ #include "CacheManager.h" #if SQUID_SNMP #include "snmp/Forwarder.h" @@ -41,8 +44,10 @@ { debugs(54, 6, HERE); Must(!isRegistered); + + HereIamMessage ann(StrandCoord(KidIdentifier, getpid())); TypedMsgHdr message; - StrandCoord(KidIdentifier, getpid()).pack(message); + ann.pack(message); SendMessage(coordinatorAddr, message); setTimeout(6, "Ipc::Strand::timeoutHandler"); // TODO: make 6 configurable? } @@ -53,13 +58,21 @@ switch (message.type()) { case mtRegistration: - handleRegistrationResponse(StrandCoord(message)); + handleRegistrationResponse(HereIamMessage(message)); break; case mtSharedListenResponse: SharedListenJoined(SharedListenResponse(message)); break; + case mtStrandSearchResponse: + IpcIoFile::HandleOpenResponse(StrandSearchResponse(message)); + break; + + case mtIpcIoNotification: + IpcIoFile::HandleNotification(message); + break; + case mtCacheMgrRequest: { const Mgr::Request req(message); handleCacheMgrRequest(req); @@ -92,10 +105,10 @@ } } -void Ipc::Strand::handleRegistrationResponse(const StrandCoord &strand) +void Ipc::Strand::handleRegistrationResponse(const HereIamMessage &msg) { // handle registration response from the coordinator; it could be stale - if (strand.kidId == KidIdentifier && strand.pid == getpid()) { + if (msg.strand.kidId == KidIdentifier && msg.strand.pid == getpid()) { debugs(54, 6, "kid" << KidIdentifier << " registered"); clearTimeout(); // we are done } else { === modified file 'src/ipc/Strand.h' --- src/ipc/Strand.h 2011-01-31 14:04:00 +0000 +++ src/ipc/Strand.h 2011-04-14 16:58:28 +0000 @@ -8,6 +8,7 @@ #ifndef SQUID_IPC_STRAND_H #define SQUID_IPC_STRAND_H +#include "ipc/forward.h" #include "ipc/Port.h" #include "mgr/forward.h" #if SQUID_SNMP @@ -33,7 +34,7 @@ private: void registerSelf(); /// let Coordinator know this strand exists - void handleRegistrationResponse(const StrandCoord &strand); + void handleRegistrationResponse(const HereIamMessage &msg); void handleCacheMgrRequest(const Mgr::Request& request); void handleCacheMgrResponse(const Mgr::Response& response); #if SQUID_SNMP === modified file 'src/ipc/StrandCoord.cc' --- src/ipc/StrandCoord.cc 2010-10-28 18:52:59 +0000 +++ src/ipc/StrandCoord.cc 2011-02-01 20:16:06 +0000 @@ -7,6 +7,7 @@ #include "config.h" +#include "Debug.h" #include "ipc/Messages.h" #include "ipc/StrandCoord.h" #include "ipc/TypedMsgHdr.h" @@ -20,14 +21,35 @@ { } -Ipc::StrandCoord::StrandCoord(const TypedMsgHdr &hdrMsg): kidId(-1), pid(0) +void +Ipc::StrandCoord::unpack(const TypedMsgHdr &hdrMsg) +{ + hdrMsg.getPod(kidId); + hdrMsg.getPod(pid); + hdrMsg.getString(tag); +} + +void Ipc::StrandCoord::pack(TypedMsgHdr &hdrMsg) const +{ + hdrMsg.putPod(kidId); + hdrMsg.putPod(pid); + hdrMsg.putString(tag); +} + + +Ipc::HereIamMessage::HereIamMessage(const StrandCoord &aStrand): + strand(aStrand) +{ +} + +Ipc::HereIamMessage::HereIamMessage(const TypedMsgHdr &hdrMsg) { hdrMsg.checkType(mtRegistration); - hdrMsg.getPod(*this); + strand.unpack(hdrMsg); } -void Ipc::StrandCoord::pack(TypedMsgHdr &hdrMsg) const +void Ipc::HereIamMessage::pack(TypedMsgHdr &hdrMsg) const { hdrMsg.setType(mtRegistration); - hdrMsg.putPod(*this); + strand.pack(hdrMsg); } === modified file 'src/ipc/StrandCoord.h' --- src/ipc/StrandCoord.h 2010-10-28 18:52:59 +0000 +++ src/ipc/StrandCoord.h 2011-02-01 05:01:43 +0000 @@ -7,6 +7,7 @@ #define SQUID_IPC_STRAND_COORD_H #include "ipc/forward.h" +#include "SquidString.h" #include namespace Ipc @@ -17,14 +18,31 @@ { public: StrandCoord(); ///< unknown location - StrandCoord(int akidId, pid_t aPid); ///< from registrant - explicit StrandCoord(const TypedMsgHdr &hdrMsg); ///< from recvmsg() + StrandCoord(int akidId, pid_t aPid); + void pack(TypedMsgHdr &hdrMsg) const; ///< prepare for sendmsg() + void unpack(const TypedMsgHdr &hdrMsg); ///< from recvmsg() public: int kidId; ///< internal Squid process number pid_t pid; ///< OS process or thread identifier -}; + + String tag; ///< optional unique well-known key (e.g., cache_dir path) +}; + +/// strand registration with Coordinator (also used as an ACK) +class HereIamMessage +{ +public: + explicit HereIamMessage(const StrandCoord &strand); ///< from registrant + explicit HereIamMessage(const TypedMsgHdr &hdrMsg); ///< from recvmsg() + void pack(TypedMsgHdr &hdrMsg) const; ///< prepare for sendmsg() + +public: + StrandCoord strand; ///< registrant coordinates and related details +}; + + } // namespace Ipc; === added file 'src/ipc/StrandSearch.cc' --- src/ipc/StrandSearch.cc 1970-01-01 00:00:00 +0000 +++ src/ipc/StrandSearch.cc 2011-03-01 02:44:52 +0000 @@ -0,0 +1,52 @@ +/* + * $Id$ + * + * DEBUG: section 54 Interprocess Communication + * + */ + + +#include "config.h" +#include "ipc/Messages.h" +#include "ipc/StrandSearch.h" +#include "ipc/TypedMsgHdr.h" + + +Ipc::StrandSearchRequest::StrandSearchRequest(): requestorId(-1) +{ +} + +Ipc::StrandSearchRequest::StrandSearchRequest(const TypedMsgHdr &hdrMsg): + requestorId(-1) +{ + hdrMsg.checkType(mtStrandSearchRequest); + hdrMsg.getPod(requestorId); + hdrMsg.getString(tag); +} + +void Ipc::StrandSearchRequest::pack(TypedMsgHdr &hdrMsg) const +{ + hdrMsg.setType(mtStrandSearchRequest); + hdrMsg.putPod(requestorId); + hdrMsg.putString(tag); +} + + +/* StrandSearchResponse */ + +Ipc::StrandSearchResponse::StrandSearchResponse(const Ipc::StrandCoord &aStrand): + strand(aStrand) +{ +} + +Ipc::StrandSearchResponse::StrandSearchResponse(const TypedMsgHdr &hdrMsg) +{ + hdrMsg.checkType(mtStrandSearchResponse); + strand.unpack(hdrMsg); +} + +void Ipc::StrandSearchResponse::pack(TypedMsgHdr &hdrMsg) const +{ + hdrMsg.setType(mtStrandSearchResponse); + strand.pack(hdrMsg); +} === added file 'src/ipc/StrandSearch.h' --- src/ipc/StrandSearch.h 1970-01-01 00:00:00 +0000 +++ src/ipc/StrandSearch.h 2011-03-01 02:44:52 +0000 @@ -0,0 +1,44 @@ +/* + * $Id$ + * + */ + +#ifndef SQUID_IPC_STRAND_SEARCH_H +#define SQUID_IPC_STRAND_SEARCH_H + +#include "ipc/forward.h" +#include "ipc/StrandCoord.h" +#include "SquidString.h" +#include + +namespace Ipc +{ + +/// asynchronous strand search request +class StrandSearchRequest +{ +public: + StrandSearchRequest(); + explicit StrandSearchRequest(const TypedMsgHdr &hdrMsg); ///< from recvmsg() + void pack(TypedMsgHdr &hdrMsg) const; ///< prepare for sendmsg() + +public: + int requestorId; ///< sender-provided return address + String tag; ///< set when looking for a matching StrandCoord::tag +}; + +/// asynchronous strand search response +class StrandSearchResponse +{ +public: + StrandSearchResponse(const StrandCoord &strand); + explicit StrandSearchResponse(const TypedMsgHdr &hdrMsg); ///< from recvmsg() + void pack(TypedMsgHdr &hdrMsg) const; ///< prepare for sendmsg() + +public: + StrandCoord strand; ///< answer matching StrandSearchRequest criteria +}; + +} // namespace Ipc; + +#endif /* SQUID_IPC_STRAND_SEARCH_H */ === modified file 'src/ipc/forward.h' --- src/ipc/forward.h 2011-01-19 10:12:54 +0000 +++ src/ipc/forward.h 2011-04-14 16:58:28 +0000 @@ -13,6 +13,8 @@ class TypedMsgHdr; class StrandCoord; +class HereIamMessage; +class StrandSearchResponse; class Forwarder; class Inquirer; class Request; === modified file 'src/main.cc' --- src/main.cc 2011-04-11 03:25:32 +0000 +++ src/main.cc 2011-05-11 22:57:17 +0000 @@ -40,6 +40,7 @@ #if USE_AUTH #include "auth/Gadgets.h" #endif +#include "base/RunnersRegistry.h" #include "base/TextException.h" #if USE_DELAY_POOLS #include "ClientDelayConfig.h" @@ -658,8 +659,8 @@ wccp2ConnectionOpen(); #endif } - // Coordinator does not start proxying services - if (!IamCoordinatorProcess()) { + // start various proxying services if we are responsible for them + if (IamWorkerProcess()) { clientOpenListenSockets(); icpConnectionsOpen(); #if USE_HTCP @@ -701,7 +702,7 @@ wccp2ConnectionClose(); #endif } - if (!IamCoordinatorProcess()) { + if (IamWorkerProcess()) { clientHttpConnectionsClose(); icpConnectionShutdown(); #if USE_HTCP @@ -997,6 +998,9 @@ #endif debugs(1, 1, "Process ID " << getpid()); + + debugs(1, 1, "Process Roles:" << ProcessRoles()); + setSystemLimits(); debugs(1, 1, "With " << Squid_MaxFD << " file descriptors available"); @@ -1228,10 +1232,10 @@ try { return SquidMain(argc, argv); } catch (const std::exception &e) { - std::cerr << "dying from an unhandled exception: " << e.what() << std::endl; + debugs(0,0, "dying from an unhandled exception: " << e.what()); throw; } catch (...) { - std::cerr << "dying from an unhandled exception." << std::endl; + debugs(0,0, "dying from an unhandled exception."); throw; } return -1; // not reached @@ -1248,6 +1252,16 @@ const size_t nameLen = idStart - (processName + 1); assert(nameLen < sizeof(TheKidName)); xstrncpy(TheKidName, processName + 1, nameLen + 1); + if (!strcmp(TheKidName, "squid-coord")) + TheProcessKind = pkCoordinator; + else + if (!strcmp(TheKidName, "squid")) + TheProcessKind = pkWorker; + else + if (!strcmp(TheKidName, "squid-disk")) + TheProcessKind = pkDisker; + else + TheProcessKind = pkOther; // including coordinator } } else { xstrncpy(TheKidName, APP_SHORTNAME, sizeof(TheKidName)); @@ -1413,6 +1427,11 @@ /* NOTREACHED */ } + debugs(1,2, HERE << "Doing post-config initialization\n"); + leave_suid(); + ActivateRegistered(rrAfterConfig); + enter_suid(); + if (!opt_no_daemon && Config.workers > 0) watch_child(argv); @@ -1487,7 +1506,7 @@ if (IamCoordinatorProcess()) AsyncJob::Start(Ipc::Coordinator::Instance()); - else if (UsingSmp() && IamWorkerProcess()) + else if (UsingSmp() && (IamWorkerProcess() || IamDiskProcess())) AsyncJob::Start(new Ipc::Strand); /* at this point we are finished the synchronous startup. */ @@ -1700,7 +1719,9 @@ Config.workers); // but we keep going in hope that user knows best } - TheKids.init(Config.workers); + TheKids.init(); + + syslog(LOG_NOTICE, "Squid Parent: will start %d kids", (int)TheKids.count()); // keep [re]starting kids until it is time to quit for (;;) { @@ -1722,7 +1743,8 @@ } kid.start(pid); - syslog(LOG_NOTICE, "Squid Parent: child process %d started", pid); + syslog(LOG_NOTICE, "Squid Parent: %s process %d started", + kid.name().termedBuf(), pid); } /* parent */ @@ -1746,19 +1768,22 @@ kid->stop(status); if (kid->calledExit()) { syslog(LOG_NOTICE, - "Squid Parent: child process %d exited with status %d", + "Squid Parent: %s process %d exited with status %d", + kid->name().termedBuf(), kid->getPid(), kid->exitStatus()); } else if (kid->signaled()) { syslog(LOG_NOTICE, - "Squid Parent: child process %d exited due to signal %d with status %d", + "Squid Parent: %s process %d exited due to signal %d with status %d", + kid->name().termedBuf(), kid->getPid(), kid->termSignal(), kid->exitStatus()); } else { - syslog(LOG_NOTICE, "Squid Parent: child process %d exited", kid->getPid()); + syslog(LOG_NOTICE, "Squid Parent: %s process %d exited", + kid->name().termedBuf(), kid->getPid()); } if (kid->hopeless()) { - syslog(LOG_NOTICE, "Squid Parent: child process %d will not" + syslog(LOG_NOTICE, "Squid Parent: %s process %d will not" " be restarted due to repeated, frequent failures", - kid->getPid()); + kid->name().termedBuf(), kid->getPid()); } } else { syslog(LOG_NOTICE, "Squid Parent: unknown child process %d exited", pid); @@ -1771,6 +1796,10 @@ #endif if (!TheKids.someRunning() && !TheKids.shouldRestartSome()) { + leave_suid(); + DeactivateRegistered(rrAfterConfig); + enter_suid(); + if (TheKids.someSignaled(SIGINT) || TheKids.someSignaled(SIGTERM)) { syslog(LOG_ALERT, "Exiting due to unexpected forced shutdown"); exit(1); @@ -1870,6 +1899,7 @@ Store::Root().sync(); /* Flush log close */ StoreFileSystem::FreeAllFs(); DiskIOModule::FreeAllModules(); + DeactivateRegistered(rrAfterConfig); #if LEAK_CHECK_MODE && 0 /* doesn't work at the moment */ configFreeMemory(); === modified file 'src/mem.cc' --- src/mem.cc 2011-04-07 00:18:17 +0000 +++ src/mem.cc 2011-04-14 16:58:28 +0000 @@ -516,7 +516,8 @@ memClean(void) { MemPoolGlobalStats stats; - MemPools::GetInstance().setIdleLimit(0); + if (Config.MemPools.limit > 0) // do not reset if disabled or same + MemPools::GetInstance().setIdleLimit(0); MemPools::GetInstance().clean(0); memPoolGetGlobalStats(&stats); === modified file 'src/mgr/FunAction.cc' --- src/mgr/FunAction.cc 2011-01-19 10:12:54 +0000 +++ src/mgr/FunAction.cc 2011-04-18 10:28:07 +0000 @@ -43,9 +43,9 @@ { debugs(16, 5, HERE); Must(entry != NULL); - if (UsingSmp() && IamWorkerProcess()) + if (UsingSmp()) storeAppendPrintf(entry, "by kid%d {\n", KidIdentifier); handler(entry); - if (atomic() && UsingSmp() && IamWorkerProcess()) + if (atomic() && UsingSmp()) storeAppendPrintf(entry, "} by kid%d\n\n", KidIdentifier); } === modified file 'src/mgr/InfoAction.cc' --- src/mgr/InfoAction.cc 2011-01-19 10:12:54 +0000 +++ src/mgr/InfoAction.cc 2011-04-18 10:28:07 +0000 @@ -175,10 +175,10 @@ Must(entry != NULL); #if XMALLOC_STATISTICS - if (UsingSmp() && IamWorkerProcess()) + if (UsingSmp()) storeAppendPrintf(entry, "by kid%d {\n", KidIdentifier); DumpMallocStatistics(entry); - if (UsingSmp() && IamWorkerProcess()) + if (UsingSmp()) storeAppendPrintf(entry, "} by kid%d\n\n", KidIdentifier); #endif if (IamPrimaryProcess()) === modified file 'src/neighbors.cc' --- src/neighbors.cc 2011-04-03 12:17:09 +0000 +++ src/neighbors.cc 2011-04-14 16:58:28 +0000 @@ -1077,6 +1077,7 @@ } if (entry->lock_count == 0) { + // TODO: many entries are unlocked; why is this reported at level 1? debugs(12, 1, "neighborsUdpAck: '" << storeKeyText(key) << "' has no locks"); neighborCountIgnored(p); return; @@ -1515,9 +1516,8 @@ cbdataReferenceDone(psstate->callback_data); - EBIT_SET(fake->flags, ENTRY_ABORTED); + fake->abort(); // sets ENTRY_ABORTED and initiates releated cleanup HTTPMSGUNLOCK(fake->mem_obj->request); - fake->releaseRequest(); fake->unlock(); HTTPMSGUNLOCK(psstate->request); cbdataFree(psstate); @@ -1823,6 +1823,7 @@ } if (e->lock_count == 0) { + // TODO: many entries are unlocked; why is this reported at level 1? debugs(12, 1, "neighborsUdpAck: '" << storeKeyText(key) << "' has no locks"); neighborCountIgnored(p); return; === modified file 'src/protos.h' --- src/protos.h 2011-04-12 00:12:32 +0000 +++ src/protos.h 2011-04-14 16:58:28 +0000 @@ -536,6 +536,14 @@ SQUIDCEXTERN void storeRebuildComplete(struct _store_rebuild_data *); SQUIDCEXTERN void storeRebuildProgress(int sd_index, int total, int sofar); +/// loads entry from disk; fills supplied memory buffer on success +bool storeRebuildLoadEntry(int fd, int diskIndex, MemBuf &buf, struct _store_rebuild_data &counts); +/// parses entry buffer and validates entry metadata; fills e on success +bool storeRebuildParseEntry(MemBuf &buf, StoreEntry &e, cache_key *key, struct _store_rebuild_data &counts, uint64_t expectedSize); +/// checks whether the loaded entry should be kept; updates counters +bool storeRebuildKeepEntry(const StoreEntry &e, const cache_key *key, struct _store_rebuild_data &counts); + + /* * store_swapin.c */ @@ -584,12 +592,16 @@ SQUIDCEXTERN bool IamCoordinatorProcess(); /// whether the current process handles HTTP transactions and such SQUIDCEXTERN bool IamWorkerProcess(); +/// whether the current process is dedicated to managing a cache_dir +bool IamDiskProcess(); /// Whether we are running in daemon mode SQUIDCEXTERN bool InDaemonMode(); // try using specific Iam*() checks above first /// Whether there should be more than one worker process running SQUIDCEXTERN bool UsingSmp(); // try using specific Iam*() checks above first /// number of Kid processes as defined in src/ipc/Kid.h SQUIDCEXTERN int NumberOfKids(); +/// a string describing this process roles such as worker or coordinator +String ProcessRoles(); SQUIDCEXTERN int DebugSignal; /* AYJ debugs function to show locations being reset with memset() */ === modified file 'src/snmp_agent.cc' --- src/snmp_agent.cc 2010-04-17 02:29:04 +0000 +++ src/snmp_agent.cc 2011-04-27 23:34:13 +0000 @@ -67,7 +67,7 @@ case SYSSTOR: Answer = snmp_var_new_integer(Var->name, Var->name_length, - store_swap_size, + Store::Root().currentSize() >> 10, ASN_INTEGER); break; @@ -131,7 +131,7 @@ case CONF_ST_SWMAXSZ: Answer = snmp_var_new_integer(Var->name, Var->name_length, - (snint) (Store::Root().maxSize() >> 10), + (snint) (Store::Root().maxSize() >> 20), ASN_INTEGER); break; @@ -531,7 +531,7 @@ case PERF_PROTOSTAT_AGGR_CURSWAP: Answer = snmp_var_new_integer(Var->name, Var->name_length, - (snint) store_swap_size, + (snint) Store::Root().currentSize() >> 10, SMI_GAUGE32); break; === modified file 'src/stat.cc' --- src/stat.cc 2011-04-11 14:08:54 +0000 +++ src/stat.cc 2011-04-27 13:26:03 +0000 @@ -546,14 +546,16 @@ stats.request_hit_disk_ratio5 = statRequestHitDiskRatio(5); stats.request_hit_disk_ratio60 = statRequestHitDiskRatio(60); - stats.store_swap_size = store_swap_size; + stats.store_swap_size = Store::Root().currentSize() / 1024.0; stats.store_swap_max_size = Store::Root().maxSize(); stats.store_mem_size = mem_node::StoreMemSize(); stats.store_pages_max = store_pages_max; stats.store_mem_used = mem_node::InUseCount(); - stats.objects_size = n_disk_objects ? (double) store_swap_size / n_disk_objects : 0.0; + stats.n_disk_objects = Store::Root().currentCount(); + stats.objects_size = stats.n_disk_objects > 0 ? + stats.store_swap_size / stats.n_disk_objects : 0.0; stats.unlink_requests = statCounter.unlink.requests; @@ -666,7 +668,6 @@ stats.store_entries = StoreEntry::inUseCount(); stats.store_mem_entries = MemObject::inUseCount(); stats.hot_obj_count = hot_obj_count; - stats.n_disk_objects = n_disk_objects; } void === modified file 'src/stmem.cc' --- src/stmem.cc 2010-12-13 11:31:14 +0000 +++ src/stmem.cc 2011-02-14 05:00:44 +0000 @@ -85,6 +85,7 @@ { nodes.destroy(SplayNode::DefaultFree); inmem_hi = 0; + debugs(19, 9, HERE << this << " hi: " << inmem_hi); } bool @@ -144,12 +145,15 @@ memcpy(aNode->nodeBuffer.data + aNode->nodeBuffer.length, source, copyLen); + debugs(19, 9, HERE << this << " hi: " << inmem_hi); if (inmem_hi <= location) inmem_hi = location + copyLen; /* Adjust the ptr and len according to what was deposited in the page */ aNode->nodeBuffer.length += copyLen; + debugs(19, 9, HERE << this << " hi: " << inmem_hi); + debugs(19, 9, HERE << this << " hi: " << endOffset()); return copyLen; } @@ -176,7 +180,7 @@ void mem_hdr::internalAppend(const char *data, int len) { - debugs(19, 6, "memInternalAppend: len " << len); + debugs(19, 6, "memInternalAppend: " << this << " len " << len); while (len > 0) { makeAppendSpace(); @@ -194,6 +198,7 @@ mem_node * mem_hdr::getBlockContainingLocation (int64_t location) const { + // Optimize: do not create a whole mem_node just to store location mem_node target (location); target.nodeBuffer.length = 1; mem_node *const *result = nodes.find (&target, NodeCompare); @@ -245,7 +250,7 @@ { assert(target.range().end > target.range().start); - debugs(19, 6, "memCopy: " << target.range()); + debugs(19, 6, "memCopy: " << this << " " << target.range()); /* we shouldn't ever ask for absent offsets */ @@ -361,7 +366,7 @@ mem_hdr::write (StoreIOBuffer const &writeBuffer) { PROF_start(mem_hdr_write); - debugs(19, 6, "mem_hdr::write: " << writeBuffer.range() << " object end " << endOffset()); + debugs(19, 6, "mem_hdr::write: " << this << " " << writeBuffer.range() << " object end " << endOffset()); if (unionNotEmpty(writeBuffer)) { debugs(19,0,"mem_hdr::write: writeBuffer: " << writeBuffer.range()); @@ -391,7 +396,9 @@ } mem_hdr::mem_hdr() : inmem_hi(0) -{} +{ + debugs(19, 9, HERE << this << " hi: " << inmem_hi); +} mem_hdr::~mem_hdr() { === modified file 'src/store.cc' --- src/store.cc 2011-03-22 12:23:25 +0000 +++ src/store.cc 2011-05-11 23:04:21 +0000 @@ -46,6 +46,7 @@ #include "mem_node.h" #include "StoreMeta.h" #include "SwapDir.h" +#include "StoreIOState.h" #if USE_DELAY_POOLS #include "DelayPools.h" #endif @@ -254,6 +255,16 @@ } + if (fd_table[fd].closing()) { + // Readers must have closing callbacks if they want to be notified. No + // readers appeared to care around 2009/12/14 as they skipped reading + // for other reasons. Closing may already be true at the delyaAwareRead + // call time or may happen while we wait after delayRead() above. + debugs(20, 3, HERE << "wont read from closing FD " << fd << " for " << + callback); + return; // the read callback will never be called + } + comm_read(fd, buf, amountToRead, callback); } @@ -303,6 +314,13 @@ * offset 0 in the memory object is the HTTP headers. */ + if (mem_status == IN_MEMORY && UsingSmp()) { + // clients of an object cached in shared memory are memory clients + return STORE_MEM_CLIENT; + } + + assert(mem_obj); + if (mem_obj->inmem_lo) return STORE_DISK_CLIENT; @@ -317,6 +335,7 @@ if (mem_obj->inmem_lo == 0 && !isEmpty()) { if (swap_status == SWAPOUT_DONE) { + debugs(20,7, HERE << mem_obj << " lo: " << mem_obj->inmem_lo << " hi: " << mem_obj->endOffset() << " size: " << mem_obj->object_sz); if (mem_obj->endOffset() == mem_obj->object_sz) { /* hot object fully swapped in */ return STORE_MEM_CLIENT; @@ -352,28 +371,43 @@ return STORE_DISK_CLIENT; } -StoreEntry::StoreEntry() +StoreEntry::StoreEntry(): + hidden_mem_obj(NULL), + swap_file_sz(0) { debugs(20, 3, HERE << "new StoreEntry " << this); mem_obj = NULL; expires = lastmod = lastref = timestamp = -1; + swap_status = SWAPOUT_NONE; swap_filen = -1; swap_dirn = -1; } -StoreEntry::StoreEntry(const char *aUrl, const char *aLogUrl) +StoreEntry::StoreEntry(const char *aUrl, const char *aLogUrl): + hidden_mem_obj(NULL), + swap_file_sz(0) { debugs(20, 3, HERE << "new StoreEntry " << this); mem_obj = new MemObject(aUrl, aLogUrl); expires = lastmod = lastref = timestamp = -1; + swap_status = SWAPOUT_NONE; swap_filen = -1; swap_dirn = -1; } +StoreEntry::~StoreEntry() +{ + if (swap_filen >= 0) { + SwapDir &sd = dynamic_cast(*store()); + sd.disconnect(*this); + } + delete hidden_mem_obj; +} + void StoreEntry::destroyMemObject() { @@ -382,6 +416,18 @@ MemObject *mem = mem_obj; mem_obj = NULL; delete mem; + delete hidden_mem_obj; + hidden_mem_obj = NULL; +} + +void +StoreEntry::hideMemObject() +{ + debugs(20, 3, HERE << "hiding " << mem_obj); + assert(mem_obj); + assert(!hidden_mem_obj); + hidden_mem_obj = mem_obj; + mem_obj = NULL; } void @@ -503,21 +549,15 @@ assert(storePendingNClients(this) == 0); if (EBIT_TEST(flags, RELEASE_REQUEST)) + { this->release(); - else if (keepInMemory()) { - Store::Root().dereference(*this); - setMemStatus(IN_MEMORY); - mem_obj->unlinkRequest(); - } else { - Store::Root().dereference(*this); - - if (EBIT_TEST(flags, KEY_PRIVATE)) - debugs(20, 1, "WARNING: " << __FILE__ << ":" << __LINE__ << ": found KEY_PRIVATE"); - - /* StoreEntry::purgeMem may free e */ - purgeMem(); + return 0; } + if (EBIT_TEST(flags, KEY_PRIVATE)) + debugs(20, 1, "WARNING: " << __FILE__ << ":" << __LINE__ << ": found KEY_PRIVATE"); + + Store::Root().handleIdleEntry(*this); // may delete us return 0; } @@ -685,6 +725,8 @@ } } + // TODO: storeGetPublic() calls below may create unlocked entries. + // We should add/use storeHas() API or lock/unlock those entries. if (mem_obj->vary_headers && !storeGetPublic(mem_obj->url, mem_obj->method)) { /* Create "vary" base object */ String vary; @@ -774,9 +816,6 @@ e->store_status = STORE_PENDING; e->setMemStatus(NOT_IN_MEMORY); - e->swap_status = SWAPOUT_NONE; - e->swap_filen = -1; - e->swap_dirn = -1; e->refcount = 0; e->lastref = squid_curtime; e->timestamp = -1; /* set in StoreEntry::timestampsSet() */ @@ -923,6 +962,8 @@ return 0; } +// TODO: remove checks already performed by swapoutPossible() +// TODO: move "too many open..." checks outside -- we are called too early/late int StoreEntry::checkCachable() { @@ -1079,16 +1120,6 @@ store_status = STORE_OK; - /* - * We assign an object length here. The only other place we assign - * the object length is in storeComplete() - */ - /* RBC: What do we need an object length for? we've just aborted the - * request, the request is private and negatively cached. Surely - * the object length is inappropriate to set. - */ - mem_obj->object_sz = mem_obj->endOffset(); - /* Notify the server side */ /* @@ -1112,8 +1143,8 @@ /* Notify the client side */ invokeHandlers(); - /* Close any swapout file */ - swapOutFileClose(); + // abort swap out, invalidating what was created so far (release follows) + swapOutFileClose(StoreIOState::writerGone); unlock(); /* unlock */ } @@ -1202,10 +1233,11 @@ /* this should be emitted by the oversize dir, not globally */ - if (store_swap_size > Store::Root().maxSize()) { + if (Store::Root().currentSize() > Store::Root().maxSize()) { if (squid_curtime - last_warn_time > 10) { - debugs(20, 0, "WARNING: Disk space over limit: " << store_swap_size << " KB > " - << Store::Root().maxSize() << " KB"); + debugs(20, 0, "WARNING: Disk space over limit: " + << Store::Root().currentSize() / 1024.0 << " KB > " + << (Store::Root().maxSize() >> 10) << " KB"); last_warn_time = squid_curtime; } } @@ -1244,31 +1276,23 @@ lock_count++; setReleaseFlag(); LateReleaseStack.push_back(this); - PROF_stop(storeRelease); - return; } else { destroyStoreEntry(static_cast(this)); + // "this" is no longer valid } + + PROF_stop(storeRelease); + return; } storeLog(STORE_LOG_RELEASE, this); if (swap_filen > -1) { + // log before unlink() below clears swap_filen + if (!EBIT_TEST(flags, KEY_PRIVATE)) + storeDirSwapLog(this, SWAP_LOG_DEL); + unlink(); - - if (swap_status == SWAPOUT_DONE) - if (EBIT_TEST(flags, ENTRY_VALIDATED)) - store()->updateSize(swap_file_sz, -1); - - if (!EBIT_TEST(flags, KEY_PRIVATE)) - storeDirSwapLog(this, SWAP_LOG_DEL); - -#if 0 - /* From 2.4. I think we do this in storeUnlink? */ - storeSwapFileNumberSet(this, -1); - -#endif - } setMemStatus(NOT_IN_MEMORY); @@ -1289,7 +1313,7 @@ } for (i = 0; i < 10; i++) { - e = LateReleaseStack.pop(); + e = LateReleaseStack.count ? LateReleaseStack.pop() : NULL; if (e == NULL) { /* done! */ @@ -1404,8 +1428,8 @@ store_pages_max = Config.memMaxSize / sizeof(mem_node); } -int -StoreEntry::keepInMemory() const +bool +StoreEntry::memoryCachable() const { if (mem_obj == NULL) return 0; @@ -1419,6 +1443,14 @@ if (!Config.onoff.memory_cache_first && swap_status == SWAPOUT_DONE && refcount == 1) return 0; + if (UsingSmp()) { + const int64_t expectedSize = mem_obj->expectedReplySize(); + // objects of unknown size are not allowed into memory cache, for now + if (expectedSize < 0 || + expectedSize > static_cast(Config.Store.maxInMemObjSize)) + return 0; + } + return 1; } @@ -1584,6 +1616,15 @@ if (new_status == mem_status) return; + if (UsingSmp()) { + assert(new_status != IN_MEMORY); // we do not call this otherwise + // This method was designed to update replacement policy, not to + // actually purge something from the memory cache (TODO: rename?). + // Shared memory cache does not have a policy that needs updates. + mem_status = new_status; + return; + } + assert(mem_obj != NULL); if (new_status == IN_MEMORY) { @@ -1593,10 +1634,10 @@ debugs(20, 4, "StoreEntry::setMemStatus: not inserting special " << mem_obj->url << " into policy"); } else { mem_policy->Add(mem_policy, this, &mem_obj->repl); - debugs(20, 4, "StoreEntry::setMemStatus: inserted mem node " << mem_obj->url); + debugs(20, 4, "StoreEntry::setMemStatus: inserted mem node " << mem_obj->url << " key: " << getMD5Text()); } - hot_obj_count++; + hot_obj_count++; // TODO: maintain for the shared hot cache as well } else { if (EBIT_TEST(flags, ENTRY_SPECIAL)) { debugs(20, 4, "StoreEntry::setMemStatus: special entry " << mem_obj->url); @@ -1628,6 +1669,14 @@ if (mem_obj) return; + if (hidden_mem_obj) { + debugs(20, 3, HERE << "restoring " << hidden_mem_obj); + mem_obj = hidden_mem_obj; + hidden_mem_obj = NULL; + mem_obj->resetUrls(aUrl, aLogUrl); + return; + } + mem_obj = new MemObject(aUrl, aLogUrl); } @@ -1769,10 +1818,9 @@ * a new reply. This eats the reply. */ void -StoreEntry::replaceHttpReply(HttpReply *rep) +StoreEntry::replaceHttpReply(HttpReply *rep, bool andStartWriting) { debugs(20, 3, "StoreEntry::replaceHttpReply: " << url()); - Packer p; if (!mem_obj) { debugs(20, 0, "Attempt to replace object with no in-memory representation"); @@ -1781,18 +1829,31 @@ mem_obj->replaceHttpReply(rep); + if (andStartWriting) + startWriting(); +} + + +void +StoreEntry::startWriting() +{ + Packer p; + /* TODO: when we store headers serparately remove the header portion */ /* TODO: mark the length of the headers ? */ /* We ONLY want the headers */ packerToStoreInit(&p, this); assert (isEmpty()); - - getReply()->packHeadersInto(&p); - - rep->hdr_sz = mem_obj->endOffset(); - - httpBodyPackInto(&getReply()->body, &p); + assert(mem_obj); + + const HttpReply *rep = getReply(); + assert(rep); + + rep->packHeadersInto(&p); + mem_obj->markEndOfReplyHeaders(); + + httpBodyPackInto(&rep->body, &p); packerClean(&p); } @@ -1813,21 +1874,85 @@ bool StoreEntry::swapoutPossible() { + if (!Config.cacheSwap.n_configured) + return false; + /* should we swap something out to disk? */ debugs(20, 7, "storeSwapOut: " << url()); debugs(20, 7, "storeSwapOut: store_status = " << storeStatusStr[store_status]); + assert(mem_obj); + MemObject::SwapOut::Decision &decision = mem_obj->swapout.decision; + + // if we decided that swapout is not possible, do not repeat same checks + if (decision == MemObject::SwapOut::swImpossible) { + debugs(20, 3, "storeSwapOut: already rejected"); + return false; + } + + // this flag may change so we must check it even if we already said "yes" if (EBIT_TEST(flags, ENTRY_ABORTED)) { assert(EBIT_TEST(flags, RELEASE_REQUEST)); - swapOutFileClose(); + // StoreEntry::abort() already closed the swap out file, if any + decision = MemObject::SwapOut::swImpossible; + return false; + } + + // if we decided that swapout is possible, do not repeat same checks + if (decision == MemObject::SwapOut::swPossible) { + debugs(20, 3, "storeSwapOut: already allowed"); + return true; + } + + // if we are swapping out already, do not repeat same checks + if (swap_status != SWAPOUT_NONE) { + debugs(20, 3, "storeSwapOut: already started"); + decision = MemObject::SwapOut::swPossible; + return true; + } + + if (!checkCachable()) { + debugs(20, 3, "storeSwapOut: not cachable"); + decision = MemObject::SwapOut::swImpossible; return false; } if (EBIT_TEST(flags, ENTRY_SPECIAL)) { debugs(20, 3, "storeSwapOut: " << url() << " SPECIAL"); + decision = MemObject::SwapOut::swImpossible; return false; } + // check cache_dir max-size limit if all cache_dirs have it + if (store_maxobjsize >= 0) { + // TODO: add estimated store metadata size to be conservative + + // use guaranteed maximum if it is known + const int64_t expectedEnd = mem_obj->expectedReplySize(); + debugs(20, 7, "storeSwapOut: expectedEnd = " << expectedEnd); + if (expectedEnd > store_maxobjsize) { + debugs(20, 3, "storeSwapOut: will not fit: " << expectedEnd << + " > " << store_maxobjsize); + decision = MemObject::SwapOut::swImpossible; + return false; // known to outgrow the limit eventually + } + if (expectedEnd < 0) { + debugs(20, 3, "storeSwapOut: wait for more info: " << + store_maxobjsize); + return false; // may fit later, but will be rejected now + } + + // use current minimum (always known) + const int64_t currentEnd = mem_obj->endOffset(); + if (currentEnd > store_maxobjsize) { + debugs(20, 3, "storeSwapOut: does not fit: " << currentEnd << + " > " << store_maxobjsize); + decision = MemObject::SwapOut::swImpossible; + return false; // already does not fit and may only get bigger + } + } + + decision = MemObject::SwapOut::swPossible; return true; } @@ -1944,7 +2069,7 @@ return matched; } -StorePointer +SwapDir::Pointer StoreEntry::store() const { assert(0 <= swap_dirn && swap_dirn < Config.cacheSwap.n_configured); @@ -1954,7 +2079,10 @@ void StoreEntry::unlink() { - store()->unlink(*this); + store()->unlink(*this); // implies disconnect() + swap_filen = -1; + swap_dirn = -1; + swap_status = SWAPOUT_NONE; } /* @@ -1973,6 +2101,13 @@ return true; } +std::ostream &operator <<(std::ostream &os, const StoreEntry &e) +{ + return os << e.swap_filen << '@' << e.swap_dirn << '=' << + e.mem_status << '/' << e.ping_status << '/' << e.store_status << '/' << + e.swap_status; +} + /* NullStoreEntry */ NullStoreEntry NullStoreEntry::_instance; === modified file 'src/store_client.cc' --- src/store_client.cc 2010-12-13 11:31:14 +0000 +++ src/store_client.cc 2011-02-15 04:02:28 +0000 @@ -591,9 +591,14 @@ storeSwapTLVFree(tlv_list); + assert(swap_hdr_sz >= 0); + assert(entry->swap_file_sz > 0); + assert(entry->swap_file_sz >= static_cast(swap_hdr_sz)); entry->mem_obj->swap_hdr_sz = swap_hdr_sz; entry->mem_obj->object_sz = entry->swap_file_sz - swap_hdr_sz; - + debugs(90, 5, "store_client::unpackHeader: swap_file_sz=" << + entry->swap_file_sz << "( " << swap_hdr_sz << " + " << + entry->mem_obj->object_sz << ")"); } void @@ -696,7 +701,7 @@ e->swapOut(); if (sc->swapin_sio != NULL) { - storeClose(sc->swapin_sio); + storeClose(sc->swapin_sio, StoreIOState::readerDone); sc->swapin_sio = NULL; statCounter.swap.ins++; } === modified file 'src/store_digest.cc' --- src/store_digest.cc 2010-10-28 18:52:59 +0000 +++ src/store_digest.cc 2011-04-27 23:40:56 +0000 @@ -507,7 +507,7 @@ * number of _entries_ we want to pre-allocate for. */ const int hi_cap = Store::Root().maxSize() / Config.Store.avgObjectSize; - const int lo_cap = 1 + store_swap_size / Config.Store.avgObjectSize; + const int lo_cap = 1 + Store::Root().currentSize() / Config.Store.avgObjectSize; const int e_count = StoreEntry::inUseCount(); int cap = e_count ? e_count :hi_cap; debugs(71, 2, "storeDigestCalcCap: have: " << e_count << ", want " << cap << === modified file 'src/store_dir.cc' --- src/store_dir.cc 2010-12-14 14:01:14 +0000 +++ src/store_dir.cc 2011-05-12 04:01:29 +0000 @@ -36,6 +36,8 @@ #include "squid.h" #include "Store.h" #include "MemObject.h" +#include "MemStore.h" +#include "mem_node.h" #include "SquidMath.h" #include "SquidTime.h" #include "SwapDir.h" @@ -73,10 +75,13 @@ int StoreController::store_dirs_rebuilding = 1; StoreController::StoreController() : swapDir (new StoreHashIndex()) + , memStore(NULL) {} StoreController::~StoreController() -{} +{ + delete memStore; +} /* * This function pointer is set according to 'store_dir_select_algorithm' @@ -87,6 +92,11 @@ void StoreController::init() { + if (UsingSmp() && IamWorkerProcess()) { + memStore = new MemStore; + memStore->init(); + } + swapDir->init(); if (0 == strcasecmp(Config.store_dir_select_algorithm, "round-robin")) { @@ -186,27 +196,19 @@ int load; RefCount sd; - ssize_t objsize = e->objectLen(); + // e->objectLen() is negative at this point when we are still STORE_PENDING + ssize_t objsize = e->mem_obj->expectedReplySize(); if (objsize != -1) objsize += e->mem_obj->swap_hdr_sz; - for (i = 0; i <= Config.cacheSwap.n_configured; i++) { + for (i = 0; i < Config.cacheSwap.n_configured; i++) { if (++dirn >= Config.cacheSwap.n_configured) dirn = 0; sd = dynamic_cast(INDEXSD(dirn)); - if (sd->flags.read_only) - continue; - - if (sd->cur_size > sd->max_size) - continue; - - if (!sd->objectSizeIsAcceptable(objsize)) - continue; - - /* check for error or overload condition */ - load = sd->canStore(*e); + if (!sd->canStore(*e, objsize, load)) + continue; if (load < 0 || load > 1000) { continue; @@ -234,8 +236,7 @@ static int storeDirSelectSwapDirLeastLoad(const StoreEntry * e) { - ssize_t objsize; - ssize_t most_free = 0, cur_free; + int64_t most_free = 0; ssize_t least_objsize = -1; int least_load = INT_MAX; int load; @@ -243,8 +244,8 @@ int i; RefCount SD; - /* Calculate the object size */ - objsize = e->objectLen(); + // e->objectLen() is negative at this point when we are still STORE_PENDING + ssize_t objsize = e->mem_obj->expectedReplySize(); if (objsize != -1) objsize += e->mem_obj->swap_hdr_sz; @@ -252,25 +253,17 @@ for (i = 0; i < Config.cacheSwap.n_configured; i++) { SD = dynamic_cast(INDEXSD(i)); SD->flags.selected = 0; - load = SD->canStore(*e); - - if (load < 0 || load > 1000) { - continue; - } - - if (!SD->objectSizeIsAcceptable(objsize)) - continue; - - if (SD->flags.read_only) - continue; - - if (SD->cur_size > SD->max_size) + + if (!SD->canStore(*e, objsize, load)) + continue; + + if (load < 0 || load > 1000) continue; if (load > least_load) continue; - cur_free = SD->max_size - SD->cur_size; + const int64_t cur_free = SD->maxSize() - SD->currentSize(); /* If the load is equal, then look in more details */ if (load == least_load) { @@ -331,39 +324,21 @@ } void -StoreController::updateSize(int64_t size, int sign) -{ - fatal("StoreController has no independent size\n"); -} - -void -SwapDir::updateSize(int64_t size, int sign) -{ - int64_t blks = (size + fs.blksize - 1) / fs.blksize; - int64_t k = ((blks * fs.blksize) >> 10) * sign; - cur_size += k; - store_swap_size += k; - - if (sign > 0) - n_disk_objects++; - else if (sign < 0) - n_disk_objects--; -} - -void StoreController::stat(StoreEntry &output) const { storeAppendPrintf(&output, "Store Directory Statistics:\n"); storeAppendPrintf(&output, "Store Entries : %lu\n", (unsigned long int)StoreEntry::inUseCount()); storeAppendPrintf(&output, "Maximum Swap Size : %"PRIu64" KB\n", - maxSize()); - storeAppendPrintf(&output, "Current Store Swap Size: %8lu KB\n", - store_swap_size); - storeAppendPrintf(&output, "Current Capacity : %"PRId64"%% used, %"PRId64"%% free\n", - Math::int64Percent(store_swap_size, maxSize()), - Math::int64Percent((maxSize() - store_swap_size), maxSize())); - /* FIXME Here we should output memory statistics */ + maxSize() >> 10); + storeAppendPrintf(&output, "Current Store Swap Size: %.2f KB\n", + currentSize() / 1024.0); + storeAppendPrintf(&output, "Current Capacity : %.2f%% used, %.2f%% free\n", + Math::doublePercent(currentSize(), maxSize()), + Math::doublePercent((maxSize() - currentSize()), maxSize())); + + if (memStore) + memStore->stat(output); /* now the swapDir */ swapDir->stat(output); @@ -384,15 +359,33 @@ return swapDir->minSize(); } +uint64_t +StoreController::currentSize() const +{ + return swapDir->currentSize(); +} + +uint64_t +StoreController::currentCount() const +{ + return swapDir->currentCount(); +} + +int64_t +StoreController::maxObjectSize() const +{ + return swapDir->maxObjectSize(); +} + void SwapDir::diskFull() { - if (cur_size >= max_size) + if (currentSize() >= maxSize()) return; - max_size = cur_size; + max_size = currentSize(); - debugs(20, 1, "WARNING: Shrinking cache_dir #" << index << " to " << cur_size << " KB"); + debugs(20, 1, "WARNING: Shrinking cache_dir #" << index << " to " << currentSize() / 1024.0 << " KB"); } void @@ -515,10 +508,19 @@ return INDEXSD(x); } +SwapDir & +StoreHashIndex::dir(const int i) const +{ + SwapDir *sd = dynamic_cast(INDEXSD(i)); + assert(sd); + return *sd; +} + void StoreController::sync(void) { - /* sync mem cache? */ + if (memStore) + memStore->sync(); swapDir->sync(); } @@ -617,13 +619,12 @@ { if (swap->swapDirs == NULL) { swap->n_allocated = 4; - swap->swapDirs = static_cast(xcalloc(swap->n_allocated, sizeof(StorePointer))); + swap->swapDirs = static_cast(xcalloc(swap->n_allocated, sizeof(SwapDir::Pointer))); } if (swap->n_allocated == swap->n_configured) { - StorePointer *tmp; swap->n_allocated <<= 1; - tmp = static_cast(xcalloc(swap->n_allocated, sizeof(StorePointer))); + SwapDir::Pointer *const tmp = static_cast(xcalloc(swap->n_allocated, sizeof(SwapDir::Pointer))); memcpy(tmp, swap->swapDirs, swap->n_configured * sizeof(SwapDir *)); xfree(swap->swapDirs); swap->swapDirs = tmp; @@ -664,35 +665,83 @@ /* Notify the fs that we're referencing this object again */ if (e.swap_dirn > -1) - e.store()->reference(e); - - /* Notify the memory cache that we're referencing this object again */ + swapDir->reference(e); + + // Notify the memory cache that we're referencing this object again + if (memStore && e.mem_status == IN_MEMORY) + memStore->reference(e); + + // TODO: move this code to a non-shared memory cache class when we have it if (e.mem_obj) { if (mem_policy->Referenced) mem_policy->Referenced(mem_policy, &e, &e.mem_obj->repl); } } -void +bool StoreController::dereference(StoreEntry & e) { + bool keepInStoreTable = false; + /* Notify the fs that we're not referencing this object any more */ if (e.swap_filen > -1) - e.store()->dereference(e); - - /* Notify the memory cache that we're not referencing this object any more */ + keepInStoreTable = swapDir->dereference(e) || keepInStoreTable; + + // Notify the memory cache that we're not referencing this object any more + if (memStore && e.mem_status == IN_MEMORY) + keepInStoreTable = memStore->dereference(e) || keepInStoreTable; + + // TODO: move this code to a non-shared memory cache class when we have it if (e.mem_obj) { if (mem_policy->Dereferenced) mem_policy->Dereferenced(mem_policy, &e, &e.mem_obj->repl); } + + return keepInStoreTable; } StoreEntry * StoreController::get(const cache_key *key) { - - return swapDir->get(key); + if (StoreEntry *e = swapDir->get(key)) { + // TODO: ignore and maybe handleIdleEntry() unlocked intransit entries + // because their backing store slot may be gone already. + debugs(20, 3, HERE << "got in-transit entry: " << *e); + return e; + } + + if (memStore) { + if (StoreEntry *e = memStore->get(key)) { + debugs(20, 3, HERE << "got mem-cached entry: " << *e); + return e; + } + } + + // TODO: this disk iteration is misplaced; move to StoreHashIndex when + // the global store_table is no longer used for in-transit objects. + if (const int cacheDirs = Config.cacheSwap.n_configured) { + // ask each cache_dir until the entry is found; use static starting + // point to avoid asking the same subset of disks more often + // TODO: coordinate with put() to be able to guess the right disk often + static int idx = 0; + for (int n = 0; n < cacheDirs; ++n) { + idx = (idx + 1) % cacheDirs; + SwapDir *sd = dynamic_cast(INDEXSD(idx)); + if (!sd->active()) + continue; + + if (StoreEntry *e = sd->get(key)) { + debugs(20, 3, HERE << "cache_dir " << idx << + " got cached entry: " << *e); + return e; + } + } + } + + debugs(20, 4, HERE << "none of " << Config.cacheSwap.n_configured << + " cache_dirs have " << storeKeyText(key)); + return NULL; } void @@ -701,6 +750,36 @@ fatal("not implemented"); } +void +StoreController::handleIdleEntry(StoreEntry &e) +{ + bool keepInLocalMemory = false; + if (memStore) { + memStore->considerKeeping(e); + // leave keepInLocalMemory false; memStore maintains its own cache + } else { + keepInLocalMemory = e.memoryCachable() && // entry is in good shape and + // the local memory cache is not overflowing + (mem_node::InUseCount() <= store_pages_max); + } + + // An idle, unlocked entry that belongs to a SwapDir which controls + // its own index, should not stay in the global store_table. + if (!dereference(e)) { + debugs(20, 5, HERE << "destroying unlocked entry: " << &e << ' ' << e); + destroyStoreEntry(static_cast(&e)); + return; + } + + // TODO: move this into [non-shared] memory cache class when we have one + if (keepInLocalMemory) { + e.setMemStatus(IN_MEMORY); + e.mem_obj->unlinkRequest(); + } else { + e.purgeMem(); // may free e + } +} + StoreHashIndex::StoreHashIndex() { if (store_table) @@ -752,8 +831,10 @@ void StoreHashIndex::create() { - for (int i = 0; i < Config.cacheSwap.n_configured; i++) - store(i)->create(); + for (int i = 0; i < Config.cacheSwap.n_configured; i++) { + if (dir(i).active()) + store(i)->create(); + } } /* Lookup an object in the cache. @@ -780,8 +861,8 @@ /* Calculate size of hash table (maximum currently 64k buckets). */ /* this is very bogus, its specific to the any Store maintaining an * in-core index, not global */ - size_t buckets = (Store::Root().maxSize() + ( Config.memMaxSize >> 10)) / Config.Store.avgObjectSize; - debugs(20, 1, "Swap maxSize " << Store::Root().maxSize() << + size_t buckets = (Store::Root().maxSize() + Config.memMaxSize) / Config.Store.avgObjectSize; + debugs(20, 1, "Swap maxSize " << (Store::Root().maxSize() >> 10) << " + " << ( Config.memMaxSize >> 10) << " KB, estimated " << buckets << " objects"); buckets /= Config.Store.objectsPerBucket; debugs(20, 1, "Target number of buckets: " << buckets); @@ -790,7 +871,7 @@ store_hash_buckets = storeKeyHashBuckets(buckets); debugs(20, 1, "Using " << store_hash_buckets << " Store buckets"); debugs(20, 1, "Max Mem size: " << ( Config.memMaxSize >> 10) << " KB"); - debugs(20, 1, "Max Swap size: " << Store::Root().maxSize() << " KB"); + debugs(20, 1, "Max Swap size: " << (Store::Root().maxSize() >> 10) << " KB"); store_table = hash_create(storeKeyHashCmp, store_hash_buckets, storeKeyHashHash); @@ -810,7 +891,8 @@ * above * Step 3: have the hash index walk the searches itself. */ - store(i)->init(); + if (dir(i).active()) + store(i)->init(); } } @@ -819,8 +901,10 @@ { uint64_t result = 0; - for (int i = 0; i < Config.cacheSwap.n_configured; i++) - result += store(i)->maxSize(); + for (int i = 0; i < Config.cacheSwap.n_configured; i++) { + if (dir(i).doReportStat()) + result += store(i)->maxSize(); + } return result; } @@ -830,8 +914,49 @@ { uint64_t result = 0; - for (int i = 0; i < Config.cacheSwap.n_configured; i++) - result += store(i)->minSize(); + for (int i = 0; i < Config.cacheSwap.n_configured; i++) { + if (dir(i).doReportStat()) + result += store(i)->minSize(); + } + + return result; +} + +uint64_t +StoreHashIndex::currentSize() const +{ + uint64_t result = 0; + + for (int i = 0; i < Config.cacheSwap.n_configured; i++) { + if (dir(i).doReportStat()) + result += store(i)->currentSize(); + } + + return result; +} + +uint64_t +StoreHashIndex::currentCount() const +{ + uint64_t result = 0; + + for (int i = 0; i < Config.cacheSwap.n_configured; i++) { + if (dir(i).doReportStat()) + result += store(i)->currentCount(); + } + + return result; +} + +int64_t +StoreHashIndex::maxObjectSize() const +{ + int64_t result = -1; + + for (int i = 0; i < Config.cacheSwap.n_configured; i++) { + if (dir(i).active() && store(i)->maxObjectSize() > result) + result = store(i)->maxObjectSize(); + } return result; } @@ -850,12 +975,16 @@ } void -StoreHashIndex::reference(StoreEntry&) -{} +StoreHashIndex::reference(StoreEntry &e) +{ + e.store()->reference(e); +} -void -StoreHashIndex::dereference(StoreEntry&) -{} +bool +StoreHashIndex::dereference(StoreEntry &e) +{ + return e.store()->dereference(e); +} void StoreHashIndex::maintain() @@ -873,10 +1002,6 @@ } void -StoreHashIndex::updateSize(int64_t, int) -{} - -void StoreHashIndex::sync() { for (int i = 0; i < Config.cacheSwap.n_configured; ++i) === modified file 'src/store_io.cc' --- src/store_io.cc 2010-10-28 18:52:59 +0000 +++ src/store_io.cc 2011-02-15 04:02:28 +0000 @@ -15,31 +15,23 @@ storeCreate(StoreEntry * e, StoreIOState::STFNCB * file_callback, StoreIOState::STIOCB * close_callback, void *callback_data) { assert (e); - ssize_t objsize; - sdirno dirn; - RefCount SD; store_io_stats.create.calls++; - /* This is just done for logging purposes */ - objsize = e->objectLen(); - - if (objsize != -1) - objsize += e->mem_obj->swap_hdr_sz; /* * Pick the swapdir * We assume that the header has been packed by now .. */ - dirn = storeDirSelectSwapDir(e); + const sdirno dirn = storeDirSelectSwapDir(e); if (dirn == -1) { - debugs(20, 2, "storeCreate: no valid swapdirs for this object"); + debugs(20, 2, "storeCreate: no swapdirs for " << *e); store_io_stats.create.select_fail++; return NULL; } - debugs(20, 2, "storeCreate: Selected dir '" << dirn << "' for obj size '" << objsize << "'"); - SD = dynamic_cast(INDEXSD(dirn)); + debugs(20, 2, "storeCreate: Selected dir " << dirn << " for " << *e); + SwapDir *SD = dynamic_cast(INDEXSD(dirn)); /* Now that we have a fs to use, call its storeCreate function */ StoreIOState::Pointer sio = SD->createStoreIO(*e, file_callback, close_callback, callback_data); @@ -63,7 +55,7 @@ } void -storeClose(StoreIOState::Pointer sio) +storeClose(StoreIOState::Pointer sio, int how) { if (sio->flags.closing) { debugs(20,3,HERE << "storeClose: flags.closing already set, bailing"); @@ -72,8 +64,8 @@ sio->flags.closing = 1; - debugs(20,3,HERE << "storeClose: calling sio->close()"); - sio->close(); + debugs(20,3,HERE << "storeClose: calling sio->close(" << how << ")"); + sio->close(how); } void === modified file 'src/store_rebuild.cc' --- src/store_rebuild.cc 2009-01-21 03:47:47 +0000 +++ src/store_rebuild.cc 2011-04-27 20:30:31 +0000 @@ -74,7 +74,6 @@ size_t statCount = 500; while (statCount-- && !currentSearch->isDone() && currentSearch->next()) { - ++validated; StoreEntry *e; e = currentSearch->currentItem(); @@ -99,7 +98,6 @@ * Only set the file bit if we know its a valid entry * otherwise, set it in the validation procedure */ - e->store()->updateSize(e->swap_file_sz, 1); if ((++validated & 0x3FFFF) == 0) /* TODO format the int with with a stream operator */ @@ -109,7 +107,7 @@ if (currentSearch->isDone()) { debugs(20, 1, " Completed Validation Procedure"); debugs(20, 1, " Validated " << validated << " Entries"); - debugs(20, 1, " store_swap_size = " << store_swap_size); + debugs(20, 1, " store_swap_size = " << Store::Root().currentSize() / 1024.0 << " KB"); StoreController::store_dirs_rebuilding--; assert(0 == StoreController::store_dirs_rebuilding); @@ -231,3 +229,187 @@ debugs(20, 1, "Store rebuilding is "<< std::setw(4)<< std::setprecision(2) << 100.0 * n / d << "% complete"); last_report = squid_curtime; } + +#include "fde.h" +#include "StoreMetaUnpacker.h" +#include "StoreMeta.h" +#include "Generic.h" + +struct InitStoreEntry : public unary_function { + InitStoreEntry(StoreEntry *anEntry, cache_key *aKey):what(anEntry),index(aKey) {} + + void operator()(StoreMeta const &x) { + switch (x.getType()) { + + case STORE_META_KEY: + assert(x.length == SQUID_MD5_DIGEST_LENGTH); + memcpy(index, x.value, SQUID_MD5_DIGEST_LENGTH); + break; + + case STORE_META_STD: + struct old_metahdr { + time_t timestamp; + time_t lastref; + time_t expires; + time_t lastmod; + size_t swap_file_sz; + u_short refcount; + u_short flags; + } *tmp; + tmp = (struct old_metahdr *)x.value; + assert(x.length == STORE_HDR_METASIZE_OLD); + what->timestamp = tmp->timestamp; + what->lastref = tmp->lastref; + what->expires = tmp->expires; + what->lastmod = tmp->lastmod; + what->swap_file_sz = tmp->swap_file_sz; + what->refcount = tmp->refcount; + what->flags = tmp->flags; + break; + + case STORE_META_STD_LFS: + assert(x.length == STORE_HDR_METASIZE); + memcpy(&what->timestamp, x.value, STORE_HDR_METASIZE); + break; + + default: + break; + } + } + + StoreEntry *what; + cache_key *index; +}; + +bool +storeRebuildLoadEntry(int fd, int diskIndex, MemBuf &buf, + struct _store_rebuild_data &counts) +{ + if (fd < 0) + return false; + + assert(buf.hasSpace()); // caller must allocate + + const int len = FD_READ_METHOD(fd, buf.space(), buf.spaceSize()); + statCounter.syscalls.disk.reads++; + if (len < 0) { + const int xerrno = errno; + debugs(47, 1, "cache_dir[" << diskIndex << "]: " << + "failed to read swap entry meta data: " << xstrerr(xerrno)); + return false; + } + + buf.appended(len); + return true; +} + +bool +storeRebuildParseEntry(MemBuf &buf, StoreEntry &tmpe, cache_key *key, + struct _store_rebuild_data &counts, + uint64_t expectedSize) +{ + int swap_hdr_len = 0; + StoreMetaUnpacker aBuilder(buf.content(), buf.contentSize(), &swap_hdr_len); + if (aBuilder.isBufferZero()) { + debugs(47,5, HERE << "skipping empty record."); + return false; + } + + if (!aBuilder.isBufferSane()) { + debugs(47,1, "Warning: Ignoring malformed cache entry."); + return false; + } + + StoreMeta *tlv_list = aBuilder.createStoreMeta(); + if (!tlv_list) { + debugs(47, 1, HERE << "failed to get swap entry meta data list"); + return false; + } + + // TODO: consume parsed metadata? + + debugs(47,7, HERE << "successful swap meta unpacking"); + memset(key, '\0', SQUID_MD5_DIGEST_LENGTH); + + InitStoreEntry visitor(&tmpe, key); + for_each(*tlv_list, visitor); + storeSwapTLVFree(tlv_list); + tlv_list = NULL; + + if (storeKeyNull(key)) { + debugs(47,1, HERE << "NULL swap entry key"); + return false; + } + + tmpe.key = key; + /* check sizes */ + + if (expectedSize > 0) { + if (tmpe.swap_file_sz == 0) { + tmpe.swap_file_sz = expectedSize; + } else if (tmpe.swap_file_sz == (uint64_t)(expectedSize - swap_hdr_len)) { + tmpe.swap_file_sz = expectedSize; + } else if (tmpe.swap_file_sz != expectedSize) { + debugs(47, 1, HERE << "swap entry SIZE MISMATCH " << + tmpe.swap_file_sz << "!=" << expectedSize); + return false; + } + } else + if (tmpe.swap_file_sz <= 0) { + debugs(47, 1, HERE << "missing swap entry size: " << tmpe); + return false; + } + + if (EBIT_TEST(tmpe.flags, KEY_PRIVATE)) { + counts.badflags++; + return false; + } + + return true; +} + +bool +storeRebuildKeepEntry(const StoreEntry &tmpe, const cache_key *key, + struct _store_rebuild_data &counts) +{ + /* this needs to become + * 1) unpack url + * 2) make synthetic request with headers ?? or otherwise search + * for a matching object in the store + * TODO FIXME change to new async api + * TODO FIXME I think there is a race condition here with the + * async api : + * store A reads in object foo, searchs for it, and finds nothing. + * store B reads in object foo, searchs for it, finds nothing. + * store A gets called back with nothing, so registers the object + * store B gets called back with nothing, so registers the object, + * which will conflict when the in core index gets around to scanning + * store B. + * + * this suggests that rather than searching for duplicates, the + * index rebuild should just assume its the most recent accurate + * store entry and whoever indexes the stores handles duplicates. + */ + if (StoreEntry *e = Store::Root().get(key)) { + + if (e->lastref >= tmpe.lastref) { + /* key already exists, old entry is newer */ + /* keep old, ignore new */ + counts.dupcount++; + + // For some stores, get() creates/unpacks a store entry. Signal + // such stores that we will no longer use the get() result: + e->lock(); + e->unlock(); + + return false; + } else { + /* URL already exists, this swapfile not being used */ + /* junk old, load new */ + e->release(); /* release old entry */ + counts.dupcount++; + } + } + + return true; +} === modified file 'src/store_swapin.cc' --- src/store_swapin.cc 2010-07-28 02:27:36 +0000 +++ src/store_swapin.cc 2011-02-04 22:18:41 +0000 @@ -50,6 +50,9 @@ return; } + if (e->mem_status != NOT_IN_MEMORY) + debugs(20, 3, HERE << "already IN_MEMORY"); + debugs(20, 3, "storeSwapInStart: called for : " << e->swap_dirn << " " << std::hex << std::setw(8) << std::setfill('0') << std::uppercase << e->swap_filen << " " << e->getMD5Text()); @@ -94,6 +97,7 @@ e->swap_dirn << " to " << sc->swapin_sio->swap_filen << "/" << sc->swapin_sio->swap_dirn); + assert(e->swap_filen < 0); // if this fails, call SwapDir::disconnect(e) e->swap_filen = sc->swapin_sio->swap_filen; e->swap_dirn = sc->swapin_sio->swap_dirn; } === modified file 'src/store_swapmeta.cc' --- src/store_swapmeta.cc 2010-12-13 11:31:14 +0000 +++ src/store_swapmeta.cc 2011-02-15 04:02:28 +0000 @@ -61,8 +61,8 @@ tlv **T = &TLV; const char *url; const char *vary; - const int64_t objsize = e->objectLen(); assert(e->mem_obj != NULL); + const int64_t objsize = e->mem_obj->expectedReplySize(); assert(e->swap_status == SWAPOUT_WRITING); url = e->url(); debugs(20, 3, "storeSwapMetaBuild: " << url ); === modified file 'src/store_swapout.cc' --- src/store_swapout.cc 2010-12-04 13:10:19 +0000 +++ src/store_swapout.cc 2011-04-27 20:30:31 +0000 @@ -85,6 +85,7 @@ if (sio == NULL) { e->swap_status = SWAPOUT_NONE; + mem->swapout.decision = MemObject::SwapOut::swImpossible; delete c; xfree((char*)buf); storeLog(STORE_LOG_SWAPOUTFAIL, e); @@ -115,6 +116,7 @@ assert(mem); assert(mem->swapout.sio == self); assert(errflag == 0); + assert(e->swap_filen < 0); // if this fails, call SwapDir::disconnect(e) e->swap_filen = mem->swapout.sio->swap_filen; e->swap_dirn = mem->swapout.sio->swap_dirn; } @@ -125,22 +127,16 @@ MemObject *mem = anEntry->mem_obj; do { - /* - * Evil hack time. - * We are paging out to disk in page size chunks. however, later on when - * we update the queue position, we might not have a page (I *think*), - * so we do the actual page update here. - */ - - if (mem->swapout.memnode == NULL) { - /* We need to swap out the first page */ - mem->swapout.memnode = const_cast(mem->data_hdr.start()); - } else { - /* We need to swap out the next page */ - /* 20030636 RBC - we don't have ->next anymore. - * But we do have the next location */ - mem->swapout.memnode = mem->data_hdr.getBlockContainingLocation (mem->swapout.memnode->end()); - } + // find the page containing the first byte we have not swapped out yet + mem_node *page = + mem->data_hdr.getBlockContainingLocation(mem->swapout.queue_offset); + + if (!page) + return; // wait for more data to become available + + // memNodeWriteComplete() and absence of buffer offset math below + // imply that we always write from the very beginning of the page + assert(page->start() == mem->swapout.queue_offset); /* * Get the length of this buffer. We are assuming(!) that the buffer @@ -150,7 +146,7 @@ * but we can look at this at a later date or whenever the code results * in bad swapouts, whichever happens first. :-) */ - ssize_t swap_buf_len = mem->swapout.memnode->nodeBuffer.length; + ssize_t swap_buf_len = page->nodeBuffer.length; debugs(20, 3, "storeSwapOut: swap_buf_len = " << swap_buf_len); @@ -161,7 +157,7 @@ mem->swapout.queue_offset += swap_buf_len; storeIOWrite(mem->swapout.sio, - mem->data_hdr.NodeGet(mem->swapout.memnode), + mem->data_hdr.NodeGet(page), swap_buf_len, -1, memNodeWriteComplete); @@ -194,6 +190,9 @@ if (!swapoutPossible()) return; + // Aborted entries have STORE_OK, but swapoutPossible rejects them. Thus, + // store_status == STORE_OK below means we got everything we wanted. + debugs(20, 7, HERE << "storeSwapOut: mem->inmem_lo = " << mem_obj->inmem_lo); debugs(20, 7, HERE << "storeSwapOut: mem->endOffset() = " << mem_obj->endOffset()); debugs(20, 7, HERE << "storeSwapOut: swapout.queue_offset = " << mem_obj->swapout.queue_offset); @@ -201,6 +200,7 @@ if (mem_obj->swapout.sio != NULL) debugs(20, 7, "storeSwapOut: storeOffset() = " << mem_obj->swapout.sio->offset() ); + // buffered bytes we have not swapped out yet int64_t swapout_maxsize = mem_obj->endOffset() - mem_obj->swapout.queue_offset; assert(swapout_maxsize >= 0); @@ -209,25 +209,29 @@ debugs(20, 7, HERE << "storeSwapOut: lowest_offset = " << lowest_offset); - /* - * Grab the swapout_size and check to see whether we're going to defer - * the swapout based upon size - */ - if ((store_status != STORE_OK) && (swapout_maxsize < store_maxobjsize)) { - /* - * NOTE: the store_maxobjsize here is the max of optional - * max-size values from 'cache_dir' lines. It is not the - * same as 'maximum_object_size'. By default, store_maxobjsize - * will be set to -1. However, I am worried that this - * deferance may consume a lot of memory in some cases. - * It would be good to make this decision based on reply - * content-length, rather than wait to accumulate huge - * amounts of object data in memory. - */ - debugs(20, 5, "storeSwapOut: Deferring starting swapping out"); - return; + // Check to see whether we're going to defer the swapout based upon size + if (store_status != STORE_OK) { + const int64_t expectedSize = mem_obj->expectedReplySize(); + const int64_t maxKnownSize = expectedSize < 0 ? + swapout_maxsize : expectedSize; + debugs(20, 7, HERE << "storeSwapOut: maxKnownSize= " << maxKnownSize); + + if (maxKnownSize < store_maxobjsize) { + /* + * NOTE: the store_maxobjsize here is the max of optional + * max-size values from 'cache_dir' lines. It is not the + * same as 'maximum_object_size'. By default, store_maxobjsize + * will be set to -1. However, I am worried that this + * deferance may consume a lot of memory in some cases. + * Should we add an option to limit this memory consumption? + */ + debugs(20, 5, "storeSwapOut: Deferring swapout start for " << + (store_maxobjsize - maxKnownSize) << " bytes"); + return; + } } +// TODO: it is better to trim as soon as we swap something out, not before trimMemory(); #if SIZEOF_OFF_T <= 4 @@ -246,11 +250,13 @@ debugs(20, 7, "storeSwapOut: swapout_size = " << swapout_maxsize); - if (swapout_maxsize == 0) { - if (store_status == STORE_OK) - swapOutFileClose(); - - return; /* Nevermore! */ + if (swapout_maxsize == 0) { // swapped everything we got + if (store_status == STORE_OK) { // got everything we wanted + assert(mem_obj->object_sz >= 0); + swapOutFileClose(StoreIOState::wroteAll); + } + // else need more data to swap out + return; } if (store_status == STORE_PENDING) { @@ -271,13 +277,7 @@ if (swap_status == SWAPOUT_NONE) { assert(mem_obj->swapout.sio == NULL); assert(mem_obj->inmem_lo == 0); - - if (checkCachable()) - storeSwapOutStart(this); - else - return; - - /* ENTRY_CACHABLE will be cleared and we'll never get here again */ + storeSwapOutStart(this); // sets SwapOut::swImpossible on failures } if (mem_obj->swapout.sio == NULL) @@ -295,22 +295,23 @@ * to the filesystem at this point because storeSwapOut() is * not going to be called again for this entry. */ + assert(mem_obj->object_sz >= 0); assert(mem_obj->endOffset() == mem_obj->swapout.queue_offset); - swapOutFileClose(); + swapOutFileClose(StoreIOState::wroteAll); } } void -StoreEntry::swapOutFileClose() +StoreEntry::swapOutFileClose(int how) { assert(mem_obj != NULL); - debugs(20, 3, "storeSwapOutFileClose: " << getMD5Text()); + debugs(20, 3, "storeSwapOutFileClose: " << getMD5Text() << " how=" << how); debugs(20, 3, "storeSwapOutFileClose: sio = " << mem_obj->swapout.sio.getRaw()); if (mem_obj->swapout.sio == NULL) return; - storeClose(mem_obj->swapout.sio); + storeClose(mem_obj->swapout.sio, how); } static void @@ -323,11 +324,11 @@ assert(e->swap_status == SWAPOUT_WRITING); cbdataFree(c); - if (errflag) { - debugs(20, 1, "storeSwapOutFileClosed: dirno " << e->swap_dirn << ", swapfile " << + // if object_size is still unknown, the entry was probably aborted + if (errflag || e->objectLen() < 0) { + debugs(20, 2, "storeSwapOutFileClosed: dirno " << e->swap_dirn << ", swapfile " << std::hex << std::setw(8) << std::setfill('0') << std::uppercase << e->swap_filen << ", errflag=" << errflag); - debugs(20, 1, "\t" << xstrerror()); if (errflag == DISK_NO_SPACE_LEFT) { /* FIXME: this should be handle by the link from store IO to @@ -337,14 +338,10 @@ storeConfigure(); } - if (e->swap_filen > 0) + if (e->swap_filen >= 0) e->unlink(); - e->swap_filen = -1; - - e->swap_dirn = -1; - - e->swap_status = SWAPOUT_NONE; + assert(e->swap_status == SWAPOUT_NONE); e->releaseRequest(); } else { @@ -352,10 +349,17 @@ debugs(20, 3, "storeSwapOutFileClosed: SwapOut complete: '" << e->url() << "' to " << e->swap_dirn << ", " << std::hex << std::setw(8) << std::setfill('0') << std::uppercase << e->swap_filen); + debugs(20, 5, HERE << "swap_file_sz = " << + e->objectLen() << " + " << mem->swap_hdr_sz); + e->swap_file_sz = e->objectLen() + mem->swap_hdr_sz; e->swap_status = SWAPOUT_DONE; - e->store()->updateSize(e->swap_file_sz, 1); + e->store()->swappedOut(*e); + // XXX: For some Stores, it is pointless to re-check cachability here + // and it leads to double counts in store_check_cachable_hist. We need + // another way to signal a completed but failed swapout. Or, better, + // each Store should handle its own logging and LOG state setting. if (e->checkCachable()) { storeLog(STORE_LOG_SWAPOUT, e); storeDirSwapLog(e, SWAP_LOG_ADD); === modified file 'src/structs.h' --- src/structs.h 2011-03-15 17:31:34 +0000 +++ src/structs.h 2011-04-27 19:05:15 +0000 @@ -143,7 +143,7 @@ class CpuAffinityMap; class RemovalPolicySettings; class external_acl; -class Store; +class SwapDir; struct SquidConfig { @@ -494,9 +494,11 @@ refresh_t *Refresh; struct _cacheSwap { - RefCount *swapDirs; + RefCount *swapDirs; int n_allocated; int n_configured; + ///< number of disk processes required to support all cache_dirs + int n_strands; } cacheSwap; /* * I'm sick of having to keep doing this .. === modified file 'src/tests/TestSwapDir.cc' --- src/tests/TestSwapDir.cc 2010-12-04 01:41:43 +0000 +++ src/tests/TestSwapDir.cc 2011-02-15 04:02:28 +0000 @@ -23,9 +23,10 @@ TestSwapDir::init() {} -int -TestSwapDir::canStore(const StoreEntry&) const +bool +TestSwapDir::canStore(const StoreEntry &, int64_t, int &load) const { + load = 0; return true; } === modified file 'src/tests/TestSwapDir.h' --- src/tests/TestSwapDir.h 2010-12-04 01:41:43 +0000 +++ src/tests/TestSwapDir.h 2011-02-15 04:02:28 +0000 @@ -17,7 +17,7 @@ virtual void reconfigure(int, char*); virtual void init(); - virtual int canStore(const StoreEntry&) const; + virtual bool canStore(const StoreEntry &e, int64_t diskSpaceNeeded, int &load) const; virtual StoreIOState::Pointer createStoreIO(StoreEntry &, StoreIOState::STFNCB *, StoreIOState::STIOCB *, void *); virtual StoreIOState::Pointer openStoreIO(StoreEntry &, StoreIOState::STFNCB *, StoreIOState::STIOCB *, void *); virtual void parse(int, char*); === modified file 'src/tests/stub_store_rebuild.cc' --- src/tests/stub_store_rebuild.cc 2009-01-21 03:47:47 +0000 +++ src/tests/stub_store_rebuild.cc 2011-02-15 04:02:28 +0000 @@ -45,3 +45,16 @@ storeRebuildComplete(struct _store_rebuild_data *dc) {} +bool +storeRebuildLoadEntry(MemBuf &buf, StoreEntry &e, cache_key *key, + struct _store_rebuild_data &counts, uint64_t expectedSize) +{ + return false; +} + +bool +storeRebuildKeepEntry(const StoreEntry &tmpe, const cache_key *key, + struct _store_rebuild_data &counts) +{ + return false; +} === modified file 'src/tests/stub_store_swapout.cc' --- src/tests/stub_store_swapout.cc 2010-10-28 18:52:59 +0000 +++ src/tests/stub_store_swapout.cc 2011-02-15 04:02:28 +0000 @@ -38,7 +38,7 @@ StoreIoStats store_io_stats; void -StoreEntry::swapOutFileClose() +StoreEntry::swapOutFileClose(int) { fatal ("Not implemented"); } === modified file 'src/tests/testStore.h' --- src/tests/testStore.h 2010-12-04 01:41:43 +0000 +++ src/tests/testStore.h 2011-05-12 03:58:16 +0000 @@ -63,9 +63,7 @@ virtual void reference(StoreEntry &) {} /* Reference this object */ - virtual void dereference(StoreEntry &) {} /* Unreference this object */ - - virtual void updateSize(int64_t size, int sign) {} + virtual bool dereference(StoreEntry &) { return true; } virtual StoreSearch *search(String const url, HttpRequest *); }; === modified file 'src/tools.cc' --- src/tools.cc 2011-04-08 00:12:34 +0000 +++ src/tools.cc 2011-04-14 16:58:28 +0000 @@ -822,7 +822,13 @@ if (opt_no_daemon || Config.workers == 0) return true; - return 0 < KidIdentifier && KidIdentifier <= Config.workers; + return TheProcessKind == pkWorker; +} + +bool +IamDiskProcess() +{ + return TheProcessKind == pkDisker; } bool @@ -834,13 +840,13 @@ bool UsingSmp() { - return !opt_no_daemon && Config.workers > 1; + return InDaemonMode() && NumberOfKids() > 1; } bool IamCoordinatorProcess() { - return UsingSmp() && KidIdentifier == Config.workers + 1; + return TheProcessKind == pkCoordinator; } bool @@ -852,7 +858,7 @@ // when there is a master and worker process, the master delegates // primary functions to its only kid - if (Config.workers == 1) + if (NumberOfKids() == 1) return IamWorkerProcess(); // in SMP mode, multiple kids delegate primary functions to the coordinator @@ -866,11 +872,27 @@ if (!InDaemonMode()) return 0; +} + +String +ProcessRoles() +{ + String roles = ""; + if (IamMasterProcess()) + roles.append(" master"); + if (IamCoordinatorProcess()) + roles.append(" coordinator"); + if (IamWorkerProcess()) + roles.append(" worker"); + if (IamDiskProcess()) + roles.append(" disker"); + return roles; } void === modified file 'src/urn.cc' --- src/urn.cc 2011-03-23 08:55:43 +0000 +++ src/urn.cc 2011-04-14 16:58:28 +0000 @@ -505,6 +505,8 @@ list[i].url = url; list[i].host = xstrdup(host); + // TODO: Use storeHas() or lock/unlock entry to avoid creating unlocked + // ones. list[i].flags.cached = storeGetPublic(url, m) ? 1 : 0; i++; }