Solr 4.2.1 LotsOfCores new options

The SOLR-5316.patch is for the lotsofcores use case: a large number of homogeneous cores with frequent loading/unloading. The new Cores options are:

• defaultLoadOnStartup : true by default
• defaultTransient : false by default
  In this way, you can change the default attributes value for all cores.
  The lotsofcores use case requires to have defaultLoadOnStartup = false and defaultTransient = true
• "numBuckets" to create a subdirectory based on a hash on the corename % numBuckets in the core Datadir, because all cores cannot live in the same directory
• "auto" with 2 differents values :
  1) createLoad: create, if not exist, and load the core on the fly on
  the first incoming request (update, select).
  2) onlyLoad: load the core on the fly on the first incoming request
  (update, select), if exist on disk
  The auto option uses an additional cores option named baseDataDir to automatically generate the dataDir of each core and uses the "numBucket" option if exists. All the cores have the same solr config defined in the solr_home/conf.
• discoverOnDemand : false by default. If enabled, the discovery and loading of cores at the solr startup are disabled and a new option named coreDiscovery in the STATUS action is available to discover on demand all the cores in the coreRootDirectory (only add the cores descriptions)
• noCorePropertiesFile : false by default. If enabled, don't use anymore the core.properties file, a core will be detected on disk based on the file segments.gen

The unload command was also modified in order to handle the non loaded transient cores, to be able to apply for example the deleteIndex, deleteDataDir options.

I added some Junit tests.

An example of solr.xml:
<solr>
<str name="shareSchema">schema.xml</str>
<int name="transientCacheSize">5000</int>
<str name="coreRootDirectory">solr_home/data</str>
<str name="baseDataDir">solr_home/data</str>
<int name="numBuckets">100</int>
<str name="auto">createLoad</str>
<str name="defaultLoadOnStartup">false</str>
<str name="defaultTransient">true</str>
<bool name="discoverOnDemand">true</bool>
<bool name="noCorePropertiesFile">true</bool>
</solr>

The patch SOLR-5316.patch in attachment is for svn solr branch_4X (revision number 1556554)

---------------

The patch solr-4.2.1.patchLOTSOFCORES is for the lotsofcores use case, including some modification :

• by default, all cores have loadOnStartup="false" and transient="true" attributes
• the create admin command can register a lazy core (to take into account the transientCacheSize option)
• add transient cores persistency
• handle unload admin command for never launched transient cores (non active cores)

To improve performance, we use this Solr patched version with the persistence disabled.
In this way, Solr is working with a solr.xml file without any core entries, because it's useless in our use case (with the new Auto option for the cores)

The new Cores options :

• "numBuckets" to create a subdirectory based on a hash on the corename
  % numBuckets in the core Datadir, because all cores cannot live in the
  same directory
• "Auto" with 3 differents values :
  1) false : default behaviour
  2) createLoad : create, if not exist, and load the core on the fly on
  the first incoming request (update, select).
  3) onlyLoad : load the core on the fly on the first incoming request
  (update, select), if exist on disk

The Auto option uses an additional cores option named baseDataDir to automatically generate the dataDir of each core and uses the "numBucket" option if exists.