Lucid Gaze - Tough Nut!

A One Stop Solution to a lot of statistical analysis for lucene internals is the all new lucid gaze for Lucene. Perhaps it has been around for a while for Solar and I'm left to wonder.. dude...where's the documentation? There aren't many places on the information superhighway where I could spot info on how to use lucid gaze. A Google for the same would prove my point.

After I did figure out how to, it seemed like a good tool, easy to use(after the eureka moment, at least for me). Here's how I analyzed various things using Lucid Gaze by Lucid Imagination.

Pre-requisites:

• lucene core jar from Lucid imagination [Here]

Write the indexing/search logic. Open the Reader/Writer/Searcher as usual.

Create a RamUsageEstimator:

• RamUsageEstimator estimator = new RamUsageEstimator();

At point where you'd want to analyze, do a

• estimator.estimateRamUsage(ir);

Where ir is an IndexReader/IndexWriter/IndexSearcher.

__Snip__

Stats s;

s = LuceneCore.getIndexStats(); //For getting IndexStats

s = LuceneCore.getStoreStats(); //For getting StoreStats

s = LuceneCore.getSearchStats(); //For getting SearchStats

s = LuceneCore.getAnalysisStats(); //For getting AnalysisStats

__Snip Ends__

Once the above step is done, s is populated with a Stats Object containing the Index/Store/Search/Analysis Stats (as per the function call).

__Snip__

HashMap h = (HashMap) s.getCurrentCounters(); // Retrieve counters accumulated since last #resetStats().

__Snip Ends__

This HashMap is populated with current stat counters. The key to these are found in the javadoc.

The following is the kind of output expected on iterating through the entire HashMap.

__Code__

IndexReader ir = IndexReader.open(indexName);

RamUsageEstimator estimator = new

estimator.estimateRamUsage(ir);

Stats s;

HashMap h;

s=LuceneCore.getIndexStats();

h = (HashMap) s.getCurrentCounters();

for(String key:h.keySet())

System.out.println("indexStats: "+key+"/"+h.get(key));

s = LuceneCore.getIndexStats();

h = (HashMap) s.getCurrentCounters();

for(String key:h.keySet())

System.out.println("storeStats: "+key+"/"+h.get(key));

s=LuceneCore.getSearchStats();

h = (HashMap) s.getCurrentCounters();

for(String key:h.keySet())

System.out.println("searchStats: "+key+"/"+h.get(key));

s=LuceneCore.getAnalysisStats();

h = (HashMap) s.getCurrentCounters();

for(String key:h.keySet())

System.out.println("analysisStats: "+key+"/"+h.get(key));

ir.close();

__Code Ends__

__Output__

indexStats: iw_adT/2000180

indexStats: ir_C/0

indexStats: ir_newC/1

indexStats: iw_C/1

indexStats: iw_segs/0

indexStats: iw_adC/15

indexStats: iw_buf/0

indexStats: iw_segC/1

indexStats: ir_ram/0

indexStats: iw_newC/1

indexStats: iw_ram/10487

storeStats: iw_adT/2000180

storeStats: ir_C/0

storeStats: ir_newC/1

storeStats: iw_C/1

storeStats: iw_segs/0

storeStats: iw_adC/15

storeStats: iw_buf/0

storeStats: iw_segC/1

storeStats: ir_ram/0

storeStats: iw_newC/1

storeStats: iw_ram/10487

analysisStats: toks/1

analysisStats: tss/30

__Output Ends__

This gives pretty much all of the desired information to optimize the search/index or any other process involving a lucene index Reader/Writer/Searcher.

Thanks to the developers @ Lucid Imagination for coming up with this.

Thanks to Jayant , Nitish for the help. :)

Download Lucid Gaze for Lucene Here [@Lucid Imagination]

Generic Code? Extensible Code?

It has been a question that pops up every time (at-least I) write code. 'How generic should this be?' By Generic, I mean the power to (re)use the same piece of code, without changing anything 'inside' the code and only changing a configuration file (xml or whatever is the implementation choice).

More often than not I end up just trying to write a code so generic, that the purpose it was primarily built for (whatever application) is complicated. Correct, everything now is in the conf file, but at the same time writing the conf file in itself is a task wherein the only kind of people who would want to write that conf instead of rewriting an application specific code would be those who are 'programming challenged'.
I've realized, perhaps if only a few questions are answered prior to writing the so generic code, the developer/designer would be at such a level of ease.
* Who asked for it?
* Would someone else ever use it? Really? Or is it just a mere assumption that someday the world would run on it?
* Assuming that the world might run on it someday, do I need to write code for all of that right now? Can I just write what I want, optimize it for what is required at the moment and a little more and then just let it be? On the lines of the early ages of internet [Design it now, and let it get corrected as it goes on, with the future users correcting it themselves]

There are a lot of other questions which should be answered before the attempt to write 'the universal machine' is made. All attempts to write code are generally towards writing a universal machine which would do all we can think of, all we can imagine, and all that the machine would be able to imagine years from now on! :)
Lets write for 'now' and write it well designed.
Let them extend it rather than configure it....

Lucene Vs Sphinx - A Showdown on a large dataset

There has long existed a battle among the "pro java" and "pro c" developer community. Not that I'd want to strictly be associated with one, but would always say the robustness/exception handling/stability vector have a better cosine value with java as compared to c [With all respect to C, for being the progenitor for Java]. Let me not move off on the tangent and rather more towards the core here. Last few weeks were spent trying to benchmark 'java' lucene as a search engine with 'C' sphinx. Though not exactly in their vanilla form, and a lot of modifications to both, we finally ran a lot of tests on both the engines.

Keeping a common playground with the following specifications:
Processor(s) : Intel Quad Core X 2
RAM : 24G
Operating System : RHEL 32 Bit
Document Corpus : 18 million+ Documents
Source Size : 90G [RDBMS Table]


On a corpus of approximately 18 million records, indexed and not all of them stored. Multiple field queries with varying boost values and some good level of complexity. Here is the result sheet:

Lucene
Index Size : 20G

Concurrency : 30 [5*6 Daemons, 2G each]
Total Searches : 64931
Slow Query Count (>=10 secs) : 3803 ( 5.86%)
Total Duration (secs) : 238094.574
Mean Duration : 3.667
Mode Duration : 0.835
Minimum Duration : 0.001
Maximum Duration : 1174.757
Duration Standard Deviation : 15.441
Search Time (secs) Distribution :-
[0,0.25) : 2770 ( 4.27%)
[0.25,0.5) : 5666 ( 8.73%)
[0.5,1) : 13515 (20.81%)
[1,1.5) : 10928 (16.83%)
[1.5,2) : 7330 (11.29%)
[2,3) : 8476 (13.05%)
[3,5) : 7222 (11.12%)
[5,10) : 5221 ( 8.04%)
[10,20) : 2335 ( 3.60%)
[20,+inf) : 1468 ( 2.26%)

Concurrency : 5 [1 Daemon * 2G]
Total Searches : 225906
Slow Query Count (>=10 secs) : 972 ( 0.43%)
Total Duration (secs) : 186700.646
Mean Duration : 0.826
Mode Duration : 0.003
Minimum Duration : 0.001
Maximum Duration : 467.647
Duration Standard Deviation : 2.864
Duration (secs) Bins :-
[0,0.25) : 64621 (28.61%)
[0.25,0.5) : 58947 (26.09%)
[0.5,1) : 56894 (25.18%)
[1,1.5) : 19836 ( 8.78%)
[1.5,2) : 9397 ( 4.16%)
[2,3) : 7941 ( 3.52%)
[3,5) : 4810 ( 2.13%)
[5,10) : 2488 ( 1.10%)
[10,20) : 684 ( 0.30%)
[20,+inf) : 288 ( 0.13%)



Sphinx
Index Size: 60G

Concurrency: 30
Total Searches : 244431
Slow Query Count (>=10 secs) : 27479 (11.24%)
Total Duration (secs) : 1243474.213
Mean Duration : 5.087
Mode Duration : 0.007
Minimum Duration : 0.001
Maximum Duration : 1869.063
Duration Standard Deviation : 17.833
Average Queries : 2.783
Duration (secs) Bins :-
[0,0.25) : 51186 (20.94%)
[0.25,0.5) : 27798 (11.37%)
[0.5,1) : 32372 (13.24%)
[1,1.5) : 20490 ( 8.38%)
[1.5,2) : 16915 ( 6.92%)
[2,3) : 21833 ( 8.93%)
[3,5) : 23550 ( 9.63%)
[5,10) : 22808 ( 9.33%)
[10,20) : 14540 ( 5.95%)
[20,+inf) : 12939 ( 5.29%)

Concurrency: 5
Total Searches : 226528
Slow Query Count (>=10 secs) : 9895 ( 4.37%)
Total Duration (secs) : 453296.517
Mean Duration : 2.001
Mode Duration : 0.007
Minimum Duration : 0.001
Maximum Duration : 164.713
Duration Standard Deviation : 4.543
Average Queries : 2.773
Duration (secs) Bins :-
[0,0.25) : 71001 (31.34%)
[0.25,0.5) : 36500 (16.11%)
[0.5,1) : 32799 (14.48%)
[1,1.5) : 20416 ( 9.01%)
[1.5,2) : 16385 ( 7.23%)
[2,3) : 13951 ( 6.16%)
[3,5) : 13330 ( 5.88%)
[5,10) : 12251 ( 5.41%)
[10,20) : 7563 ( 3.34%)
[20,+inf) : 2332 ( 1.03%)

As per the analysis, for the dataset analyzed, Lucene was found to win convincingly over its rival.
More details on the same to come soon!

P.S: Though lucene works great for a lot of cases, so does sphinx. Here, lucene seemed to have an upper hand