#
# DGS Search
# $Id: SPECS,v 1.1 2002/08/26 01:13:40 sella Exp $
# Copyright (c) 2000-2001 Digital Genesis Software, LLC. All Rights Reserved.
# Released under the GPL Version 2 License.
# http://www.digitalgenesis.com
#

-----------------------
  Index Specification
-----------------------

[Index Format Overview]

MEM
---------------------------------------------------------------------------------------------------------------
    |[4B seek to next][4B #pairs][XB letters][4B offset to letters].. [4B offset to first filename]
0000|216              26         aNNN        216                      5000
    |[4B seek to next][4B #files with][4B keyword total][?B string][4B filename + seek][4B #keywords in file]..
0216|34               40              82                alphaN     0                   10
0250|35               15              52                betaN      35                  8
.   |.
.   |.
.   |.
49??|5000             1               1                 zetaN      700                 1
    |[4B seek to next (first file at 0)][?B string   ]
5000|14                                 /x/index.htmlN
5014|13                                 /y/index.phpN
---------------------------------------------------------------------------------------------------------------

- Legend -

X = Number of bytes needed to handle the # of quick indexes configured. (We are assuming 4B in this example.)
? = Unknown number of bytes. It varies from field to field.
N = Means NULL (the end of the strings)
seek = Current file location + seek
offset = the exact location in file from the beginning of the file.
filename+seek = First filename location + seek.


[Index Format Desc]

Quick index (Line 0)
--------------------
Used to quickly locate the general location of a keyword in the Keyword Hash. Quick indexes are sorted alphanumerically.

Quick Index is of this format:

[(1) 4B seek][(2) 4B #pairs][(3) XB index][(4) 4B offset to index].. [(5) 4B offset to first filename]

1) Seek the end of the quick index (4 Bytes)
2) Number of quick index pairs (letters). (4 Bytes)
3) Quick index key (letters) ie: 'a' (X Bytes, based on # of pairs. X is the same for all pairs)
4) Offset to first keyword with this index (4 Bytes)
5) Offset to first filename in file. Allows us to jump to the filenames. (4 Bytes)

NOTE: Item 2 is *not* a NULL terminated string, but will end in NULLs if and only if there is extra room in the cell.

NOTE: Item 3 and 4 are a pair, and will repeat. You can calculate X (the size of Item 3) with:

              [Item 1] - 12B
             ----------------  -  4B = [Size of Item 3]
                 [Item 2]


Keyword Hash (Lines 1 -> ?)
---------------------------
Keyword hash stores how often an in what files the keyword is found on your site. Each line in the keyword hash is sorted alphanumerically.. and the entries within a line are sorted by pages with the highest count of the keyword.

Keyword Hash is of this format:

[(1) 4B seek][(2) 4B #files with][(3) 4B keyword total][(4) ?B string][(5) 4B filename + seek][(6) 4B #keyword in file]..

1) Seek to end of keyword hash line. It will be the address of the next entry. (4 Bytes)
2) Number of file that have this keyword. There will be this number of FilenameSeek/Count pairs. (4 Bytes)
3) Total count of this keyword for the whole site. The sum of all filename totals. (4 Bytes)
4) Keyword as a NULL terminated string. (? Bytes)
5) Seek to filename from first filename. Used by jumping to Quick Index Item 5 and then seeking to the filename. (4 Bytes)
6) Total count of this keyword in this file. (4 Bytes)

NOTE: Item 5 and 6 are a pair, and will repeat. Item 2 specifies how many times it will repeat.


Filename List (Lines ? + 1 -> $)
--------------------------------
Filenames that were indexed. This list is in no particular order, but can't change after the Keyword Hash is built, since it tracks the location of the file via an seek.

Filename List is of this format:

[(1) 4B seek][?B string]

1) Seek to end of filename line. It will be the address of the next filename. The seek is based from the location of the first filename. Offset to the first filename is kept in the Quick Index. (4 Bytes)
2) Filename as a NULL terminated string. (? Bytes)


[Functions]

Function Flow (fs index module)
------------------------------------
Search for all filenames. (Caching is disabled when indexing is on.)
Function to strip unwanted content of file, and return the contents.
Func to count instances of words for the contents of a file, ignoring the stop words. Returns an array of keyword/count pairs.. hash by keyword.
Merge files keyword/count into master keyword/count/fileindex array.
If files keyword/count was new, then write out keyword/count into its own file.
Function to write out keywork:count fileoffset:count entries.
Function to write out filenames at the end of the file.
Function to make quick index. (Goes at top of file, since we know the offset.. )

Read Functions (dgsutil libs)
-----------------------------
-High Level-
Gets total matches given a boolean query.
Gets filenames and counts for a range (ie: 10 thru 20) for a boolean query.

-Low Level-
X Gets total matches given a keyword
X Gets filename count given a keyword
Func to generate quick index locations
X Gets the offset from quick index, given a keyword
X Gets filename given an offset
X Gets the offset of the first filename
X Gets <filenmae offset:count> array given keyword, or returns FALSE if not found
