Amberfish Documentation

A typical command for creating a new index looks something like:

The -i option is used for building indexes. The files to be added to an index are listed at the end of the command line, in this case *.txt.

The -d option specifies a database name. The index will be written to a set of files that begin with this name. In this example, files are created with names such as mydb.db, mydb.dt, mydb.fd, etc.

The -C option creates a new database, overwriting any existing database with the same name. This option should not be used when adding to an existing index.

The -v option prints names of the files as they are processed.

Another option, -m (not shown above), can be used to specify how much memory in megabytes will be used. Increasing memory usage can significantly reduce indexing time.

Once the index has been created, we can use it to run queries:

The -s option is used for searching.

The -q option specifies a free-text search query, which is effectively a list of words to search for.

The search command above is roughly equivalent to this Boolean search:

The -Q option specifies a Boolean search query, in which "or" acts as a Boolean operator. This query searches for all documents that contain "cat" or "dog" or "mouse".

A more interesting Boolean query might be:

This searches for all documents that contain "cat" and also contain "dog" or "mouse".

Both free-text and Boolean search queries are case-insensitive, meaning that uppercase and lowercase characters are interchangeable.

The output of these searches is a list of documents taking the form:

where <score> is an estimate of the document’s relevance to the query, <dbname> is the database name, <docid> is an unique number identifying the document within the database, <parent> is the docid of a document that contains this document (or 0 if no such relationship exists), <filename> is the name of the file where the document is located, and <begin> and <end> are byte offsets of the beginning and ending of the document within the file.

The words in a query can end with an asterisk (*):

In this example, car* finds all documents containing the word car, cars, carpet, or any other word that begins with the prefix car.

Phrase searching finds a sequence of words:

The phrase is surrounded by quotation marks ("). Individual words in a phrase can end with an asterisk for wildcard searching.

Phrase searching must be enabled when the database is created by using the --phrase option.

A document can consist of an entire file or a portion of a file. Documents are identified in an index by their file name and beginning and ending byte offsets. By default a file is considered to be a single document.

The --split index option is a basic way of dividing files into multiple documents, for example:

In the above example, any occurrences of the string ==== are interpreted as the beginning of a new document.

The list of documents in an index can be viewed with, for example:

A document can be extracted using the --fetch option:

where <filename>, <begin>, and <end> are taken from the output of af -s or af -l.

The --split option does not work with the xml document type (described below), which uses a different method of dividing files into documents.

Many documents contain fields, such as Title, Author, Subject, etc., which add structure to the text. For XML files (and potentially other file types in the future), Amberfish queries can be restricted to specific fields as needed. This is enabled when creating an index by specifying -t xml, for example:

where xml is the "document type" for XML files. The default document type is text which does not support field search.

An example of querying within a "Title" field:

This searches for documents that contain "cat" in the "Title" field. Note that field names may be case-sensitive, depending on the document type.

We can search across multiple indexes:

Each database is queried and the results are merged into a single result set.

To list documents that have been added to an index:

The output looks like:

where <docid> is an unique number identifying the document within the database, <parent> is the docid of a document that contains this document (or 0 if no such relationship exists), <filename> is the name of the file where the document is located, <begin> and <end> are byte offsets of the beginning and ending of the document within the file, and <doctype> is the name of the document type associated with the document.