Nextpoint EDA Search Guide

Like in a Nextpoint database, Nextpoint EDA uses boolean searching for text searches. A "boolean" search request consists of a group of words or phrases linked by connectors such as AND and OR that indicate the relationship between them. 

Examples:

"test phrase" OR single OR word

If you use more than one connector (and, or, contains, etc.), you should use parentheses to indicate precisely what you want to search for.  For example, apple and pear or orange could mean (apple and pear) or orange, or it could mean apple and (pear or orange).  For best results, always enclose expressions with connectors in parenthesis.  Example:

(apple and pear) or (name contains smith)

(field_name contains "content in the field")

(email_from contains ("jsmith@nextpoint.com" or "jjohnson@nextpoint.com))

(mailbox_path contains "Inbox")

(email_subject contains "Urgent")

//text contains ("search request" or term)

//text contains ("John Smith")

mailbox_path
author
subject
email_from
email_to
email_cc
email_bcc
email_subject

Field Filtering (in the Slice Section)

The following metadata fields can be used to filter for hits, but they must be applied in the "Slices" section of your Nextpoint EDA app. Fields can vary based on the data type. For emails, we generally extract the following field information (if available):

import_path
ancestry
file_type
file_size
md5
s3_path
status
project_id
batch_id
searchability
content_type
creation_date
creator
language
email_date
email_content_type
email_message_id
has_children
family_date
file_id
family_id

• Any file extracted from another file (loose files from a zip, attachments from emails, etc.) will have an ancestry field
• Any file that has other files extracted from it will have a has_children field (value is true/false)
• Any file directly or indirectly extracted from a mailbox will have a mailbox_path field
• Most (if not all) text-based files will have author,content_type, creation_date, and language fields.
• Emails and their attachments will have a family_date field. This is like Nextpoint's "master_date" field. It uses the family parent's creation_date value, but it's inherited by all children in the family. 

All files should have the following metadata fields which can be searched on:

import_path
file_type
file_size
md5
s3_path
status
searchability
project_id
batch_id
file_id
family_id

Search terms may include the following special characters:

Fuzzy Searching

Fuzzy searching will find a word even if it is misspelled.  For example, a fuzzy search for apple will find appple.  Fuzzy searching can be useful when you are searching text that may contain typographical errors (such as emails), or for text that has been scanned using optical character recognition (OCR). 

Add fuzziness selectively using the % character.  The number of % characters you add determines the number of differences dtSearch will ignore when searching for a word.  The position of the % characters determines how many letters at the start of the word have to match exactly.  Examples:

ba%nana

Word must begin with ba and have at most one difference between it and banana.

b%%anana

Word must begin with b and have at most two differences between it and banana.

Phonic Searching

Phonic searching looks for a word that sounds like the word you are searching for and begins with the same letter.  For example, a phonic search for Smith will also find Smithe and Smythe.

To ask dtSearch to search for a word phonically, put a # in front of the word in your search request. Examples:

#smith

#johnson

Stemming

Stemming extends a search to cover grammatical variations on a word.  For example, a search for fish would also find fishing.  A search for applied would also find applying, applies, and apply. 

To add stemming selectively, add a ~ at the end of words that you want stemmed in a search.  Example: apply~

The stemming rules included with dtSearch are designed to work with the English language.  

Synonym Searching

Synonym searching finds synonyms of a word that you include in a search request.  For example, a search for fast would also find quickly. You can enable synonym searching selectively by adding the & character after certain words in your request.  Example:

improve& w/5 search

Numeric Range Searching

A numeric range search is a search for any numbers that fall within a specified range.  To add a numeric range component to a search request, enter the upper and lower bounds of the search separated by ~~ like this:

apple w/5 12~~17

This request would find any document containing apple within 5 words of a number between 12 and 17.

Notes

• A numeric range search includes the upper and lower bounds (so 12 and 17 would be retrieved in the above example).
• Numeric range searches only work with integers greater than or equal to zero, and less than 2,147,483,648
• For purposes of numeric range searching, decimal points and commas are treated as spaces and minus signs are ignored. For example, -123,456.78 would be interpreted as: 123 456 78 (three numbers).  

Regular Expressions

Regular expression searching provides a way to search for advanced combinations of characters. A regular expression included in a search request must be quoted and must begin with ##.

Examples:

Apple and "##199[0-9]"

This would hit on a file containing the word "Apple" and the number 1994 (or 1990, 1991...1999). 

Apple and "##19[0-9]+"

This would hit on a file containing the word "Apple" and the number 194 (or 1964 or 1983302002...).

Special characters in a regular expression are:

Limitations

• A regular expression must match a single whole word. For example, a search for "##app.*ie" would not find "apple pie".
• Only letters and numbers are searchable. Characters that are not indexed as letters are not searchable even using regular expressions, because the index does not contain any information about them.
• Because the dtSearch index does not store information about line breaks, searches that include begining-of-line or end-of-line regular expression criteria (^ and $) will not work.
• No case or other conversion is done on regular expressions, so a regular expression must match the case of the information stored in the index.  If an index is case-insensitive, all letters in the regular expression must be lower-case.  If a character is not searchable in the index, then it cannot be included as a searchable character in the regular expression.  Non-searchable characters in a regular expression are not ignored as they are in other search expressions.

Performance

A regular expression is like the * wildcard character in its effect on search speed: the closer to the front of a word the expression is, the more it will slow searching. "appl.*" will be nearly as fast as "apple", while ".*pple" will be much slower.

Searching for numbers

The = wildcard, which matches a single digit, is faster than regular expressions for matching patterns of numbers. For example, to search for a social security number, you could use "=== == ====" instead of the equivalent regular expression.