Search
Search Builder
- Building your search:
- In the search builder input field (1), you can manually enter your searches or paste from your external documentation into the field.
- In the search builder, each line item is equal to one search. If you would like to start a new search within the input field simply press enter/return on your keyboard.
- Note: We strongly suggest running searches in sets together, rather than individually when possible. This will be the most time and cost efficient way to search.
- For example it is considerably faster to run 100 searches together than it is to run them individually.
- You can also work out the syntax in any outside text editor and copy/paste them into the search builder.
- Applying conditions to your search:
- Conditions set in the builder will apply to all searches within the input field.
- Date Range (2) searches are inclusive of the input dates (so 10/29/2022 to 10/31/2022 would include 3 total days)
- Custodian (3) and File type (4) searches that include more than one value allow for any listed value to return a hit. So a custodian parameter that includes John Smith and Jane Doe only requires that John OR Jane are the custodian of the file in question.
- Conditions set in the builder will apply to all searches within the input field.
- Assigning your search to a slice:
- Before creating your search, you will need to assign your search to a slice. By selecting the “Assign to Slice” button (5), a modal will appear prompting you to either create a new slice for this set of search terms or to assign them to an existing slice.
- This is a required step that will help you group searches, and further filter your data to compare and contrast slices in order to make educated decisions on what you would like to export out of Data Mining later in your process.
- Once a slice is assigned, the "Create" button (6) will activate allowing the user to run the search.
Search Table
The search table showcases a list of all of the searches you have created within this project.
- Here you can see how each line item within the search builder (as mentioned in the previous step) appears as its own row with related data, conditions applied, and slice assigned.
- Each line item includes specific data relating to that search including file and family count, uniqueness and search proportion. See Glossary.
- "0" results means that there were no hits for that search.
- Empty results means that the search has not run yet. The user should hit the “calculate results” button to view the results of the search.
- "Error calculating results" means that an internal error occurred on this search. Users should reach out to the support team to identify the issue and possible next steps. If you would like to retry these searches, please copy them to builder, edit as needed, and run them again,
- If you select a slice in the “Assigned Slice(s)” column of the search page, you will be shown the Slice Details modal which give you more detailed insight into how your slice as well as the conditions applied, affect your overall data counts.
- You can review, compare, and contrast these slices and the data that they yield for context as to what you may want to export later on.
- Each line item includes specific data relating to that search including file and family count, uniqueness and search proportion. See Glossary.
- The search table is a list of all of the “searches," the individual sets of searches on each row of a search set. Each term includes specific information about that search on the data with those conditions and the counts will update every time your searches update. New searches will only calculate hits after clicking the "Calculate Results" button (2).
- The "Calculate Results" button refreshes both new and old searches with updated hit counts based on all documents currently in the database. If you have multiple searches (or even multiple slices) to run you should add them all to the search table before clicking the "Calculate Results" button.
- If you click on a slice in the “Assigned Slice(s)” column of the search page, you will be shown the details of this slice including conditions applied, total file and family counts, and counts for each of the terms included in the slice.
- The "Copy to Builder" button will copy selected terms to the builder where they can be edited and rerun with modified conditions. This button will only be active when one or more terms is selected.
- The "Export CSV" button will generate a CSV file with all of the search terms and count data from the chart. This button is only active when no specific terms are selected as it will return data for the entire chart.
- Searches refresh after new searches run, new files are imported, and whenever the "Calculate Results" button is pushed. The "Last Updated" date/time lets the user know the last time that the searches were updated.
Next up: Data Mining Search Guide
Or view one of the other support resources in the Data Mining series:
Data Mining - Project Dashboard
Data Mining – Uploading and Importing Data
Data Mining - Searching and Slicing Data
Data mining uses a powerful search syntax called dtSearch. There are differences between dtSearch and the search syntax employed in Nextpoint databases, so some translation may be required.
Documents are searchable with scans after processing is completed. Consultation on terms and syntax is available for an additional hourly charge.
Like in a Nextpoint database, Data Mining uses boolean searching for text searches. A "boolean" search request consists of a group of words or phrases linked by connectors such as AND and OR that indicate the relationship between them.
Examples:
|
Search Request |
Meaning |
|
apple and pear |
both words must be present |
|
apple or pear |
either word can be present |
|
apple w/5 pear |
"apple" must occur within 5 words of "pear" |
|
apple not w/12 pear |
"apple" must occur, but not within 12 words of "pear" |
|
apple and not pear |
"apple" must be present and "pear" cannot be present. |
|
name contains smith |
the field name must contain smith |
|
apple w/5 xfirstword |
apple must occur in the first five words of the document |
|
apple w/5 xlastword |
apple must occur in the last five words of the document |
Warning
Exact phrases should be off set by quotation marks. If pasting from a text editor that uses curly quotation marks, you will need to replace these with straight quotation marks for your search to run correctly.
“test phrase” >> "test phrase"
If you use more than one connector (and, or, contains, etc.), you should use parentheses to indicate precisely what you want to search for. For example, apple and pear or orange could mean (apple and pear) or orange, or it could mean apple and (pear or orange). For best results, always enclose expressions with connectors in parenthesis. Example:
(apple and pear) or (name contains smith)
Search terms may include the following special characters:
|
Character |
Meaning |
|
? |
matches any character |
|
= |
matches any single digit |
|
* |
matches any number of characters |
|
% |
|
|
# |
|
|
~ |
|
|
& |
|
|
~~ |
|
|
## |
Fuzzy Searching
Fuzzy searching will find a word even if it is misspelled. For example, a fuzzy search for apple will find appple. Fuzzy searching can be useful when you are searching text that may contain typographical errors (such as emails), or for text that has been scanned using optical character recognition (OCR).
Add fuzziness selectively using the % character. The number of % characters you add determines the number of differences dtSearch will ignore when searching for a word. The position of the % characters determines how many letters at the start of the word have to match exactly. Examples:
ba%nana
Word must begin with ba and have at most one difference between it and banana.
b%%anana
Word must begin with b and have at most two differences between it and banana.
Phonic Searching
Phonic searching looks for a word that sounds like the word you are searching for and begins with the same letter. For example, a phonic search for Smith will also find Smithe and Smythe.
To ask dtSearch to search for a word phonically, put a # in front of the word in your search request. Examples:
#smith
#johnson
Stemming
Stemming extends a search to cover grammatical variations on a word. For example, a search for fish would also find fishing. A search for applied would also find applying, applies, and apply.
To add stemming selectively, add a ~ at the end of words that you want stemmed in a search. Example: apply~
The stemming rules included with dtSearch are designed to work with the English language.
Synonym Searching
Synonym searching finds synonyms of a word that you include in a search request. For example, a search for fast would also find quickly. You can enable synonym searching selectively by adding the & character after certain words in your request. Example:
improve& w/5 search
Numeric Range Searching
A numeric range search is a search for any numbers that fall within a specified range. To add a numeric range component to a search request, enter the upper and lower bounds of the search separated by ~~ like this:
apple w/5 12~~17
This request would find any document containing apple within 5 words of a number between 12 and 17.
Notes
- A numeric range search includes the upper and lower bounds (so 12 and 17 would be retrieved in the above example).
- Numeric range searches only work with integers greater than or equal to zero, and less than 2,147,483,648
- For purposes of numeric range searching, decimal points and commas are treated as spaces and minus signs are ignored. For example, -123,456.78 would be interpreted as: 123 456 78 (three numbers).
Regular Expressions
Regular expression searching provides a way to search for advanced combinations of characters. A regular expression included in a search request must be quoted and must begin with ##.
Examples:
Apple and "##199[0-9]"
This would hit on a file containing the word "Apple" and the number 1994 (or 1990, 1991...1999).
Apple and "##19[0-9]+"
This would hit on a file containing the word "Apple" and the number 194 (or 1964 or 1983302002...).
Special characters in a regular expression are:
|
Regular expression |
Effect |
|
. (period) |
Matches any single character. Example: "sampl." would match "sample" or "samplZ" |
|
\ |
Treat next character literally. Example: in "\$100", the \ indicates that the pattern is "$100", not end-of-line ($) followed by "100" |
|
[abc] |
Brackets indicate a set of characters, one of which must be present. For example, "sampl[ae]" would match "sample" or "sampla", but not "samplx" |
|
[a-z] |
Inside brackets, a dash indicates a range of characters. For example, "[a-z]" matches any single lower-case letter. |
|
[^a-z] |
Indicates any character except the ones in the bracketed range. |
|
.* (period, asterisk) |
An asterisk means "0 or more" of something, so .* would match any string of characters, or nothing |
|
.+ (period, plus) |
A plus means "1 or more" of something, so .+ would match any string of at least one character |
|
[a-z]+ |
Any sequence of one or more lower-case letters. |
Limitations
- A regular expression must match a single whole word. For example, a search for "##app.*ie" would not find "apple pie".
- Only letters and numbers are searchable. Characters that are not indexed as letters are not searchable even using regular expressions, because the index does not contain any information about them.
- Because the dtSearch index does not store information about line breaks, searches that include begining-of-line or end-of-line regular expression criteria (^ and $) will not work.
- No case or other conversion is done on regular expressions, so a regular expression must match the case of the information stored in the index. If an index is case-insensitive, all letters in the regular expression must be lower-case. If a character is not searchable in the index, then it cannot be included as a searchable character in the regular expression. Non-searchable characters in a regular expression are not ignored as they are in other search expressions.
Performance
A regular expression is like the * wildcard character in its effect on search speed: the closer to the front of a word the expression is, the more it will slow searching. "appl.*" will be nearly as fast as "apple", while ".*pple" will be much slower.
Searching for numbers
The = wildcard, which matches a single digit, is faster than regular expressions for matching patterns of numbers. For example, to search for a social security number, you could use "=== == ====" instead of the equivalent regular expression.
For additional information about dtSearch syntax, review the following documentation (from which this search guide was adapted): https://support.dtsearch.com/webhelp/dtsearch/search_requests_overview.htm
Next up: Data Mining - Exporting Reports and Data
Or view one of the other support resources in the Data Mining series:
Data Mining - Project Dashboard
Data Mining – Uploading and Importing Data