Understanding Deduplication


Deduplication at the time of import prevents existing documents and email families from entering your database multiple times. The deduplication settings selected in the import workflow determine the definition of 'Duplicate' for the import batch at-hand. 

When deduplication is turned off at the time of import, no deduplication will occur and all files will be imported.

How do I set Deduplication Settings?

When importing data via, Nextpoint will make pre-set recommendations for Deduplication settings in Step 2 of the import workflow.  The dedupe selections will be populated based on which type of data is detected from the File Room (or selected in Step 1).

If you would like to modify the recommended settings, make sure the applicable toggle is turned ON ImportWizard_toggle.png and click the gear to open the settings pop-up.

Location of Deduplication Settings


How to Modify Recommended Deduplication Settings

Upon clicking on the gear icon mceclip0.png, you will be presented with the option to toggle File match criteria and Context Criteria ON mceclip1.png  or OFF mceclip2.png.  Read more below on how File Match and Context Criteria factor into the deduplication process.


Default Deduplication Settings per Import Type

Outlined below is a list of the various import types and their associated deduplication settings:

Import Type Deduplication Setting Image from Import Data Settings
Manual Dedupe - OFF mceclip2.png
Single Mailbox Dedupe - ON , File Match - ON , Context - ON



Multiple Files Dedupe - ON , File Match - ON , Context - ON



Production with load file Dedupe - OFF  mceclip1.png


How does Deduplication work?





1  |  First, you queue your files for processing by initiating an import.  See the three different file types queued for processing on the far left in Figure 1 (above).

2  |  Once you initiate an import, Nextpoint begins processing by extracting files from their containers (zips, pst, box), extracting any attachments from their parent emails, and extracting metadata.



3  |  Next, is the First Deduplication Pass on all three file types where we look for a matching expansive hash for all the file types.

  1. If a loose file, expansive hash is the MD5 hash value.
  2. If an email family, expansive hash is formulated from the MD5s of all members of the email family.

Once an expansive hash is identified, we compare to other files being processed and existing in the database.  Any matches are placed in a Dedupe Queue.  Loose files without a match are imported.

4  |  Also during the First Deduplication Pass, all remaining email files not placed in the dedupe queue due to matching expansive hash are checked for matching Message ID's.

Again, we look for matches against what already exists in the database and other files currently being processed. If we find a matching set, we add to the Dedupe Queue.  If no match is found, the file is imported.

Note: Turn File Match Criteria ON mceclip1.png for more aggressive deduplication using expansive hash and email message ID, as describe below.  Turn OFF mceclip2.png to deduplicate more conservatively and only consider Content Hash matches duplicates.



5  |  Next, we address the Dedupe Queue.

  1. If Context is OFF, we take everything in the Dedupe Queue and merge field values which may conflict (e.g. file_path of file A is different than file B).  We keep the first copy of the file which entered the database and discard the other(s).
  2. If Context is ON, we take any sets of duplicates* and handle field value conflicts accordingly:
    1. If fields from our Conflict Field List do not match in a set of duplicates, we keep both files, remove from the Dedupe Queue and import both.

      If there are no conflicts present in the Conflict fields:
    2. We evaluate fields from our Merge Field List.  If any Merge Field does not match in a set of duplicates*, we keep one copy of the file, merge the mismatched values into the respective field, and discard the last copy to enter the database.
    3. We evaluate fields from our Ignore Field List.  If any Ignore Field does not match in a set of duplicates, we do nothing with the fields and only keep the first copy of the file which entered the database.

*Duplicates can be considered two files within the same import OR a single file in an import being compared to a file existing in the database.

Conflict Field List Merge Field List Ignore Field List
author document_last_author email_sent
bcc document_subject email_subject
created_date_time email_author last_print_date
document_author email_message_id modified_date_time
document_date email_reply_id  

Note: All deduplication is considered at a family-level.  If after a loose file is added to your case, that same file is added, but as part of a larger email family (or vice versa), no deduplication will occur.  

2 out of 2 found this helpful



Please sign in to leave a comment.

Articles in this section

See more