Produced Data Imports
- Nextpoint Default and Protected Fields
- How to Import a CSV Overlay
- How to Import Produced Data with a Load File
- How to Import Produced Data for Manual Import Type
- Produced Data Structure and Relative Path Information
- Produced Data Import Required Specifications
- Produced Data Imports, Exhibit A - Default Fields & Document Attributes
- Produced Data Imports, Exhibit B - Protected System Fields
- Linking Email Families in Produced Data Imports
- Load Files
The following fields exist in Nextpoint without needing to explicitly add them to your database. Some are default fields to which you can map data during an import (Exhibit A), and others are protected fields to which you can not push data. While it is not necessary to map data to our default fields, it is best practice because it can reduce the number of fields you may need to search in your database
For instance, if you map "From" data in your load file to a custom "From" field instead of the default field "Email_Author," then to search your clients emails and those produced by Opposing Counsel for those authored by "John Smith," you would need to search:
email_author:"John Smith" OR from:"John Smith"
If you map the "From" field to "Email_Author," you only need to search:
email_author:"John Smith"
to return hits in both sets of data.
Warning
One field not listed below that is very important to most load file imports is the field "begattach." This field allows Nextpoint users to link parent emails with their attachments in produced data imports. In this field the bates_start value of the parent email appears in the attachment's "begattach" field. The "begattach" field must be an exact match to the character string "begattach" and must be created in a database before family linking is possible.
Default Fields & Document Attributes
For any of the below-listed fields, you do not need to set up a new Field under SETTINGS > Coding. Instead, if you have a header value in Row 1 of your load file, make sure that the load file value matches the below default fields exactly.
Values can be imported by load file headers to the provided values. App Name indicates what your load file header should read. Visible vs. Hidden indicates if a field can be seen under SETTINGS Coding Fields (Visible), or if it isn't seen in the field list but can accept your load file information (Hidden).
| App Name | Visible vs. Hidden |
|---|---|
| Author | Hidden |
| Bates_Start | Hidden |
| Bates_End | Hidden |
| Bates_Range_Start | Hidden |
| Bates_Range_End | Hidden |
| BCC | Visible |
| CC | Visible |
| Created_Date_Time | Visible |
| Custodian | Hidden |
| Custodians | Hidden |
| Date | Hidden |
| Document_Author | Visible |
| Document_Last_Author | Visible |
| Document_Subject | Visible |
| Document_Title | Visible |
| Document_Type | Hidden |
| Document_Date | Hidden |
| Email_Author | Visible |
| Email_Received | Visible |
| Email_Sent | Visible |
| Email_Subject | Visible |
| Email_Thread_Index | Hidden |
| Encrypted | Visible |
| File_Name | Visible |
| File_Path | Visible |
| Image_File | Hidden |
| Last_Print_Date | Visible |
| Mailbox_File | Visible |
| Mailbox_Path | Visible |
| Modified_Date_Time | Visible |
| Native_File | Hidden |
| Recipients | Visible |
| Root_Folder | Visible |
| Shortcut | Hidden |
| Tags | Hidden |
| Text_File | Hidden |
| Title | Hidden |
Important Field Notes:
- Custodian/Custodians are visible under Settings > Import > Custodians
- Have existing/historical Tags you want to migrate to Nextpoint's Additional Tags field? Your column header should read Tags and the values should be semicolon delimited (e.g. Bob Randolph; hard copy document; Production 1; 10/22/2019).
Protected System Fields
Data cannot be imported into any of the below fields because they are generated by the Nextpoint application.
If you’d like to map any of the below values into your database, you will be required to setup a field with a different name.
Note: Common fields replaced and suggested replacement values provided in parenthesis:
| Protected Field | Suggested Alternative |
|---|---|
| Filename | (use existing File Name) |
| Filepath | (use existing File Path) |
| batch | |
| batch_id | |
| batch_ids | |
| Bates | (use existing Bates_start and Bates_end as applicable) |
| Bates_stamped | |
| Confidentiality | (Conf_Status) |
| Confidentiality_Status | (Conf_Status) |
| Content_hash | (ContentHash) |
| Created_at | (use existing Created_Date_Time) |
| Created_at_gmt | (Created_Date_Time) |
|
Created_on |
|
| Description | (Desc) |
| Delete_at_gmt | |
| Deposition_id | |
| Deposition_names | |
| Deposition_volume_id | |
| Display_name | |
| Doc_props | |
| Document_properties | |
| Domain | (EmailDomain) |
| Email_message_id | (MessageID) |
| Email_reply_id | (ReplyID) |
| Email_Thread | (ThreadID) |
| Es_exh_designations | |
| Es_tags | |
| Es_updated_at_gmt | |
| Exhibit_stamped_as | |
| Expansive_hash | |
| Export_ids | |
| File_extension | (Fileext) |
| File_size | (Size) |
| Has_markups | (Has_Annotations) |
| Has_native_placeholder | |
| Highlight_issues | (Annotation_Issues) |
| Highlight_notes | (Annotation_Notes) |
| Id | |
| Incoming_wire_id | |
| Issues | (Document_Issues); (DocIssues) |
| Marked_for_delete | |
| Non_imaged_placeholder | |
| Notes | (Annotations) |
| Npdcase_id | |
| Original_filename | (use existing File Name) |
| Outgoing_wire_ids | |
| Page_count | (Pgcount) |
| Privileged | (Isprivileged) |
| Privileged_status | |
| Redacted | (Isredacted) |
| Relevancy_Status | (Relevancy) |
| Responsive_Issues | (Resp_Issues) |
| Review_status | |
| Updated_at | (Timeupdated) |
| Updated_at_gmt | |
| Updated_on | |
| Verified_page_count | |
| Wire_transfer_tags |
Nextpoint Default and Protected Fields
CSV Overlays
What is an overlay? An overlay is an index file formatted as a CSV that, when applied to your Discovery or Litigation database via the Overlay workflow, can change the metadata or coding of a document which has been previously imported.
Using the load file mapper, users will be provided the opportunity to easily match fields from your csv overlay file to the fields displayed in your Nextpoint database, from within the application. We’ve also included several instances of load file and field validation to ensure your overlay is successful.
Some notable highlights of this feature:
- Ability to perform CSV overlays via the new CSV Overlay Import Type.
- Document mapping key, providing Nextpoint with instructions for locating documents in your database which will be updated with information during the overlay.
- Effortless field matching with 'exact match' recommendations and the ability to search alternative existing fields or create new fields 'on the fly'.
- Backup Overlay File makes reverting data prior to the overlay stress-free.
This functionality is available for Advanced users only.
Formatting Requirements for a CSV Overlay
Your Overlay file should be formatted so that the first row of the CSV contains column headers (field names) and any subsequent row reflects a specific document.
Coding overlay column headers fall into one of three categories:
- Keys: Used to locate and target a specific document to which data will be applied
- Nextpoint System Fields: always available in all databases, not configurable
- Nextpoint Custom Fields: user-created fields which can be configured
We suggest supplying column headers as lowercase, underscore separated values. The system will do its best to coerce values like "Document Date" to "document_date", but we recommend formatting as much as possible before uploading to the File Room, as described in the steps below.
Each CSV row must contain at least one document key value. The Document Key provides Nextpoint with instructions for locating documents in your database which will be updated with information during the overlay. Available keys include:
| CSV Header | Description |
|---|---|
| npid | The Nextpoint ID of a document. This can be exported in bulk from Nextpoint using the "Nextpoint ID" export field. For individual documents, this is listed under "System Data" as DocID. Tip: This is the most accurate and preferred key for an overlay; it is guaranteed to be unique and will result in the fastest execution speed. |
| bates_start | The Bates value that corresponds with the first page of a document in Nextpoint. This can be exported in bulk from Nextpoint using the "Bates Range Start" export field. |
After your Document Key is defined in the first column of your overlay, all other fields which you intend to overlay in your database will fill in to the right in the first row. Subsequently, each row defines the data which will be overlaid in each particular field. See illustration above.
Outlined below is a list of Default System Fields to which you can overlay data to in your Nextpoint database. The only other fields which can be overlaid in your database are Custom (user-created) fields as described in the tab to the right.
| CSV Header | Description |
|---|---|
| app_name | Appears as "App Name" under coding fields. |
| author | Does not appear under coding fields, but is available for mapping. Often a merged field of document_author and email_author. |
| bcc | Appears as "BCC" under coding fields. |
| cc | Appears as "CC" under coding fields. |
| created_date_time | This field must be a date and will be parsed as such. Note: American date formatting is always utilized, it is recommended that dates are provided in the format MM/DD/YYYY. |
| document_author | Appears as "Document Author" under coding fields. |
| document_date | Appears as "Document Date" under coding fields. This field must be a date and will be parsed as such. Note: American date formatting is always utilized, it is recommended that dates are provided in the format MM/DD/YYYY. |
| document_last_author | Appears as "Document Last Author" under coding fields. |
| document_subject | Appears as "Document Subject" under coding fields. |
| document_title | Appears as "Document Title" under coding fields. |
| document_type | Appears as "Document Type" under coding fields. |
| email_author | Appears as "Email Author" under coding fields. |
| email_received | This field must be a date and will be parsed as such. Note: American date formatting is always utilized, it is recommended that dates are provided in the format MM/DD/YYYY. |
| email_sent | This field must be a date and will be parsed as such. Note: American date formatting is always utilized, it is recommended that dates are provided in the format MM/DD/YYYY. |
| email_subject | Appears as "Email Subject" under coding fields. |
| encrypted | Appears as "Encrypted" under coding fields. |
| file_name | Appears as "File Name" under coding fields. |
| file_path | Appears as "File Path" under coding fields. |
| last_print_date | This field must be a date and will be parsed as such. Note: American date formatting is always utilized, it is recommended that dates are provided in the format MM/DD/YYYY. |
| mailbox_file | Appears as "Mailbox File" under coding fields. |
| mailbox_path | Appears as "Mailbox Path" under coding fields. |
| modified_date_time | This field must be a date and will be parsed as such. Note: American date formatting is always utilized, it is recommended that dates are provided in the format MM/DD/YYYY. |
| np_folder_prefix | Does not appear under coding fields, but is available for mapping. Provides user with ability to add documents to a folder via an Overlay file. See coordinating np_folder_position field. Applicable value(s) which can be mapped via Overlay are any existing folder abbreviation/prefix. This article outlines how to use it. |
| np_folder_position | Does not appear under coding fields, but is available for mapping. Provides user with ability to add documents to a folder in a particular order via an Overlay file. See coordinating np_folder_prefix field. Applicable value(s) which can be mapped via Overlay are digits outlining which position you would like the document to take in a particular folder (e.g. a cell value of 6 would be the 6th document in the folder). This article outlines how to use it. |
| recipients | Appears as "Recipients" under coding fields. |
| root_folder | Appears as "Root Folder" under coding fields. |
| shortcut | Appears as "Shortcut" under coding fields. |
| title | Does not appear under coding fields, but is available for mapping. Often a merged field of document_title and email_subject. |
In addition to Nextpoint's default system field described in the previous tab, you can also overlay information to any custom fields which you create in your database.
All column headers should be lowercase with any spaces in the custom field names replaced by underscores. For example: “City Name” is represented in the csv header row as “city_name”
The way to format the CSV cells containing your document-level data will depend on the type of field being overlaid.
| CSV Header | Description |
|---|---|
| Freeform & Paragraph | Any value can be provided. |
| Date |
The coding overlay tool will attempt to parse data provided for this field into YYYY-MM-DD HH:MM:SS 24-hour format, which is how we require users to enter dates into our custom date fields, as it is sortable. If the provided value cannot be parsed into that format, the originally provided value will be overlaid. Note: American date formatting is always utilized, it is recommended that dates are provided in the format indicated above. |
| List & Locked List | The exact list option you want to be applied. If this list option does not exist it will be created. |
| Picklist |
These should be provided as a semi-colon delimited list of picklist options that should be checked. For example: "Option 1;option2; option 3" Trailing and preceding whitespace will be trimmed from the delimited values. This will go into Nextpoint as:
Keep in mind, all original picklist coding (the selections, not the options themselves) will be removed and replaced with what is provided in this field. |
| Single Checkbox | If any data is provided in this field, the checkbox will be checked. If an empty cell is provided, the checkbox will not be checked. |
Overlay Formatting Notes
Bates, Custodians, Native_File and Text_File cannot be overlaid. Please contact the Nextpoint support team for assistance with this action.
Additionally, please ensure your load file is saved as a CSV with UTF-8 Encoding.
Importing a CSV Overlay with the Load File Mapper
Before initiating any Overlay, it’s important to note that the documents to which you would like to overlay data must already be imported into your Nextpoint database.
1 | Upload CSV Overlay load file to File Room
-
Navigate to the File Room: In Discovery databases, via DATA File Room. In Litigation via MORE Data File Room.
-
Upload your CSV Overlay load file(s) to the File Room via one of the four following options:
Upload a folder of files via Drag & DropSelect the folder(s) on your desktop, thumb drive, or other location, and Drag & Drop into your File Room. All contents and subdirectory information will be maintained.
Upload loose file(s) via Drag & DropWithin the File Room, click the green Create Folder name your folder and select OK.
Maintaining an organizational system for your data Overlay files will help ensure you can best track your various overlays as time progresses.
Click into your newly created folder, and drag and drop your Overlay file into the folder location.
Upload an individual file via Upload Files buttonWithin the File Room, click the blue Upload Files button and select your Overlay file from the directory prompt.
Note: This option only allows upload of one CSV Overlay file at a time.
Request file upload from third partyIf you are requesting a CSV Overlay file from a third party (e.g. counsel, etc.), you can securely Request Files from your Nextpoint File Room.
It is a simple process in which you ("requestor") request files from a specified third-party, the "recipient" of that request receives a secure link to upload their respective file(s), and then you can access the uploaded files right away.
Read more on requesting from third parties here >>
Best Practices for Uploading
- Overlay files must be in CSV format (no excel or .dat)
- We recommend organizing your CSV Overlay files with the data set you are overlaying or in a specific Overlay Files folder in the File Room. The better organized you are here, the easier it will be to cross reference later, as needed.
2 | Initiate Overlay Workflow
Once your Overlay file has been uploaded to the File Room, navigate to the Overlays tab, also under the top-level DATA tab.
Click the blue Start an Overlay button to initiate the overlay workflow.
3 | Confirm Overlay Settings
Once you initiate the overlay workflow, you will be navigated to the next step, Overlay Settings. Here, you will verify and/or outline settings applicable to your current overlay.
The Overlay Settings you will need to set are as follows:
-
Type of Import: Verify Overlay is selected. Click the pencil icon
to edit, as needed.
-
Overlay File Selection: Click the folder icon
to access the File Room contents and select your overlay file.
- Overlay Name: Recommended for most efficient tracking once the overlay is complete.
- Backup restored data: On by default, this setting will ensure existing field values are preserved in a separate CSV file before any information is overlaid. The resulting CSV file will be helpful if you ever need to revert an overlay to the previous state of coding/metadata.
4 | Review & Set Overlay Mappings
Load File Mapper at a Glance
When the Overlay Import Type is selected, the third step in the guided workflow is where you review and set your overlay file mapping specifications in the load file mapper.
The key objective of the load file mapper is to ensure the fields from your overlay file are mapping to the appropriate Nextpoint database fields. If the mapping is not made, Nextpoint will not know where to send the load file information during processing.
Once in the load file mapper, you will be presented with four core columns which provide you with data from your overlay file, a list of database fields to which load file data can be mapped, and a status indicator of your field mapping. These four columns are described in detail below.
A | The first column, Field from your load file, will be populated with a list of the fields contained in the header row of your selected overlay file.
B | The second column, Destination field, contains a list of available fields in your Nextpoint database to which corresponding overlay file field data can be mapped. Nextpoint will attempt to auto-map exact matches. You can also edit the auto-mapped recommendation, create a new destination field 'on the fly', or choose to skip the field within the field drop-down.
C | The third column, Load file preview, contains a three row preview of the data within the load file for each field. This information can provide assurance the appropriate values correspond with a particular field.
D | The fourth column, Field status, provides confirmation if a load file field has been successfully mapped to a destination field or if additional user action is required prior to import.
A Closer Look at the Destination Field Column
Now that you have an overview of four columns within the mapper, let’s take a closer look at the Destination field column and the different opportunities for mapping information from your Overlay load file to your Nextpoint database.
The first row in your Destination field column will be populated with document mapping key. The Document mapping key provides Nextpoint with instructions for locating documents in your database which will be updated with information during the overlay. See Formatting Requirements for a CSV Overlay above for further information on document mapping keys.
Click on the gear icon to validate the load file field which Nextpoint should use to locate the documents for your overlay.
Note: In the above visualization, you will note the duplicative ProdBeg (Bates_start) is skipped by the user. Bates cannot be overlaid, so there is no need to map that field beyond the document mapping key selection.
Auto-Mapping
For each load file field in column 1, Nextpoint will first look for an exact match with an existing Destination field. If an exact match is found, the field match will be auto-mapped.
For example, in the below image, BegAttach and BCC were an exact match so they are auto-mapped.
User-Mapping
For each load file filed in column 1, if an exact match can’t be made, fields can be user-mapped to existing or new fields.
For example, Tags were not auto-mapped in the below image because no exact match was present (and tags cannot be mapped). Therefore, we ask you as the user to select where that information should be mapped to in your Nextpoint database.
If a field needs to be user-mapped, each Destination field input contains a drop-down with a comprehensive list of all the fields which are available for mapping in your current database.
Varying text styles within the drop-down help to distinguish different types of fields.
- Normal (unbolded) field names are default fields which exist in every Nextpoint database. When possible, we recommend mapping to these default fields for streamlined searching later on.
- Grayed out fields are already in use within the current mappings and cannot be used twice. Remove/change the other field mapping in order to utilize a grayed out field.
- Bolded fields are user-created (or custom) fields which can be used in the mapper.
Create a New Field for Mapping
If the field you would like to use for mapping isn't available in the Destination field list drop-down, you are provided with the option to create a new database field on the fly. To create a new field while in the mapper, click the icon to the right of the applicable Destination field list.
Upon clicking the icon, confirm the auto-populated name or set your preferred field title, then click Create.
As soon as you click Create, that field will now be in your database and data can be mapped to that field moving forward.
You can skip a field mapping by selecting Skip Field in the Destination field drop-down, but skipping a field during mapping means skipping data going into your database. Therefore, we do require users explicitly skip a field for mapping via the Skip Field selection.
You can also skip all unmapped fields as a last selection before moving to the next step, processing.
5 | Initiate Your Overlay
Once all load file mappings are complete, click the blue Overlay button at the bottom right and processing will begin. Nextpoint will take your original overlay file, make a copy with any mapping requirements, and make any necessary conversions on the back-end during processing.
Warning!
All fields provided will be entirely overwritten by what is in the provided CSV. This means that if a field has an empty value in the CSV, that field will have its existing data removed in Nextpoint.
Before initiating your Overlay, we recommend ensuring Backup restored data is Enabled in when confirming your overlay settings. Doing so will ensure you are able to revert your overlay changes, if needed.
6 | Review Overlay Results
After your overlay is complete, we recommend reviewing your overlay results to ensure your overlay mapped as expected. You can review any overlay batch details via DATA > Overlays > click on the name of the overlay in the overlay batch list.
Several reports/files are generate during processing which will be helpful in your review:
- The overlay file used for processing will be added to your original folder selected from the File Room for import. Any selections made in the load file mapper will be reflected in this resulting file.
- Field Mapping Report: Fields used for mapping will be stored as a report in the individual overlay details page.
- Processing Results Report: Outlines the document mapping key utilized, which Nextpoint DocIDs the key matched, and the associated coding/metadata applied during overlay.
- Backup Overlay Load File: An CSV export of documents in your overlay batch and the coding/metadata in place prior to processing the overlay. Helpful in the event you need to revert your overlay for any reason.
7 | Family Link
Note: This step is only applicable if you were overlaying email family relationships via the combination of Bates_start document mapping key and BegAttach field.
The very last step once processing is complete is to Family Link your overlay batch. Family linking is the process which visually links imported (produced) emails and attachments using a set of two fields: "Unique ID" and "Family ID".
This takes place after import processing is complete and via DATA Overlays click on the overlay name to view Batch Details. At the bottom right of your Batch Summary page, click . A sliding menu will appear with checkboxes & headers for Batch, User, Uploaded, Docs (batch number, importing user, upload date, batch document count), and a selection drop-down for your Linking Options.
- Select the overlay batch(es) you would like to Family Link.
- Select the fields Nextpoint should use to link parent emails to their attachments via the Linking Options drop-down.
- Click Family Link Documents
- Confirm your selection by clicking Run Family Linking.
Read more on family linking here >>
After processing is complete, you will receive a green Complete, yellow Complete With Errors, or red Error status. Read more on Common Overlay Errors and Solutions here >>
How to Import a CSV Overlay
Amazingly Simple Load File Imports.
At the beginning of 2021, we rolled out the much anticipated second installment of enhancements to the Nextpoint import experience with the introduction of a load file mapper for produced data imports.
This new load file mapper provides the opportunity to easily match fields from your load file to the fields displayed in your Nextpoint database, from within the application. We’ve also included several instances of load file and field validation to ensure your import is successful.
Note: Imports available to users with Advanced user permissions, only.
Prefer a video presentation on this topic? You can find one here
Table of Contents
- Upload data to Nextpoint File Room
- Select files for import from Nextpoint File Room
- Confirm Import Data Settings
- Review & Set Load File Mappings
- Initiate Your Import
- Review Import Results and Family Link
Importing Produced Data with the Load File Mapper
1 - Upload Data to Nextpoint File Room
The first step to importing in Nextpoint, is to upload your data to the File Room. File Room is a secure ‘data-bank’ for storing all your confidential files in your database, and comes with a built-in, high-speed, multi-file uploader to get data into Nextpoint quickly and efficiently.
To get started uploading your data to the File Room:
It is important to note, we recommend you upload your production data folder to the File Room unzipped. This will ensure all subdirectories and load files are accessible during the guided import workflow and processing.
-
Navigate to the File Room: In Discovery databases, via DATA File Room. In Litigation via MORE Data File Room.
-
Upload your files to the File Room via one of the four following options:
Upload a folder of files via Drag & DropThe primary, and recommended, function for uploading data to your File Room.
Select the folder(s) on your desktop, thumb drive, or other location, and Drag & Drop into your File Room. All contents and subdirectory information will be maintained.
Upload loose files via Drag & DropWithin the File Room, click the green Create Folder name your folder and select OK.
Click into your newly created folder, and drag and drop your loose files into the folder location.
Maintaining an organizational system for your data uploads will help ensure you can best track your various imports as time progresses.
Upload an individual file via Upload Files buttonWithin the File Room, click the blue Upload Files button and select the file from the directory prompt.
Note: This option only allows upload of one file at a time (loose file, .zip, mailbox archive).
Request file upload from third partyYou can securely Request Files from any third party (clients, counsel, etc...) from your Nextpoint File Room. It is a simple process in which you ("requestor") request files from a specified third-party, the "recipient" of that request receives a secure link to upload their respective files, and then you can access the uploaded files right away.
Read more on requesting from third parties here >>
-
After you initiate your upload in the step above, the data will begin to upload and the status of the upload will be displayed on your screen.
It is important you do not navigate away from the File Room during an active upload. This will cancel the ongoing upload, and you will need to delete all files from the File Room which were interrupted during upload and begin again.
Need to keep working on other action items? Open a new tab, or duplicate your current tab and you are set!
Read our File Room Best Practices here >>
2 - Select Files for Import
Once your data has been successfully uploaded to the File Room, you can select that data directly from the File Room to initiate the guided import sequence.
To select your data for import click the blue Import button next to any production folder in the File Room.
When initiating your import from the File Room, Nextpoint will detect the type of data you selected and will automatically set such in the first step of the guided import sequence. If a load file is present anywhere in your production folder, your import type will be recognized as Production with Load File.
Alternative option for selecting files for import
While selecting files for Import from the File Room, as described above, is the recommended workflow for initiating your imports, you may also initiate your import via DATA Imports.
The difference you will notice in starting from this location will be the added Import Type selection in Step 1. After making this selection, you will meet the 'Import from File Room' sequence at Step 2, Import Data Settings.
Read more on Import Types here >>
3 - Confirm Import Data Settings
Once your files have been selected for import, you will be navigated to the next step of the import sequence, Import Data Settings. Here, you will verify and/or outline settings applicable to your current import.
For Production with Load File imports, common Import Data Settings you will see are as follows:
-
Type of Import: Verify Production with Load File is selected. Click the pencil icon
to edit, as needed.
-
Selected Files for Import: If you initiated your import from the File Room, verify the selected files or modify by clicking the folder icon
to access the File Room contents.
- Load File Selection: Nextpoint will use the first load file found, but if there is a particular load file you would like to use, you can change such in the drop-down.
- Batch Name: Recommended for most efficient tracking once the data has been imported.
- Assign Custodian on Import: For produced data sets with an accompanying load file, Custodian information is typically provided in the load file (and will be mapped in the next step).
-
Add to Folder on Import: Search list of existing folders or add new via the folder + icon
.
- Deduplication and DeNIST Detection: Deduplication and DeNIST settings are turned OFF as a pre-set recommendation for this type of import to help mitigate risk of deduping any data produced TO you.
FAQ: Production with Load File Import Type not recognized?
If you reach the Import Data Settings page and the Import Type does not indicated "Production with load file", then a load file was not detected during the selection process. If this is the case, return to the File Room and make sure that a DAT or CSV load file is in your production folder. We typically see load files present in the DATA folder or at the root of the main production folder.
4 - Review & Set Load File Mappings
Load File Mapper at a Glance
When Production with Load File Import Type is selected, a new step has been added to the guided workflow in which you can review and set your load file mapping specifications, the load file mapper.
The key objective of the load file mapper is to ensure the fields from your load file are mapping to the appropriate Nextpoint database fields. If the mapping is not made, Nextpoint will not know where to send the load file information during processing.
Once in the load file mapper, you will be presented with four core columns which provide you with data from your load file, a list of database fields to which load file data can be mapped, and a status indicator of your field mapping. These four columns are described in detail below.
A | The first column, Field from your load file, will be populated with a list of the fields contained in the header row of your selected load file.
B | The second column, Destination field, contains a list of available fields in your Nextpoint database to which corresponding load file field data can be mapped. Nextpoint will attempt to auto-map matches between your load file and the database fields based on common field name formats. You can also edit the auto-mapped recommendation, create a new destination field 'on the fly', or choose to skip the field within the field drop-down.
C | The third column, Load file preview, contains a three row preview of the data within the load file for each field. This information can provide assurance the appropriate values correspond with a particular field.
D | The fourth column, Field status, provides confirmation if a load file field has been successfully mapped to a destination field or if additional user action is required prior to import.
A Closer Look at the Destination Field Column
Now that you have an overview of four columns within the mapper, let’s take a closer look at the Destination field column and the different opportunities for mapping information from your load file to your Nextpoint database.
The first one or two rows in your Destination field column will be populated with image mapping criteria. Image mapping criteria provides Nextpoint with instructions for locating your production images during processing.
Click on the gear icon to validate the load file field which Nextpoint should use to locate the images for your production.
Auto-Mapping
For each load file field in column 1, Nextpoint will first look for an exact match with an existing Destination field. If an exact match is found, the field match will be auto-mapped.
For example, in the below image, BegAttach and EndAttach were an exact match so they are auto-mapped.
User-Mapping
For each load file filed in column 1, if an exact match can’t be made, fields can be user-mapped to existing or new fields.
For example, ProdBeg and ProdEnd were not auto-mapped in the below image because no exact match was present. Therefore, we ask you as the user to select where that information should be mapped to in your Nextpoint database.
If a field needs to be user-mapped, each Destination field input contains a drop-down with a comprehensive list of all the fields which are available for mapping in your current database.
Varying text styles within the drop-down help to distinguish different types of fields.
- Normal (unbolded) field names are default fields which exist in every Nextpoint database. When possible, we recommend mapping to these default fields for streamlined searching later on. For more information on linking fields in a load file to Nextpoint's system fields, please review the Nextpoint System and Protected Fields support article.
- Grayed out fields are already in use within the current mappings and cannot be used twice. Remove/change the other field mapping in order to utilize a grayed out field.
- Bolded fields are user-created (or custom) fields which can be used in the mapper.
The following fields are VERY important to map correctly if they exist in your load file.
bates_start/bates_end - usually called "Prodbeg"/"Prodend" or something similar, these fields map your Bates numbers to your documents and are usually used for imaging (if the images are named by Bates numbers).
begattach - This fields must be created as custom fields in Nextpoint and should contain the metadata in the load file associated with family relationships. Usually called Production Begin Attachment or some abbreviation of that. If parent ID exists in the load file, that usually could work for begattach if the parent ID's are the parents' Bates numbers.
native file - If your production contains a "Natives" folder and native files, you need to map the paths to these natives onto the "native file" field and validate the paths by clicking on the gear icon next to the field and selecting the folder containing the native files.
text file - If your production contains a "Text" or "OCR" folder containing the search text for each document, you need to map the paths to these text files onto the "text file" field and validate the paths by clicking on the gear icon next to the field and selecting the folder containing the text files.
Create a New Field for Mapping
If the field you would like to use for mapping isn't available in the Destination field list drop-down, you are provided with the option to create a new database field on the fly. To create a new field while in the mapper, click the icon to the right of the applicable Destination field list.
Upon clicking the icon, confirm the auto-populated name or set your preferred field title, then click Create.
As soon as you click Create, that field will now be in your database and data can be mapped to that field moving forward.
You can skip a field mapping by selecting Skip Field in the Destination field drop-down, but skipping a field during mapping means skipping data going into your database. Therefore, we do require users explicitly skip a field for mapping via the Skip Field selection.
You can also skip all unmapped fields as a last selection before moving to the next step, processing.
Before finalizing your mapping, the last, but critical decision point concerns validating pathing information for Natives and Text. This validation point ensures your load file is giving Nextpoint the correct instructions to find the different pieces of your data set to create unique documents during processing.
To validate, click the gear icon, and confirm the relative path of the location of the respective. In other words, the top-level folder in which Nextpoint should start looking for text or native files during processing. Most often, this will be the top-level TEXT and NATIVES folders in your production set.
Load File Mapper Tips + Tricks
In the event "something doesn't seem right" when you are in the mapper (spinning, no text/native paths showing, etc.), we recommend the following troubleshooting tips and tricks
- If the native/text file path listings are not displaying when you go to validate, your parent production folder in the File Room and the folder in the native/text paths may be mismatched. Update your File Room folder to correlate with the native/text paths or update the native/text paths to match the File Room folder, then re-upload.
- If your load file is not being read and/or has uncommon characters contained within, make sure your load file is saved with UTF-8 Encoding.
5 - Initiate Your Import
Once all load file mappings are complete, click import and processing will begin. Nextpoint will take your original load file, make a copy and make any necessary conversions on the back-end during processing.
6 - Review Import Results and Family Link
After your import is complete, the converted load file copy will be added to your original folder selected from the File Room for import. Fields used for mapping will be stored as a report in the individual import details page.
The very last step once processing is complete is to Family Link your import batch. Family linking is the process which visually links imported (produced) emails and attachments using a set of two fields: "Unique ID" and "Family ID". This takes place after import processing is complete and via DATA Import and by clicking on the batch name to view Batch Details Family Linking.
Read more on family linking here >>
How to Import Produced Data with a Load File
Note: This functionality is available for Advanced users only.
In early 2021, we rolled out the much anticipated second installment of enhancements to the Nextpoint import experience with the introduction of a load file mapper for produced data imports. This new and improved workflow will accommodate most produced data imports with a load file, but we understand some users may still want to utilize the workflow they were previously used to. We refer to this as a Manual Import - one in which you modify your load file before uploading to the File Room and do not use the load file mapper.
In addition to serving existing workflows, this Manual Import Type is helpful in the event you have data for import which has already been processed. In particular, we see this most often with a native production import which already has attachments extracted from their parent emails. This approach allows you to choose to disable child processing which prevents attachments from extracting from their parent emails a second time.
All steps outlined below based on the assumption of a ranged image import. See below for a breakdown of the typical format for such an import:
What Does a Ranged Image Import Look Like?
Your produced data set will contain up to 4 folders. At the very least, this type of data set must contain an "Images" or "IMAGES" folder):
IMAGES - This folder contains the document pages, each a one-page image file. Image file pages will be in the .tif or .jpg format, and the files will be named by their Bates numbers.
TEXT - This folder contains the OCR text information, and can be either one text file per page, or one text file per document. If included, the OCR text files will be named by the corresponding Bates start numbers of the document they represent.
NATIVES - This folder contains any native files that accompany each document. If included, the Native files will be named by the corresponding Bates start numbers of the document they represent.
DATA - This folder contains load files associated with your data set. All information contained within the load file serves as an instruction manual for Nextpoint during import so the correct IMAGES, NATIVES, and TEXT files are combined to create unique documents and those documents can be viewed, searched, and filtered.
These load files will, at the minimum, contain document boundary information outlining which image files should be extracted from the aforementioned IMAGES directory and combined to create unique documents during import. Additionally, load files will oftentimes include data about the documents (Author, Recipients, Bates range, etc..) and have the paths to text and native files within the TEXT and NATIVES directories.
Below is what a graphic representation of a what a common ranged image data set looks like (when it is expanded):
Produced Data Import Instructions
The following steps have been strategically organized to help guide you through the import of produced data using the Manual Import Type selection. We recommend navigating the instructions one step at a time.
It is also recommended you review these required considerations for produced data imports prior to proceeding with the below Produced Data Import Instructions.
Open your production folder and review the subdirectories for the following specifications:
- DATA: Check for a .CSV and/or .DAT load files. If you have a .DAT, you must convert this file type to a more user-friendly .CSV format in Step 2. Additional file types such as .lfp, .log, .opt, may be discarded for purposes of these instructions.
- IMAGES: Check for single-page tiffs/jpgs named by their Bates number. In some cases, the image files are provided as PDFs named by their Bates start and that is acceptable.
- TEXT: Check for per-document text files. These are named by the Bates start of each document.
- NATIVES: Check for any native file types which have been provided. These are named by the Bates start of the document. Oftentimes Excel, Powerpoint, and other file types which do not image well (or at all).
A reminder of what you should anticipate seeing in a ranged image import:
FAQ: My TEXT (.txt) files are located in my IMAGES folder. What do I do?
If your text (.txt) files are not in a separate TEXT folder, that is typically OK. You will need to supplement a field in your load file titled image_extension. We further explain how to add this field when you get to Step 3 | Load File Configuration --> Document Boundaries.
For successful import to Nextpoint, load files must be saved in a CSV (comma-separated value) format.
If you located a .DAT in your DATA folder in Step 1, you must first convert the file to a .CSV before proceeding. Not only is it required for the Manual Import Type in Nextpoint, but making the conversion from .DAT .CSV allows you to more easily view your load file and make necessary modifications (in Excel).
VS 
Steps to convert a .DAT to .CSV:
- Open your .DAT file in a text editor such as TextPad or Sublime Text (this is the application preferred by Nextpoint Data Analysts).
To open, right click on the .DAT file Select "Open With" click on the text editor program you would like to use.
- Once opened in your text editor, the values in your .DAT file will likely be separated by the symbols þ□þ (thorn, □ (ASCII 20), and another thorn). So, the text displayed resembles this:
þProdBegþ□þProdEndþ□þBegAttachþ□þENDAttachþ□þCustodiansþ
If using Sublime text, your beginning state may look like this:
þProdBegþ<0x14>þProdEndþ<0x14>þBegAttachþ<0x14>þENDAttachþ - Using the functions in TextPad or Sublime, you must complete a find and replace on the aforementioned characters to convert the load file format. Use the following sequence:
- Find-replace all " (quote) with "" (double quote) - This will ensure your line breaks remain consistent.
- Find-replace all þ (thorn) with " (quote)
- Find-replace all □ (ASCII 20) with , (comma) - □ may read as <0x14> if working in Sublime Text.
- Find-replace all " (quote) with "" (double quote) - This will ensure your line breaks remain consistent.
- The find and replace actions you take above in Step 3 will result in all values being now separated by "," (quote comma quote, or in other words, comma separated values / CSV). The text displayed should resemble this:
- Though it is often not necessary, at times DAT files are not provided in UTF-8 format. Therefore, it is best practice to save the file with UTF-8 encoding. Select "File" > "Save with Encoding" > "UTF-8".
- Save your file as: nextpoint_load_file.csv.
- Locate your nextpoint_load_file.csv in your production folder and open the file in Excel. Your values will be separated into columns, and much easier to work with moving forward.
Prefer a video tutorial on converting your load file from .DAT to .CSV? Watch here >>
Troubleshooting: "My find-replace is not working correctly."
Some clients using Textpad or Notepad have had trouble in the past with certain characters being unexplainably replaced during this process, thus altering the desired results of their ending CSV (e.g. "" replacing the character sequence of "th" ). If you notice nuances in your resulting CSV, some users have found Sublime Text to be a helpful text editor alternative.
Now, you will make modifications to your .CSV load file AND Nextpoint database fields to ensure all information in your load file has somewhere to go when imported into your Nextpoint database.
Once your load file is converted in Step 2, open your resulting .CSV in Excel, if you have not already. Each row in your load file (spreadsheet) contains the information for one document. 
DOCUMENT BOUNDARIES
- In your load file, review the first two columns to confirm Bates_start and Bates_end or a general control number is present (e.g. Bates_begin/Bates_end OR prod_beg/prod_end)
- These beginning and ending numbers will identify which ranges of single page tiff/jpgs should be pulled from the IMAGES folder and combined to create individual documents.
- These beginning and ending numbers will identify which ranges of single page tiff/jpgs should be pulled from the IMAGES folder and combined to create individual documents.
- If these two columns are named something other than Bates_start and Bates_end, you must rename these columns in your load file to Bates_start and Bates_end in order for your import to be successful. *Capitalization and underscores (vs. a space) do not matter.
If your Image files were provided as PDF's named by their Bates start in Step 1
If you have singular images for documents, such as a PDF (likely named by Bates start), you must add a column to your load file titled image_file. After you add the image_file column to your load file, you must then populate the cells below with the appropriate files' relative path information.
Read more here on how to populate the image_file column.
FAMILY RELATIONSHIPS
- In your load file, review your column headers to confirm Begattach and Endattach column headers are present. Sometimes these are titled production_begin and production_end, etc...
- These beginning and ending attachment numbers are vital to establishing email family relationships after your import is complete.
- Establishing these visual relationships after import is referred to as Family Linking and is further covered below in step 7.
- In your load file, if named anything other than begattach and endattach (e.g. prod_begin/end, production_beg_attach/production_end_attach, etc..), you must rename these columns to Begattach and Endattach in order for your import to be successful. *Capitalization and underscores (vs. a space) do not matter.
- In your Nextpoint Discovery database, if begattach and endattach fields are not present under Settings Coding Fields, then you must add begattach and endattach as Freeform fields.
- Importing to a Litigation database? You can add these fields under MORE Settings Coding Fields.
If your text (.txt) files were contained in your IMAGES folder in Step 1
If your image and text files are contained within the same folder (most oftentimes, IMAGES), you must add a column header in your load file titled image_extension and add the value tif|tiff|jpg|jpeg in all of the rows. Adding explicit file extensions for images will limit what sorts of files Nextpoint should consider as potential page images (and keep it from mistaking text files for images).
METADATA FIELDS
Once your document boundaries and family relationships are verified in the steps above, the next step is to verify the metadata fields in your load file. It is recommended you address the remaining load file fields in the following order:
- DEFAULT FIELDS
- First, see this list of Default Fields and Document Attributes, Exhibit A ("Exhibit A"). Fields on this list already exist in each Nextpoint database.
- If a column header in your load file matches, or is similar to, a field listed in Exhibit A, you must rename your load file header to match the field name from Exhibit A.
IMPORTANT: Your load file column header must exactly match the field name in Default Fields and Document Attributes, Exhibit A in order for your import to be successful.
FAQ re: Default Fields
Q: I'm concerned if I use a header in my load file that is similar to, but not exactly the same, as a Default Field, I will mess up my import.
A: Don't worry, you will not "break" the import! Sometimes, it may be necessary to create similar fields to what is existing, and that is OK. For example, Nextpoint has a default Email_Sent field which incorporates the date AND time an email was sent. If you have Email_Sent_Date and Email_Sent_Time in your load file, it is perfectly fine to keep your load file fields named as such. However, if you do this, please make sure to set up the corresponding field in your Nextpoint Discovery database by navigating to Settings Coding Fields Create New Select Freeform Field enter the field name click "Create". This gives your load file data somewhere to go in Nextpoint once imported.
- PROTECTED SYSTEM FIELDS
- Next, see this list of Protected System Fields, Exhibit B ("Exhibit B"). Data cannot be imported into any of these fields because they are generated by the Nextpoint application.
- If a column header in your load file matches a field listed in Exhibit B, you must rename the fields in your load file to something different AND ALSO set up a corresponding field in your database.
- Renaming the field in your load file eliminates the use of a protected system field, and setting up the corresponding field in your database gives the information from your load file "somewhere to go" when your load file is ultimately imported into Nextpoint.
- Suggestions for what to rename the field in your load file (and subsequently add to your database) are provided in parentheses next to each field in Exhibit B.
RENAME FIELD IN YOUR LOAD FILE TO SOMETHING OTHER THAN PROTECTED SYSTEM FIELD:
- If you change a column header in your load file (in 2.2 above), you must also set up the corresponding field in your Nextpoint Discovery database by navigating to Settings Coding Fields Create New Select Freeform Field enter the field name click "Create".
- Importing to a Litigation database? You can add these fields under MORE Settings Coding Fields.
- TEXT & NATIVE PATHS
Next up? Text and Native Paths.
During import, Nextpoint needs to know which text and native files to pull from their respective TEXT and NATIVE folders, and line up with their corresponding document image(s) from the IMAGES folder. This is accomplished by using the text_file and/or native_file column headers in your load file, which contain the path to, and name of, the text and native files, respectively.
Oftentimes, you will not be provided with ALL natives, but rather a select set of file types which do not image well (or at all). If you notice your native_file column lacking information in many of the cells, there is likely nothing wrong with your import, and rather only certain file types were provided natively (e.g. Excels, Powerpoints, Autocad drawings, etc).
- In your load file, make sure these two column headers are named text_file and native_file, respectively. They must be named as such in order for the import to work correctly.
- Then, in your load file, check to make sure the paths are relative to where the load file will be saved. It is critical to your import's success that the nextpoint_load_file.csv includes native and text path information relative to where the load file resides in your production set's base folder.
- The relative path for any native file starts with NATIVES. For example, the first value of the native_file column in the image above is NATIVES\00001\NXP000149.xlsx
- The relative path for any text file starts with TEXT. For example, the first value of the text_file column in the image above is TEXT\00001\NXP_0000127.txt
- See more information on relative paths here.
- In your load file, make sure these two column headers are named text_file and native_file, respectively. They must be named as such in order for the import to work correctly.
- EVERYTHING ELSE
- After addressing the Default Fields, Protected System Fields, and Text/Native Paths above, check for any remaining headers in your load file without a corresponding field in your Nextpoint database.
- If remaining fields are found, create a new Field in your database named exactly the same as your load file header(s).
- Working in a Discovery database? Navigate to Settings Coding Fields Create New Select Freeform Field and enter the field name.
- Working in a Litigation database? Navigate to MORE first, then Settings Coding Fields Create New Select Freeform Field and enter the field name.
After your load file is configured in Step 3, it is critical to the import’s success to save your load file as nextpoint_load_file.csv. This is what indicates to Nextpoint during import "this is a load file and you should use it for this import".
- First, move your load file to the root of your production folder.
- Then, upload your production folder to your Nextpoint File Room by dragging and dropping the unzipped folder.
Initiate the Import Workflow via DATA Import Import Files Select Manual Import.
Once in the guided import workflow, take the following steps:
- Select the Manual Import type in Step 1 of the Import Workflow click Next.
- You will be navigated to Import Data Settings. Here, select your files for import by first clicking on the folder icon next to the Files text.
Then, select your parent production folder click orange Add selected to import list button click blue Import button.
After you've selected your file for import, complete the remainder of the Import Data Settings. This includes naming your import batch, selecting a folder to which you would like your documents added after processing is complete, deduplication settings(we recommend deduplication be off for produced data), and Child Processing settings.
What is Child Processing? When enabled, children of parent emails or contents of zip files will be extracted and process in their standard format as stand-alone documents after import.When disabled, children will not be extracted from their parents. Disabling this feature may be leveraged in use cases such as native production imports with a load file, or in the event that children have been processed separately from the import at hand.
Once the aforementioned settings are determined, click the blue Next button to initiate your import.
My Import Didn't Work, Why?
Every import is different, and sometimes even the smallest nuance can affect the outcome of your import. We have compiled a list of common import errors and solutions for general troubleshooting, and are further developing more specific troubleshooting options for produced data imports, specifically.
Once your Import is has completed processing, you will receive an email notification, and subsequently run “Family Linking" from your Batch Summary page. This is vital for visually establishing parent-email relationships in your import.
- You will receive an email notification stating your import is Complete. Click on the link contained within the email, or navigate back to Data Imports and click on the title of the batch you just imported to view the Batch Summary. Working in a Litigation database? Navigate here via More Data Imports.
- At the bottom right of your Batch Summary page, click
. A sliding menu will appear with checkboxes & headers for Batch, User, Uploaded, Docs (batch number, importing user, upload date, batch document count), and a selection drop-down for your Linking Options.
- Select the batch(es) you would like to Family Link. You can link more than one batch at the same time.
- Select the fields Nextpoint should use to link parent emails to their attachments via the "Linking Options" drop-down. If you utilized Begattach/Endattach in Step 3/Family Relationships, select that option from the drop-down.
- Click "Family Link Documents"
- Confirm your selection by clicking "Run Family Linking"
- Once linking is complete you will receive an email with a link back to the Batch Summary page where you can download and review a report to ensure the emails and attachments were linked without issue.
Have Questions?
Produced data and migration imports can be difficult. After reviewing the above Produced Data Import Approach, if you need additional training or assistance from the Nextpoint Engagement, please contact your Client Success Director or the Nextpoint Support Team.
Default Fields & Document Attributes
For any of the below-listed fields, you do not need to set up a new Field under SETTINGS > Coding. Instead, if you have a header value in Row 1 of your load file, make sure that the load file value matches the below default fields exactly.
Values can be imported by load file headers to the provided values. App Name indicates what your load file header should read. Visible vs. Hidden indicates if a field can be seen under SETTINGS Coding Fields (Visible), or if it isn't seen in the field list but can accept your load file information (Hidden).
| App Name | Visible vs. Hidden |
|---|---|
| Author | Hidden |
| Bates_Start | Hidden |
| Bates_End | Hidden |
| Bates_Range_Start | Hidden |
| Bates_Range_End | Hidden |
| BCC | Visible |
| CC | Visible |
| Created_Date_Time | Visible |
| Custodian | Hidden |
| Custodians | Hidden |
| Date | Hidden |
| Document_Author | Visible |
| Document_Last_Author | Visible |
| Document_Subject | Visible |
| Document_Title | Visible |
| Document_Type | Hidden |
| Document_Date | Hidden |
| Email_Author | Visible |
| Email_Received | Visible |
| Email_Sent | Visible |
| Email_Subject | Visible |
| Email_Thread_Index | Hidden |
| Encrypted | Visible |
| File_Name | Visible |
| File_Path | Visible |
| Image_File | Hidden |
| Image_Range_Start | Hidden |
| Image_Range_End | Hidden |
| Last_Print_Date | Visible |
| Mailbox_File | Visible |
| Mailbox_Path | Visible |
| Modified_Date_Time | Visible |
| Native_File | Hidden |
| Recipients | Visible |
| Root_Folder | Visible |
| Shortcut | Hidden |
| Tags | Hidden |
| Text_File | Hidden |
| Title | Hidden |
Important Field Notes:
- Custodian/Custodians are visible under Settings > Import > Custodians
- Have existing/historical Tags you want to migrate to Nextpoint's Additional Tags field? Your column header should read Tags and the values should be semicolon delimited (e.g. Bob Randolph; hard copy document; Production 1; 10/22/2019).
Protected System Fields
Data cannot be imported into any of the below fields because they are generated by the Nextpoint application.
If you’d like to map any of the below values into your database, you will be required to setup a field with a different name.
Note: Common fields replaced and suggested replacement values provided in parenthesis:
| Protected Field | Suggested Alternative |
|---|---|
| Filename | (use existing File Name) |
| Filepath | (use existing File Path) |
| batch | |
| batch_id | |
| batch_ids | |
| Bates | (use existing Bates_start and Bates_end as applicable) |
| Bates_stamped | |
| Confidentiality | (Conf_Status) |
| Confidentiality_Status | (Conf_Status) |
| Content_hash | (ContentHash) |
| Created_at | (use existing Created_Date_Time) |
| Created_at_gmt | (Created_Date_Time) |
| Created_on | |
| Delete_at_gmt | |
| Deposition_id | |
| Deposition_names | |
| Deposition_volume_id | |
| Display_name | |
| Doc_props | |
| Document_properties | |
| Domain | (EmailDomain) |
| Email_message_id | (MessageID) |
| Email_reply_id | (ReplyID) |
| Email_Thread | (ThreadID) |
| Es_exh_designations | |
| Es_tags | |
| Es_updated_at_gmt | |
| Exhibit_stamped_as | |
| Expansive_hash | |
| Export_ids | |
| File_extension | (Fileext) |
| File_size | (Size) |
| Has_markups | (Has_Annotations) |
| Has_native_placeholder | |
| Highlight_issues | (Annotation_Issues) |
| Highlight_notes | (Annotation_Notes) |
| Id | |
| Incoming_wire_id | |
| Issues | (Document_Issues); (DocIssues) |
| Marked_for_delete | |
| Non_imaged_placeholder | |
| Notes | (Annotations) |
| Npdcase_id | |
| Original_filename | (use existing File Name) |
| Outgoing_wire_ids | |
| Page_count | (Pgcount) |
| Privileged | (Isprivileged) |
| Privileged_status | |
| Redacted | (Isredacted) |
| Relevancy_Status | (Relevancy) |
| Responsive_Issues | (Resp_Issues) |
| Review_status | |
| Updated_at | (Timeupdated) |
| Updated_at_gmt | |
| Updated_on | |
| Verified_page_count | |
| Wire_transfer_tags |
How to Import Produced Data for Manual Import Type
Produced Data Structure at a Glance
While every import can be the slightest bit different, there are several structural components of a produced data set which are helpful to know as you approach your first (or 100th) import.
The production shown below is located on your local computer in the absolute path of C:\Nextpoint\Smith_v_Jones\Defense Productions\UHCProd001
The base folder of this production set is UHCProd001, and the base folder includes subdirectories of IMAGES, NATIVES, TEXT, and DATA (read: load file) associated with the production.
Your nextpoint_load_file.csv load file goes in this base folder.
Relative Path
It is critical to your import's success that the nextpoint_load_file.csv includes native and text path information relative to where the load file resides in your production set's base folder.
The relative path for any native files starts with NATIVES. For example, the first value of the native_file column in the load file would be NATIVES\NATIVES001\UHC0000001.xlsx
The relative path for any text files starts with TEXT. For example, the first value of the text_file column in the load file would be TEXT\TEXT001\UHC0000001.txt
Ready to import?
Have you:
- Followed the instructions outlined in the Produced Data Import Instructions?
- Ensured all data is in your Nextpoint File Room?
- Verified your load file fields, relative path information, and the placement of your load file are correct?
If so, you are set to initiate the import of your base folder. In the aforementioned example, you would select UHCProd001 for import.
Produced Data Structure and Relative Path Information
Outlined below are required considerations for produced data imports, specifically those which utilize a load file.
| Nextpoint Required Specifications |
|---|
| Load file in CSV or DAT formats in UTF-8 format. Ensure that the load file does NOT contain a BOM. |
| Preferred production format includes single page TIFF/JPG, any included natives, and text files named by starting bates number, and a CSV delimited load file with proper relative pathing to native and text files. Image files must be consistently named by their bates numbers without suffixes such as "_CONFIDENTIAL". Confidential coding should be indicated in the load file and stamped onto the face of each image. If page number suffixes (e.g. _0001) are required in image names, then all image files should contain the suffix including the first page such that the image names are a consistent length and sequential. |
| PDF image files with load files are also accepted and should be named by the bates start value of the document without any additional characters (e.g. the suffix "_CONFIDENTIAL"). If PDF image files are not named by bates but the load file contains a path to the image file PDF, the images may be imported, but this may increase load file complexity and limit the ability to import search text. |
| Load file paths included and accurate as to the location of natives (if applicable) and text files. If importing using multipage, document level image files (PDFs), paths to image files must be relative to where the load file resides and should not include starting periods or ending slashes. |
| Text files contain page breaks and are encoded as UTF-8 or ASCII. |
| Replace special characters in field headers with underscores. |
| Field headers are NOT case sensitive. |
See more information on the steps to handle a produced data with load file import here >>
Produced Data Import Required Specifications
Outlined below are the load file header values which are available in every Nextpoint database for produced data imports (with a load file).
These Default Fields & Document Attributes do not need to be created as new Fields under SETTINGS > Coding. Instead, if you have a header value in Row 1 of your load file similar to a value listed below, make sure that the load file value matches the below default fields exactly.
Note: These fields correlate with Produced Data Import instructions, 3 | Load File Configuration > Metadata Fields > Default Fields.
| App_Name | Attribute |
|---|---|
| Image_File | File Loading |
| Author | Nextpoint Default |
| Bates_Start | File Loading and Imaging |
| Bates_End | File Loading |
| BCC | Nextpoint Default |
| CC | Nextpoint Default |
| Created_Date_Time | Nextpoint Default |
| Custodian | Structural |
| Custodians | Structural |
| Document_Author | Nextpoint Default |
| Document_Date | Structural |
| Document_Last_Author | Nextpoint Default |
| Document_Subject | Nextpoint Default |
| Document_Title | Nextpoint Default |
| Document_Type | Structural |
| Email_Author | Nextpoint Default |
| Email_Received | Nextpoint Default |
| Email_Sent | Nextpoint Default |
| Email_Subject | Nextpoint Default |
| Email_Thread_Index | Structural |
| Encrypted | Nextpoint Default |
| File_Name | Nextpoint Default |
| File_Path | Nextpoint Default |
| Last_Print_Date | Nextpoint Default |
| Mailbox_File | Nextpoint Default |
| Mailbox_Path | Nextpoint Default |
| Modified_Date_Time | Nextpoint Default |
| Recipients | Nextpoint Default |
| Root_Folder | Nextpoint Default |
| Shortcut | Structural |
| Title | Structural |
| Text_File | File Loading |
| Native_File | File Loading |
NOTES:
- Custodian/Custodians are visible under Settings > Import > Custodians
- Have existing/historical Tags you want to migrate to Nextpoint's Additional Tags field? Your column header should read Tags and the values should be semicolon delimited (e.g. Bob Randolph; hard copy document; Production 1; 10/22/2019).
Produced Data Imports, Exhibit A - Default Fields & Document Attributes
Outlined below is a list of Protected System Fields in every Nextpoint database.
Data cannot be imported into any of the below fields because they are generated by the Nextpoint application. If you’d like to map any of the below values into your database, you must set up a field with a different name.
Common fields replaced and suggested replacement values provided in parenthesis.
NOTE: All Default Fields and Document Attributes can also be considered Protected fields as we will prohibit creation of duplicative fields (because they already exist in every database). See the full list here >>
| Protected Field | Suggested Alternative |
|---|---|
| Filename | (use existing File Name) |
| Filepath | (use existing File Path) |
| batch | |
| batch_id | |
| batch_ids | |
| Bates | (use existing Bates_start and Bates_end as applicable) |
| Bates_stamped | |
| Confidentiality | (Conf_Status) |
| Confidentiality_Status | (Conf_Status) |
| Content_hash | (ContentHash) |
| Created_at | (use existing Created_Date_Time) |
| Created_at_gmt | (Created_Date_Time) |
| Created_on | |
| Delete_at_gmt | |
| Deposition_id | |
| Deposition_names | |
| Deposition_volume_id | |
| Display_name | |
| Doc_props | |
| Document_properties | |
| Domain | (EmailDomain) |
| Email_message_id | (MessageID) |
| Email_reply_id | (ReplyID) |
| Email_Thread | (ThreadID) |
| Es_exh_designations | |
| Es_tags | |
| Es_updated_at_gmt | |
| Exhibit_stamped_as | |
| Expansive_hash | |
| Export_ids | |
| File_extension | (Fileext) |
| File_size | (Size) |
| Has_markups | (Has_Annotations) |
| Has_native_placeholder | |
| Highlight_issues | (Annotation_Issues) |
| Highlight_notes | (Annotation_Notes) |
| Id | |
| Issues | (Document_Issues); (DocIssues) |
| Incoming_wire_id | |
| Marked_for_delete | |
| Non_imaged_placeholder | |
| Notes | (Annotations) |
| Npdcase_id | |
| Original_filename | (use existing File Name) |
| Outgoing_wire_ids | |
| Page_count | (Pgcount) |
| Privileged | (Isprivileged) |
| Privileged_status | |
| Redacted | (Isredacted) |
| Relevancy_Status | (Relevancy) |
| Responsive_Issues | (Resp_Issues) |
| Review_status | |
| Updated_at | (Timeupdated) |
| Updated_at_gmt | |
| Updated_on | |
| Verified_page_count | |
| Wire_transfer_tags |
Produced Data Imports, Exhibit B - Protected System Fields
Introduction to Family Linking
When importing Produced data, you will likely be provided a metadata load file outlining Bates information, document boundaries, email family relationships, and associated metadata for any documents included in the data set. The information provided for email family relationships can be found in the "Unique ID" and "Family ID" (see examples below), and in order to effectively review the data set in Nextpoint, this information must be formatted in a specific manner and then the families linked after import.
The following instructions outline how to properly map family relationships when importing, and visually link emails and attachments in your Nextpoint database. First, a couple of definitions that will help you as you prepare to approach using the Family Linking functionality:
Term |
Description |
|---|---|
Family Linking |
Family linking is the process which links imported (produced) emails and attachments using a set of two fields: "Unique ID" and "Family ID". This takes place after import processing is complete and via DATA Import and by clicking on the batch name to view Batch Details Family Linking. See further instructions below. |
Load File |
An index outlining for Nextpoint processors, which documents to look for, where they are located and allows for the import of all metadata. The two main elements of a load file are the column headers and the coding values. The column headers correspond with document fields in your Nextpoint database which need to be populated with metadata. The coding values in the load file contain the information that will be mapped (sent) to those fields for each document. |
Ranged Image Import |
The typical format is single-page tiff/jpg image files named by their Bates number, together with document-level text files, any included natives, and document breaks & metadata contained in a load file. |
Unique_ID |
A Nextpoint specific value necessary for successful Ranged Image Imports which include email-attachment relationships. When Unique ID and Family ID are an exact match, the document is recognized as the "parent". All other documents with the same Family ID will be linked as children. Unique ID often = Bates Start. |
Family_ID |
A Nextpoint specific value necessary for successful Ranged Image Imports which include email-attachment relationships. This information indicates to Nextpoint where an email family begins and ends. When Unique ID and Family ID are an exact match, the document is recognized as the "parent". All other documents with the same Family ID will be linked as children. Family ID often = BegAttach |
Unique ID/Family ID Pairs |
For the above Unique ID and Family ID, below are the options for custom field pairs for linking. UNIQUE ID / FAMILY ID Alternatively, the built-in bates start field can be used if the begattach custom field is set-up in your database. UNIQUE ID / FAMILY ID |
How to Family Link Load File Imports
Assumptions
- You are importing your produced data set to a Nextpoint Litigation database. You can also import to a Nextpoint Discovery database, but we more commonly see produced data imported to Litigation.
- Your production folder, along with the associated load file has been uploaded to the appropriate database's File Room.
Family Linking
- Follow the steps to select your data for import, navigate the load file mapper, and initiate your import as described in our support topic How to Import Produced Data with a Load File, linked here. If you prefer a video tutorial, click here >>
- Once your import has been initiated, you will receive an email notification when your import is Complete. Click on the link contained within the email, or navigate back to More Data Imports click on the title of the batch you just imported to view the Batch Summary.
- At the bottom right of your Batch Summary page, click
. A sliding menu will appear with checkboxes & headers for Batch, User, Uploaded, Docs (batch number, importing user, upload date, batch document count), and a selection drop-down for your Linking Options.
- Select the batch(es) you would like to Family Link.
- Select the fields Nextpoint should use to link parent emails to their attachments via the "Linking Options" drop-down.
- Click "Family Link Documents"
- Confirm your selection by clicking "Run Family Linking"
- Once linking is complete you will receive an email with a link back to the Batch Summary page where you can download and review a report to ensure the emails and attachments were linked without issue.
A Closer Look at the Family Linking Report
- Nextpoint Database ID and Name in which Family Linking was run.
- The Unique ID and Family ID Linking Option utilized.
- Batches on which Family Linking was run.
- Child linked to its parent document - Will show for all relationships created.
- Nextpoint Database ID and Name in which Family Linking was run.
- The Unique ID and Family ID Linking Option utilized.
- Batches on which Family Linking was run.
- Issues Found
- Details of Issues found
Linking Email Families in Produced Data Imports
Topics Below:
- What is a load file?
- ESI and self-contained files
- Data processed to images, including previously produced data
- Important load file column headers
- Compiling a list of files to create a load file
What is a load file?
A load file tells the Cloud processors what documents to look for, where they are located and allows for the import of all metadata.
The content for your load file will come from either an existing load file which you will need to convert, or by manually generating a directory listing of your files.
Modify your load file in Excel, then "Save As" a .csv file. Your load file must be named nextpoint_load_file.csv in order for Nextpoint to recognize it as the load file for your import.
The two main elements of a load file are the column headers and the coding values. The column headers correspond with document coding fields in your Nextpoint case, which need to be populated with metadata. The coding values on the load file contain the information that will be mapped (sent) to those fields for each document. Each row of a load file = 1 document in Nextpoint.
* Spaces are not recommended in load file headers, since different character encoding can create inconsistency among text. So, where a coding field in Nextpoint has a space, the corresponding column header must have an underscore.
Load files can be very simple or quite complex, depending on the complexity of your data set. To start constructing your load file, first identify what type of data you are dealing with:
- ESI and self-contained files - emails/PST/mailbox containers, PDFs and other file types where each file = 1 document
- Data processed to images, including previously produced data - single-page image files (TIFF or JPG) that get assembled into documents in Nextpoint, based on boundaries defined in a load file
* see processes below
ESI and self-contained files
1. Which documents need to be imported, and where are they?
Nextpoint needs to know what files it is looking for, and where to find them. This is accomplished by using an image_file column header. The information in the image_file column contains the path (location) of the file, and the file name itself.
2. What title should Nextpoint give the document?
Nextpoint needs to know what to call the document, or what value to place in the subject/title field. This is accomplished by using a title column header.
So in order to import a set of multi-page files to Nextpoint, your load file would need to look like this. Row 1, the column headers. Rows 2-7, the coding values that correspond to each column header for the 6 documents being imported. These are the minimum required load file commands for such a document import:
In this example, you would save the load file and place it in the same location as the "Evidence" folder in your data set. When you select the folder "Import" as your upload, the path information in the image_file column of your load file tells Nextpoint application where to start digging for each source file:
Beyond the required column headers illustrated above, you can add any other coding information that you would like to be mapped to your documents in Nextpoint. Remember in order to send additional coding values to your documents in Nextpoint during import, the coding fields need to be created first in your Nextpoint database. There are exceptions, or standard fields that are already built in to Nextpoint are:
- Bates
- Shortcut
- Author
- Document Type
- Title
- Document Date
* all this means is that you don't need to add these fields to your Nextpoint database - they still need to be part of your load file
Here is an example of a load file with additional coding, and the way that the first processed document (from Row 2) would look after the import:
3. Loading native files with a placeholder: You can also use a load file to import native files with a placeholder image. Just use the image_file column (for your placeholder image file), then include a native_file column for the native/original file. The native file will be uploaded to Nextpoint with the placeholder file as its image page:
Data processed to images, including previously produced data
First, the data set will contain up to 3 folders (at the very least, this type of data set must contain an IMAGES folder):
IMAGES - this folder contains the document pages, each a one-page image file
TEXT - this folder contains the OCR text information, and can be either one text file per page, or one text file per document
NATIVES - this folder contains any native files that accompany each document
Image file pages will be in the .tif or .jpg format, and the files will be named by their bates numbers. If included, the OCR text and native files will be also be named by the corresponding bates numbers. Here is what a common single-page image data set looks like:
This data structure is constant among most imports of this type, so Nextpoint has built in some automation to allow for more efficient utilization of load files:
1. Which documents need to be imported, and where are they?
When single-page image files are located in a folder called IMAGES and are named by their Bates values, you only need to use the column headers bates_start and bates_end. The combination of these two headers serves two functions:
- To tell Nextpoint which page files to use to assemble each document image (the page boundaries for each document). Nextpoint will automatically find the image files based on the Bates numbers, since they are in an IMAGES folder.
- To assign the appropriate Bates values to each document as it is processed into Nextpoint.
2. What title should Nextpoint give the document?
Nextpoint needs to know what to call the document, or what value to place in the subject/title field. This is accomplished by using a title column header.
3. Which OCR text file corresponds to each document?
Nextpoint needs to know what text file to grab, and apply to each document image as the OCR text. This is accomplished by using a text_file column header, which contains the path to and name of the text file.
4. Which native file corresponds to each document (where applicable)?
Nextpoint needs to know what native file to grab, and apply to each document image that requires an accompanying native. This is accomplished by using a native_file column header, which contains the path to and name of the native file.
5. Make sure that no OCR text (.txt) files are used as image page files:
Occasionally, text files are mixed into the same folder with image files. Since they are all named by Bates number, that means the only difference between an OCR text file and an image page is the file extension. Therefore you need to tell Nextpoint to use only files with a certain extension as document image pages. This is easily accomplished with a column header called image_extension, containing the value tif|jpg in every row.
*other important Nextpoint load file column headers
So in order to import a set of single-page image files to Nextpoint, your load file would need to look like this. Row 1, the column headers. Rows 2-5, the coding values that correspond to each column header for the 4 documents being imported. These are the minimum, required load file commands for such a document import:
In this example, you would save the load file and place it in the same location as the IMAGES, NATIVES, TEXT folders in your data set. When you select the folder "Image_Import" as your upload, the information in the bates_start column of your load file tells Nextpoint application how to find each document image page (IMAGES), native file (NATIVE), and OCR text file (TEXT):
Important Load File Column Headers
| IMAGE FIELDS | |
| image_dir | The folder where the image files are located |
| image_extension | The extension that the image files have, especially useful if text files are in the same folder as image files |
| image_file | The name of the image file, for multipage documents |
| image_range_start | The name of the first image file for the document. Used when several single page images (TIFFs & JPGs usually) comprise a document. * not necessary if pages are named by Bates number, and you have bates_start column |
| image_range_end | The name of the last image file for the document. Used when several single page images (TIFFs & JPGs usually) comprise a document. * not necessary if pages are named by Bates number, and you have bates_end column |
| NATIVE FILE FIELDS | |
| native_dir | The folder where the native files are located |
| native_file | The name of the native file |
| OCR FIELDS | |
| text | This column header is used when the OCR text is contained in the loadfile itself, rather than in separate text files |
| text_dir | The folder where the text files are located |
| text_file | The name of the text file, where a single text file contains text for all pages of the document |
| text_range_start | The name of the first text file for the document. Used when a single page text file exists for each page of a document |
| text_range_end | The name of the last text file for the document. Used when a single page text file exists for each page of a document |
| OTHER FIELDS | |
| bates_start | The Bates number of the first page of the document. Also will specify the image range start when pages are named by Bates |
| bates_end | The Bates number of the last page of the document. Also will specify the image range end when pages are named by Bates |
Compiling a List of Files to Create a Load File
To create a load file, you will need a list of files to include, and if applicable, the folders that they reside in. If you don't have this information in a list, you can easily create one.
Windows
1. Right click on the Root Folder, select "Open Command Window Here".
(If your PC does not have this option, download the plugin here.)
In Windows 7, you may need to hold the Shift key to get this option.
2. A command window will appear. On the Command Line, type the following: dir /b /s /a-d >>list.txt
3. This will create a text file called "list.txt" in the root folder, that contains all the path information for each file and folder within the root folder that looks like this:
4. Open "list.txt" in a text editor and do a Find and Replace to remove the extra pathing information, and you'll have a list of all the files in the folder, and their subfolder pathing, if necessary.
5. Use this list to create a load file for your documents. Click here for information on how to create a simple load file.
Mac
1. Open the utility, Terminal.
2. Type "find " (be sure to include the space) then drag the folder you want a list from.
3. Terminal will give you a list of all the file names and pathing information for the folder, that you can copy and paste into a text editor to get in the format needed for your load file.