How to Verify Search Definitions or Scan Playbook Sensitivity

Verification is typically performed across 4 stages:

1. Configuration Review
2. Controlled Test Scans
3. Results & Action Validation
4. Log/Threshold Tuning

1. Verify Playbook Logic and Decision Points

In the SDP Console, navigate to Scans > Scan Playbooks:

1. Inspect Decision Point Criteria:
2. • Select the edit (pencil) icon on the decision point.
   • Verify the condition definition:
   • • Attribute: Set to Data Types.
     • Operator: Set to Contains (or your intended logical operator).
     • Selected Data Types: Confirm the specific data types
     • • For example, SSN, Credit Card Numbers, Credentials, custom regex patterns) are added to the right-hand selection box.
2. Verify Branching and Action Sensitivity:
3. • Yes Branch (Matched Criteria): Confirm the intended action is selected (e.g., Action: Classification > Action Options: Perform Action on Database Only or persistent file tagging > Classification Type: Add Classification with the appropriate label like Confidential or Secret).
   • No Branch (Unmatched Criteria): Confirm whether the flow routes to another decision point or ends with Take No Action.
   • Automation Flags: Verify whether Automate Action is checked if the playbook is expected to classify or remediate without manual analyst intervention.

2. Verify Search Definition & Target Scope

A search definition combines what to find (Data Types & Playbooks) with where to look (Targets & Scan Policies):

• Target Filters & Exclusions: Review the target definition under Targets to confirm that file extensions (e.g., .docx, .xlsx, .pdf), directory paths, database tables, or mailboxes are not inadvertently filtered out by exclusion rules.
• Scan Agent & Credentials: Ensure the target is bound to an active agent pool with appropriate read/write privileges to prevent Done, Errors, No Results outcomes.
• Scan Policy Settings: Check performance throttling, deep archive scanning (e.g., zip/tar extraction depth), and file size boundaries.

3. Run a Controlled Verification Scan

To test sensitivity and detection accuracy:

1. Deploy a Synthetic Test Target:
2. • Create a test "staging" Target (for example, a test directory or file share) populated with the following:
   • • Known Positive Samples: Test files containing valid format data
     • • For example, test credit cards passing Luhn checks, synthetic Social Security numbers, phone numbers, addressed, other PII, etc..
     • Known Negative / Borderline Samples: Files containing similar formatting that should not trigger
     • • For example, 9-digit account numbers, or phone numbers that resemble Social Security numbers.
2. Run the Scan:
3. • Run the scan linked to the verified playbook and Target.
   • Navigate to the All Scans page (Scans > All Scans).
   • Locate the test scan and confirm the scan completed with the status “Done, With Results” (and no blocking errors).

4. Validate Results and Tune Sensitivity

1. Evaluate Match Accuracy in the Console:
2. • Check Scans > Scan Results or Dashboards:
   • • Match Counts: Verify whether LocationsWithMatches reflects the exact number of test items.
     • Applied Classifications: Check whether the designated classification Tags were successfully written or recorded.
2. Diagnose False Positives or False Negatives:
3. • False Positives (Oversensitive): If benign files triggered detections, review the Data Type definition to enable stricter validation
   • • For example, check-digit algorithms, proximity/context keywords like "SSN:" or "Cardholder"
   • False Negatives (Undersensitive): If real data was missed, verify whether file types are supported, whether files were password-protected or corrupted, or if Regex boundaries are too strict.
3. Inspect Agent Search Engine Logs:
4. • On the scanning Agent, check the engine logs (such as SystemSearch / IDF or EPS services) to confirm that the file was fully parsed and whether matches were filtered out during post-processing.