Spirion PostgreSQL and PgBouncer Agent Queuing (v13.6+)

New in version 13.6 is the Spirion PostgreSQL and PgBouncer Agent queuing system.

Old Architecture (Version 13 to Version 13.5)

The new PostgreSQL database architecture replaces the Erlang/RabbitMQ architecture of the v13-13.5 greyhound Agents, shown below:

This diagram below shows the old (v13-v13.5) path of full journey of scan work and scan results—from an endpoint (client environment) all the way into the server-side systems.

New Architecture vs Old: Two Internal Queues

• Old: Spirion Agents prior to version 13.6 relied on RabbitMQ for two internal queues.
• New: Those queues now run on a small local PostgreSQL database (with pgBouncer as a lightweight connection “traffic cop”) resident on each agent machine.

• Benefit of new queueing architecture: The new queueing architecture simplifies installation, reduces resource usage, and makes recovery more predictable when a machine or network has a hiccup.

What Does “queue” Mean?

A “queue” is a table in the local PostgreSQL database.

• Each “item in the queue” is a row in that table.
• Items are “moved” through the queue by changing the row’s state/status (example: Pending → Processing → Failed → PermFailed).

Queue = table, item = row, progress = status change

Job Queue and Shipper Queue

1. Job Queue (per scan): A list of “places to search” that search Agent workers pull from.
2. Shipper Queue (always on): A durable list of “results waiting to be sent” to Spirion cloud services (Ingress).

In the Spirion Windows agent architecture (introduced in version 13.6), the legacy RabbitMQ messaging system has been replaced by a local PostgreSQL database. This database uses two primary types of queues to manage the flow of a scan.

The high-level difference is their purpose: the Job Queue manages the work to be done (Discovery to Search), while the Shipper Queue manages the results waiting to be delivered (Search to Console).

Job Queue (search_queue_<scanId>)

This queue acts as the "To-Do List" for a specific scan.

• Used in both single-agent and distributed scans
• Scope: Per-scan (Temporary)
• • A new table is created for every unique scan (for example, search_queue_GUID).
  • The table is scan-specific (the <scanId> suffix is the actual scan GUID) and is dropped/cleaned up after the scan completes
• Producer: The Discovery Agent (thread) populates this queue with "jobs" (specific files, folders, or locations that need to be searched).
• • One row per discovered location, inserted as Pending
• Consumer: One or more Search Agents pull jobs from this queue to perform the actual inspection.
• • Search Agents claim rows using FOR UPDATE SKIP LOCKED and process them
• Lifecycle: It is temporary. It exists for the duration of the scan and is typically cleaned up after the scan completes and a grace period (for example, 15 minutes) expires.
• Distributed Scanning: In a distributed scan, the Job Queue lives on the "Discovery" machine. Remote Search Agents connect to this central Postgres instance via pgBouncer to claim work.

Shipper Queue (shipper_queue)

This queue acts as a "Durable Staging Area" for scan results.

• Scope: Permanent
• • This is a single, constant table that exists as long as the Agent is installed.
  • This is a permanent table that persists across scans
• Producer: Search Agents write rows into this table as soon as they find sensitive data or complete a file scan.
• • After a Search Agent finishes scanning a location, it stages the results here via the stored procedure sp_publish_to_shipper_queue — before the job is marked "Processed"
• Consumer: The Shipper Process (IDFMessagingSvc.exe) reads from this queue and attempts to post the data to the Spirion Ingress Service (Cloud).
• • IDFMessagingSvc.exe (the Shipper process) drains this queue, POSTing JSON payloads to the Services Ingress API
• Resilience:
• • Durable by design: survives restarts, network interruptions, and agent crashes
  • Its primary job is to protect data against network interruptions.
  • If the endpoint loses internet connectivity, results stay safely in this local Postgres table until the connection is restored.
• Lifecycle: Rows progress from Pending to Processing and are only deleted (or moved to a processed state) once the Ingress service confirms receipt.

The Key Relationship Between Them

• A job in search_queue_<scanId> is only marked Processed after its results have been durably written to shipper_queue.
• • This ordering eliminates the "processed but not shipped" failure class that could occur on crashes.
• Both queues are backed by PostgreSQL + pgBouncer (the RabbitMQ replacement introduced in 13.6), and both use the same Pending → Processing → Processed / Failed / PermFailed status model with heartbeat-driven recovery.

Comparison Summary

Key Statuses for Both Queues

In both tables, you can track progress by looking at the status column:

• 0 (Pending): Waiting to be claimed.
• 1 (Processing): Currently being worked on by a process.
• 2 (Processed): Successfully completed.
• 3 (Failed): Encountered an error; may be retried.
• 4 (PermFailed): Retries exhausted; will not be attempted again.

Services/Processes

These are the “moving parts.”

These names appear in logs and in troubleshooting.

On the endpoint

• PostgreSQL (local) – Stores queue rows durably (replaced RabbitMQ).
• pgBouncer – Connection pooling to keep resource usage small.
• idfEndpoint.exe (EPS) – Ensures the database schema exists and runs background queue cleanup/retry logic.
• identityfinderCMD.exe (IDF / SystemSearch) – Does discovery/search work and writes results into the shipper queue.
• IDFMessagingSvc.exe (IFS / Shipper) – Sends queued results to server-side Ingress.

Key Terms

• “Pending / Processing / Failed / Permfailed: Basic queue states (waiting / in progress / had trouble / permanently failed).
• “Heartbeat”: a periodic “I’m alive” update from an agent; used to auto-recover abandoned work.

Best Practices for Common Issues

Confirm Queue Components are Running

Check services.

Are the following services running?

• PostgreSQL
• pgBouncer
• IDFMessagingSvc (shipper)

Provide the Following Logs to archTIS Support

• SystemSearch / IDF logs (identityfinderCMD)
• Endpoint Service / EPS logs (idfEndpoint.exe)
• Shipper / IFS logs (IDFMessagingSvc.exe)

Typical locations (may vary by version):

• Windows:
• • %ProgramData%\Identity Finder\Logs\
• Provide Gather Data ZIP file if necessary
• Services Side logs: Ingress and Search Persistence Services logs

Recommended Troubleshooting Methods

Follow data through the pipeline to find the root cause

Symptoms often show up “later” than the cause.

When troubleshooting, trace the data through each stage:

• Discovery → Job Queue → Search → Shipper Queue → Shipper → Ingress → Server storage

Find the last stage that is working and the first stage that is failing. The root cause is typically between those two points.

• See the following page for instructions on conducting this investigation: How to Follow Data Through the Search Pipeline

Port 5671

• Port 5671 is a RabbitMQ message.
• Newer logging shows Postgres/pgBouncer ports.
• Port 5671 appearing in logs and messages can indicate older components/log messages.

Distributed scans failing due to “can’t reach the queue host”

• In a distributed scan, search agent endpoints must reach the discovery endpoint’s pgBouncer/Postgres connection.
• Firewalls or port configuration issues can make search agents time out waiting for the queue.

“Waiting for queue to be empty” is not typically a failure

Spirion Search Agents may legitimately wait, especially when the following is true:

• Discovery is still analyzing locations and hasn’t posted eligible work yet (filters/criteria can make this seem slower).
• A distributed scan is finishing up and search agents must wait for all agents to complete so jobs don’t get re-queued.

“Processing” isn’t always bad—there’s auto-recovery

If an agent crashes mid-job, the system uses heartbeats to detect staleness and re-queue abandoned work automatically (often within about a minute).

Common Causes of Issues

Port Mismatches (most common configuration issue)

Default ports:

• PostgreSQL: 5433 (used instead of PostgreSQL default port to avoid collision with any possible traffic on port 5432)
• pgBouncer: 6433 (used instead of pgBouncer's default to avoid collision with any possible traffic on port 6432)

Some environments may use 5432/6432 (older/special configs).

Ensure consistency across your Agents and any required firewall rules.

Search Agents “waiting for queue”

Search agents can wait up to a configured window of time for the per-scan job queue to appear. This is usually not a problem unless search agents time out and exit.

Note: If the discovery endpoint is offline, the job queue is offline too.

Shipper logs may be less obvious in some versions

Shipper logging may not always show detailed “destination URL / message count” information depending on build. Logs are still essential if results are missing.