3.6 Data retention, legal hold, and audit export pack

Data has mass. Accumulating terabytes of high-frequency sensor data without a disciplined purge strategy often leads to system paralysis. Conversely, deleting compliance records prematurely creates significant legal liability. It is highly recommended to implement a Data Lifecycle Management (DLM) strategy that balances performance, cost, and legal risk.

Retention policy by data class

Not all bytes are created equal. Data must be partitioned into classes with distinct expiration dates.

Class a: compliance & genealogy (the “forever” data)

Content: Serial Numbers, Pass/Fail Results, Operator IDs, RMA History.
Retention: Aim for Warranty Period + 1 Year (Consumer) or up to 15 Years (Automotive/Medical).
Action: Ensure these records are never deleted until the defined policy expires.

Class b: high-frequency telemetry (the “engineering” data)

Content: Waveforms, Torque curves, Temperature profiles (at 100ms resolution).
Retention: 12 Months is a typical standard.
Logic: If an engineering team hasn’t analyzed a temperature spike within a year, the data is likely just noise. After 12 months, design the system to down-sample to “Min/Max/Avg” summaries and safely purge the raw, high-frequency rows.

Class c: IT & security logs (the “forensic” data)

Content: Login attempts, API calls, Firewall logs, User Access changes.
Retention: 1 Year (Standard Cybersecurity requirement).
Action: Rolling purge.

Class d: large binary objects (BLOBs)

Content: AOI Images, X-Ray Tiffs, PCB Schematics.
Retention: 6 Months (unless specifically linked to a confirmed Defect or failure analysis).
Logic: Storing terabytes of images for purely “Good” results is an inefficient use of capital. Prioritize keeping the “Fail” images for the same duration as Class A data.

Storage strategy: hot, warm, cold

Keeping 10 years of detailed history in the primary production SQL database should be avoided. This practice almost universally kills query performance.

Hot (Tier 1): Production DB (NVMe SSD).
- Age: 0 – 12 Months.
- Purpose: Instant operations, dashboards, active reporting.
Warm (Tier 2): Data Warehouse / Read-Replica (HDD).
- Age: 1 – 3 Years.
- Purpose: Monthly/Yearly analytics.
Cold (Tier 3): Archive (S3 Glacier / Tape).
- Age: 3+ Years.
- Purpose: “In case of lawsuit.” Low cost, high latency (hours to retrieve).
Automation via a Purge Script: Automate the data movement process. When a specific record exceeds 12 months in age, the database script should systematically move it to cold storage and delete it from the hot tier to preserve speed.

The legal hold (the “stop shredding” button)

When litigation or a major recall begins, automated purging becomes evidence destruction. A “Kill Switch” is required.

Trigger: The Legal Department officially notifies IT regarding a specific Lot, Product Line, or Date Range.
Mechanism:
1. The administrator creates a “Legal_Hold” flag in the database.
2. Logic: When the “Legal_Hold” flag is active, the system should automatically bypass the automated purge script, protecting the records.
Scope: The hold must apply to all associated data (Emails, Logs, Genealogy, Telemetry).

The audit export pack

When an auditor (such as the FDA, ISO, UL, or a Customer representative) requests data, they are effectively testing your retrieval capability. They generally expect a coherent, readable story, rather than a raw SQL data dump.

Service Level Agreement (SLA): A robust operation should generate this comprehensive pack in < 30 Minutes.

The structure

The “Export Pack” is a zipped folder containing four distinct artifacts for a specific Serial Number (SN):

1. The Genealogy Report (PDF)

Tree view of all child components (Lots/SNs).
List of all equipment used (Asset IDs).
List of all operators involved (User IDs).

2. The Process History (CSV)

Parametric data for every step.
Columns: Step, Machine, Timestamp, Setpoint, Actual_Value, Result, Upper_Limit, Lower_Limit.
Why CSV? So the auditor can load it into Minitab/Excel for their own analysis.

3. The Master Data Snapshot (PDF)

Evidence of the configuration at that moment in time.
BOM Revision and Process Routing Revision active on the production date.

4. The Compliance Certificates (PDF)

Links to Calibration Certs for the tools used.
Links to Training Records for the operators involved.

Validation: the “restore” drill

The effectiveness of backups is only confirmed through successful restore operations.

Frequency: Conduct these drills Quarterly.
Drill: The IT team selects a random, archived Lot from several years ago.
Objective: Successfully retrieve the data from the Cold Storage environment and generate the complete Audit Export Pack.
Fail Condition: Consider the drill a failure if the retrieval process takes > 24 hours or if the supplied data is corrupt.

Recap: Data Retention, Legal Hold, and Audit Export Pack

Data Class	Retention Period	Storage Tier & Age	Legal Hold Action	Audit Export SLA
Class A (Compliance/Genealogy)	Warranty + 1 year, up to 15 years	Cold (3+ years)	Suspend purge for all associated data	Generate pack in <30 minutes
Class B (High-Freq Telemetry)	12 months (raw), then down-sample	Hot (0-12 months), then archive	Suspend purge for all associated data	Generate pack in <30 minutes
Class C (IT/Security Logs)	1 year (rolling purge)	Warm (1-3 years)	Suspend purge for all associated data	Generate pack in <30 minutes
Class D (BLOBs/Images)	6 months (good), match Class A (defects)	Per strategy (see Class A for defects)	Suspend purge for all associated data	Generate pack in <30 minutes