5.2 Change & release management

A manufacturing facility operates with different constraints than a software startup. A “Move fast and break things” approach can unfortunately translate to stopping the line and impacting revenue. The primary goal of Release Management is Stability. Every change to the MES or ERP should be treated with the utmost care: carefully packaged, systematically routed, and easily reversible.

The three-tier environment

It is standard practice to avoid developing or testing directly on the production server. Strict isolation between environments is fundamental to preventing data corruption and downtime.

DEV (development)

Purpose: The Sandbox. Developers write code and break things here.
Data: Synthetic / Dummy data.
Access: Full Admin rights for Developers.
SLA: None.

UAT (user acceptance testing / staging)

Purpose: The Mirror. This environment serves as an exact replica of the Production hardware and software configuration.
Data: An anonymized copy of Production data (ideally refreshed monthly).
Access: Read-only for Developers; Read/Write for Key Users (Testers).
Logic: When a change succeeds in DEV but fails in UAT, the release is rejected. This typically indicates environment configuration drift.

PROD (production)

Purpose: The Live Manufacturing Environment.
Data: Live, real-time master data and daily transactions.
Access: Generally requires Zero Write access for Developers. Changes are applied securely via automated deployment scripts or managed by System Admins.
Rule: “Hot-fixing” directly in PROD introduces significant risk and should be heavily restricted.

The release gate: governance logic

Code should not migrate from UAT to PROD based solely on verbal assurance. It moves based on documented Evidence.

The request for change (RFC)

Every release must have a ticket containing:

Impact Analysis: Which lines/modules are affected?
Test Evidence: Screenshots/Logs of the pass in UAT.
Rollback Plan: The exact script to undo the change if it fails.
Timing: Estimated downtime.

The approval matrix

Minor Patch (Bug fix): IT Manager Approval.
Feature Release (New Logic): IT Manager + Operations Manager Approval.
Major Upgrade (Architecture): CIO + Plant Director Approval.

Release Windows & blackout periods

Timing is critical. Deploying updates during periods when the factory is highly vulnerable or lacks sufficient support must be avoided.

The “safe” window

Time: Tuesday, Wednesday, or Thursday. 09:00 – 11:00 or 14:00 – 16:00.
Why: IT Support is in the office. Operations leadership is present.
Logic: Deployments require “All Hands on Deck.”

The restriction zone (blackouts)

Fridays: Deploying on Friday increases the risk of requiring emergency weekend support.
Shift Change: (e.g. 06:00, 14:00, 22:00). These periods inherently have high operational noise and distraction.
Peak Season / End of Quarter: System changes must be limited when maximum throughput is required to meet customer commitments.

Versioning strategy

Semantic Versioning (Major.Minor.Patch) must be adopted to communicate risk to the business.

Major (v2.0.0): Breaking Change. Requires downtime and Operator Retraining.
Minor (v1.1.0): New Feature. Backward compatible. No downtime required.
Patch (v1.0.1): Bug Fix. Invisible to the user.

Pro-Tip: The current Version Number should be displayed in the footer of every MES screen. If a user reports a bug, the first question is “What version are you on?”

Rollback rules (the “undo” button)

A deployment plan that lacks a reliable rollback plan introduces significant operational risk. The team must possess the capability to return the system to its previous known-good state, ideally within < 15 Minutes.

The snapshot rule

Virtual Machines: A full VM Snapshot must be taken before applying the update.
Database: A Transaction Log backup must be taken immediately before the script runs.

The 15-minute timer

Trigger: Deployment begins.
Check: At T+15 minutes, execute the “Smoke Test” (e.g. Print one label, successfully complete one cycle).
Logic:
- When the Smoke Test Passes, commit the change.
- When the Smoke Test Fails, or if overall Performance degrades significantly (e.g. > 20%), the team should Execute the Rollback Immediately. It is generally safer to revert than to attempt live troubleshooting during production hours.

Final Checkout: Change & release management

Category	Metric / Control	Threshold / Rule
Isolation	Environment Gap	0% Developer Write Access to PROD Database.
Gate	Evidence	No Release without UAT Sign-off attached to RFC.
Timing	Scheduling	No Deploys on Fridays or during Shift Handover.
Safety	Rollback	Undo script/Snapshot tested before the release window.
Visibility	Versioning	UI displays Active Version (vX.Y.Z) on all terminals.
Data	UAT Freshness	UAT data refreshed from PROD < 30 days ago.
Migration	Database	All SQL scripts must be idempotent (safe to run twice).