Skip to main content

Operations

Upgrade​

  1. Read the release notes and back up PostgreSQL and object storage.
  2. Update the pinned chart version in the GitOps release declaration.
  3. Review any required changes to values.production.yaml.
  4. Render and review the proposed manifests in CI.
  5. Merge through the customer's normal approval process.
  6. Let the GitOps controller reconcile the release.
  7. Confirm health, license state, sign-in, discovery, and a representative download.

Migrations run automatically under a PostgreSQL advisory lock when API replicas start. Only one replica applies them. Do not run different application versions against the same database during an upgrade.

Rollback​

Revert the Git commit that changed the chart version or values, then let the GitOps controller reconcile the previous desired state. Application rollback does not reverse a database migration. If release notes mark a migration incompatible, restore the matching database and object-storage recovery point instead.

Backup and restore​

Back up PostgreSQL, the complete S3 bucket, redacted Helm values, Secret references, and the active license delivery record as one recovery set. Valkey is not the system of record.

Restore PostgreSQL and object storage from the same recovery point, recreate Secrets, and pin the same application/chart version that produced the backup. Verify core flows before upgrading. Rehearse the complete restore procedure before relying on the recovery set.

Monitoring​

Probe /health/live for liveness and /health/ready for PostgreSQL, Valkey, and storage readiness. Alert on readiness failures, restart loops, sustained 5xx responses, queue failures, license expiry, dependency capacity, backup failures, and certificate or OIDC-secret expiry.

Scaling​

API and web scale horizontally. Change replica counts in values.production.yaml, review the rendered manifests, and merge through GitOps. Enable the PodDisruptionBudget with multiple replicas. Increase worker concurrency gradually while monitoring dependencies. Every replica must use the same release and license source.

Uninstall​

Remove the LoadoutHQ release declaration from the GitOps repository and let the controller prune chart-managed workloads. External dependencies, retained data, and operator-managed Secrets are not removed automatically. Confirm retention and backups before deleting those systems.