Troubleshooting

Start with health and workload state:

curl -fsS https://loadouthq.example.com/health/live
curl -fsS https://loadouthq.example.com/health/ready
kubectl -n loadouthq get pods,deploy,svc,ingress
kubectl -n loadouthq get events --sort-by=.lastTimestamp
kubectl -n loadouthq logs deploy/loadouthq-api --since=15m

HTTP 423

Read /api/v1/license/status: missing requires activation; invalid means the file, signature, key ID, or claims were rejected; not_yet_valid requires checking dates and cluster time; expired requires renewal. A mounted file takes precedence over an uploaded license.

Readiness failure

The response identifies database, cache, or storage. Confirm routes, TLS, credentials, DNS, and dependency health. For S3-compatible storage, verify endpoint, region, path-style mode, bucket, and list permission.

For bundled Valkey, confirm the dependency pod and Service, then verify that REDIS_URL uses the current release name and that REDIS_PASSWORD exists in valkey.auth.usersExistingSecret:

kubectl -n loadouthq get pod,svc,pvc -l app.kubernetes.io/name=valkey
kubectl -n loadouthq logs deploy/loadouthq-valkey --since=15m

OIDC failure

Confirm public base URL, issuer, client ID, client secret, and the exact registered callback. Verify forwarding headers, certificate trust, and clock synchronization.

Configurable validation unavailable

In Skill validation, OpenAI validation not configured means that either OPENAI_API_KEY is absent from the runtime Secret or OPENAI_VALIDATION_MODEL is empty. Confirm the Secret key mapping and rendered non-secret values without printing the key:

kubectl -n loadouthq get secret loadouthq-runtime -o jsonpath='{.data.OPENAI_API_KEY}' | wc -c
kubectl -n loadouthq get configmap loadouthq-loadouthq-config -o jsonpath='{.data.OPENAI_VALIDATION_MODEL}'
kubectl -n loadouthq rollout restart deployment/loadouthq-loadouthq-api

For failed runs, use the check's Audit page to inspect the error code, attempt count, token usage, and provider latency. Verify egress and TLS access to OPENAI_BASE_URL. A provider timeout is controlled by VALIDATION_OPENAI_TIMEOUT_MS; retries are capped by VALIDATION_OPENAI_MAX_ATTEMPTS.

Artifact pull failure

Authenticate Helm and the cluster image-pull Secret, confirm the release tag exists, and verify repositories. OCI chart versions omit the leading v used by Git tags.

Safe support bundle

helm -n loadouthq list
helm -n loadouthq get manifest loadouthq
kubectl -n loadouthq get pods,deploy,svc,ingress -o wide
kubectl -n loadouthq get events --sort-by=.lastTimestamp

Before sharing, remove Secret data, environment values, headers, database URLs, credentials, tokens, and the raw license. Include timestamps and non-sensitive error codes.