Description
A database schema mismatch was introduced in the 'default/policyadministration' service following a deployment of version v2.3.0, which enforced a 'strict' schema mode that the underlying database did not yet support. This incompatibility triggered 'PolicyDatabaseSchemaMismatchException' errors and 'relation does not exist' log entries, causing the service to fail health checks and return 500 errors for incoming requests. The impact extends to 9 entities, including 'default/policyadministration-5c94f76cbd-rfw2h' and 'default/policydecision', resulting in widespread service degradation. Remediation should focus on applying the required database migration '2025-12-15.2' before re-deploying the service.
Context
Database Schema Incompatibility
The service deployment enforced a strict schema mode requiring version '2025-12-15.2', but the database remained at '2025-12-15.1', causing the application to fail during startup and request processing.
Key Evidence:
[Event | Info] ConfigMap updated schema mode | First seen: 2026-05-27 22:31:09 | Count: 1
View detailsKubernetes event: ConfigMap updated {"resource":"configmap/policyadministration-config","namespace":"default","field":"POLICY_RULES_SCHEMA_MODE","new_value":"strict","note":"rollout_requires_migration_first"}[Event | Info] Deployment rollout v2.3.0 | First seen: 2026-05-27 22:31:12 | Count: 1
View detailsKubernetes event: Deployment rollout {"resource":"deployment/policyadministration","namespace":"default","image":"policyadministration:v2.3.0","strategy":"rollingUpdate","revision":"12"}[Log | Error] DB schema guard failed | First seen: 2026-05-27 22:32:10 | Count: 3
View detailsDB schema guard failed (code-level remediation: enforce schema compatibility on startup and rollback safely) {"request_id":"wl-req-1","caller":"storage/schema_guard.go:54","func":"CheckMinimumSchemaVersion","db":"postgres","expected_min_schema":"2025-12-15.2","actual_schema":"2025-12-15.1","action":"fail_fast_or_enter_readonly","deploy_order_hint":"apply_migration_before_rollout","service":"policyadministration"}[Exception] PolicyDatabaseSchemaMismatchException | First seen: 2026-05-27 22:32:19 | Count: 1
View detailsPolicyDatabaseSchemaMismatchException: expected policy_rules_v2 schema but DB is behind at repository.LoadPolicyRulesV2(repository/policy_store.go:143) at repository.LoadPolicyRulesV2(repository/policy_store.go:143) at repository.LoadPolicyRules(repository/policy_store.go:88) at service.RefreshPolicyCache(service/policy_cache.go:57) at http.Authorize(http/handlers/authorize.go:121) at http.Middleware.RequestID(http/middleware/request_id.go:33)
Impact
svc
policydecision
Tier: SLOns: defaultcl: bettingst: Deployment
This Service was experiencing a high number of errors, impacting request processing and data integrity.
The success rate objective have been violated, its error budget is fully exhausted.
svc
policyadministration
Tier: SLOns: defaultcl: bettingst: Deployment
This Service was experiencing a high number of errors, impacting request processing and data integrity.
The success rate objective have been violated, its error budget is fully exhausted.
svc
policyenforcement
Tier: SLOns: defaultcl: bettingst: Deployment
This Service was experiencing a high number of errors, impacting request processing and data integrity.
The success rate objective have been violated, its error budget is fully exhausted.
svc
PolicyQueries
Tier: SLOcl: betting
This Service was experiencing a high number of errors, impacting request processing and data integrity.
The success rate objective have been violated, its error budget is fully exhausted.
Suggested Remediation
Remediations are based on the runtime telemetry currently available. For more specific steps, pass this to an agent with access to your configuration, code, or runbooks.
Apply Database Migration and Revert Schema Mode
Consider applying the required database migration '2025-12-15.2' to your PostgreSQL instance to align the database schema with the expectations of the v2.3.0 deployment. Since the service is currently failing due to a 'strict' schema mode enforcement, you might want to temporarily revert the 'POLICY_RULES_SCHEMA_MODE' configuration to its previous value or a 'permissive' state if the migration cannot be applied immediately. This would allow the service to stabilize while you prepare the necessary database changes.
For Kubernetes environments, you could update the ConfigMap to disable the strict mode or point to the correct schema version. You might also consider using a Kubernetes Job to execute the migration script before the deployment rollout, ensuring the database is ready for the new application version.
apiVersion: v1
kind: ConfigMap
metadata:
name: policyadministration-config
namespace: default
data:
POLICY_RULES_SCHEMA_MODE: "permissive" # Revert to allow startup while migrating
Once the database migration is successfully applied, you could then update the ConfigMap to set the mode back to 'strict' and perform a rolling restart of the deployment to ensure the service correctly initializes with the updated schema.
Source: https://kubernetes.io/docs/concepts/configuration/configmap/
Evidence
10 Observed Signals
No evidence in Alerts
Exceptions
2026-05-27T22:32:19.141462ZERRORdefault/policyadministration-5c94f76cbd-rfw2h/policyadministrationFailed to update policy {"error":"failed to update policy after 3 attempts","policy_id":"policy_abc123","admin_user":"admin_001","status_code":503,"note":"secondary symptom during schema mismatch / DB instability"}
at github.com/Causely/causely/pkg/pap.(*PolicyAdminService).HandleUpdate(PolicyAdminService.go:148)
at github.com/Causely/causely/pkg/pap.(*PolicyAdminService).processRequest(PolicyAdminService.go:98)
at runtime.goexit(runtime/asm_arm64.s:1136)
2026-05-27T22:32:19.141462ZERRORdefault/policyadministration-5c94f76cbd-rfw2h/policyadministrationPolicyDatabaseSchemaMismatchException: expected policy_rules_v2 schema but DB is behind at repository.LoadPolicyRulesV2(repository/policy_store.go:143)
at repository.LoadPolicyRules(repository/policy_store.go:88)
at service.RefreshPolicyCache(service/policy_cache.go:57)
at http.Authorize(http/handlers/authorize.go:121)
at http.Middleware.RequestID(http/middleware/request_id.go:33)
Logs
2026-05-27T22:32:19.335ZERRORdefault/policyenforcement-5d56f979f7-lhczl/policyenforcementError handler mapped auth errors to 500 (code path) {"request_id":"http-req-42","caller":"http/error_mapping.go:41","func":"MapErrorToHTTP","mapped_status":500,"expected_statuses":[401,403],"action":"map_OIDCVerificationFailed_to_401_and_AuthorizationDenied_to_403","suggested_kubectl":["kubectl -n default logs deploy/policyenforcement --tail=200"],"service":"policyenforcement"}
2026-05-27T22:32:19.275ZERRORControlDistributionGateway.allocation[1]/defaultRequestErrorRate spike due to dependency chain {"request_id":"oauth-req-7","dependency_chain":["graph.facebook.com","SocialFeedAgg"],"status_code":503}
2026-05-27T22:32:19.245ZERRORdefault/policyadministration-5c94f76cbd-rfw2h/policyadministrationRequest failed due to schema mismatch (code-level remediation: surface actionable error + add startup readiness gate) {"request_id":"wl-req-1","caller":"http/handlers/authorize.go:121","func":"Authorize","route":"/v1/authorize","status_code":500,"error_class":"PolicySchemaMismatch","suggested_fix":["run_db_migration_2025-12-15.2","add_schema_version_check_on_startup","make_v2_query_backward_compatible"],"service":"policyadministration"}
2026-05-27T22:32:17.335ZWARNINGdefault/policyenforcement-5d56f979f7-lhczl/policyenforcementIssuer claim mismatch due to trailing slash (config normalization required) {"request_id":"http-req-42","caller":"auth/oidc_verifier.go:121","func":"VerifyToken","issuer_claim":"https://auth.local/realms/causely","configured_issuer":"https://auth.local/realms/causely/","action":"normalize_issuer_or_fix_env","suggested_kubectl":["kubectl -n default set env deploy/policyenforcement OIDC_ISSUER=https://auth.local/realms/causely","kubectl -n default rollout restart deploy/policyenforcement"],"service":"policyenforcement"}
2026-05-27T22:32:16.245ZWARNINGdefault/policyadministration-5c94f76cbd-rfw2h/policyadministrationRefusing to fallback to legacy schema due to strict mode (code-level remediation: add controlled fallback with telemetry) {"request_id":"wl-req-1","caller":"config/policy_rules.go:88","func":"ResolvePolicySchema","config_key":"POLICY_RULES_SCHEMA_MODE","config_value":"strict","expected":"graceful_fallback_with_alerting","service":"policyadministration"}
2026-05-27T22:32:15.335ZERRORdefault/policyenforcement-5d56f979f7-lhczl/policyenforcementJWKS fetch failed (config mismatch) {"request_id":"http-req-42","caller":"auth/oidc_verifier.go:58","func":"FetchJWKS","jwks_url":"https://auth.local/realms/causely/protocol/openid-connect/certs","http_status":404,"error":"unexpected status code","action":"fix_OIDC_JWKS_URL_or_ingress_path","suggested_kubectl":["kubectl -n default get configmap policyenforcement-config -o yaml","kubectl -n default describe ing policyenforcement"],"service":"policyenforcement"}
2026-05-27T22:32:15.275ZERRORControlDistributionGateway.allocation[1]/defaultDownstream service requests failing {"request_id":"oauth-req-7","downstream":["CampaignSync"],"status_code":503}
2026-05-27T22:32:13.335ZERRORdefault/policyenforcement-5d56f979f7-lhczl/policyenforcement/access returned 500 for auth failure (should be 401/403) {"request_id":"http-req-42","route":"/access","caller":"http/access_handler.go:92","func":"HandleAccess","error_class":"OIDCVerificationFailed","status":500,"expected_status":401,"config_hint":{"oidc_issuer_env":"OIDC_ISSUER","oidc_jwks_env":"OIDC_JWKS_URL","audience_env":"OIDC_AUDIENCE"},"suggested_kubectl":["kubectl -n default set env deploy/policyenforcement OIDC_ISSUER=https://auth.local/realms/causely OIDC_JWKS_URL=https://auth.local/realms/causely/protocol/openid-connect/certs OIDC_AUDIENCE=policyenforcement","kubectl -n default rollout restart deploy/policyenforcement","kubectl -n default rollout status deploy/policyenforcement"],"service":"policyenforcement"}
2026-05-27T22:32:13.245ZERRORdefault/policyadministration-5c94f76cbd-rfw2h/policyadministrationQuery failed: relation does not exist (code-level remediation: add backward-compatible query path + migration gating) {"request_id":"wl-req-1","caller":"repository/policy_store.go:143","func":"LoadPolicyRulesV2","sqlstate":"42P01","error":"pq: relation \\\"policy_rules_v2\\\" does not exist","feature_flag":"policy_rules_v2","feature_flag_value":true,"action":"disable_flag_or_fallback_to_v1","service":"policyadministration"}
2026-05-27T22:32:11.275ZWARNINGControlDistributionGateway.allocation[1]/defaultDistribution using cached feed snapshot {"request_id":"oauth-req-7","fallback":"cached_feed","cache_age":"30m"}
2026-05-27T22:32:10.245ZERRORdefault/policyadministration-5c94f76cbd-rfw2h/policyadministrationDB schema guard failed (code-level remediation: enforce schema compatibility on startup and rollback safely) {"request_id":"wl-req-1","caller":"storage/schema_guard.go:54","func":"CheckMinimumSchemaVersion","db":"postgres","expected_min_schema":"2025-12-15.2","actual_schema":"2025-12-15.1","action":"fail_fast_or_enter_readonly","deploy_order_hint":"apply_migration_before_rollout","service":"policyadministration"}
2026-05-27T22:32:07.275ZERRORControlDistributionGateway.allocation[1]/defaultControl distribution failed - feed data unavailable {"request_id":"oauth-req-7","dependency":"SocialFeedAgg","status_code":503,"cause":"dependency_unavailable"}
Events
2026-05-27T22:31:49.273ZINFOdefault/policyadministration-5c94f76cbd-rfw2hKubernetes event: Readiness probe succeeded {"pod":"policyadministration-5c94f76cbd-rfw2h","route":"/healthz","note":"recovered after restart"}
2026-05-27T22:31:36.273ZWARNINGdefault/policyadministration-5c94f76cbd-rfw2hKubernetes event: Restarting container {"pod":"policyadministration-5c94f76cbd-rfw2h","container":"policyadministration","reason":"failed liveness probe"}
2026-05-27T22:31:32.273ZERRORdefault/policyadministration-5c94f76cbd-rfw2hKubernetes event: Liveness probe failing {"pod":"policyadministration-5c94f76cbd-rfw2h","reason":"HTTP probe failed with statuscode: 500","route":"/healthz","note":"schema mismatch / DB failure path"}
2026-05-27T22:31:24.273ZWARNINGdefault/policyadministration-5c94f76cbd-rfw2hKubernetes event: Readiness probe failing {"pod":"policyadministration-5c94f76cbd-rfw2h","reason":"HTTP probe failed with statuscode: 500","route":"/healthz","note":"should_block_ready_until_schema_ok"}
2026-05-27T22:31:12.273ZINFOdefault/policyadministration-5c94f76cbd-rfw2hKubernetes event: Deployment rollout {"resource":"deployment/policyadministration","namespace":"default","image":"policyadministration:v2.3.0","strategy":"rollingUpdate","revision":"12"}
2026-05-27T22:31:09.273ZINFOdefault/policyadministration-5c94f76cbd-rfw2hKubernetes event: ConfigMap updated {"resource":"configmap/policyadministration-config","namespace":"default","field":"POLICY_RULES_SCHEMA_MODE","new_value":"strict","note":"rollout_requires_migration_first"}