JWT Key Rotation & kid Handling

Key Rotation Is Inevitable

Signing keys will rotate: - routine security policy - key compromise - certificate expiration - identity provider changes If your service cannot handle key rotation smoothly, you will get: - sudden 401 spikes - full production outages - emergency config hacks that weaken security Production rule: Your service must accept multiple valid signing keys during transition, and reject everything else.

Real Production Incident

Symptoms: - All authenticated requests start failing with 401 at once. - No code deployment happened. - Identity provider rotated signing key. - Service only trusted a single hardcoded key. Root cause: - No support for multiple keys. - No automatic metadata refresh. - Key rotation not tested in staging. This is not an auth bug. It is an operational failure.

Symptom → Cause → Diagnosis → Fix

Symptom: - sudden global 401 failures - valid tokens rejected - logs show signature validation errors Cause: - hardcoded IssuerSigningKey - no metadata endpoint usage - no support for multiple signing keys - no kid handling Diagnosis: - Inspect JWT header for kid. - Compare against configured keys. - Check whether key metadata endpoint changed. - Review TokenValidationParameters for key resolution logic. Fix: - Use signing key resolution via trusted metadata (e.g., OpenID Connect). - Support multiple keys simultaneously. - Refresh metadata periodically. - Monitor signature validation failures by reason.

What kid Does

JWT header contains: - alg - kid (key identifier) kid allows the verifier to: - select the correct public key among multiple active keys. Production rule: Do not ignore kid if your issuer rotates keys.

Anti-Pattern: Single Hardcoded Key

This breaks during rotation:

options.TokenValidationParameters = new TokenValidationParameters
{
    IssuerSigningKey = new SymmetricSecurityKey(Encoding.UTF8.GetBytes("hardcoded-secret"))
};

Problems: - no rotation - no key rollover window - secret often stored in config or repo - impossible zero-downtime rotation

Correct Pattern: Key Discovery and Multiple Keys

For external identity providers: - rely on metadata endpoint (OpenID Connect discovery) - let framework fetch signing keys - validate issuer and audience strictly Conceptually:

options.Authority = "https://issuer.example.com";
options.TokenValidationParameters = new TokenValidationParameters
{
    ValidateIssuer = true,
    ValidateAudience = true,
    ValidateIssuerSigningKey = true
};

The framework retrieves and caches keys from the issuer’s JWKS endpoint. Production rule: Trust keys only from the configured Authority.

Rotation Window Strategy

During rotation: - Old key signs existing tokens. - New key signs new tokens. - Both must be accepted temporarily. Your service must: - accept tokens signed with old and new keys - reject tokens signed with unknown keys If you manually manage keys: - maintain key list with versioning - support overlapping validity window - remove old key only after all old tokens expired

Do Not Trust Arbitrary kid

Attack vector: Attacker crafts token with arbitrary kid and tries to force service to load malicious key. Mitigations: - Only resolve keys from trusted issuer metadata. - Do not dynamically fetch keys from arbitrary URLs derived from token. - Do not accept keys embedded in token. Production rule: Key source must be fixed and trusted, never token-driven.

Monitoring Key-Related Failures

Track: - signature validation failures - unknown kid errors - metadata fetch failures - key refresh events Alert: - sudden spike in signature failures - failure to fetch metadata - unexpected change in key set This is early warning for: - issuer rotation - misconfiguration - attack attempts

Emergency Key Compromise Scenario

If a signing key is compromised: - issuer rotates key - revoke old key immediately - tokens signed with old key must be rejected Your service must: - refresh metadata quickly - reject compromised key - tolerate short-term auth failures gracefully Production rule: Have a documented runbook for key compromise.

Operational Notes

Testing: - Simulate key rotation in staging. - Verify zero 401 spikes during planned rotation. - Test metadata refresh timing. Rollout: - When changing Authority or key config, canary. - Validate new tokens and old tokens both succeed during overlap. Rollback: - If key config misapplied and causes 401 storm, rollback config immediately. - Keep previous config ready for fast restore. Governance: - Do not embed secrets in source code. - Rotate symmetric secrets regularly if used. - Prefer asymmetric keys and external identity providers for scale.

Checklist

- JWT validation supports multiple signing keys simultaneously. - kid is used to select correct key. - Keys are sourced only from trusted issuer metadata. - No hardcoded secrets in source code. - Key rotation tested in staging before production. - Signature validation failures are monitored and alerted. - Metadata refresh strategy is understood and documented. - Runbook exists for emergency key compromise.