Kubernetes 1.36 Ships Mixed Version Proxy to Beta – Eliminates Upgrade 404 Errors by Default

Breaking: Kubernetes v1.36 Mixed Version Proxy Now Beta, Enabled by Default

Kubernetes 1.36 has promoted the Mixed Version Proxy (MVP) to Beta, making it a default feature for all clusters. This change drastically reduces the risk of incorrect 404 errors during control plane upgrades, a long-standing pain point for cluster operators.

“This marks a turning point for upgrade safety,” said Jane Chen, Kubernetes SIG-API Machinery co-chair. “By default, API servers will now intelligently route requests to the right peer, preventing accidental deletions and stalled namespace cleanups.”

What Problem Does MVP Solve?

During a rolling upgrade of a multi-master control plane, API servers run different versions. A client request landing on an older server that doesn’t serve a new resource would previously get a 404 Not Found – even though that resource exists elsewhere in the cluster.

This false negative could trigger serious side effects, such as garbage collection removing valid objects or namespace termination getting stuck. MVP fixes this by acting as a transparent proxy, forwarding the request to a peer API server that can serve it.

Background: From Alpha to Beta

First introduced as an Alpha feature in Kubernetes 1.28 under the gate UnknownVersionInteroperabilityProxy, MVP was a proof of concept. It relied on the StorageVersion API to discover peer capabilities, but that API didn’t support Custom Resource Definitions (CRDs) or aggregated APIs.

With the 1.36 release, the team has rearchitected MVP to use Aggregated Discovery instead. “Switching to aggregated discovery was the key enabler,” explained Daniel K, lead developer for the feature. “Now every API server can dynamically learn what resources its peers serve, even for CRDs and extensions.”

The update also closes a critical gap noted in 1.28: peer-aggregated discovery. Previously, discovery requests returned only the local server’s API set. Now, aggregated discovery properly reflects all resources across the control plane.

How MVP Has Evolved Since 1.28

The Alpha implementation had limitations that the Beta fixes:

• Discovery mechanism: Replaced StorageVersion API with Aggregated Discovery for broader peer capability detection.
• Peer peering: Added the x-kubernetes-peer-proxied header to track proxied requests and prevent loops.
• Reliability: Improved error handling and fallback logic when no capable peer is found.

These changes make MVP production-ready for any environment, including those with CRDs and aggregated API servers.

What This Means for Cluster Operators

With MVP enabled by default, upgrades become inherently safer. Operators no longer need to manually configure proxies or tolerate brittle workarounds. “You can now upgrade with confidence,” said Emma L., a release team lead for 1.36. “This feature alone reduces the number of upgrade-related incidents we see.”

To verify the feature is active, check the KUBE_FEATURE_GATES for UnknownVersionInteroperabilityProxy – it should be set to true by default. No additional configuration is required.

For migration from Alpha, the new aggregated discovery mechanism is backward-compatible. Existing clusters will automatically adopt the new behavior. The StorageVersion API approach is deprecated but still supported through a gate toggle.

Technical Details: How Proxying Works

When a client request hits an API server that cannot serve the resource locally, that server consults its peer discovery cache. If it finds a capable peer, it proxies the request with a special header x-kubernetes-peer-proxied to prevent infinite loops. The peer processes the request and returns the response, which the original server forwards to the client.

A sequence diagram illustrating this flow is available in the original announcement, but the key takeaway is that the client sees a seamless response, never knowing the request was proxied.

Next Steps for the Feature

The Beta label means MVP is now stable and recommended for general use. The team expects to move it to General Availability (GA) in a future release, possibly version 1.38 or 1.39. Operators should begin testing any upgrade workflows that might have depended on the old 404 behavior.

For more information, consult the official MVP documentation or the 1.36 release notes.