Argo Rollouts Canary Not Progressing? Step-by-Step Fix Guide

Introduction

Canary deployments are a primary reason teams adopt Argo Rollouts. You define a sequence of traffic weights and pauses, and the controller gradually shifts traffic from the old version to the new one while you watch metrics and behavior. When everything works, it feels like a safe, controlled way to ship. Sometimes, though, the canary seems to stall. The rollout may get stuck at an early weight and refuse to move on, or it may pause and never resume. In other cases, you run a promotion command and nothing changes. From the outside, it looks like your canary is simply “not progressing.”

This guide walks through the most common causes of stuck canaries and gives you a structured way to debug them. By the end, you will know where to look, AnalysisRuns, traffic routing, pod health, and configuration, so you can bring the rollout back to a healthy, predictable flow.

How Canary Progression Works in Argo Rollouts

At a high level, canary progression in Argo Rollouts is driven by steps. Each step tells the controller what to do next: set a weight, pause, run an analysis, or execute a specific action.

A typical canary strategy might look like this:

strategy:

  canary:

    steps:

            - setWeight: 20

            - pause: { duration: 120 }

            - setWeight: 50

            - pause: {}

            - setWeight: 100

The controller evaluates these steps in order. After it completes one step, it moves to the next. Pauses can be time‑based (a fixed duration) or manual (waiting for you to explicitly resume). Analysis steps can run metrics checks using providers like Prometheus or Datadog and decide whether to continue or roll back.

When a canary gets stuck, it usually means the controller is waiting for something: a pause to end, an analysis to finish, pods to become healthy, or traffic routing to converge. Your job is to determine which condition is blocking the next step.

Step-by-Step Guide to Fix a Stalled Argo Rollouts Canary

Step 1: Look at Rollout Status and Current Step

Start by asking Argo Rollouts what it thinks is happening. The kubectl argo rollouts plugin provides a detailed view of the rollout’s status.

Run:

kubectl argo rollouts get rollout <rollout-name> -n <namespace>

The output shows:

• The current step index.
• The set weight and actual traffic weight (if integrated with ingress or service mesh).
• Whether the rollout is paused and why.
• Any AnalysisRuns associated with the rollout and their status.

If the status indicates Paused, read the reason carefully. It may say that the rollout is paused at a certain step, waiting for a manual resume or for an analysis to finish. This single view often tells you immediately why progression has stopped.

Step 2: Distinguish Between Manual and Time‑Based Pauses

Canary steps that include pause can behave very differently depending on their configuration.

A time‑based pause looks like this:

- pause: { duration: 120 }

The rollout remains paused for the specified number of seconds, then automatically continues. If your canary seems stuck on a time‑based pause, check whether the controller is still counting down or if the pause has already elapsed.

A manual pause is defined without a duration:

- pause: {}

In this case, the rollout stays paused indefinitely until you explicitly resume it. If your get command shows a pause with no duration, the fix is simple: you must tell Argo Rollouts to continue.

Use:

kubectl argo rollouts resume <rollout-name> -n <namespace>

After resuming, re‑run the get command and watch the canary move to the next step.

Step 3: Investigate AnalysisRuns and Metric Checks

If your canary includes analysis steps, a failing or hanging AnalysisRun can stop progression. In your rollout spec, analysis steps may look like:

- analysis:

    templates:

            - templateName: canary-check

    args:

            - name: stable-hash

        valueFrom:

          fieldRef:

            fieldPath: metadata.annotations['rollouts.argoproj.io/stable-pod-hash']

When such a step executes, Argo Rollouts creates an AnalysisRun resource. Its status determines what happens next: success allows progression; failure usually triggers a rollback; inconclusive or error states may also pause the rollout depending on your configuration.

List AnalysisRuns for your rollout:

kubectl get analysisruns -n <namespace>
Then inspect the one associated with the stuck step:
kubectl describe analysisrun <name> -n <namespace>

Look for:

• Metric failures or thresholds being crossed.
• Errors connecting to metric providers.
• Timeouts or inconclusive results.

If the AnalysisRun consistently fails, you may need to adjust your metrics, thresholds, or data collection windows. If it never finishes, check connectivity to your monitoring backend and ensure the analysis template is configured correctly.

Step 4: Confirm Pod Health and Readiness

Even without analysis, Argo Rollouts will not progress a canary if the new pods are unhealthy. After a step sets a new weight, the controller waits for the desired number of canary pods to become ready.

Check the pods for your rollout:

kubectl get pods -n <namespace> -l <your-rollout-selector>

Then describe problematic pods:

kubectl describe pod <pod-name> -n <namespace>

Look for events indicating:

• Image pull failures.
• Crashes or restarts.
• Readiness or liveness probe failures.
• Resource constraints such as insufficient CPU or memory.

If pods cannot become ready, the rollout has no healthy canary capacity to shift traffic to. Fixing these underlying pod issues, by correcting images, adjusting probes, or right‑sizing resources, allows the canary to move forward.

Step 5: Check Traffic Routing With Ingress or Service Mesh

In many setups, Argo Rollouts integrates with an ingress controller or service mesh (such as Istio or Linkerd) to implement actual traffic splitting. The rollout spec includes references to this routing layer. If the integration is misconfigured, the controller may think it is changing weights while real traffic remains pinned to the old version.

Examine the traffic routing configuration in your rollout:

strategy:

  canary:

    canaryService: my-app-canary

    stableService: my-app-stable

    trafficRouting:

      istio:

        virtualService:

          name: my-app-vs

          routes:

            - primary

Verify that:

• The services exist and point at the correct pods.
• The virtual service or ingress object is present in the namespace.
• Annotations and routing rules match the expectations for your ingress or mesh.

You can also inspect the configuration directly in Istio or your ingress controller to ensure weights update as Argo Rollouts changes them. If the external routing layer is out of sync, fix that configuration so the controller’s intended steps actually affect real traffic.

Step 6: Look for Conflicting Manual Actions

Sometimes, canaries get stuck because humans intervene in ways the controller does not expect. For example, someone might edit the rollout by hand, scale deployments directly, or modify services outside of the normal workflow.

Review recent changes in the namespace:

• Have any Deployment resources with overlapping selectors been created?
• Has someone edited the rollout spec manually, removing or reordering steps?
• Did a GitOps tool apply a different version of the manifest partway through the canary?

If you suspect interference, align everyone on a single, consistent workflow and treat the rollout manifest as the single source of truth. Clear up conflicting resources, re‑apply the intended spec, and let the controller resume from a clean state.

Step 7: Use Events and Logs for Extra Clarity

Kubernetes events and Argo Rollouts controller logs are valuable when the basic checks do not reveal the cause of a stuck canary.

Describe the rollout to see events:

kubectl describe rollout <rollout-name> -n <namespace>

Events often include messages such as “Paused due to analysis run,” “Waiting for rollouts to be healthy,” or “Resumed rollout after manual command.” These hints tell you what condition the controller is waiting on.

For deeper insight, inspect the controller logs:

kubectl logs deploy/argo-rollouts -n argo-rollouts

Search for your rollout name and look for warnings or errors around step processing, analysis, or traffic routing. While you do not need to read every log line, a few targeted searches can highlight the underlying blocker.

Step 8: Consider Simplifying the Canary for Initial Debugging

If your canary strategy is very complex, many steps, multiple analysis runs, intricate traffic routing rules, it may be harder to pinpoint the specific source of the stall. In these situations, temporarily simplifying the strategy can help.

You might:

• Remove analysis steps and test a simpler progression first.
• Reduce the number of weight changes to just two or three.
• Disable integration with mesh or ingress and use service‑only routing.

Once a simpler canary flows correctly, gradually reintroduce complexity. This iterative approach makes it easier to identify which addition introduces the problem.

Conclusion: Turn “Stuck” Canaries Into Predictable Workflows

When a canary in Argo Rollouts stops progressing, it is rarely a mysterious failure. Almost always, the controller is doing exactly what you asked, it is just waiting for a condition that has not been satisfied yet. That might be a manual pause, an unfinished analysis, unhealthy pods, or misaligned traffic routing.

By systematically checking rollout status, pauses, AnalysisRuns, pod health, and routing configuration, you can quickly locate the blocker and decide whether to fix the underlying issue, resume progression, or roll back.

Over time, you can build runbooks and dashboards that make these signals obvious. Instead of treating stuck canaries as emergencies, your team can view them as intentional safeguards that surface real issues before they affect all users.

FAQs: Canary Progression Issues in Argo Rollouts

Latest Post:

• How Argo Rollouts Works: Canary & Blue-Green Strategies
• Argo Rollouts Features: Blue-Green, Canary & Traffic Control
• Argo Rollouts vs Kubernetes Deployments: Key Differences & Benefits
• Is Argo Rollouts Safe? Security & Production Readiness Explained
• Business Use Cases for Argo Rollouts: Safe Deployments, KPIs & Continuous Delivery