Fleet OCI storage: multiple registries and per-cluster routing

UI prototype and design notes for rancher/fleet 5711, which covers 4843 and 4844. Built from f3dcae8ae on 2026-11-04, on top of dashboard master, proxying the mf Rancher.

Try it

1. Open the prototype and log in with your mf Rancher account. The sample data is in workspace fleet-default. The page is at Continuous Delivery, Resources, OCI Storage.
2. Debug a failure: Clusters tab, click edge-store-0203.
3. See replication wait: Cluster Routing, open apac-clusters, set Replication wait to 2. Effect on Clusters now predicts Failover. Save it and check the Clusters tab.
4. See route order matter: Add Route, pick eu-staging-01 and any registry. The form warns that eu-clusters matches it first. Save, then move it up with the arrows.
5. See the simple case: switch the workspace to fleet-local and add one registry.
6. Other pages touched: App Bundles, Create, Git Repo, Advanced step. A Fleet cluster's OCI Storage tab. A workspace's Default OCI Registry tab.
7. Reset mock data in the yellow banner puts everything back.

The problem

Fleet stores a bundle in one registry and every target cluster pulls from that same place. That breaks when the management cluster and downstream clusters can't use the same registry, namespace or credentials, or when a site needs failover. The RFEs ask for per-cluster registry, credentials and namespace (4844) and for multiple registries with failover (4843). The triage comment on 4843 sets two constraints: mirroring stays out of Fleet's scope, and agents must retry with back-off when an artifact hasn't replicated yet.

What Fleet does today, which shaped the design:

• fleet apply pushes to the secret named in ociRegistrySecret, or to the ocistorage fallback, and writes a copy of those settings to an internal secret named after the artifact (apply.go). With neither secret, the bundle stays in etcd (apply.go).
• The bundle controller clones that internal secret into each cluster's namespace (bundle_controller.go). That copy is why changing a secret has no effect until the next update. It is also where per-cluster credentials would have to be picked, as the triage comment suggests.
• The agent pulls with agentUsername if set, otherwise with the push credentials (ociwrapper.go). It then checks that the content matches the artifact ID (deployer.go). That check is what makes failover between registries safe.
• A named secret that is missing or invalid doesn't fall back to etcd (docs).
• ociRegistrySecret only exists on GitRepo. HelmOps pull their chart directly, so the HelmOp form isn't touched.

The model the UI assumes

• Registry: today's secret (reference, credentials, TLS) plus a role. The source is where Fleet pushes, one per workspace. A mirror is only pulled from, and replicating to it is your job.
• Route: these clusters use these registries, in this order. First match wins, and a default route catches the rest. Clusters are selected the same way as GitRepo targets, by name, group or label.
• Status: agents report what happened. Fleet can't see inside a mirror it may not reach, so availability comes from agents asking for artifacts.
• Different credentials or namespace on the same registry is simply a second registry entry with the same host. No separate concept is needed.
• Two failure paths: an auth, TLS or connection error moves the agent to the next registry straight away. A missing artifact means replication is behind, so the agent retries with back-off for the route's replication wait first.

Use cases

Walkthrough

What data is worth showing

Exists Fleet has it today, though the dashboard reads neither contentsId nor ociReference yet. New needs a backend change. Derived the UI can compute it once the new data exists.

What the UI would need from the backend

1. A place for routing with the fields in the first example below, wherever it ends up (options).
2. Status on resources that aren't Secrets. Someone debugging a pull shouldn't need to read credentials. Aggregate on Cluster.status as well: the Fleet cluster page already loads every BundleDeployment, and that doesn't scale to a table of thousands of clusters.
3. A distinct, non-error state for "waiting for replication". Fleet recently added WaitingForDependency, and the dashboard needed fixes because it painted the new state red. Emit it in status.display.state and in the summary counts from day one, and we add it to the UI state tables alongside.
4. A failure class, not only a message. Unauthorized, TLS, Unreachable, NotFound and IntegrityMismatch. The agent needs it to decide between failing over and backing off anyway, and the UI uses it for the state and the next step.
5. Artifact metadata on the Bundle: size and pushed-at. Without pushed-at nobody can see replication lag.
6. Later: an agent-side connectivity probe behind Test Connection. Today that button is simulated.

The field names below are only there to make the idea concrete:

apiVersion: fleet.cattle.io/v1alpha1
kind: OCIStorageRouting            # hypothetical name
metadata:
  name: default
  namespace: fleet-default
spec:
  routes:                          # first match wins
    - name: eu-clusters
      clusterSelectors:            # same vocabulary as GitRepo targets
        - matchLabels: { region: eu }
      registries: [harbor-eu, harbor-central]   # existing OCI storage secrets
      replicationTimeout: 10m
    - name: air-gapped-stores
      clusterGroups: [airgap-stores]
      registries: [zot-stores]
  default:
    registries: [harbor-central]

# BundleDeployment.status, reported by the agent (hypothetical)
ociPull:
  registry: harbor-apac
  state: WaitingForArtifact        # Pulled | WaitingForArtifact | Failed
  reason: NotFound                 # Unauthorized | TLS | Unreachable | NotFound | IntegrityMismatch
  attempts: 3
  nextRetry: "2026-11-04T16:21:00Z"
  lastSuccess: "2026-11-04T10:12:00Z"

# Fleet Cluster.status, aggregated so the UI doesn't have to load every BundleDeployment
ociStorage:
  route: apac-clusters
  chain: [harbor-apac, harbor-central]
  active: harbor-apac
  waiting: 1
  failed: 0

Where routing could live

The first three are from the issue. The UI only depends on "registries, routes, status", so changing this later only touches the prototype's data layer.

My recommendation from the UI side is the new resource. The secret option can work if a new CRD is a hard no, but then status on Cluster and BundleDeployment matters even more, because the routing itself won't be readable for many users. A per-cluster override doesn't need its own field: it's a route that names one cluster (lab-sandbox in the sample data).

Open questions

For the Fleet team

1. Should a routing change need an update at all? The prototype follows today's rule (changes apply on the next Git update or force update) and shows Update Pending. But that delay comes from the per-bundle copy of the settings. If the controller reads the current route when it clones credentials, a routing change could apply on the next pull, and nothing needs redeploying because artifact IDs are content-addressed. That would remove a state, a banner and a checkbox from this UI. I'd prefer that.
2. Is the replication wait per route or global? The prototype has it per route because a store on a slow link differs from a region. A single default may be enough to start with.
3. What cleans up mirrors? Fleet deletes old artifacts from the source (apply.go). Mirrors only lose them if the replication tool propagates deletes.
4. Hundreds of sites with their own registry. One entry per host doesn't scale. The sample data avoids it with a local DNS name that every store shares. If that isn't realistic, the reference may need to template on cluster labels.
5. Does Fleet ever push to more than one registry? The prototype assumes one source, following the triage comment. Sites with no replication tool would have no way to fill a mirror.

For UX

1. Words: Source and Mirror, Route, Pull Order, Waiting. Do these read well to someone who isn't a registry expert?
2. Placement: it sits under Resources next to Bundles because OCI storage is opt-in. Top level would be easier to find. Should the Clusters tab also be a column on the main Fleet Clusters list?
3. Banners: a page in trouble can show the prototype banner, a failure banner and a pending-update banner together. Without the prototype banner it's two, and one goes away if question 1 goes the way I'd prefer.
4. The workspace's Default OCI Registry tab overlaps with "source". It prefills the GitRepo form through a UI-only annotation. I'd drop it once a source exists. The registry form could also serve dashboard issue 14433 (create OCI secrets from the workspace form), which is labelled requires-prototyping.

What the prototype doesn't do

• Nothing reaches the server. Passwords aren't kept, Test Connection is simulated, and a pinned registry on the GitRepo form isn't saved.
• Real clusters on the mf server have no region labels, so they land on the default route. The richer scenarios use made-up clusters that only exist on this page.
• Tables aren't paginated and RBAC isn't modelled, so read-only users see the same page. Both need real thought before this ships.
• Only the unit tests for the routing logic exist (37 cases). There are no e2e tests, since this is a prototype.