harbor-o11y

Grafana dashboards and alerts Harbor ships to the FUTO o11y cluster (yucca-o11y, Model A). CI renders dashboards/*.json into GrafanaDashboard CRs under the harbor folder, adds alerts/*.yaml as-is, and pushes the result as a Flux OCI artifact:

registry.futo.org/harbor/o11y-manifests:master

The registry allows anonymous pulls under harbor/, so no pull credential is needed. The artifact is signed by digest with a cosign key pair; keyless is out because Fulcio does not trust a self-hosted forge. The key is the one harbor/harbor signs its images and charts with: harbor-infra's terraform (shared/futo-git/org) mints it, sets the private half as the COSIGN_PRIVATE_KEY Actions secret on both repos and commits the public half here as cosign.pub. The publish workflow fails the push if the signature does not verify against that file.

cosign verify --key cosign.pub registry.futo.org/harbor/o11y-manifests:master

Dashboards

File (= uid)	Covers
harbor-server.json	Traffic, latency, active users, workers, Kafka lag, DB pool, autoscaling (API on CPU, workers on Kafka lag via KEDA); $namespace picks harbor-server and/or -alt
harbor-scraper.json	Scrape outcomes and latency, upstream statuses, image fetches, Chromium memory
harbor-moderation.json	Throughput, labels published, CSAM matches, Kafka lag of the moderation group, worker health, KEDA autoscaling; $namespace picks harbor-moderation and/or -alt
harbor-push-notifications.json	Delivery volume, token lifecycle, worker health, autoscaling (Kafka lag and CPU)
harbor-verifier-bot.json	Per-platform health checks (up/down timeline, time since last pass), verify outcomes and latency by platform, HTTP API, process
node-pools.json	Per-pool CPU, memory and disk, OOMKills, restarts, nodes per pool and pending pods (cluster autoscaler)
harbor-flux.json	Flux: failing and suspended objects with reasons, Warning events and controller errors, readiness over time, reconcile rates and durations, controller resources
harbor-billing.json	Cloud spend from the harbor-infra billing collector: month to date, forecast and last month per provider and env, daily spend and trends (35 days), DigitalOcean by project, Azure by resource group, Cloudflare subscriptions and R2 per bucket, collector health, and per-resource line items from VictoriaLogs; $provider and $env pickers
harbor-valkey.json	Valkey pub/sub: messages published, subscriber clients and channels, subscription churn, output buffers and dropped slow subscribers (with their log lines), replication lag and events, memory, network, latency, resources and logs
harbor-cloudnative-pg.json	Upstream CNPG board with local fixes: vars scoped to the cnpg jobs, CNPG cluster read from pg_cluster, slow-log panels from VictoriaLogs, and a PgBouncer row for CNPG Poolers (clients, server connections, wait, throughput, prepared statements, resources)

Imported by scripts/import-upstream.py (re-run to refresh; it overwrites local edits and rewrites them to the conventions below):

File (= uid)	Upstream
k8s-views-{global,namespaces,nodes,pods}.json	dotdc/grafana-dashboards-kubernetes
kafka-exporter.json	strimzi/strimzi-kafka-operator examples (consumer lag, topics)
kafka.json, kafka-kraft.json	strimzi/strimzi-kafka-operator examples (broker JMX: disk, throughput, requests, JVM; KRaft quorum)

The o11y store is shared by every FUTO cluster, so every dashboard:

• pins $ds_metrics to VictoriaMetrics Fleet. o11y's default datasource only serves its own cluster.
• uses $ds_logs for VictoriaLogs.
• has a hidden $cluster variable (regex /harbor-infra-.*/) on every PromQL selector (cluster=~"$cluster") and LogsQL query (cluster:$cluster). Each o11y store holds one Harbor cluster, so it resolves on its own. On the CNPG board the Postgres cluster picker is $pg_cluster, after the label, so it cannot collide with $cluster (upstream issue cloudnative-pg/grafana-dashboards#2).
• keeps uid equal to the file name.

Alerts

alerts/harbor.yaml is a GrafanaAlertRuleGroup with folderRef: harbor; notification routing on o11y follows the folder. Rules pin datasourceUid: VictoriaMetricsFleet (never the default VictoriaMetrics, which only sees o11y's own series) and scope on cluster=~"harbor-infra-.*".

Rule	Fires when
HarborPVCFilling / HarborPVCNearlyFull	any PVC above 85% (warning) / 95% (critical): postgres and kafka volumes
HarborServerHighLatency	p95 above 10s for 15m on any route except PutEvents (uploads run 30s+ by nature)
HarborServer5xx	5xx ratio above 5% for 10m
HarborModerationLagging	moderation group (<server>.moderation) more than 100k messages behind for 30m
HarborConsumerStalled	any consumer group lagging with no offset movement for 30m
HarborKafkaUnderReplicated	under-replicated partitions for 10m
HarborPostgresBackupStale / HarborPostgresWALArchiveBacklog	base backup older than 36h / more than 200 WAL segments queued for archive for 15m
HarborDeploymentDown	a harbor or kafka deployment with zero available replicas
HarborPodCrashLooping	more than 3 restarts in 30m
HarborVerifierPlatformDown / HarborVerifierPlatformStale	a platform's health check failing for 15m (warning) / no passing check for 1h (critical)
HarborVerifier5xx	verifier bot API 5xx ratio above 5% for 10m
HarborFluxSourceFailing	a GitRepository, OCIRepository, HelmRepository, HelmChart or Bucket not ready for 10m (critical): nothing sourced from it deploys
HarborFluxReconcileFailing	a Kustomization or HelmRelease not ready for 15m
HarborFluxStale	a Kustomization ready at a revision other than the flux-system GitRepository's for 30m: ready but not reconciling
HarborFluxSuspended	any Flux object suspended for 2h

The verifier rules carry the series' platform and verifier labels, so a platform that is expected to fail (no OAuth app yet, for example) is muted with a Grafana silence matching platform=<slug>, without touching the rules.

Editing

Export from Grafana, save over the file keeping the uid, then:

mise install
mise run build   # validate, lint, render, kustomize build

Linter exclusions live in dashboards/.lint.

Consuming

On o11y, add base/harbor-o11y/ and include it from each env's o11y overlay. The Secret holds cosign.pub:

apiVersion: v1
kind: Secret
metadata:
  name: harbor-o11y-cosign
  namespace: flux-system
stringData:
  cosign.pub: |
    <contents of cosign.pub>
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: harbor-o11y
  namespace: flux-system
spec:
  interval: 1m
  url: oci://registry.futo.org/harbor/o11y-manifests
  ref:
    tag: master
  verify:
    provider: cosign
    secretRef:
      name: harbor-o11y-cosign
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: harbor-o11y
  namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: OCIRepository
    name: harbor-o11y
  path: ./
  prune: true
  targetNamespace: o11y
  dependsOn:
    - name: grafana-operator