harbor-o11y

Grafana dashboards and alerts Harbor ships to the FUTO o11y cluster (yucca-o11y, Model A). CI renders dashboards/*.json into GrafanaDashboard CRs under the harbor folder, adds alerts/*.yaml as-is, and pushes the result as a Flux OCI artifact:

gitlab.futo.org:5050/harbor/harbor-o11y/o11y-manifests:master

The project is public, so no pull credential is needed. The artifact is signed by digest with a cosign key pair; keyless is out because Fulcio does not trust self-hosted GitLab. harbor-infra's terraform (shared/gitlab/group) mints the pair, sets the private half as the protected COSIGN_PRIVATE_KEY variable and commits the public half here as cosign.pub. CI fails the push if the signature does not verify against that file.

cosign verify --key cosign.pub gitlab.futo.org:5050/harbor/harbor-o11y/o11y-manifests:master

Dashboards

File (= uid)	Covers
harbor-server.json	Traffic, latency, active users, workers, Kafka lag, DB pool; $namespace picks harbor-server and/or -alt
harbor-scraper.json	Scrape outcomes and latency, upstream statuses, image fetches, Chromium memory
harbor-moderation.json	Throughput, labels published, CSAM matches, Kafka lag of the moderation group, worker health; $namespace picks harbor-moderation and/or -alt
harbor-push-notifications.json	Delivery volume, token lifecycle, worker health
harbor-verifier-bot.json	Per-platform health checks (up/down timeline, time since last pass), verify outcomes and latency by platform, HTTP API, process
node-pools.json	Per-pool CPU, memory and disk, OOMKills, restarts
cloudnative-pg.json	Upstream CNPG board with local fixes: vars scoped to the cnpg jobs, CNPG cluster read from pg_cluster, slow-log panels from VictoriaLogs

Imported by scripts/import-upstream.py (re-run to refresh; it overwrites local edits and rewrites them to the conventions below):

File (= uid)	Upstream
k8s-views-{global,namespaces,nodes,pods}.json	dotdc/grafana-dashboards-kubernetes
kafka-exporter.json	strimzi/strimzi-kafka-operator examples (consumer lag, topics)
kafka.json, kafka-kraft.json	strimzi/strimzi-kafka-operator examples (broker JMX: disk, throughput, requests, JVM; KRaft quorum)

The o11y store is shared by every FUTO cluster, so every dashboard:

• pins $ds_metrics to VictoriaMetrics Fleet. o11y's default datasource only serves its own cluster. harbor-ops names its datasource the same so one JSON works in both Grafanas.
• uses $ds_logs for VictoriaLogs.
• has a hidden $cluster variable (regex /harbor-infra-.*/) on every PromQL selector (cluster=~"$cluster") and LogsQL query (cluster:$cluster). Each o11y store holds one Harbor cluster, so it resolves on its own. On the CNPG board the Postgres cluster picker is $pg_cluster, after the label, so it cannot collide with $cluster (upstream issue cloudnative-pg/grafana-dashboards#2).
• keeps uid equal to the file name.

Alerts

alerts/harbor.yaml is a GrafanaAlertRuleGroup with folderRef: harbor; notification routing on o11y follows the folder. Rules pin datasourceUid: VictoriaMetricsFleet (never the default VictoriaMetrics, which only sees o11y's own series) and scope on cluster=~"harbor-infra-.*".

Rule	Fires when
HarborPVCFilling / HarborPVCNearlyFull	any PVC above 85% (warning) / 95% (critical): postgres and kafka volumes
HarborServerHighLatency	p95 above 10s for 15m on any route except PutEvents (uploads run 30s+ by nature)
HarborServer5xx	5xx ratio above 5% for 10m
HarborModerationLagging	moderation group (<server>.events) more than 100k messages behind for 30m
HarborConsumerStalled	any consumer group lagging with no offset movement for 30m
HarborKafkaUnderReplicated	under-replicated partitions for 10m
HarborPostgresBackupStale / HarborPostgresWALArchiveStuck	base backup older than 36h / WAL segments queued for archive
HarborDeploymentDown	a harbor or kafka deployment with zero available replicas
HarborPodCrashLooping	more than 3 restarts in 30m

Delivery ships in the bundle too: alerts/contactpoint-mattermost.yaml is a Slack-type GrafanaContactPoint (Mattermost webhooks take Slack payloads) and alerts/route-harbor.yaml is a GrafanaNotificationPolicyRoute sending the harbor folder to it. o11y merges the route through its policy's routeSelector (notification-route: "true") and supplies the webhook as Secret grafana-mattermost-webhook (key url). harbor-ops applies the same two CRs against its own Grafana while the delivery is being verified.

Editing

Export from Grafana, save over the file keeping the uid, then:

mise install
mise run build   # validate, lint, render, kustomize build

Linter exclusions live in dashboards/.lint.

Consuming

harbor-ops applies this bundle to its own Grafana (kubernetes/apps/base/harbor-o11y). On o11y, add base/harbor-o11y/ and include it from each env's o11y overlay. The Secret holds cosign.pub:

apiVersion: v1
kind: Secret
metadata:
  name: harbor-o11y-cosign
  namespace: flux-system
stringData:
  cosign.pub: |
    <contents of cosign.pub>
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: harbor-o11y
  namespace: flux-system
spec:
  interval: 1m
  url: oci://gitlab.futo.org:5050/harbor/harbor-o11y/o11y-manifests
  ref:
    tag: master
  verify:
    provider: cosign
    secretRef:
      name: harbor-o11y-cosign
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: harbor-o11y
  namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: OCIRepository
    name: harbor-o11y
  path: ./
  prune: true
  targetNamespace: o11y
  dependsOn:
    - name: grafana-operator