14. Observability

Alerts, audit trail, PanFS events, logs, and metrics, each as a point-in-time snapshot endpoint and a matching Server-Sent Events (SSE) stream endpoint for live updates.

14.1 Metrics and Logs Behind These Endpoints

The endpoints in this chapter are windows onto the data stores of the VDURA observability stack, not a separate data model. Knowing what those stores hold, and how it is named, is what makes the query endpoints useful. This section summarizes it; the full catalog of metric names, labels, alert definitions, and dashboards is in the PanFS Metrics Reference Guide.

Where each endpoint group reads from

Metric naming families

Four naming families coexist in the metrics database. Recognizing the family tells you where a metric comes from and how to read it.

Common labels

Nearly every series carries five labels that identify the emitting node, so any metric can be sliced by node or node role without a join. Realm-level panfs_* inventory metrics describe realm objects rather than the emitting host and carry only node_name.

Reading values correctly

• Counters (host disk and network totals, File Manager operation counts) increase monotonically; use rate() or increase().
• Gauges are read directly.
• Info metrics have a constant sample value and carry their content in labels; read them with last_over_time() or group by the label, never with rate().
• Interval gauges (the measure_ group under magent_local_sw_iopath_) already represent the last collection interval; sum them across nodes for a realm figure, and do not apply rate() to them.
• Management agent and realm metrics arrive about every 20 seconds.

Example queries

Pass any of these as the query parameter of GET /api/v1/observability/metrics/query:

curl -sG "https://<realm-address>/api/v1/observability/metrics/query" \
  -H "Authorization: Bearer $TOKEN" \
  --data-urlencode 'query=count by (node_type) (system_uptime)'

Logs and the audit trail

The audit and event endpoints read structured log lines from VictoriaLogs. The fields of an AuditRecord are the fields of the API audit log (/var/log/panapi-audit.log): client_ip, method, request_id, result, status_code, action, resource, target_id, tenant_id, and user_id. Every log line also carries stream identity fields such as _time, _msg, node_name, host_id, and filename (or unit for journald entries). To go beyond what the audit endpoints offer, query the logs directly through the proxy with LogsQL. For example, to list recent failed API operations:

curl -sG "https://<realm-address>/api/v1/observability/logs/query" \
  -H "Authorization: Bearer $TOKEN" \
  --data-urlencode 'query=filename:="/var/log/panapi-audit.log" result:=failed' \
  --data-urlencode 'limit=20'

14.2 Endpoint Reference

Endpoint Summary

GET /api/v1/observability/alerts

Alerts snapshot (keyed set)

The current alert set of the caller’s own tenant, most severe first. A platform caller widens to one tenant with tenant_id, or to every tenant with all_tenants=true; its own tenant is the realm’s telemetry, so the default read is the realm’s health rather than the tenants’ records. A tenant-filtered read returns only records that carry a tenant, so during a rolling upgrade a node that has not taken the new configuration yet produces records no tenant-filtered read returns. That fails closed: it is a completeness limit, never a disclosure.

Authentication: required (PasswordLogin or BearerJWT).

Parameters

Responses

GET /api/v1/observability/alerts/stream

Alerts SSE stream (added/resolved)

Deltas for the same feed the alerts snapshot reads, taking the same tenant_id scope. The stream ends itself with one terminal end event whose reason is credential_expired (refresh the token and reconnect), credential_revoked (sign in again), permission_revoked (the caller no longer holds the grant), lifetime_expired (the maximum stream lifetime; reconnect) or recheck_unavailable (the credential could not be re-checked several times running; reconnect with backoff). The checks run on stream ticks, which on a quiet feed happen every 20 seconds, so an end can lag by about that.

Authentication: required (PasswordLogin or BearerJWT).

Parameters

Responses

GET /api/v1/observability/audit

Audit trail snapshot (last N)

Newest API audit records, newest first, one row per request. The server owns the LogsQL query, so clients pass no query text. Records are deduplicated by their request_id - one record per API call. A tenant caller reads its own tenant’s records; and so does a platform caller by default; a platform caller widens with tenant_id or all_tenants=true. user_id reads one user’s records; the filters compose. only keeps just the records recorded under the named actions, and skip drops them - skip=list,view hides the read-only bulk, leaving the records that changed something, while only=create,delete reads the lifecycle alone. since and until bound the record time inclusively (RFC 3339; a value with no timezone reads as UTC); an inverted range matches nothing. A tenant-filtered read returns only records that carry a tenant, so during a rolling upgrade a node that has not taken the new configuration yet produces records no tenant-filtered read returns. That fails closed: it is a completeness limit, never a disclosure.

Authentication: required (PasswordLogin or BearerJWT).

Parameters

Responses

GET /api/v1/observability/audit/stream

Audit SSE stream (append deltas)

Deltas for the same feed the audit snapshot reads, taking the same tenant_id, user_id, skip and only filters - pass the values the snapshot was taken with, so the stream continues it. since/until have no stream counterpart: a stream starts at its baseline and appends from there. The stream ends itself with one terminal end event whose reason is credential_expired (refresh the token and reconnect), credential_revoked (sign in again), permission_revoked (the caller no longer holds the grant), lifetime_expired (the maximum stream lifetime; reconnect) or recheck_unavailable (the credential could not be re-checked several times running; reconnect with backoff). The checks run on stream ticks, which on a quiet feed happen every 20 seconds, so an end can lag by about that.

Authentication: required (PasswordLogin or BearerJWT).

Parameters

Responses

GET /api/v1/observability/audit/{request_id}

Audit record with failure diagnostics

One audit record plus the correlated pan-api log detail for that request. diagnostics is parsed from the correlated ERROR line when one is found, and is always present, empty when no correlated lines exist. 404 for an unknown or aged-out id, and for a tenant caller also for a record of another tenant or one carrying no tenant, so a tenant cannot probe which ids exist. There is no all-tenants form here, because a request id already names one record: a platform caller reads another tenant’s record by naming it with tenant_id, and gets the same 404 otherwise.

Authentication: required (PasswordLogin or BearerJWT).

Parameters

Responses

Response Body (200, AuditRecordDetail)

GET /api/v1/observability/events

PanFS events snapshot (last N)

Newest PanFS management events, newest first, one row per event. The server owns the query, so clients pass no query text. Records are deduplicated by their PanFS event_id, which the feed repeats whenever the events file is re-read, or by their request_id when they come from the audit trail and carry no event_id. A tenant caller reads its own tenant’s events, and so does a platform caller by default; a platform caller widens to one tenant’s with tenant_id. A tenant-filtered read returns only records that carry a tenant, so during a rolling upgrade a node that has not taken the new configuration yet produces records no tenant-filtered read returns. That fails closed: it is a completeness limit, never a disclosure.

Authentication: required (PasswordLogin or BearerJWT).

Parameters

Responses

GET /api/v1/observability/events/stream

PanFS events SSE stream (append deltas)

Deltas for the same feed the snapshot reads, taking the same tenant_id filter. Pass the value the snapshot was taken with, so the stream continues it. The stream ends itself with one terminal end event whose reason is credential_expired (refresh the token and reconnect), credential_revoked (sign in again), permission_revoked (the caller no longer holds the grant), lifetime_expired (the maximum stream lifetime; reconnect) or recheck_unavailable (the credential could not be re-checked several times running; reconnect with backoff). The checks run on stream ticks, which on a quiet feed happen every 20 seconds, so an end can lag by about that.

Authentication: required (PasswordLogin or BearerJWT).

Parameters

Responses

GET /api/v1/observability/logs

Logs snapshot (last N)

Newest log records matching a LogsQL query. A query the backend cannot parse answers 400 with Invalid query: and the backend’s own diagnosis, which never echoes the query itself. A backend that is down answers 503.

Authentication: required (PasswordLogin or BearerJWT).

Parameters

Responses

GET /api/v1/observability/logs/stream

Logs SSE stream (append deltas)

Deltas for the same feed the logs snapshot reads. A query the backend cannot parse answers 400 with Invalid query: and the backend’s own diagnosis, which never echoes the query itself. A backend that is down answers 503. The stream ends itself with one terminal end event whose reason is credential_expired (refresh the token and reconnect), credential_revoked (sign in again), permission_revoked (the caller no longer holds the grant), lifetime_expired (the maximum stream lifetime; reconnect) or recheck_unavailable (the credential could not be re-checked several times running; reconnect with backoff). The checks run on stream ticks, which on a quiet feed happen every 20 seconds, so an end can lag by about that.

Authentication: required (PasswordLogin or BearerJWT).

Parameters

Responses

GET /api/v1/observability/logs/{logsql_endpoint}

VictoriaLogs /select/logsql proxy

Proxies to VictoriaLogs /select/logsql/{endpoint} on the local vlselect instance (co-located on the director node). All query parameters are forwarded as-is (GET only). See VictoriaLogs querying docs for supported endpoints and their parameters.

Authentication: required (PasswordLogin or BearerJWT).

Parameters

Responses

GET /api/v1/observability/metrics/query

Prometheus instant query proxy

Forwards only Prometheus instant-query parameters (query, time, timeout, step) to the Victoria Metrics /api/v1/query endpoint; other query parameter names are rejected. Upstream status, body, and end-to-end headers are forwarded; hop-by-hop headers (RFC 7230) and any names listed in Connection are stripped. Content-Length and Content-Encoding are omitted because the response body is returned already decoded. If Content-Type is missing on a 200 response, application/json is assumed; other status codes are not given a default type. Query parameters are validated with PrometheusInstantQueryParams; extra names yield 422.

Authentication: required (PasswordLogin or BearerJWT).

Parameters

Responses

GET /api/v1/observability/metrics/query_range

Prometheus range query proxy

Forwards Prometheus range-query parameters (query, start, end, step, timeout) to the Victoria Metrics /api/v1/query_range endpoint; other query parameter names are rejected. Upstream status, body, and end-to-end headers are forwarded; hop-by-hop headers (RFC 7230) and any names listed in Connection are stripped. Content-Length and Content-Encoding are omitted because the response body is returned already decoded. If Content-Type is missing on a 200 response, application/json is assumed; other status codes are not given a default type. Query parameters are validated with PrometheusRangeQueryParams; extra names yield 422.

Authentication: required (PasswordLogin or BearerJWT).

Parameters

Responses

GET /api/v1/observability/metrics/stream

Grouped metric SSE stream (append deltas for several keys)

Streams append deltas for every key in k over a single connection. Each event carries {"key": "<metric key>", "samples": [...]} so the client can route it to the right chart. Snapshots stay per-key (GET /observability/metrics/{key}).

Grouping keeps concurrent SSE connections low: browsers cap HTTP/1.1 connections at six per host, so one stream per chart would starve ordinary REST calls. Each key is still subscribed as its own broker feed, so clients requesting overlapping sets share one upstream poller per common key.

Authentication: required (PasswordLogin or BearerJWT).

Parameters

Responses

GET /api/v1/observability/metrics/targets

List metrics targets

Returns a list of cluster nodes grouped by their type and state in Prometheus HTTP Service Discovery format. Each target is <node management IP>:<port> where the port is the observability agent (telegraf) Prometheus exporter serving /metrics on that node - 9273 by default. __meta_node_state is the PanFS node state, not the health of the scrape endpoint. Nodes that do not run the observability agent are still listed.

Authentication: required (PasswordLogin or BearerJWT).

Responses

GET /api/v1/observability/metrics/{key}

Metric snapshot (range window)

Range-query snapshot for a named metric over the lookback window.

Authentication: required (PasswordLogin or BearerJWT).

Parameters

Responses

GET /api/v1/observability/metrics/{key}/stream

Metric SSE stream (append deltas)

SSE: append events as new metric samples arrive (deltas only).

Authentication: required (PasswordLogin or BearerJWT).

Parameters

Responses

GET /api/v1/observability/storagesets/status

Storage-set alert status

Active-alert presence per storage set, resolved from one query over all PanFS alerts. Only sets carrying at least one critical or warning alert are listed - an absent set means no alerts. Callers combine this with the set’s node/spare counts to render its status badge.

Authentication: required (PasswordLogin or BearerJWT).

Responses