3. Metric reference by subsystem
Tables list families rather than every flattened name; Appendix B has the complete index. “Extra labels” are those beyond the five common node labels.
3.1 Realm inventory and state (panfs_*)
These metrics are produced by a collector that reads the PanFS realm database on the director nodes (the same source the GUI and PanCLI use). The inventory families (panfs_node, panfs_volume, panfs_storageset, panfs_nic, panfs_alerts_info, panfs_sysstat_error_event and the scrape_success gauges) carry only node_name, not the five host labels, because they describe realm objects rather than the emitting host; join them to host level metrics on node_name. They follow the Prometheus “info metric” pattern: The sample value is a state code or a constant, and the interesting content is in the labels. Every panfs_* family has a companion <family>_scrape_success gauge (1 = the last collection of that family succeeded, 0 = it failed), emitted once per realm rather than per node. Dashboards read these with last_over_time(…[5m]) and group by the labels they need.
| Metric/family | Type | Extra labels | Meaning | Example query | Conf |
|---|---|---|---|---|---|
| panfs_node | gauge (state code) | name, node_name, node_role (Director, Storage), node_state_name (online), node_state_description, node_data_serial, node_hardware_serial, node_ip_address, enclosure_name, enclosure_hardware_serial, model_number, display_model_number, software_revision, cpu_model, cpu_hw_cores, cpu_hw_threads, cpu_active_cores, memory_bytes, bios_date, bios_revision, platform_id (cloudvm), platform_variant, system_model, system_vendor, slot_id, activated, upgrading, reinit_eligible, remove_eligible, leds_blinking, primary_nic_mode, primary_link_count, secondary_nic_mode, secondary_link_count, nic_physical_port_mode, cpu_temperature_celsius, internal_temperature_celsius, incoming_air_temperature_celsius | One series per realm node. Value observed = 6 on every node, which lines up with node_state_name=online, so the value is the numeric node state code. The labels are a full hardware and software inventory of the node. The Realm Overview “Cluster Nodes” and “Versions” tables are built from this metric. | group by (node_name, node_role, software_revision) (last_over_time(panfs_node[5m])) | C |
| panfs_node_repset | gauge (0/1) | node_name, node_data_serial, node_ip_address, model_number, activated | Whether a node is a member of the realm manager replica set. 1 = member, 0 = not. Storage nodes are always 0, so join with panfs_node{node_role=“Director”} before alerting. The vmalert rule PanFSDirectorNotInReplicaSet fires from this. | panfs_node_repset == 0 and on(node_name) panfs_node{node_role=“Director”} | C |
| panfs_president | gauge (0/1) | datasn, slot, president_slot | Which node currently holds the realm “president” role (the elected realm manager among directors). 1 on the president, 0 elsewhere. datasn is the node data serial. | panfs_president == 1 | C |
| panfs_master_mserver_up | gauge (0/1) | master_ip | Whether the master mserver (the realm management server) at master_ip is reachable from this node. 1 = up. | min(panfs_master_mserver_up) | C |
| panfs_storageset | gauge (state code) | name, bladeset_id, state (“6/6 OSDs online”), capacity_balancing_enabled, balancing_rate_bytes_per_second, imbalance_bytes, blade_imbalance_bytes, spare_osd_count, desired_spare_osd_count, brickset_flags, mlec, osd_filesystem_version, sed_key_generation_number, correctable_error_count, uncorrectable_error_count | One series per StorageSet (formerly BladeSet). Value is a state code: Observed 1 and 2 for healthy sets and -1 for the “Mixed” pseudo set; the dashboard filters with > 0 to drop the placeholder. Labels carry the capacity balancing status, spare OSD counts and error counters. | count(group by (name) (panfs_storageset > 0)) | C |
| panfs_volume | gauge (state code) | name, volume_id, bladeset_id, volume_state_display (Online), volume_state_description, volume_raid (Object RAID6+), raid_layout, raid_group_osd_width, raid_group_stripe_depth, stripe_unit_bytes, stripe_osd_count_max, max_failures_files, directory_replicas, aggregate_layout, compression, compression_type (lz4), efsa_mode, recovery_priority, soft_quota_bytes, hard_quota_bytes, quota_consumed_bytes, space_used, space_available, user_and_group_quotas_policy, down_storage_node_count, corrupt_object_count, corrupt_parity_object_count, corrupt_log_count, object_id, vsm_id, volume_flags, volume_dynamic_flags, volume_mgmt_flags, online_since, offline_since | One series per volume. Value observed = 3 for all online volumes (state code). Labels carry the volume configuration, RAID layout, quota settings and integrity counters. Note quota_consumed_bytes and space_used are labels (strings) not sample values, so they cannot be graphed without label_value(); treat them as snapshot inventory. Tenant volumes appear under /.tenants/<id>/… | topk(1, last_over_time(panfs_volume[5m])) by (volume_id) | C |
| panfs_nic | gauge (index) | interface (eth0, bond0..2), nic_type, node_name, node_data_serial, node_ip_address, active, in_use, mode_of_operation, uplink_down, warnings_disabled, firmware | One series per NIC per director (12 series = 4 NICs x 3 directors). Value is the NIC index (1..4). active/in_use/uplink_down labels give link status. | panfs_nic{uplink_down=“true”} | C |
| panfs_events_dropped_total | counter | (common only) | Number of realm events the metrics collector dropped (queue overflow or parse failure) since it started. | increase(panfs_events_dropped_total[1h]) | I |
| panfs_*_scrape_success | gauge (0/1) | none | One per collected family: alert_occurrences, alerts_active, alerts_info, drive, drive_bytes_read, drive_bytes_written, fan, ipmi, nic, node, node_repset, power_supply, storageset, switch, sysstat_error_event, volume. Value 1 = the last collection succeeded. Note drive, drive_bytes_read/written, fan, ipmi, power_supply and switch report success but emit no data series in this realm, because the nodes are cloud VMs with no physical enclosure; on hardware realms expect panfs_drive*, panfs_fan, panfs_ipmi, panfs_power_supply and panfs_switch families to appear. | min by (__name__) ({__name__=~“panfs_.*_scrape_success”}) == 0 | C |
| observability_node_type | gauge (1) | role (director_l, vpod), dev_type (10, 11) | Emitted by the VDURA metrics agent on every node to declare its role. role and dev_type match the node_type label stamped on all other series (director_l = director, vpod = virtual storage node/OSD). dev_type 10/11 is the internal device type code. | count by (role) (observability_node_type) | C |
3.2 Alerts and events
Two alerting layers are visible. The PanFS realm has its own alert and event system (alert codes, the same ones shown in the PanFS GUI), exported as info metrics. On top of that, vmalert evaluates rule groups (panfs.alerts, panfs.processes) and writes the standard ALERTS and ALERTS_FOR_STATE series, which are what the PanFS Alerts dashboard reads. The vmalert rule definitions cannot be read from Grafana (the vmselect /api/v1/rules endpoint returns no groups, and the test Grafana instance defined no managed alert rules), so the rule expressions live in the vmalert configuration on the observability VM.
| Metric/family | Type | Extra labels | Meaning | Example query | Conf |
|---|---|---|---|---|---|
| ALERTS | gauge (1) | alertname, alertstate (firing/pending), alertgroup, severity, category, source, message, node_name, activated | One series per active vmalert alert. Observed alertnames: PanFSDirectorNotInReplicaSet (group panfs.alerts) and ProcessDown (group panfs.processes). The message label is the human readable text (for example “Software Process Stopped: The process mserver…”). severity values: Critical, Warning, Informational, None. | count by (severity) (ALERTS{alertstate=“firing”}) | C |
| ALERTS_FOR_STATE | gauge (unix time) | same as ALERTS minus alertstate | Standard vmalert companion series: Value is the unix timestamp at which the alert condition first became true. Used to restore “for” durations after a restart. time() - ALERTS_FOR_STATE gives the age of an alert. | time() - ALERTS_FOR_STATE | C |
| panfs_alerts_info | info (1) | alert_code, alert_name (PAN_MGMT_EV_*), severity, error_level_name (info/warn/crit/none), category (hardware/volume/bladeset), is_manually_clearable, action_message, node_name | The realm alert catalog: One series per alert definition known to the realm (703 in this release). It is static reference data, not active alerts. It is the best source for decoding an alert_code seen elsewhere; Appendix A lists it. | panfs_alerts_info{alert_code=“8455”} | C |
| panfs_alert_occurrences | gauge (count) | alert_code, alert_name, category, severity, message, node_name | Number of times a given realm alert is currently raised (active occurrences). Only alerts with at least one occurrence appear. In the test realm: One Warning, “ntpd monitor: No synchronization with external NTP server”. | sum by (severity) (panfs_alert_occurrences) | C |
| panfs_alerts_active (family) | info | see scrape_success | Only the panfs_alerts_active_scrape_success series exists; no active alert series were present at collection time. Expect a panfs_alerts_active info metric to appear when realm alerts are active. Needs confirmation. | I | |
| panfs_sysstat_error_event | info (1) | alert_code, alert_name, category, severity, error_level_name, is_manually_clearable, message, action_message, alert_details (JSON, e.g. {“volume_id”: 1}), last_updated (timestamp), node_name | Realm system status events (the “sysstat” event log). Each event is re emitted every collection with a fresh last_updated label, so a single logical event produces a new series every minute (382 series were live for 30 events over one hour). | count by (alert_name) (last_over_time(panfs_sysstat_error_event[5m])) | C |
3.3 Host operating system metrics (Telegraf style)
The vdura-metrics-agent service on every node ships host level metrics using Telegraf input plugin naming (measurement_field). All of them carry the five common labels. Sample rate is high: About 30 samples per 10 minutes for most agent metrics and up to 180 per 10 minutes for system_uptime. These are the metrics behind the Realm Node dashboard.
| Metric/family | Type | Extra labels | Meaning | Example query | Conf |
|---|---|---|---|---|---|
| cpu_usage_{user,system,idle,iowait,irq,softirq,steal,nice,guest,guest_nice} | gauge (percent) | cpu (cpu-total, cpu0..cpuN) | Per CPU and aggregate CPU time breakdown in percent over the collection interval. Utilization is 100 - cpu_usage_idle. steal is meaningful here because the nodes in the test realm are cloud VMs. | 100 - avg by (host) (cpu_usage_idle{cpu=“cpu-total”}) | C |
| system_load1, system_load5, system_load15 | gauge | none | Standard load averages. | system_load5/system_n_cpus | C |
| system_n_cpus, system_n_physical_cpus | gauge | none | Logical and physical CPU count (4/2 on directors, 2/1 on storage VMs). | C | |
| system_uptime | gauge (seconds) | none | Seconds since boot. Used by the dashboards as the “node is reporting” heartbeat and as the source of the $Node variable. | count by (node_type) (system_uptime) | C |
| system_n_users, system_n_unique_users | gauge | none | Logged in user sessions. | C | |
| mem_total, mem_used, mem_free, mem_available, mem_used_percent, mem_available_percent, mem_active, mem_inactive, mem_buffered, mem_cached, mem_shared, mem_slab, mem_sreclaimable, mem_sunreclaim, mem_mapped, mem_dirty, mem_write_back, mem_write_back_tmp, mem_page_tables, mem_commit_limit, mem_committed_as, mem_swap_total, mem_swap_free, mem_swap_cached, mem_huge_page_size, mem_huge_pages_total, mem_huge_pages_free, mem_vmalloc_total, mem_vmalloc_used, mem_vmalloc_chunk, mem_high_*, mem_low_* | gauge (bytes, percent) | none | /proc/meminfo fields in bytes. Directors have 16 GiB, storage VMs are mixed (memory_bytes label on panfs_node shows 16 GiB and 2.7 GiB models). Note the Realm Node dashboard also queries mem_wired and mem_laundry, which are FreeBSD fields and do not exist on these Linux nodes; those panels render empty. | mem_used_percent | C |
| disk_total, disk_used, disk_free, disk_used_percent, disk_inodes_total, disk_inodes_used, disk_inodes_free, disk_inodes_used_percent | gauge (bytes, count, percent) | device, fstype (ext4, btrfs, zfs, vfat, efivarfs, panfs), path, mode, label | Filesystem usage per mount. This includes the PanFS mount itself (device=“panfs://<director-ip>/”, fstype=“panfs”) on nodes that mount the realm, which is a convenient realm wide capacity number, and the ZFS datapool datasets on storage nodes (datapool-0/pandata-0/…). | disk_used_percent{fstype=“panfs”} | C |
| diskio_reads, diskio_writes, diskio_read_bytes, diskio_write_bytes, diskio_read_time, diskio_write_time, diskio_io_time, diskio_weighted_io_time, diskio_merged_reads, diskio_merged_writes, diskio_iops_in_progress | counter (except iops_in_progress: Gauge) | name (sda, sda1.., sdb..) | Block device counters from /proc/diskstats, per whole disk and per partition. Times are milliseconds. Rate them for throughput and IOPS. The dashboards exclude name=“pass0” (a FreeBSD device name) which is harmless here. | sum by (host) (rate(diskio_read_bytes[$__rate_interval]) + rate(diskio_write_bytes[$__rate_interval])) | C |
| diskio_io_util, diskio_io_await, diskio_io_svctm | gauge (percent, ms) | name | Derived iostat style values (utilization percent, average wait, average service time). Present only for whole disks (84 series vs 213 for the counters). | max by (host) (diskio_io_util) | C |
| net_bytes_sent, net_bytes_recv, net_packets_sent, net_packets_recv, net_err_in, net_err_out, net_drop_in, net_drop_out | counter | interface (eth0) | Interface counters. In the cloud test realm only eth0 is reported (the gve virtual NIC); bond0..2 exist in the PanFS NIC inventory but carry no host counters. | sum by (node_type) (rate(net_bytes_recv[$__rate_interval]) * 8) | C |
| net_speed | gauge (Mbit/s) | interface | Link speed reported by the driver (example from the cloud test realm: 8000 on directors, 4000 on storage VMs, which reflect the cloud provider’s per vCPU bandwidth caps). | C | |
| netstat_tcp_{established,listen,time_wait,close_wait,syn_sent,syn_recv,fin_wait1,fin_wait2,closing,last_ack,close,none}, netstat_udp_socket | gauge (count) | none | Socket counts by TCP state. | netstat_tcp_established | C |
| nstat_* (149 metrics) | counter | none | Kernel SNMP/netstat counters from /proc/net/snmp and /proc/net/netstat with their kernel names (nstat_TcpRetransSegs, nstat_TcpExtTCPTimeouts, nstat_IpInReceives, nstat_UdpInErrors, nstat_Icmp*…). Useful ones for a storage network: TcpRetransSegs, TcpExtTCPLostRetransmit, TcpExtTCPTimeouts, TcpExtListenDrops, TcpExtTCPOFOQueue, UdpRcvbufErrors, IpReasmFails. | rate(nstat_TcpRetransSegs[5m])/rate(nstat_TcpOutSegs[5m]) | C |
| ethtool_* (108 metrics) | counter/gauge | interface, driver (gve) | NIC driver statistics from ethtool -S for the Google gve driver: Per queue rx/tx bytes, descriptors, drops (ethtool_rx_dropped_pkt_N, rx_queue_drop_cnt_N, rx_no_buffers_posted_N), XDP counters, admin queue command counts, plus link state (ethtool_link, ethtool_speed, ethtool_duplex, ethtool_autoneg, ethtool_interface_up). Suffix _0/_1 is the queue index. On physical hardware the set will differ with the driver. | rate(ethtool_rx_dropped[5m]) | C |
| processes_{total,running,sleeping,idle,blocked,stopped,zombies,dead,paging,unknown}, processes_total_threads | gauge (count) | none | Process table summary by state. | processes_blocked | C |
| procstat_cpu_usage, procstat_memory_rss, procstat_memory_vms, procstat_num_fds, procstat_num_threads, procstat_created_at | gauge | process_name, pid, pattern | Per process resource usage for processes matched by the agent’s configured patterns (pattern label shows the regex: (vdura|pan)-api-venv, /pan/bin/(startsrv|mserver/mserver), /pan/bin/pan_mgr_hw_agent, /pan/bin/pan_sqlite_server, /pan/bin/iopathv4, /pan/bin/dhclient, and a catch all “.*”). Because pid is a label and one pattern is “.*”, short lived processes (sleep, kworker) create a new series each time: 15,487 series for procstat_created_at, which is 15 percent of the whole database. | topk(10, procstat_cpu_usage{host=~“$Node”}) | C |
3.4 Management agent: Hardware inventory (magent_local_hw_*)
The magent_* namespace is the PanFS management agent statistics tree (the same counters PanCLI and the realm manager read from each node), flattened into metric names: magent_local_<subsystem>_<stat>. String valued stats are encoded as a series with sample value 1 and the text in a value label. All hw metrics are emitted by every node (15 series).
| Metric/family | Type | Extra labels | Meaning | Example query | Conf |
|---|---|---|---|---|---|
| magent_local_hw_{bios_date, bios_version, cpu_model, platform_id, variant_id, system_model, system_vendor, sysname_str, sn, version, nic_phys_ports} | info (value label) | value | Hardware identity strings: BIOS date/vendor, CPU model, platform (cloudvm), system model (Google Compute Engine), node serial (sn), model/version code (1d4034001 = storage VM model, 3f4018001 = director VM model), physical port list (eth0:10). | magent_local_hw_cpu_model | C |
| magent_local_hw_cpu_active_cores, cpu_hw_cores, cpu_hw_threads | gauge | none | Core and thread counts (same as the panfs_node labels). | C | |
| magent_local_hw_mem | gauge (MiB) | none | Physical memory in MiB (15690 on 16 GiB nodes, 2741 on the small storage VMs). | magent_local_hw_mem * 1048576 | C |
| magent_local_hw_status | gauge | none | Overall hardware status code (1 observed = OK). | magent_local_hw_status != 1 | I |
| magent_local_hw_{internal_temp, external_temp, incoming_air_temp} | gauge (Celsius) | none | Enclosure temperatures. All 0 on cloud VMs (no sensors); meaningful on physical enclosures. | C | |
| magent_local_hw_{high_temp_alert, trip_temp_alert, ff_trip_temp_alert, temperature_shutdown, drives_over_temp} | gauge (0/1 or count) | none | Thermal alarm flags: High temperature warning, trip point reached, fan failure trip, thermal shutdown state, number of drives over temperature. | max(magent_local_hw_trip_temp_alert) | I |
| magent_local_hw_nic{1..4}_{desc, name, type, status, firmware, nic_warnings} | info/gauge | value (for desc, name, firmware) | Per NIC slot inventory: The desc label is a descriptor string such as “10 GE Network Interface 1<physical,active,…>”, name is the Linux device (eth0, bond0..2), type is a code (6 = physical 10 GE observed, 0 = none), status 0 = OK, nic_warnings = count of link warnings. |
magent_local_hw_nic1_status != 0 | I |
3.5 Management agent: Node services (hagent, monitor, mcproxy, pm, sys, shelf)
Agent level counters describing the node’s management daemons. Emitted by all nodes unless noted.
| Metric/family | Type | Extra labels | Meaning | Example query | Conf |
|---|---|---|---|---|---|
| magent_local_sw_hagent_{version, hwserial, slot_id, shelf_master, thermal_state, check_realm_status, reload_config, local_asset_string, shelf_asset_string} | info/gauge | value (version, hwserial) | Hardware agent (pan_mgr_hw_agent): Software version string (2026.35.0-pioneer_robot.1898425), hardware serial, slot in shelf, whether this node is the shelf master (0/1), thermal state code, and counters of check_realm_status and reload_config invocations. | count by (value) (magent_local_sw_hagent_version) | I |
| magent_local_sw_monitor_{runlevel, runevent, checkmodule, startmodule, stopmodule, restartmodule, corefile, ledstate, property, setproperty, clearproperty, brick_property, brick_property_set, brick_property_clr} | info/gauge | value | Process monitor (pan_monitor2) state: Runlevel string (normal), last run event (fsck-complete, panfs-mounted), last module checked/started/stopped/restarted (module names such as iopathv4, pan_config, init-pan-api), last core file recorded (“0 <epoch> <size> <process>”), LED state, and brick property bit fields. The last-* values are strings so they only tell you the most recent action, not counts. | magent_local_sw_monitor_corefile | I |
| magent_local_sw_mcproxy_{primary,secondary}_lagg_config_{mode, link_count, flags}, magent_local_sw_mcproxy_nic_warning_disabled | gauge | none | Multicast/management proxy view of link aggregation: LAGG mode code (3 = LACP observed on primary), number of links in the aggregate (2 primary, 0 secondary), flags, and whether NIC warnings are suppressed. | I | |
| magent_local_sw_pm_{cpty_size, cpty_used, cpty_util, disk_util, proc_util, pm_status, text_info} | gauge | none | Performance monitor summary: Local disk capacity size/used in bytes (the OS/system disk, about 37 GB here), capacity and disk utilization percent, processor utilization percent, overall status code, and a text info string. | magent_local_sw_pm_cpty_used/magent_local_sw_pm_cpty_size | I |
| magent_local_sys_devinfo_{boot_epoch, version}, magent_local_sys_keys_{mkey, wkey}, magent_local_sys_scripts_do_script, magent_shelf_sn, magent_shelf_version | info | none | Miscellaneous device info (boot time, version), encryption key status (directors only: Master key and wrapping key present = 1), script runner, and shelf serial/version. Values are all 1 in this realm. | I |
3.6 Directors: File Manager operations (magent_local_sw_fm_*)
The File Manager (FM) is the PanFS metadata service running on directors; these counters are director only (3 series each). Every operation name appears three times with the suffixes op_count, op_lifetime and op_occurred. Observed behaviour: op_count grows slowly and never resets over 6 hours (values such as 66 for obj_get_r_capability), so it is a cumulative count since the FM started; op_lifetime is the accumulated time spent in that operation (microseconds, inferred); op_occurred is 0/1, apparently whether the operation happened during the last collection interval. The fm_mgr_* set is the management facing operations (volume and directory lifecycle) and the plain fm_* set is the client facing metadata protocol.
| Metric/family | Type | Extra labels | Meaning | Example query | Conf |
|---|---|---|---|---|---|
| magent_local_sw_fm_op_count_<op> | counter | none | Client metadata operations: lookup, readdir, rename, dir_create, dir_delete, dir_check, dir_compact, file_create, file_create_special, file_delete, obj_setattr, obj_link, obj_start_write, obj_end_write, obj_get_r_capability (read capability grant), obj_report_io_error, obj_report_parity, ogroup_info_get, renew_callback, cancel_callback, cancel_callback_multi, cancel_all_callbacks (client cache callbacks/leases), userquota_get, id_identity_translate, id_identity_get_name. | sum by (host) (rate(magent_local_sw_fm_op_count_lookup[5m])) | I |
| magent_local_sw_fm_op_lifetime_<op> | counter (usec) | none | Accumulated service time for the operation. lifetime/count gives mean latency. | rate(magent_local_sw_fm_op_lifetime_lookup[5m])/rate(magent_local_sw_fm_op_count_lookup[5m]) | I |
| magent_local_sw_fm_op_occurred_<op> | gauge (0/1) | none | Whether the operation occurred in the last interval. | I | |
| magent_local_sw_fm_mgr_op_{count,lifetime,occurred}_<op> | counter/gauge | none | Management operations on the FM: config_get, debug, dir_create_realm_root, dir_create_volume_root, dir_delete_volume_root, dir_batch_sync_all, dir_flush_for_fsrc, dir_update_ent_map_hint, volume_flush_dirs, notify_new_volume_info, notify_ogroup_deleted, obj_attrs_updated, obj_recall_caps, obj_update_map_hint, obj_update_parent_info, set_defer_delete_enable, set_defer_dir_write_enable. | magent_local_sw_fm_mgr_op_count_dir_create_volume_root | I |
| magent_local_sw_fm_global_op_{count,lifetime,occurred} | counter/gauge | none | Totals across all FM operations (442 on the busiest director). | rate(magent_local_sw_fm_global_op_count[5m]) | I |
3.7 Directors: Metadata caches (magent_local_sw_gen_cache_*)
The generic cache subsystem on directors. Two views: Per cache statistics (cache_<name>_*) and per memory partition statistics (partition_<name>_*). Caches observed: FM_dir_cache (directory entries), FM_object_attribute_cache, FM_ogroup_cache (object groups), SM_object_cache (Storage Manager objects) and pan_id_user_info_cache (identity mapping). Partitions: fm_dir, fm_attr, fm_ogroup, sm_obj and default. Director only except the default partition, which every node reports.
| Metric/family | Type | Extra labels | Meaning | Example query | Conf |
|---|---|---|---|---|---|
| magent_local_sw_gen_cache_cache_<cache>_{lookup, get, miss}_{count, occurred} | counter/gauge | none | Lookups, successful gets and misses per cache. Hit ratio = 1 - miss/lookup. | 1 - rate(magent_local_sw_gen_cache_cache_FM_dir_cache_miss_count[5m])/rate(magent_local_sw_gen_cache_cache_FM_dir_cache_lookup_count[5m]) | I |
| magent_local_sw_gen_cache_cache_<cache>_{load_io, reload_io, insert_io, store_io}_{count, lifetime, occurred} | counter/gauge | none | Backing store I/O performed on behalf of the cache: Load (fill on miss), reload (refresh), insert (new entry written), store (dirty entry written back). lifetime is accumulated microseconds. | rate(magent_local_sw_gen_cache_cache_FM_object_attribute_cache_load_io_lifetime[5m]) | I |
| magent_local_sw_gen_cache_partition_<part>_{bytes_cur, bytes_hiwat, bytes_limit, ents_cur, ents_hiwat, evicted_ent_count, eviction_event_count, eviction_event_occurred} | gauge/counter | none | Memory partition sizing: Current bytes, high water mark, configured limit (fm_attr about 1.67 GB and sm_obj about 1.38 GB on 16 GiB directors), current and peak entry counts, and eviction counters. bytes_cur/bytes_limit is cache pressure. | magent_local_sw_gen_cache_partition_fm_attr_bytes_cur/magent_local_sw_gen_cache_partition_fm_attr_bytes_limit | I |
3.8 Storage nodes: OSD I/O path (magent_local_sw_iopath_*)
iopathv4 is the OSD (object storage device) service on storage nodes; these metrics are storage node only (12 series each). This is the largest family (about 900 names) because several statistics are arrays flattened with a numeric suffix. Two index spaces exist: _0.._37 (38 slots) on counts/durations/timings, which is indexed by OSD command type, and _0.._34 (35 slots) on measure_client_times_*, which is a per client table whose IP is given by measure_client_times_ip_addr_N. The command index to name mapping is not exposed in the data; the heat_* names (append, clear, clone_object, clone_object_group, close, create_object, create_object_group, get_attribute, list, list_marked_objects, move, open, read, remove_object, set_attribute, write, write_ret_xor, xor_write) are the likely command set. Values in the “measure” group are per interval (they go back to zero and stay small), while counts_N and timings_N look like per interval too (max 0 on an idle realm). Treat all of these as interval gauges rather than counters until confirmed.
| Metric/family | Type | Extra labels | Meaning | Example query | Conf |
|---|---|---|---|---|---|
| magent_local_sw_iopath_counts_N, success_counts_N | gauge (count per interval) | none | Number of commands of type N received/completed successfully in the interval. | sum(magent_local_sw_iopath_counts_0) | I |
| magent_local_sw_iopath_duration{all, active, queue, wire, osdfs}_N | gauge (usec) | none | Latency breakdown per command type: Total (all), time actively processing (active), time queued before processing (queue), network transfer time (wire), and time in the OSD filesystem (osdfs). Values observed are microseconds (for example all=1044, osdfs=616, queue=106, wire=24). The metric names have no underscore between “duration” and the suffix (for example magent_local_sw_iopath_durationall_0). | magent_local_sw_iopath_durationall_0 - magent_local_sw_iopath_durationqueue_0 | I |
| magent_local_sw_iopath_timings_{lifetime, active_time, inactive_time, wire_time, diskreads}_N | gauge (usec/count) | none | Second timing table with the same command index: Lifetime of the command, active vs inactive time, wire time, and number of disk reads it caused. | I | |
| magent_local_sw_iopath_measure_{bytes_in, bytes_out, cur_bytes_transferred, io_occured, io_duration_usec, rd_duration_usec, wr_duration_usec, rw_duration_usec, ra_duration_usec, read_size, write_size, op_count_write, op_count_non_write, sol_data_write, sol_data_non_write, task_queue_depth, lowpri_count} | gauge (per interval) | none | Whole OSD throughput and latency measurements per interval: Bytes in/out, number of I/Os, total and per class (read, write, read+write, readahead) durations in microseconds, last read/write sizes, write vs non write op counts, solicited data counts, current task queue depth and low priority I/O count. Realm Overview sums these with last_over_time() for the DirectFlow panels. To convert to rates divide by the collection interval (about 20 s) rather than using rate(). | sum(magent_local_sw_iopath_measure_bytes_in)/20 | I |
| magent_local_sw_iopath_measure_df_{bytes_in, bytes_out, io_occured, io_duration_usec} | gauge (per interval) | none | Same measurements restricted to DirectFlow client traffic (df). In the test realm they equal the totals, because all traffic is DirectFlow. | sum(last_over_time(magent_local_sw_iopath_measure_df_bytes_out[$__rate_interval])) | C |
| magent_local_sw_iopath_measure_client_times_{ip_addr, read_active_times, read_wire_times, write_active_times, write_wire_times}_N | info/gauge (usec) | value (ip_addr) | Per client latency table (35 slots). ip_addr_N gives the client IP in the value label; the four timing metrics give read/write active and wire time for that client. Join on the suffix to attribute latency to a client. | magent_local_sw_iopath_measure_client_times_ip_addr_0 | I |
| magent_local_sw_iopath_measure_{capacity_bytes, remaining_capacity_bytes, bucket0_capacity_bytes, bucket0_remaining_capacity_bytes, bucket1_capacity_bytes, bucket1_remaining_capacity_bytes} | gauge (bytes) | none | OSD raw capacity and free space (224.7 GB per storage VM here) and the same split across two capacity buckets (bucket0/bucket1, presumably tiers or SSD/HDD pools). These are the right metrics for realm free space: sum(remaining_capacity_bytes). | 1 - sum(magent_local_sw_iopath_measure_remaining_capacity_bytes)/sum(magent_local_sw_iopath_measure_capacity_bytes) | I |
| magent_local_sw_iopath_measure_{num_failed_allocations, num_timeout_cmds, num_timeout_sessions} | counter | none | Error counters: Failed space allocations, commands that timed out, and client sessions that timed out (about 50 per node in the test realm). | increase(magent_local_sw_iopath_measure_num_timeout_cmds[1h]) | I |
| magent_local_sw_iopath_measure_{group_heat, top_n_heat_density, lowest_n_heat_density}, magent_local_sw_iopath_heat_* | gauge | none | Heat (access frequency) tracking used for data placement and tiering: heat_<command>_a/_b are the per command heat weights (constant 10, set_attribute_heat 0), heat_bucket_constant (100000) is the bucket scale, and the measure_* values are the current group heat and the density of the hottest and coldest object groups. | I | |
| magent_local_sw_iopath_measure_writev_{from_tm, from_non_tm, size, from_tm_size} | gauge | none | Vectored write statistics split by whether the write came through the transaction manager (tm). | I | |
| magent_local_sw_iopath_osdfs_{op_count, op_cache_hit, onode_cache_count, onode_cache_hit, bytes_per_sync, access_count_*, hit_count_*, repeat_hit_count_*} | counter | none | OSD filesystem (OSDFS) internal cache statistics by object component: Attr, cg_hdr (capacity group header), data, oid, onode. access/hit/repeat_hit per component, onode cache size and hits, and bytes written per sync. All zero on the idle test realm. | I | |
| magent_local_sw_iopath_fs{0,1,2}_{purpose, data_capacity, metadata_capacity, free_capacity, total_inodes}, magent_local_sw_iopath_hd{0..14}_fs | info/gauge | value (purpose) | The OSD’s local filesystems (up to 3) and the drive to filesystem map (15 drive slots). purpose carries a UUID identifying the filesystem, capacities are bytes, hdN_fs is the index of the filesystem the drive belongs to (4294967295 = slot empty). Realm Overview sums fsN_metadata_capacity as “Metadata Used” and fs0_free_capacity as “Total Capacity (Free)”. | sum(magent_local_sw_iopath_fs0_free_capacity) | C |
| magent_local_sw_iopath_iscsi_{num_cur_conns, config_rcv_buf_size, config_snd_buf_size} | gauge | none | iSCSI transport for the OSD protocol: Current connections and socket buffer sizes (128 KiB receive, 512 KiB send). | sum(magent_local_sw_iopath_iscsi_num_cur_conns) | C |
| magent_local_sw_iopath_{info_iopath_state, info_df_client_array, bad_object_info, dump_object_list} | info/gauge | none | State of the iopath service, the DirectFlow client array, bad object records and the object list dump trigger. Values 1/0 in this realm. | I |