.. _monitoring: =================== Monitoring overview =================== This document explains the Ceph monitoring stack and a number of important Ceph metrics. Ceph admins can explore the rich observability stack deployed by Ceph, and can leverage Prometheus, Alertmanager, Grafana, and scripting to create customized monitoring tools. Ceph Monitoring stack ===================== Ceph deploys an integrated monitoring stack as described in the :ref:`Monitoring Services ` section of the ``cephadm`` documentation. Deployments with external fleetwide monitoring and observability systems using these or other tools may choose to disable the stack that Ceph deploys by default. Ceph metrics ============ Many Ceph metrics are gathered from the performance counters exposed by each Ceph daemon. These :doc:`../dev/perf_counters` are native Ceph metrics. Performance counters are rendered into standard Prometheus metrics by the ``ceph_exporter`` daemon. This daemon runs on every Ceph cluster host and exposes an endpoint where performance counters exposed by Ceph daemons running on that host are presented in the form of Prometheus metrics. In addition to the ``ceph_exporter`` the Ceph Manager ``prometheus`` module exposes metrics relating to the Ceph cluster as a whole. Ceph provides a Prometheus endpoint from which one can obtain the complete list of available metrics, or against which admins, Grafana, and Alertmanager can exeute queries. Prometheus (and related systems) accept data queries formatted as PromQL expressions. Expansive documentation of PromQL can be viewed [here](https://prometheus.io/docs/prometheus/latest/querying/basics/) and several excellent books can be found at the usual sources of digital and print books. We will explore a number of PromQL queries below. Use the following command to obtain the Prometheus endpoint for your cluster: Example: .. code-block:: bash # ceph orch ps --service_name prometheus NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID prometheus.cephtest-node-00 cephtest-node-00.cephlab.com *:9095 running (103m) 50s ago 5w 142M - 2.33.4 514e6a882f6e efe3cbc2e521 With this information you can connect to ``http://cephtest-node-00.cephlab.com:9095`` to access the Prometheus server interface, which includes a list of targets, an expression browser, and metrics related to the Prometheus service itself. The complete list of metrics (with descriptions) is available at the URL of the below form: in: ``http://cephtest-node-00.cephlab.com:9095/api/v1/targets/metadata`` The Ceph Dashboard provides a rich set of graphs and other panels that display the most important cluster and service metrics. Many of the examples in this document are taken from Dashboard graphics or extrapolated from metrics exposed by the Ceph Dashboard. Ceph daemon health metrics ========================== The ``ceph_exporter`` provides a metric named ``ceph_daemon_socket_up`` that indicates the health status of a Ceph daemon based on its ability to respond via the admin socket, where a value of ``1`` means healthy, and ``0`` means unhealthy. Although a Ceph daemon might still be "alive" when it reports ``ceph_daemon_socket_up=0``, this status indicates a significant issue in its functionality. As such, this metric serves as an excellent means of detecting problems in any of the main Ceph daemons. The ``ceph_daemon_socket_up`` Prometheus metrics also have labels as described below: * ``ceph_daemon``: Identifier of the Ceph daemon exposing an admin socket on the host. * ``hostname``: Name of the host where the Ceph daemon is running. Example: .. code-block:: bash ceph_daemon_socket_up{ceph_daemon="mds.a",hostname="testhost"} 1 ceph_daemon_socket_up{ceph_daemon="osd.1",hostname="testhost"} 0 To identify any Ceph daemons that were not responsive at any point in the last 12 hours, you can use the following PromQL expression: .. code-block:: bash ceph_daemon_socket_up == 0 or min_over_time(ceph_daemon_socket_up[12h]) == 0 Performance metrics =================== Below we explore a a number of metrics that indicate Ceph cluster performance. All of these metrics have the following labels: * ``ceph_daemon``: Identifier of the Ceph daemon from which the metric was harvested * ``instance``: The IP address of the exporter instance exposing the metric. * ``job``: Prometheus scrape job name Below is an example Prometheus query result showing these labels: .. code-block:: bash ceph_osd_op_r{ceph_daemon="osd.0", instance="192.168.122.7:9283", job="ceph"} = 73981 *Cluster throughput:* Query ``ceph_osd_op_r_out_bytes`` and ``ceph_osd_op_w_in_bytes`` to obtain cluster client throughput: Example: .. code-block:: bash # Writes (B/s): sum(irate(ceph_osd_op_w_in_bytes[1m])) # Reads (B/s): sum(irate(ceph_osd_op_r_out_bytes[1m])) *Cluster I/O (operations):* Query ``ceph_osd_op_r``, ``ceph_osd_op_w`` to obtain the rates of client operations (IOPS): Example: .. code-block:: bash # Writes (ops/s): sum(irate(ceph_osd_op_w[1m])) # Reads (ops/s): sum(irate(ceph_osd_op_r[1m])) *Latency:* Query ``ceph_osd_op_latency_sum`` to measure the delay before OSD transfers of data begins in respose to client requests: Example: .. code-block:: bash sum(irate(ceph_osd_op_latency_sum[1m])) OSD performance =============== The cluster performance metrics described above are gathered from OSD metrics. By specifying an appropriate label value or regular expression we can retrieve performance metrics for one or a subset of the cluster's OSDs: Examples: .. code-block:: bash # OSD 0 read latency irate(ceph_osd_op_r_latency_sum{ceph_daemon=~"osd.0"}[1m]) / on (ceph_daemon) irate(ceph_osd_op_r_latency_count[1m]) # OSD 0 write IOPS irate(ceph_osd_op_w{ceph_daemon=~"osd.0"}[1m]) # OSD 0 write thughtput (bytes) irate(ceph_osd_op_w_in_bytes{ceph_daemon=~"osd.0"}[1m]) # OSD.0 total raw capacity available ceph_osd_stat_bytes{ceph_daemon="osd.0", instance="cephtest-node-00.cephlab.com:9283", job="ceph"} = 536451481 Physical storage drive performance: =================================== By combining Prometheus ``node_exporter`` metrics with Ceph cluster metrics we can derive performance information for physical storage media backing Ceph OSDs. Example: .. code-block:: bash # Read latency of device used by osd.0 label_replace(irate(node_disk_read_time_seconds_total[1m]) / irate(node_disk_reads_completed_total[1m]), "instance", "$1", "instance", "([^:.]*).*") and on (instance, device) label_replace(label_replace(ceph_disk_occupation_human{ceph_daemon=~"osd.0"}, "device", "$1", "device", "/dev/(.*)"), "instance", "$1", "instance", "([^:.]*).*") # Write latency of device used by osd.0 label_replace(irate(node_disk_write_time_seconds_total[1m]) / irate(node_disk_writes_completed_total[1m]), "instance", "$1", "instance", "([^:.]*).*") and on (instance, device) label_replace(label_replace(ceph_disk_occupation_human{ceph_daemon=~"osd.0"}, "device", "$1", "device", "/dev/(.*)"), "instance", "$1", "instance", "([^:.]*).*") # IOPS of device used by osd.0 # reads: label_replace(irate(node_disk_reads_completed_total[1m]), "instance", "$1", "instance", "([^:.]*).*") and on (instance, device) label_replace(label_replace(ceph_disk_occupation_human{ceph_daemon=~"osd.0"}, "device", "$1", "device", "/dev/(.*)"), "instance", "$1", "instance", "([^:.]*).*") # writes: label_replace(irate(node_disk_writes_completed_total[1m]), "instance", "$1", "instance", "([^:.]*).*") and on (instance, device) label_replace(label_replace(ceph_disk_occupation_human{ceph_daemon=~"osd.0"}, "device", "$1", "device", "/dev/(.*)"), "instance", "$1", "instance", "([^:.]*).*") # Throughput for device used by osd.0 # reads: label_replace(irate(node_disk_read_bytes_total[1m]), "instance", "$1", "instance", "([^:.]*).*") and on (instance, device) label_replace(label_replace(ceph_disk_occupation_human{ceph_daemon=~"osd.0"}, "device", "$1", "device", "/dev/(.*)"), "instance", "$1", "instance", "([^:.]*).*") # writes: label_replace(irate(node_disk_written_bytes_total[1m]), "instance", "$1", "instance", "([^:.]*).*") and on (instance, device) label_replace(label_replace(ceph_disk_occupation_human{ceph_daemon=~"osd.0"}, "device", "$1", "device", "/dev/(.*)"), "instance", "$1", "instance", "([^:.]*).*") # Physical drive utilization (%) for osd.0 in the last 5 minutes. Note that this value has limited mean for SSDs label_replace(irate(node_disk_io_time_seconds_total[5m]), "instance", "$1", "instance", "([^:.]*).*") and on (instance, device) label_replace(label_replace(ceph_disk_occupation_human{ceph_daemon=~"osd.0"}, "device", "$1", "device", "/dev/(.*)"), "instance", "$1", "instance", "([^:.]*).*") Pool metrics ============ Ceph pool metrics have the following labels: * ``instance``: The IP address of the exporter providing the metric * ``pool_id``: Numeric identifier of the Ceph pool * ``job``: Prometheus scrape job name Pool-specific metrics include: * ``ceph_pool_metadata``: Information about the pool that can be used together with other metrics to provide more information in query resultss and graphs. In addition to the above three common labels this metric provides the following: * ``compression_mode``: Compression type enabled for the pool. Values are ``lz4``, ``snappy``, ``zlib``, ``zstd``, and ``none`). Example: ``compression_mode="none"`` * ``description``: Brief description of the pool data protection strategy including replica number or EC profile. Example: ``description="replica:3"`` * ``name``: Name of the pool. Example: ``name=".mgr"`` * ``type``: Data protection strategy, replicated or EC. ``Example: type="replicated"`` * ``ceph_pool_bytes_used``: Total raw capacity (after replication or EC) consumed by user data and metadata * ``ceph_pool_stored``: Total client data stored in the pool (before data protection) * ``ceph_pool_compress_under_bytes``: Data eligible to be compressed in the pool * ``ceph_pool_compress_bytes_used``: Data compressed in the pool * ``ceph_pool_rd``: Client read operations per pool (reads per second) * ``ceph_pool_rd_bytes``: Client read operations in bytes per pool * ``ceph_pool_wr``: Client write operations per pool (writes per second) * ``ceph_pool_wr_bytes``: Client write operation in bytes per pool **Useful queries**: .. code-block:: bash # Total raw capacity available in the cluster: sum(ceph_osd_stat_bytes) # Total raw capacity consumed in the cluster (including metadata + redundancy): sum(ceph_pool_bytes_used) # Total client data stored in the cluster: sum(ceph_pool_stored) # Compression savings: sum(ceph_pool_compress_under_bytes - ceph_pool_compress_bytes_used) # Client IOPS for a specific pool reads: irate(ceph_pool_rd[1m]) * on(pool_id) group_left(instance,name) ceph_pool_metadata{name=~"testrbdpool"} writes: irate(ceph_pool_wr[1m]) * on(pool_id) group_left(instance,name) ceph_pool_metadata{name=~"testrbdpool"} # Client throughput for a specific pool reads: irate(ceph_pool_rd_bytes[1m]) * on(pool_id) group_left(instance,name) ceph_pool_metadata{name=~"testrbdpool"} writes: irate(ceph_pool_wr_bytes[1m]) * on(pool_id) group_left(instance,name) ceph_pool_metadata{name=~"testrbdpool"} RGW metrics ================== These metrics have the following labels: * ``instance``: The IP address of the exporter providing the metric * ``instance_id``: Identifier of the RGW daemon instance * ``job``: Orometheus scrape job name Example: .. code-block:: bash ceph_rgw_req{instance="192.168.122.7:9283", instance_id="154247", job="ceph"} = 12345 Generic metrics --------------- * ``ceph_rgw_metadata``: Provides generic information about an RGW daemon. This can be used together with other metrics to provide contextual information in queries and graphs. In addtion to the three common labels, this metric provides the following: * ``ceph_daemon``: Name of the RGW daemon instance. Example: ``ceph_daemon="rgw.rgwtest.cephtest-node-00.sxizyq"`` * ``ceph_version``: Version of the RGW daemon. Example: ``ceph_version="ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)"`` * ``hostname``: Name of the host where the daemon runs. Example: ``hostname:"cephtest-node-00.cephlab.com"`` * ``ceph_rgw_req``: Number of requests processed by the daemon (``GET``+``PUT``+``DELETE``). Useful for detecting bottlenecks and optimizing load distribution. * ``ceph_rgw_qlen``: Operations queue length for the daemon. Useful for detecting bottlenecks and optimizing load distribution. * ``ceph_rgw_failed_req``: Aborted requests. Useful for detecting daemon errors. GET operation metrics --------------------- * ``ceph_rgw_op_global_get_obj_lat_count``: Number of ``GET`` requests * ``ceph_rgw_op_global_get_obj_lat_sum``: Total latency for ``GET`` requests * ``ceph_rgw_op_global_get_obj_ops``: Total number of ``GET`` requests * ``ceph_rgw_op_global_get_obj_bytes``: Total bytes transferred for ``GET`` requests PUT operation metrics ------------------------------- * ``ceph_rgw_op_global_put_obj_lat_count``: Number of get operations * ``ceph_rgw_op_global_put_obj_lat_sum``: Total latency time for ``PUT`` operations * ``ceph_rgw_op_global_put_obj_ops``: Total number of ``PUT`` operations * ``ceph_rgw_op_global_get_obj_bytes``: Total bytes transferred in ``PUT`` operations Additional Useful queries ------------------------- .. code-block:: bash # Average GET latency rate(ceph_rgw_op_global_get_obj_lat_sum[30s]) / rate(ceph_rgw_op_global_get_obj_lat_count[30s]) * on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata # Average PUT latency rate(ceph_rgw_op_global_put_obj_lat_sum[30s]) / rate(ceph_rgw_op_global_put_obj_lat_count[30s]) * on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata # Requests per second rate(ceph_rgw_req[30s]) * on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata # Total number of "other" operations (``LIST``, ``DELETE``, etc) rate(ceph_rgw_req[30s]) - (rate(ceph_rgw_op_global_get_obj_ops[30s]) + rate(ceph_rgw_op_global_put_obj_ops[30s])) # GET latency per RGW instance rate(ceph_rgw_op_global_get_obj_lat_sum[30s]) / rate(ceph_rgw_op_global_get_obj_lat_count[30s]) * on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata # PUT latency per RGW instance rate(ceph_rgw_op_global_put_obj_lat_sum[30s]) / rate(ceph_rgw_op_global_put_obj_lat_count[30s]) * on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata # Bandwidth consumed by GET operations sum(rate(ceph_rgw_op_global_get_obj_bytes[30s])) # Bandwidth consumed by PUT operations sum(rate(ceph_rgw_op_global_put_obj_bytes[30s])) # Bandwidth consumed by RGW instance (PUTs + GETs) sum by (instance_id) (rate(ceph_rgw_op_global_get_obj_bytes[30s]) + rate(ceph_rgw_op_global_put_obj_bytes[30s])) * on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata # HTTP errors and other request failures rate(ceph_rgw_failed_req[30s]) CephFS Metrics ============== These metrics have the following labels: * ``ceph_daemon``: The name of the MDS daemon * ``instance``: The IP address and port of the exporter exposing the metric * ``job``: Prometheus scrape job name Example: .. code-block:: bash ceph_mds_request{ceph_daemon="mds.test.cephtest-node-00.hmhsoh", instance="192.168.122.7:9283", job="ceph"} = 1452 Important metrics ----------------- * ``ceph_mds_metadata``: Provides general information about the MDS daemon. It can be used together with other metrics to provide contextual information in queries and graphs. The following extra labels are populated: * ``ceph_version``: MDS daemon version * ``fs_id``: CephFS filesystem ID * ``hostname``: Name of the host where the MDS daemon runs * ``public_addr``: Public address of the host where the MDS daemon runs * ``rank``: Rank of the MDS daemon Example: .. code-block:: bash ceph_mds_metadata{ceph_daemon="mds.test.cephtest-node-00.hmhsoh", ceph_version="ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)", fs_id="-1", hostname="cephtest-node-00.cephlab.com", instance="cephtest-node-00.cephlab.com:9283", job="ceph", public_addr="192.168.122.145:6801/118896446", rank="-1"} * ``ceph_mds_request``: Total number of requests for the MDS * ``ceph_mds_reply_latency_sum``: Reply latency total * ``ceph_mds_reply_latency_count``: Reply latency count * ``ceph_mds_server_handle_client_request``: Number of client requests * ``ceph_mds_sessions_session_count``: Session count * ``ceph_mds_sessions_total_load``: Total load * ``ceph_mds_sessions_sessions_open``: Sessions currently open * ``ceph_mds_sessions_sessions_stale``: Sessions currently stale * ``ceph_objecter_op_r``: Number of read operations * ``ceph_objecter_op_w``: Number of write operations * ``ceph_mds_root_rbytes``: Total number of bytes managed by the daemon * ``ceph_mds_root_rfiles``: Total number of files managed by the daemon Useful queries: --------------- .. code-block:: bash # Total MDS read workload: sum(rate(ceph_objecter_op_r[1m])) # Total MDS daemons workload: sum(rate(ceph_objecter_op_w[1m])) # Read workload for a specific MDS sum(rate(ceph_objecter_op_r{ceph_daemon=~"mdstest"}[1m])) # Write workload for a specific MDS sum(rate(ceph_objecter_op_r{ceph_daemon=~"mdstest"}[1m])) # Average reply latency rate(ceph_mds_reply_latency_sum[30s]) / rate(ceph_mds_reply_latency_count[30s]) # Total requests per second rate(ceph_mds_request[30s]) * on (instance) group_right (ceph_daemon) ceph_mds_metadata Block metrics ============= By default RBD metrics for images are not gathered, as their cardinality may be high. This helps ensure the performance of the Manager's ``prometheus`` module. To produce metrics for RBD images, configure the Manager option ``mgr/prometheus/rbd_stats_pools``. For more information see :ref:`prometheus-rbd-io-statistics` These metrics have the following labels: * ``image``: Name of the image (volume) * ``instance``: Node where the exporter runs * ``job``: Name of the Prometheus scrape job * ``pool``: RBD pool name Example: .. code-block:: bash ceph_rbd_read_bytes{image="test2", instance="cephtest-node-00.cephlab.com:9283", job="ceph", pool="testrbdpool"} Important metrics ------------------ * ``ceph_rbd_read_bytes``: RBD bytes read * ``ceph_rbd_write_bytes``: RBD image bytes written * ``ceph_rbd_read_latency_count``: RBD read operation latency count * ``ceph_rbd_read_latency_sum``: RBD read operation latency total time * ``ceph_rbd_read_ops``: RBD read operation count * ``ceph_rbd_write_ops``: RBD write operation count * ``ceph_rbd_write_latency_count``: RBD write operation latency count * ``ceph_rbd_write_latency_sum``: RBD write operation latency total Useful queries -------------- .. code-block:: bash # Average read latency rate(ceph_rbd_read_latency_sum[30s]) / rate(ceph_rbd_read_latency_count[30s]) * on (instance) group_left (ceph_daemon) ceph_rgw_metadata Hardware monitoring =================== See :ref:`hardware-monitoring`