doc/mgr/dashboard.rst
.. _mgr-dashboard:
The Ceph Dashboard is a web-based Ceph management-and-monitoring tool that can
be used to inspect and administer resources in the cluster. It is implemented
as a :ref:ceph-manager-daemon module.
The original Ceph Dashboard shipped with Ceph Luminous and was a simple read-only view into the run-time information and performance data of Ceph clusters. It had a simple architecture. However, demand grew for richer, web-based management capabilities for users who prefer a WebUI over the CLI.
The :term:Ceph Dashboard module adds web-based monitoring and administration
to the Ceph Manager. The architecture and functionality of this new module are
derived from the openATTIC Ceph management and monitoring tool <https://openattic.org/>. Development was originally driven by the openATTIC
team at SUSE <https://www.suse.com/>, with support from members of the Ceph
community and from companies including Red Hat <https://redhat.com/>_.
The dashboard module's backend code uses the CherryPy framework, and implements
a custom REST API. The WebUI implementation is based on Angular/TypeScript and
includes both functionality from the original dashboard and new features
originally developed for the standalone version of openATTIC. The Ceph
Dashboard module is implemented as an application that provides a graphical
representation of information and statistics through a web server hosted by
ceph-mgr.
Feature Overview ^^^^^^^^^^^^^^^^
The dashboard provides the following features:
dashboard-user-role-management for details.dashboard-ssl-tls-support for details.PUT, POST
and DELETE API requests in the Ceph audit log. See :ref:dashboard-auditing
for instructions on how to enable this feature.The Ceph Dashboard offers the following monitoring and management capabilities:
Grafana_ dashboards may be embedded in external applications and web pages
to surface information and performance metrics gathered by
the :ref:mgr-prometheus module. See
:ref:dashboard-grafana for details on how to configure this functionality.ceph-iscsi). Display the
iSCSI gateway status and info about active initiators.
See :ref:dashboard-iscsi-management for instructions on how to configure
this feature.dashboard-enabling-object-gateway for configuration instructions.dashboard-nfs-ganesha-management for details on how to
enable this functionality.Overview of the Dashboard Landing Page ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The landing page of Ceph Dashboard serves as the home page and features metrics such as the overall cluster status, performance, and capacity. It provides real-time updates on any changes in the cluster and allows quick access to other sections of the dashboard.
.. image:: dashboard-landing-page.png
Details """"""" Provides an overview of the cluster configuration, displaying various critical aspects of the cluster.
.. image:: details-card.png
.. _dashboard-landing-page-status:
Status """""" Provides a visual indication of cluster health, and displays cluster alerts grouped by severity.
.. image:: status-card-open.png
.. _dashboard-landing-page-capacity:
Capacity """"""""
nearfull threshold of the OSDsfull threshold of the OSDs.. image:: capacity-card.png
.. _dashboard-landing-page-inventory:
Inventory """"""""" An inventory for all assets within the cluster. Provides direct access to subpages of the dashboard from each item of this card.
.. image:: inventory-card.png
.. _dashboard-landing-page-performance:
Cluster Utilization """""""""""""""""""
.. image:: cluster-utilization-card.png
Supported Browsers ^^^^^^^^^^^^^^^^^^
Ceph Dashboard is primarily tested and developed using the following web browsers:
+---------------------------------------------------------------+---------------------------------------+
| Browser | Versions |
+===============================================================+=======================================+
| Chrome <https://www.google.com/chrome/>_ and | latest 2 major versions |
| Chromium <https://www.chromium.org/>_ based browsers | |
+---------------------------------------------------------------+---------------------------------------+
| Firefox <https://www.mozilla.org/firefox/>_ | latest 2 major versions |
+---------------------------------------------------------------+---------------------------------------+
| Firefox ESR <https://www.mozilla.org/firefox/enterprise/>_ | latest major version |
+---------------------------------------------------------------+---------------------------------------+
While Ceph Dashboard might work in older browsers, we cannot guarantee compatibility and recommend keeping your browser up to date.
If you have installed ceph-mgr-dashboard from distribution packages, the
package management system should take care of installing all required
dependencies.
If you're building Ceph from source and want to start the dashboard from your
development environment, please see the files README.rst and HACKING.rst
in the source directory src/pybind/mgr/dashboard.
Within a running Ceph cluster, the Ceph Dashboard is enabled with:
.. prompt:: bash $
ceph mgr module enable dashboard
.. _dashboard-ssl-tls-support:
SSL/TLS Support ^^^^^^^^^^^^^^^
All HTTP connections to the dashboard are secured with SSL/TLS by default.
To get the dashboard up and running quickly, you can generate and install a self-signed certificate:
.. prompt:: bash #
ceph dashboard create-self-signed-cert
Note that most web browsers will complain about self-signed certificates and require explicit confirmation before establishing a secure connection to the dashboard.
To properly secure a deployment and to remove the warning, a certificate that is issued by a certificate authority (CA) should be used.
For example, a key pair can be generated with a command similar to:
.. prompt:: bash #
openssl req -new -nodes -x509
-subj "/O=IT/CN=ceph-mgr-dashboard" -days 3650
-keyout dashboard.key -out dashboard.crt -extensions v3_ca
.. note::
Currently, the Ceph Dashboard supports only RSA private keys for SSL/TLS certificates. If you attempt to configure the dashboard with an ECDSA/EC key, the module will fail to start with an error similar to:
MGR_MODULE_ERROR: Module 'dashboard' has failed: key type unsupported
This limitation exists because the verification routine in the Ceph Manager
uses pyOpenSSL, which supports only RSA keys in its
PKey.check() method. Until this restriction is lifted, generate or
request certificates with RSA keys.
The dashboard.crt file should then be signed by a CA. Once that is done, you
can enable it for Ceph manager instances by running the following commands:
.. prompt:: bash #
ceph dashboard set-ssl-certificate -i dashboard.crt ceph dashboard set-ssl-certificate-key -i dashboard.key
If unique certificates are desired for each manager instance,
the name of the instance can be included as follows (where $name is the name
of the ceph-mgr instance, usually the hostname):
.. prompt:: bash #
ceph dashboard set-ssl-certificate $name -i dashboard.crt ceph dashboard set-ssl-certificate-key $name -i dashboard.key
SSL can also be disabled by setting this configuration value:
.. prompt:: bash #
ceph config set mgr mgr/dashboard/ssl false
This might be useful if the dashboard will be running behind a proxy which does
not support SSL for its upstream servers or other situations where SSL is not
wanted or required. See :ref:dashboard-proxy-configuration for more details.
.. warning::
Use caution when disabling SSL as usernames and passwords will be sent to the dashboard unencrypted.
.. note::
You must restart Ceph manager processes after changing the SSL
certificate and key. This can be accomplished by either running ceph mgr fail mgr or by disabling and re-enabling the dashboard module (which also
triggers the manager to respawn itself):
.. prompt:: bash #
ceph mgr module disable dashboard
ceph mgr module enable dashboard
.. _dashboard-host-name-and-port:
Host Name and Port ^^^^^^^^^^^^^^^^^^
Like most web applications, the dashboard binds to a TCP/IP address and TCP port.
By default, the ceph-mgr daemon hosting the dashboard (i.e., the currently
active manager) will bind to TCP port 8443 or 8080 when SSL is disabled.
If no specific address has been configured, the web app will bind to ::,
which corresponds to all available IPv4 and IPv6 addresses.
These defaults can be changed via the configuration key facility on a cluster-wide level (so they apply to all manager instances) as follows:
.. prompt:: bash #
ceph config set mgr mgr/dashboard/server_addr $IP ceph config set mgr mgr/dashboard/server_port $PORT ceph config set mgr mgr/dashboard/ssl_server_port $PORT
Since each ceph-mgr hosts its own instance of the dashboard, it may be
necessary to configure them separately. The IP address and port for a specific
manager instance can be changed with the following commands:
.. prompt:: bash #
ceph config set mgr mgr/dashboard/$name/server_addr $IP ceph config set mgr mgr/dashboard/$name/server_port $PORT ceph config set mgr mgr/dashboard/$name/ssl_server_port $PORT
Replace $name with the ID of the ceph-mgr instance hosting the dashboard.
.. note::
The command ceph mgr services will show you all endpoints that are
currently configured. Look for the dashboard key to obtain the URL for
accessing the dashboard.
Username and Password ^^^^^^^^^^^^^^^^^^^^^
In order to be able to log in, you need to create a user account and associate
it with at least one role. We provide a set of predefined system roles that
you can use. For more details please refer to the User and Role Management_
section.
To create a user with the administrator role you can use the following commands:
.. prompt:: bash #
ceph dashboard ac-user-create <username> -i <file-containing-password> administrator
Account Lock-out ^^^^^^^^^^^^^^^^
It disables a user account if a user repeatedly enters the wrong credentials for multiple times. It is enabled by default to prevent brute-force or dictionary attacks. The user can get or set the default number of lock-out attempts using these commands respectively:
.. prompt:: bash #
ceph dashboard get-account-lockout-attempts ceph dashboard set-account-lockout-attempts value:int
.. warning::
This feature can be disabled by setting the default number of lock-out attempts to 0. However, by disabling this feature, the account is more vulnerable to brute-force or dictionary based attacks. This can be disabled by:
.. prompt:: bash #
ceph dashboard set-account-lockout-attempts 0
Enable a Locked User ^^^^^^^^^^^^^^^^^^^^
If a user account is disabled as a result of multiple invalid login attempts, then it needs to be manually enabled by the administrator. This can be done by the following command:
.. prompt:: bash #
ceph dashboard ac-user-enable <username>
Accessing the Dashboard ^^^^^^^^^^^^^^^^^^^^^^^
You can now access the dashboard using your (JavaScript-enabled) web browser, by
pointing it to any of the host names or IP addresses and the selected TCP port
where a manager instance is running: e.g., http(s)://<$IP>:<$PORT>/.
The dashboard page displays and requests a previously defined username and password.
.. _dashboard-enabling-object-gateway:
Enabling the Object Gateway Management Frontend ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When RGW is deployed with cephadm, the RGW credentials used by the dashboard will be automatically configured. You can also manually force the credentials to be set up with:
.. prompt:: bash #
ceph dashboard set-rgw-credentials
This will create an RGW user with uid dashboard for each realm in
the system.
If you've configured a custom 'admin' resource in your RGW admin API, you should set it here also:
.. prompt:: bash #
ceph dashboard set-rgw-api-admin-resource <admin_resource>
If you are using a self-signed certificate in your Object Gateway setup, you should disable certificate verification in the dashboard to avoid refused connections, e.g. caused by certificates signed by unknown CA or not matching the host name:
.. prompt:: bash #
ceph dashboard set-rgw-api-ssl-verify False
To set a custom hostname or address for an RGW gateway, set the value of RGW_HOSTNAME_PER_DAEMON
accordingly:
.. prompt:: bash #
ceph dashboard set-rgw-hostname <gateway_name> <hostname>
The setting can be unset using:
.. prompt:: bash #
ceph dashboard unset-rgw-hostname <gateway_name>
If the Object Gateway takes too long to process requests and the dashboard runs into timeouts, you can set the timeout value to your needs:
.. prompt:: bash #
ceph dashboard set-rest-requests-timeout <seconds>
The default value is 45 seconds.
.. _dashboard-iscsi-management:
Enabling iSCSI Management ^^^^^^^^^^^^^^^^^^^^^^^^^
The Ceph Dashboard can manage iSCSI targets using the REST API provided by the
rbd-target-api service of the :ref:ceph-iscsi. Please make sure that it is
installed and enabled on the iSCSI gateways.
.. note::
The iSCSI management functionality of Ceph Dashboard depends on the latest
version 3 of the ceph-iscsi <https://github.com/ceph/ceph-iscsi>_ project.
Make sure that your operating system provides the correct version, otherwise
the dashboard will not enable the management features.
If the ceph-iscsi REST API is configured in HTTPS mode and its using a self-signed
certificate, you need to configure the dashboard to avoid SSL certificate
verification when accessing ceph-iscsi API.
To disable API SSL verification run the following command:
.. prompt:: bash #
ceph dashboard set-iscsi-api-ssl-verification false
The available iSCSI gateways must be defined using the following commands:
.. prompt:: bash #
ceph dashboard iscsi-gateway-list
ceph dashboard iscsi-gateway-add -i <file-containing-gateway-url> [<gateway_name>] ceph dashboard iscsi-gateway-rm <gateway_name>
.. _dashboard-grafana:
Enabling the Embedding of Grafana Dashboards ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Grafana_ pulls data from Prometheus <https://prometheus.io/>. Although
Grafana can use other data sources, the Grafana dashboards we provide contain
queries that are specific to Prometheus. Our Grafana dashboards therefore
require Prometheus as the data source. The Ceph :ref:mgr-prometheus
module exports its data in the Prometheus exposition format. These Grafana
dashboards rely on metric names from the Prometheus module and Node exporter <https://prometheus.io/docs/guides/node-exporter/>. The Node exporter is a
separate application that provides machine metrics.
.. note::
Prometheus' security model presumes that untrusted users have access to the Prometheus HTTP endpoint and logs. Untrusted users have access to all the (meta)data Prometheus collects that is contained in the database, plus a variety of operational and debugging information.
However, Prometheus' HTTP API is limited to read-only operations. Configurations can not be changed using the API and secrets are not exposed. Moreover, Prometheus has some built-in measures to mitigate the impact of denial of service attacks.
Please see Prometheus' Security model <https://prometheus.io/docs/operating/security/> for more detailed
information.
Installation and Configuration using cephadm """"""""""""""""""""""""""""""""""""""""""""
Grafana and Prometheus can be installed using :ref:cephadm. They will
automatically be configured by cephadm. Please see
:ref:mgr-cephadm-monitoring documentation for more details on how to use
cephadm for installing and configuring Prometheus and Grafana.
Manual Installation and Configuration """""""""""""""""""""""""""""""""""""
The following process describes how to configure Grafana and Prometheus manually. After you have installed Prometheus, Grafana, and the Node exporter on appropriate hosts, proceed with the following steps.
#. Enable the Ceph Exporter which comes as Ceph Manager module by running:
.. prompt:: bash #
ceph mgr module enable prometheus
More details can be found in the documentation of the :ref:`mgr-prometheus`.
#. Add the corresponding scrape configuration to Prometheus. This may look like::
global:
scrape_interval: 5s
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'ceph'
static_configs:
- targets: ['localhost:9283']
- job_name: 'node-exporter'
static_configs:
- targets: ['localhost:9100']
.. note::
Please note that in the above example, Prometheus is configured
to scrape data from itself (port 9090), the Ceph manager module
`prometheus` (port 9283), which exports Ceph internal data, and the Node
Exporter (port 9100), which provides OS and hardware metrics for each host.
Depending on your configuration, you may need to change the hostname in
or add additional configuration entries for the Node
Exporter. It is unlikely that you will need to change the default TCP ports.
Moreover, you don't *need* to have more than one target for Ceph specific
data, provided by the `prometheus` mgr module. But it is recommended to
configure Prometheus to scrape Ceph specific data from all existing Ceph
managers. This enables a built-in high availability mechanism, so that
services run on a manager host will be restarted automatically on a different
manager host if one Ceph Manager goes down.
#. Add Prometheus as data source to Grafana using the Grafana Web UI <https://grafana.com/docs/grafana/latest/features/datasources/add-a-data-source/>_.
.. IMPORTANT:: The data source must be named "Dashboard1".
#. Install the vonage-status-panel and grafana-piechart-panel plugins using:
.. prompt:: bash #
grafana-cli plugins install vonage-status-panel
grafana-cli plugins install grafana-piechart-panel
#. Add Dashboards to Grafana:
Dashboards can be added to Grafana by importing dashboard JSON files.
Use the following command to download the JSON files:
.. prompt:: bash #
wget https://raw.githubusercontent.com/ceph/ceph/main/monitoring/ceph-mixin/dashboards_out/<Dashboard-name>.json
You can find various dashboard JSON files `here <https://github.com/ceph/ceph/tree/
main/monitoring/ceph-mixin/dashboards_out>`_.
For Example, for ceph-cluster overview you can use:
.. prompt:: bash #
wget https://raw.githubusercontent.com/ceph/ceph/main/monitoring/ceph-mixin/dashboards_out/ceph-cluster.json
You may also author your own dashboards.
#. Configure anonymous mode in /etc/grafana/grafana.ini::
[auth.anonymous]
enabled = true
org_name = Main Org.
org_role = Viewer
In newer versions of Grafana (starting with 6.2.0-beta1) a new setting named
``allow_embedding`` has been introduced. This setting must be explicitly
set to ``true`` for the Grafana integration in Ceph Dashboard to work, as the
default is ``false``.
::
[security]
allow_embedding = true
Enabling RBD-Image monitoring """""""""""""""""""""""""""""
Monitoring of RBD images is disabled by default, as it can significantly impact
performance. For more information please see :ref:prometheus-rbd-io-statistics.
When disabled, the overview and details dashboards will be empty in Grafana and
metrics will not be visible in Prometheus.
Configuring Dashboard """""""""""""""""""""
After you have set up Grafana and Prometheus, you will need to configure the connection information that the Ceph Dashboard will use to access Grafana.
You need to tell the dashboard on which URL the Grafana instance is running/deployed:
.. prompt:: bash #
ceph dashboard set-grafana-api-url <grafana-server-url> # default: ''
The format of url is : <protocol>:<IP-address>:<port>
.. note::
The Ceph Dashboard embeds Grafana dashboards via iframe HTML elements.
If Grafana is configured without SSL/TLS support, most browsers will block the
embedding of insecure content if SSL support is
enabled for the dashboard (which is the default). If you
can't see the embedded Grafana dashboards after enabling them as outlined
above, check your browser's documentation on how to unblock mixed content.
Alternatively, consider enabling SSL/TLS support in Grafana.
If you are using a self-signed certificate for Grafana, disable certificate verification in the dashboard to avoid refused connections, which can be a result of certificates signed by an unknown CA or that do not match the host name:
.. prompt:: bash #
ceph dashboard set-grafana-api-ssl-verify False
You can also access Grafana directly to monitor your cluster.
.. note::
Ceph Dashboard configuration information can also be unset. For example, to clear the Grafana API URL we configured above:
.. prompt:: bash #
ceph dashboard reset-grafana-api-url
Alternative URL for Browsers """"""""""""""""""""""""""""
The Ceph Dashboard backend requires the Grafana URL to be able to verify the existence of Grafana Dashboards before the frontend even loads them. Due to the nature of how Grafana is implemented in Ceph Dashboard, this means that two working connections are required in order to be able to see Grafana graphs in Ceph Dashboard:
Now, it might be the case that your environment makes it difficult for the
user's browser to directly access the URL configured in Ceph Dashboard. To solve
this issue, a separate URL can be configured which will solely be used to tell
the frontend (the user's browser) which URL it should use to access Grafana.
This setting won't ever be changed automatically, unlike the GRAFANA_API_URL
which is set by :ref:cephadm (only if cephadm is used to deploy monitoring
services).
To change the URL that is returned to the frontend issue the following command:
.. prompt:: bash #
ceph dashboard set-grafana-frontend-api-url <grafana-server-url>
If no value is set for that option, it will simply fall back to the value of the GRAFANA_API_URL option. If set, it will instruct the browser to use this URL to access Grafana.
.. _dashboard-saml2-sso-support:
Enabling SAML2 Single Sign-On (SSO) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The Ceph Dashboard supports external authentication of users via the
SAML 2.0 <https://en.wikipedia.org/wiki/SAML_2.0>_ protocol. You need to
first create user accounts and associate them with desired roles, as
authorization is performed by the Dashboard. However, the authentication
process can be performed by an existing Identity Provider (IdP).
.. note::
Ceph Dashboard SSO support relies on onelogin's
python-saml <https://pypi.org/project/python-saml/>_ library.
Please ensure that this library is installed on your system, either by using
your distribution's package management or via Python's pip installer.
To configure SSO on Ceph Dashboard, you should use the following command:
.. prompt:: bash #
ceph dashboard sso setup saml2 <ceph_dashboard_base_url> <idp_metadata> {<idp_username_attribute>} {<idp_entity_id>} {<sp_x_509_cert>} {<sp_private_key>}
Parameters:
https://cephdashboard.local)http://, https://) or local (file://) path or content of the IdP metadata XML (e.g., https://myidp/metadata, file:///home/myuser/metadata.xml).uid... note::
The issuer value of SAML requests will follow this pattern: <ceph_dashboard_base_url>/auth/saml2/metadata
To display the current SAML 2.0 configuration, use the following command:
.. prompt:: bash #
ceph dashboard sso show saml2
.. note::
For more information about onelogin_settings, please check the onelogin documentation <https://github.com/onelogin/python-saml>_.
To disable SSO:
.. prompt:: bash #
ceph dashboard sso disable
To check if SSO is enabled:
.. prompt:: bash #
ceph dashboard sso status
To enable SSO:
.. prompt:: bash #
ceph dashboard sso enable saml2
.. _dashboard-oauth2-sso-support:
Enabling OAuth2 Single Sign-On (SSO) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The Ceph Dashboard supports external authentication of users via the
OAuth <https://en.wikipedia.org/wiki/OAuth>_ protocol. You need to
have :ref:cephadm enabled as your orchestrator with an active
:ref:deploy-cephadm-mgmt-gateway and :ref:deploy-cephadm-oauth2-proxy services.
From the IDP of choice, Keycloak is the current recomendation and tested solution,
configure the IDP's client used in the oauth2-proxy service configuration to validate the following redirect URLs
for login_url: https://<host_name>|<IP_address>/oauth2/callback and
the following logout_url: https://<host_name>|<IP_address>/ /oauth2/sign_out
Again, from the IDP, we will need a user with a valid role, this user will be the one to perform authorization against, we can create a role like: 'administator' to give admin level access to the user.
Make certain that the enable_auth flag has been included in the ceph orch apply mgmt-gateway command and that it has been set to true by running a
command of the following form:
.. prompt:: bash $
ceph orch apply mgmt-gateway --enable_auth=true --placement=<ceph-node-02>
To disable SSO:
.. prompt:: bash $
ceph dashboard sso disable
To check if SSO is enabled:
.. prompt:: bash $
ceph dashboard sso status
To enable SSO:
.. prompt:: bash $
ceph dashboard sso enable oauth2
.. _dashboard-alerting:
Enabling Prometheus Alerting ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
To use Prometheus for alerting you must define alerting rules <https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules>.
These are managed by the Alertmanager <https://prometheus.io/docs/alerting/alertmanager>.
If you are not yet using the Alertmanager, install it <https://github.com/prometheus/alertmanager#install>_ as it receives
and manages alerts from Prometheus.
Alertmanager capabilities can be consumed by the dashboard in three different ways:
#. Use the notification receiver of the dashboard.
#. Use the Prometheus Alertmanager API.
#. Use both sources simultaneously.
All three methods notify you about alerts. You won't be notified twice if you use both sources, but you need to consume at least the Alertmanager API in order to manage silences.
This allows you to get notifications as configured <https://prometheus.io/docs/alerting/configuration/>_ from the Alertmanager.
You will get notified inside the dashboard once a notification is send out,
but you are not able to manage alerts.
Add the dashboard receiver and the new route to your Alertmanager configuration. This should look like::
route:
receiver: 'ceph-dashboard'
...
receivers:
- name: 'ceph-dashboard'
webhook_configs:
- url: '<url-to-dashboard>/api/prometheus_receiver'
Ensure that the Alertmanager considers your SSL certificate in terms
of the dashboard as valid. For more information about the correct
configuration checkout the <http_config> documentation <https://prometheus.io/docs/alerting/configuration/#%3Chttp_config%3E>_.
This allows you to manage alerts and silences and will enable the "Active Alerts", "All Alerts" as well as the "Silences" tabs in the "Monitoring" section of the "Cluster" menu entry.
Alerts can be sorted by name, job, severity, state and start time. Unfortunately it's not possible to know when an alert was sent out through a notification by the Alertmanager based on your configuration, that's why the dashboard will notify the user on any visible change to an alert and will notify the changed alert.
Silences can be sorted by id, creator, status, start, updated and end time. Silences can be created in various ways, it's also possible to expire them.
#. Create from scratch
#. Based on a selected alert
#. Recreate from expired silence
#. Update a silence (which will recreate and expire it (default Alertmanager behaviour))
To use it, specify the host and port of the Alertmanager server:
.. prompt:: bash #
ceph dashboard set-alertmanager-api-host <alertmanager-host:port> # default: ''
For example:
.. prompt:: bash #
ceph dashboard set-alertmanager-api-host 'http://localhost:9093'
To be able to see all configured alerts, you will need to configure the URL to the Prometheus API. Using this API, the UI will also help you in verifying that a new silence will match a corresponding alert.
.. prompt:: bash #
ceph dashboard set-prometheus-api-host <prometheus-host:port> # default: ''
For example:
.. prompt:: bash #
ceph dashboard set-prometheus-api-host 'http://localhost:9090'
After setting up the hosts, refresh your browser's dashboard window or tab.
The behaviors of both methods are configured in a way that they should not disturb each other, through annoying duplicated notifications may pop up.
If you are using a self-signed certificate in your Prometheus or your Alertmanager setup, you should disable certificate verification in the dashboard to avoid refused connections caused by certificates signed by an unknown CA or that do not match the host name.
.. prompt:: bash #
ceph dashboard set-prometheus-api-ssl-verify False
.. prompt:: bash #
ceph dashboard set-alertmanager-api-ssl-verify False
.. _dashboard-user-role-management:
Password Policy ^^^^^^^^^^^^^^^
By default the password policy feature is enabled, which includes the following checks:
The password policy feature can be switched on or off completely:
.. prompt:: bash #
ceph dashboard set-pwd-policy-enabled <true|false>
The following individual checks can also be switched on or off:
.. prompt:: bash #
ceph dashboard set-pwd-policy-check-length-enabled <true|false> ceph dashboard set-pwd-policy-check-oldpwd-enabled <true|false> ceph dashboard set-pwd-policy-check-username-enabled <true|false> ceph dashboard set-pwd-policy-check-exclusion-list-enabled <true|false> ceph dashboard set-pwd-policy-check-complexity-enabled <true|false> ceph dashboard set-pwd-policy-check-sequential-chars-enabled <true|false> ceph dashboard set-pwd-policy-check-repetitive-chars-enabled <true|false>
Additionally the following options are available to configure password policy.
.. prompt:: bash #
ceph dashboard set-pwd-policy-min-length <N>
Minimum password complexity (defaults to 10):
.. prompt:: bash #
ceph dashboard set-pwd-policy-min-complexity <N>
Password complexity is calculated by classifying each character in the password. The complexity count starts by 0. A character is rated by the following rules in the given order.
!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~.A list of comma separated words that are not allowed to be used in a password:
.. prompt:: bash #
ceph dashboard set-pwd-policy-exclusion-list <word>[,...]
User Accounts ^^^^^^^^^^^^^
The Ceph Dashboard supports multiple user accounts. Each user account
consists of a username, a password (stored in encrypted form using bcrypt),
an optional name, and an optional email address.
If a new user is created via the Web UI, it is possible to set an option that the user must assign a new password when they log in for the first time.
User accounts are stored in the monitors' configuration database, and are
available to all ceph-mgr instances.
We provide a set of CLI commands to manage user accounts:
Show User(s):
.. prompt:: bash #
ceph dashboard ac-user-show [<username>]
Create User:
.. prompt:: bash #
ceph dashboard ac-user-create [--enabled] [--force-password] [--pwd_update_required] <username> -i <file-containing-password> [<rolename>] [<name>] [<email>] [<pwd_expiration_date>]
To bypass password policy checks use the force-password option.
Add the option pwd_update_required so that a newly created user has
to change their password after the first login.
Delete User:
.. prompt:: bash #
ceph dashboard ac-user-delete <username>
Change Password:
.. prompt:: bash #
ceph dashboard ac-user-set-password [--force-password] <username> -i <file-containing-password>
Change Password Hash:
.. prompt:: bash #
ceph dashboard ac-user-set-password-hash <username> -i <file-containing-password-hash>
The hash must be a bcrypt hash and salt, e.g. $2b$12$Pt3Vq/rDt2y9glTPSV.VFegiLkQeIpddtkhoFetNApYmIJOY8gau2.
This can be used to import users from an external database.
Modify User (name, and email):
.. prompt:: bash #
ceph dashboard ac-user-set-info <username> <name> <email>
Disable User:
.. prompt:: bash #
ceph dashboard ac-user-disable <username>
Enable User:
.. prompt:: bash #
ceph dashboard ac-user-enable <username>
User Roles and Permissions ^^^^^^^^^^^^^^^^^^^^^^^^^^
User accounts are associated with a set of roles that define which dashboard functionality can be accessed.
The Dashboard functionality/modules are grouped within a security scope. Security scopes are predefined and static. The current available security scopes are:
Hosts menu
entry.A role specifies a set of mappings between a security scope and a set of permissions. There are four types of permissions:
See below for an example of a role specification, in the form of a Python dictionary::
{ 'role': 'my_new_role', 'description': 'My new role', 'scopes_permissions': { 'pool': ['read', 'create'], 'rbd-image': ['read', 'create', 'update', 'delete'] } }
The above role dictates that a user has read and create permissions for features related to pool management, and has full permissions for features related to RBD image management.
The Dashboard provides a set of predefined roles that we call system roles, which can be used right away by a fresh Ceph Dashboard installation.
The list of system roles are:
The list of available roles can be retrieved with the following command:
.. prompt:: bash #
ceph dashboard ac-role-show [<rolename>]
You can also use the CLI to create new roles. The available commands are the following:
Create Role:
.. prompt:: bash #
ceph dashboard ac-role-create <rolename> [<description>]
Delete Role:
.. prompt:: bash #
ceph dashboard ac-role-delete <rolename>
Add Scope Permissions to Role:
.. prompt:: bash #
ceph dashboard ac-role-add-scope-perms <rolename> <scopename> <permission> [<permission>...]
Delete Scope Permission from Role:
.. prompt:: bash #
ceph dashboard ac-role-del-scope-perms <rolename> <scopename>
To assign roles to users, the following commands are available:
Set User Roles:
.. prompt:: bash #
ceph dashboard ac-user-set-roles <username> <rolename> [<rolename>...]
Add Roles To User:
.. prompt:: bash #
ceph dashboard ac-user-add-roles <username> <rolename> [<rolename>...]
Delete Roles from User:
.. prompt:: bash #
ceph dashboard ac-user-del-roles <username> <rolename> [<rolename>...]
Example of User and Custom Role Creation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In this section we show a complete example of the commands that create a user account that can manage RBD images, view and create Ceph pools, and has read-only access to other scopes.
Create the user:
.. prompt:: bash #
ceph dashboard ac-user-create bob -i <file-containing-password>
Create role and specify scope permissions:
.. prompt:: bash #
ceph dashboard ac-role-create rbd/pool-manager ceph dashboard ac-role-add-scope-perms rbd/pool-manager rbd-image read create update delete ceph dashboard ac-role-add-scope-perms rbd/pool-manager pool read create
Associate roles to user:
.. prompt:: bash #
ceph dashboard ac-user-set-roles bob rbd/pool-manager read-only
.. _dashboard-proxy-configuration:
In a Ceph cluster with multiple ceph-mgr instances, only the dashboard
running on the currently active ceph-mgr daemon will serve incoming requests.
Connections to the dashboard's TCP port on standby ceph-mgr instances
will receive an HTTP redirect (303) to the active manager's dashboard URL.
This enables you to point your browser to any ceph-mgr instance in
order to access the dashboard.
If you want to establish a fixed URL to reach the dashboard or if you don't want
to allow direct connections to the manager nodes, you could set up a proxy that
automatically forwards incoming requests to the active ceph-mgr
instance.
Configuring a URL Prefix ^^^^^^^^^^^^^^^^^^^^^^^^
If you are accessing the dashboard via a reverse proxy,
you may wish to service it under a URL prefix. To get the dashboard
to use hyperlinks that include your prefix, you can set the
url_prefix setting:
.. prompt:: bash #
ceph config set mgr mgr/dashboard/url_prefix $PREFIX
so you can access the dashboard at http://$IP:$PORT/$PREFIX/.
Disable the redirection ^^^^^^^^^^^^^^^^^^^^^^^
If the dashboard is behind a load-balancing proxy like HAProxy <https://www.haproxy.org/>_
you might want to disable redirection to prevent situations in which
internal (unresolvable) URLs are published to the frontend client. Use the
following command to get the dashboard to respond with an HTTP error (500 by default)
instead of redirecting to the active dashboard:
.. prompt:: bash #
ceph config set mgr mgr/dashboard/standby_behaviour "error"
To reset the setting to default redirection, use the following command:
.. prompt:: bash #
ceph config set mgr mgr/dashboard/standby_behaviour "redirect"
Configure the error status code ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When redirection is disabled, you may want to customize the HTTP status code of standby dashboards. To do so you need to run the command:
.. prompt:: bash #
ceph config set mgr mgr/dashboard/standby_error_status_code 503
Resolve IP address to hostname before redirect ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Redirection from a standby dashboard to the active dashboard is done via the manager's IP address, not via the manager's hostname. In virtualized environments, IP-address-based redirection reduces the incidence of error as compared to hostname-based resolution. Because of the increased risk of error due to hostname-based resolution, the option for hostname resolution is disabled by default.
However, in some situations it might be helpful to redirect via the hostname. For example, if the configured TLS certificate matches only the hostnames and not the IP addresses of those hosts, hostname redirection would be preferable.
To activate redirection from standby dashboards to active dashboards via the manager's hostname, run the following command:
.. prompt:: bash #
ceph config set mgr mgr/dashboard/redirect_resolve_ip_addr True
Disable hostname redirection by running the following command:
.. prompt:: bash #
ceph config set mgr mgr/dashboard/redirect_resolve_ip_addr False
.. warning::
If you attempt to activate redirection by using the command above and you
get the error message EINVAL: unrecognized config option 'mgr/dashboard/redirect_resolve_ip_addr', then you might be running a
release of Ceph prior to version 17.2.6. This feature was introduced in
17.2.6, in this commit: https://github.com/ceph/ceph/pull/48219.
HAProxy example configuration ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Below you will find an example configuration for SSL/TLS passthrough using
HAProxy <https://www.haproxy.org/>_.
Please note that this configuration works under the following conditions. If the dashboard fails over, the front-end client might receive a HTTP redirect (303) response and will be redirected to an unresolvable host. This happens when failover occurs between two HAProxy health checks. In this situation the previously active dashboard node will now respond with a 303 which points to the new active node. To prevent that situation you should consider disabling redirection on standby nodes.
::
defaults log global option log-health-checks timeout connect 5s timeout client 50s timeout server 450s
frontend dashboard_front mode http bind *:80 option httplog redirect scheme https code 301 if !{ ssl_fc }
frontend dashboard_front_ssl mode tcp bind *:443 option tcplog default_backend dashboard_back_ssl
backend dashboard_back_ssl mode tcp option httpchk GET / http-check expect status 200 server x <HOST>:<PORT> check check-ssl verify none server y <HOST>:<PORT> check check-ssl verify none server z <HOST>:<PORT> check check-ssl verify none
.. _dashboard-auditing:
The REST API can log PUT, POST and DELETE requests to the Ceph audit log. This feature is disabled by default, but can be enabled with the following command:
.. prompt:: bash #
ceph dashboard set-audit-api-enabled <true|false>
If enabled, the following parameters are logged per each request:
The logging of the request payload (the arguments and their values) is enabled by default. Execute the following command to disable this behaviour:
.. prompt:: bash #
ceph dashboard set-audit-api-log-payload <true|false>
A log entry may look like this::
2018-10-22 15:27:01.302514 mgr.x [INF] [DASHBOARD] from='https://[::ffff:127.0.0.1]:37022' path='/api/rgw/user/klaus' method='PUT' user='admin' params='{"max_buckets": "1000", "display_name": "Klaus Mustermann", "uid": "klaus", "suspended": "0", "email": "[email protected]"}'
.. _dashboard-nfs-ganesha-management:
The dashboard requires enabling the NFS module which will be used to manage
NFS clusters and NFS exports. For more information check :ref:mgr-nfs.
Plug-ins extend the functionality of the Ceph Dashboard in a modular and loosely coupled fashion.
.. _Grafana: https://grafana.com/
.. include:: dashboard_plugins/feature_toggles.inc.rst .. include:: dashboard_plugins/debug.inc.rst .. include:: dashboard_plugins/motd.inc.rst
Locating the Dashboard ^^^^^^^^^^^^^^^^^^^^^^
If you are unsure of the location of the Ceph Dashboard, run the following command:
.. prompt:: bash #
ceph mgr services | jq .dashboard
::
"https://host:port"
The command returns the URL where the Ceph Dashboard is located: https://<host>:<port>/
.. note::
Many Ceph tools return results in JSON format. We suggest that
you install the `jq <https://stedolan.github.io/jq>`_ command-line
utility to facilitate working with JSON data.
Accessing the Dashboard ^^^^^^^^^^^^^^^^^^^^^^^
If you are unable to access the Ceph Dashboard, run the following commands:
#. Verify the Ceph Dashboard module is enabled:
.. prompt:: bash #
ceph mgr module ls | jq .enabled_modules
Ensure the Ceph Dashboard module is listed in the return value of the command. Example snipped output from the command above::
[
"dashboard",
"iostat",
"restful"
]
#. If it is not listed, activate the module with the following command:
.. prompt:: bash #
ceph mgr module enable dashboard
#. Check the Ceph Dashboard and/or ceph-mgr log files for any errors.
Check if ceph-mgr log messages are written to a file by:
.. prompt:: bash #
ceph config get mgr log_to_file
::
true
Get the location of the log file (it's /var/log/ceph/<cluster-name>-<daemon-name>.log
by default):
.. prompt:: bash #
ceph config get mgr log_file
::
/var/log/ceph/$cluster-$name.log
#. Ensure the SSL/TLS support is configured properly:
Check if the SSL/TLS support is enabled:
.. prompt:: bash #
ceph config get mgr mgr/dashboard/ssl
If the command returns true, verify a certificate exists by:
.. prompt:: bash #
ceph config-key get mgr/dashboard/crt
and:
.. prompt:: bash #
ceph config-key get mgr/dashboard/key
If it doesn't return true, run the following command to generate a self-signed
certificate or follow the instructions outlined in
:ref:dashboard-ssl-tls-support:
.. prompt:: bash #
ceph dashboard create-self-signed-cert
Trouble Logging into the Dashboard ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you are unable to log into the Ceph Dashboard and you receive the following error, run through the procedural checks below:
.. image:: ../images/dashboard/invalid-credentials.png :align: center
#. Check that your user credentials are correct. If you are seeing the notification message above when trying to log into the Ceph Dashboard, it is likely you are using the wrong credentials. Double check your username and password, and ensure that your keyboard's caps lock is not enabled by accident.
#. If your user credentials are correct, but you are experiencing the same error, check that the user account exists:
.. prompt:: bash #
ceph dashboard ac-user-show <username>
This command returns your user data. If the user does not exist, it will print::
Error ENOENT: User <username> does not exist
#. Check if the user is enabled:
.. prompt:: bash #
ceph dashboard ac-user-show <username> | jq .enabled
::
true
Check if enabled is set to true for your user. If not the user is
not enabled, run:
.. prompt:: bash #
ceph dashboard ac-user-enable <username>
Please see :ref:dashboard-user-role-management for more information.
A Dashboard Feature is Not Working ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When an error occurs on the backend, you will usually receive an error notification on the frontend. Run through the following scenarios to debug.
#. Check the Ceph Dashboard and ceph-mgr logfile(s) for any errors. These can
found by searching for keywords, such as 500 Internal Server Error,
followed by traceback. The end of a traceback contains more details about
what exact error occurred.
#. Check your web browser's JavaScript Console for any errors.
Ceph Dashboard Logs ^^^^^^^^^^^^^^^^^^^
Dashboard Debug Flag """"""""""""""""""""
With this flag enabled, error traceback is included in backend responses.
To enable this flag via the Ceph Dashboard, navigate from Cluster to Manager modules. Select Dashboard module and click the edit button. Click the debug checkbox and update.
To enable it via the CLI, run the following command:
.. prompt:: bash #
ceph dashboard debug enable
Setting Logging Level of Dashboard Module """""""""""""""""""""""""""""""""""""""""
Setting the logging level to debug makes the log more verbose and helpful for debugging.
#. Increase the logging level of manager daemons:
.. prompt:: bash #
ceph tell mgr config set debug_mgr 20
#. Adjust the logging level of the Ceph Dashboard module via the Dashboard or CLI:
Navigate from Cluster to Manager modules. Select Dashboard module
and click the edit button. Modify the log_level configuration.
To adjust it via the CLI, run the following command:
.. prompt:: bash #
ceph config set mgr mgr/dashboard/log_level debug
High log levels can result in considerable log volume, which can easily fill up your filesystem or central log store. Set a calendar reminder for an hour, a day, or a week in the future to revert a temporary logging increase. Any current, non-default setting may be shown with the below command:
.. prompt:: bash #
ceph config dump | grep mgr/dashboard/log_level
:: ... mgr advanced mgr/dashboard/log_level debug ...
.. prompt:: bash #
ceph config reset 11
.. _centralized-logging:
Enable Centralized Logging in Dashboard """""""""""""""""""""""""""""""""""""""
To learn more about centralized logging, see :ref:cephadm-monitoring-centralized-logs
Create the Loki service on any particular host using "Create Services" option.
Similarly create the Promtail service which will be by default deployed on all the running hosts.
To see debug-level messages as well as info-level events, run the following command via CLI:
.. prompt:: bash #
ceph config set mgr mgr/cephadm/log_to_cluster_level debug
To enable logging to files, run the following commands via CLI:
.. prompt:: bash #
ceph config set global log_to_file true ceph config set global mon_cluster_log_to_file true
Click on the Daemon Logs tab under Cluster -> Logs.
You can find some pre-defined labels there on clicking the Log browser button such as filename, job etc that can help you query the logs at one go.
You can query the logs with LogQL for advanced search and perform some calculations as well - https://grafana.com/docs/loki/latest/logql/.
Reporting issues from Dashboard """""""""""""""""""""""""""""""
Ceph-Dashboard provides two ways to create an issue in the Ceph Issue Tracker, either using the Ceph command line interface or by using the Ceph Dashboard user interface.
To create an issue in the Ceph Issue Tracker, a user needs to have an account
on the issue tracker. Under the my account tab in the Ceph Issue Tracker,
the user can see their API access key. This key is used for authentication
when creating a new issue. To store the Ceph API access key, in the CLI run:
.. prompt:: bash #
ceph dashboard set-issue-tracker-api-key -i <file-containing-key>
Then on successful update, you can create an issue using:
.. prompt:: bash #
ceph dashboard create issue <project> <tracker_type> <subject> <description>
The available projects to create an issue on are: #. dashboard #. block #. object #. file_system #. ceph_manager #. orchestrator #. ceph_volume #. core_ceph
The available tracker types are: #. bug #. feature
The subject and description are then set by the user.
The user can also create an issue using the Dashboard user interface. The settings
icon drop down menu on the top right of the navigation bar has the option to
Raise an issue. On clicking it, a modal dialog opens that has the option to
select the project and tracker from their respective drop down menus. The subject
and multiline description are added by the user. The user can then submit the issue.