Lab 2: Deploying the Ceph Cluster
Introduction
In this Lab, you will learn how to deploy the ceph cluster using Cephadm tool.
There are different ways to deploy Ceph Cluster:
Cephadm: Installs and Manages a Ceph cluster using containers and systemd.
Rook: Deploys and Manages Ceph clusters running in Kubernetes
ceph-ansible: Deploys and Manages Ceph clusters using Ansible.
ceph-deploy: Is a tool for quickly deploying clusters.
DeepSea: Installs Ceph using Salt.
jaas.ai/ceph-mon: Installs Ceph using Juju.
github.com/openstack/puppet-ceph: Installs Ceph via Puppet.
Ceph can also be installed manually.
Note: Ensure that the pre-requisites installed on all of the nodes from Lab1 before proceeding
with this lab.
1. Ensure that you have logged-in as root user with password as linux on ceph-mon1 node.
1.1 Let us first fetch the most recent version of the cephadm standalone script from github
using curl.
# curl --silent --remote-name --location
https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm
1.2 Add execute permission to cephadm script
# chmod +x cephadm
Student Material – Do Not Re-distribute. For any queries contact:
[email protected] or https://www.linkedin.com/in/naushadpasha/
1.3 Add the octopus release repository and Install the cephadm command on the host.
# ./cephadm add-repo --release octopus
output:
1.4 Installing the cephadm packges
# ./cephadm install
output
1.5 Confirm cephadm command installed and is your path.
# which cephadm
output:
1.6 Let us bootstrap a new cluster.
# mkdir -p /etc/ceph
# cephadm bootstrap --mon-ip 192.168.100.51
Student Material – Do Not Re-distribute. For any queries contact:
[email protected] or https://www.linkedin.com/in/naushadpasha/
Output:
Note: At the end of installation, the dashboard credentials are displayed, please save the details for
future use.
This command will:
1. Create a monitor and manager daemon for the new cluster on the local host.
2. Generate a new SSH key for the Ceph cluster and adds it to the root user’s
/root/.ssh/authorized_keys file.
3. Write a minimal configuration file needed to communicate with the new cluster to
/etc/ceph/ceph.conf.
4. Write a copy of the client.admin administrative (privileged!) secret key to
/etc/ceph/ceph.client.admin.keyring.
5. Write a copy of the public key to /etc/ceph/ceph.pub.
6. Bootstrap writes the files needed to access the new cluster to /etc/ceph for
convenience, so that any Ceph packages installed on the host can easily find them.
7. cephadm bootstrap -h to see all available options
1.7 Create an alias to the cephadm shell for ease of access
# alias ceph='cephadm shell -- ceph' >> ~/.alias
Note: The cephadm shell command launches a bash shell in a container with all of the Ceph packages
installed. By default, if configuration and keyring files are found in /etc/ceph on the host, they are
passed into the container environment so that the shell is fully functional.
1.8 Verify that in the backend there are containers created and running.
# ceph orch ps
Student Material – Do Not Re-distribute. For any queries contact:
[email protected] or https://www.linkedin.com/in/naushadpasha/
1.9 Install the ceph-common package, which contains all of the ceph commands, including ceph, rbd,
mount.ceph (for mounting CephFS file systems).
# cephadm add-repo --release octopus
# yum clean all
# yum clean dbcache
# yum repolist
output:
# cephadm install ceph-common
1.10 Verify the command is accessible
# ceph -v
Output:
Student Material – Do Not Re-distribute. For any queries contact:
[email protected] or https://www.linkedin.com/in/naushadpasha/
1.11 Confirm that the ceph command can connect to the cluster by checking the status.
# ceph status
Output:
1.12 Add hosts to the cluster
To add each new host to the cluster, Install the cluster’s public SSH key in the new host’s root user’s
authorized_keys file:
# ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-mon2
# ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-mon3
# ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-osd1
# ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-osd2
# ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-osd3
# ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-osd4
# ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-client
Note: Enter root’s password “linux” when prompted.
Student Material – Do Not Re-distribute. For any queries contact:
[email protected] or https://www.linkedin.com/in/naushadpasha/
1.13 Let us limit the number of monitors to 3. Default number of monitors is 5
# ceph orch apply mon 3
1.14 Tell Ceph that the new node is part of the cluster:
# ceph orch host add ceph-mon2
# ceph orch host add ceph-mon3
1.15 Let us add labels to the hosts for our convenience and ease of deployment
# ceph orch host label add ceph-mon1 mon
# ceph orch host label add ceph-mon1 mgr
# ceph orch host label add ceph-mon2 mon
# ceph orch host label add ceph-mon2 mgr
# ceph orch host label add ceph-mon3 mon
1.16 To view the current hosts and labels:
# ceph orch host ls
Output:
Student Material – Do Not Re-distribute. For any queries contact:
[email protected] or https://www.linkedin.com/in/naushadpasha/
1.17 Deploy additional monitor to the cluster, this can be done by explicitly mentioning the
hostnames or by applying the monitors by label
# ceph orch apply mon "ceph-mon1 ceph-mon2 ceph-mon3"
----OR----
# ceph orch apply mon label:mon
Note: Be sure to include the first (bootstrap) host in this list. It takes around 5 minutes for all the
daemons to spawn up. Be patient!
Cephadm will be automatically deploy monitors on the hosts labeled as mon, this feature can be
disabled by passing --unmanaged option. We will see this is next step.
Output:
----OR----
1.18 Let us see how to disable the automated deployment of manager
# ceph orch apply mgr --unmanaged
1.19 Let us manually apply manager to 2 nodes
# ceph orch apply mgr "ceph-mon1 ceph-mon2"
1.20 Let us add the osd nodes to the cluster
# ceph orch host add ceph-osd1
# ceph orch host add ceph-osd2
# ceph orch host add ceph-osd3
Student Material – Do Not Re-distribute. For any queries contact:
# ceph orch host add ceph-osd4
[email protected] or https://www.linkedin.com/in/naushadpasha/
1.21 Let us add labels to the hosts for our convenience and ease of deployment
# ceph orch host label add ceph-osd1 osd
# ceph orch host label add ceph-osd2 osd
# ceph orch host label add ceph-osd3 osd
# ceph orch host label add ceph-osd4 osd
1.22 To view the current hosts and labels:
# ceph orch host ls
Output:
1.23 An inventory of storage devices on all cluster hosts can be displayed with:
# ceph orch device ls
Output:
Student Material – Do Not Re-distribute. For any queries contact:
[email protected] or https://www.linkedin.com/in/naushadpasha/
A storage device is considered available if all of the following conditions are met:
• The device must have no partitions.
• The device must not have any LVM state.
• The device must not be mounted.
• The device must not contain a file system.
• The device must not contain a Ceph BlueStore OSD.
• The device must be larger than 5 GB.
Ceph refuses to provision an OSD on a device that is not available.
1.24 Create an OSD from a specific device on a specific host:
# ceph orch daemon add osd ceph-osd1:/dev/sdb
# ceph orch daemon add osd ceph-osd2:/dev/sdb
# ceph orch daemon add osd ceph-osd3:/dev/sdb
# ceph orch daemon add osd ceph-osd4:/dev/sdb
Note: Alternatively we can tell ceph to consume any available and unused storage device by specifying
the command “ceph orch apply osd --all-available-devices”.
1.25 Verify the ceph cluster status to verify is all the monitors and osd daemons are reflecting.
# ceph status
Output:
Note: It take around 5 minutes for all the daemons to spawn up. Be patient! To see HEALTH_OK.
Student Material – Do Not Re-distribute. For any queries contact:
[email protected] or https://www.linkedin.com/in/naushadpasha/
