Ceph Deployment

#INSTALLATION (MANUAL)

##INSTALL PACKAGES WITH CEPHADM

prepare env

yum install python3 yum-utils -y
    yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo
    yum install docker-ce docker-ce-cli containerd.io
    systemctl enable docker && service docker start
    systemctl disable firewalld &&  service firewalld stop

Download the cephadm script

curl --silent --remote-name --location https://github.com/ceph/ceph/raw/pacific/src/cephadm/cephadm
    chmod +x cephadm
    
Configure the Ceph repository based on the release name:

./cephadm add-repo --release Octopus
    #https://docs.ceph.com/en/latest/releases/index.html
    
Install the appropriate packages. You can install them using your package management tool (e.g., APT, Yum) directly, or you can also use the cephadm wrapper. For example:

sudo rpm --import 'https://download.ceph.com/keys/release.asc'
    ./cephadm install
    ./cephadm install ceph-common

##BOOTSTRAP A NEW CLUSTER

The first step in creating a new Ceph cluster is running the cephadm bootstrap command on the Ceph cluster’s first host.

cephadm bootstrap --mon-ip *<mon-ip>*
    cephadm bootstrap --mon-ip 192.168.0.131

Cephadm does not require any Ceph packages to be installed on the host. However, we recommend enabling easy access to the ceph command. There are several ways to do this

cephadm shell
    
##ADDING HOSTS

Hosts must have these Requirements installed. Hosts without all the necessary requirements will fail to be added to the cluster.

To add each new host to the cluster, perform two steps:

Install the cluster’s public SSH key in the new host’s root user’s authorized_keys file

ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph02
    ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph03
    
Tell Ceph that the new node is part of the cluster:

ceph orch host add *<newhost>* [*<ip>*] [*<label1> ...*]
    #eg
    ceph orch host add ceph02 192.168.0.132
    ceph orch host add ceph03 192.168.0.133

One or more labels can also be included to immediately label the new host. For example, by default the _admin label will make cephadm maintain a copy of the ceph.conf file and a client.admin keyring file in /etc/ceph:

ceph orch host label add *<host>* _admin
    #eg
    ceph orch host label add ceph02 _admin
    ceph orch host label add ceph03 _admin
    
finally

ceph orch host ls
    scp -r /etc/ceph/ ceph02:/etc/
    scp -r /etc/ceph/ ceph03:/etc/
    
    
##ADDING ADDITIONAL MONS

A typical Ceph cluster has three or five monitor daemons spread across different hosts. We recommend deploying five monitors if there are five or more nodes in your cluster.

ceph orch apply mon *<number-of-monitors>*
    #eg
    ceph orch apply mon 3
    ceph orch apply mon --placement="ceph01,ceph02,ceph03"
    
##ADDING STORAGE

Run this command to display an inventory of storage devices on all cluster hosts:

ceph orch device ls
    
Create an OSD from a specific device on a specific host:

ceph orch daemon add osd *<host>*:*<device-path>*
    #eg 
    ceph orch daemon add osd ceph01:/dev/sdb
    ceph orch daemon add osd ceph02:/dev/sdb
    ceph orch daemon add osd ceph03:/dev/sdb
    
#CEPH BLOCK DEVICE

Unless specified, the rbd command will access the Ceph cluster using the ID admin. This ID allows full administrative access to the cluster. It is recommended that you utilize a more restricted user wherever possible.

scp 192.168.0.131:/etc/ceph/ceph.conf /etc/ceph
    ssh 192.168.0.131 ceph auth get client.admin > /etc/ceph/ceph.client.admin.keyring
    
To list block devices in the rbd pool, execute the following (i.e., rbd is the default pool name):

rbd ls
    rbd ls {poolname}
    
To retrieve information from a particular image, execute the following, but replace {image-name} with the name for the image:

rbd info {image-name}

##KERNEL MOUNT

for centos7

rbd feature disable workload02 object-map fast-diff deep-flatten

To mount a block device image, first return a list of the images.

rbd list
    
Use rbd to map an image name to a kernel module.

rbd device map {pool-name}/{image-name} --id {user-name}
    rbd device map rbd/myimage --id admin --keyring /etc/ceph/ceph.client.admin.keyring
    
To show block device images mapped to kernel modules with the rbd, specify device list arguments.

rbd device list
    
To unmap a block device image with the rbd command

rbd device unmap /dev/rbd/{poolname}/{imagename}
    
##MOUNT BOOT TIME

vi /etc/ceph/rbdmap
    rbd/workload01    id=admin,keyring=/etc/ceph/ceph.client.admin.keyring
    
    systemctl enable rbdmap.service && service rbdmap start
    
    vi /etc/fstab
    /dev/rbd/rbd/workload02 /mnt/workload xfs _netdev,defaults 0 2
    /dev/rbd/rbd/nextcloud02 /mnt/nextcloud xfs _netdev,defaults 0 2
    
##DOCKER DEPENDENCY

systemctl list-units --type=mount
    vi /usr/lib/systemd/system/docker.service
    After=network-online.target firewalld.service containerd.service mnt-workload.mount mnt-nextcloud.mount
    
    systemctl daemon-reload
    service docker restart

#OPERATION
    
##DASHBOARD

ceph dashboard set-grafana-api-url https://ceph-grafana.louyj.top
    ceph config set mgr mgr/dashboard/standby_behaviour "error"
    
##RESET WARNNING

To display a list of messages:

ceph crash ls

If you want to read the message:

ceph crash info <id>

then:

ceph crash archive <id>

or:

ceph crash archive-all
    
##REMOVE OSD

Before you remove an OSD, it is usually up and in. You need to take it out of the cluster so that Ceph can begin rebalancing and copying its data to other OSDs.

ceph osd out 4
    
After you take an OSD out of the cluster, it may still be running. That is, the OSD may be up and out. You must stop your OSD before you remove it from the configuration.

ceph orch daemon rm osd.4 --force
    
Once you have taken your OSD out of the cluster, Ceph will begin rebalancing the cluster by migrating placement groups out of the OSD you removed. You can observe this process with the ceph tool.

ceph osd purge 4 --yes-i-really-mean-it
    ceph orch device zap ceph01 /dev/sdb  --force
    
speed up recovery

ceph config set osd osd_max_backfills 3
    ceph config set osd osd_recovery_max_active 6
    ceph config set osd osd_recovery_op_priority 1
    ceph config set osd osd_recovery_sleep_hdd 0
    #To set back to default run
    ceph config rm osd osd_recovery_max_active
    ceph config rm osd osd_max_backfills
    ceph config rm osd osd_recovery_op_priority
    ceph config rm osd osd_recovery_sleep_hdd
    
stop recovery

ceph osd set norecover
    ceph osd unset norecover
    
##SHOW RBD SIZE

rbd du pool/rbd
    
##Fast Release Space

```
rbd map nextcloud01
mount /dev/rbd4 /mnt/temp
fstrim -v /mnt/temp
```

##Resize Rbd

```
rbd resize rbd-low/xxx -s 2T
xfs_growfs /dev/rbd2
```

##Memory Usage Limit

```
ceph config set osd osd_memory_target 1073741824  #default 4294967296
ceph config set osd mds_cache_memory_limit 1073741824 #default 4294967296

ceph config set mon mon_memory_target 536870912 #default 2147483648

ceph config set osd osd_client_message_size_cap 209715200# 524288000
```

see https://support.huaweicloud.com/tngg-kunpengsdss/kunpengcephblock_05_0013.html
see https://docs.ceph.com/en/latest/rados/configuration/mon-config-ref/?

```
rbd_cache_writethrough_until_flush false
```

##Disable Rebalance

```
ceph osd set noout
ceph osd set norebalance

ceph osd unset noout
ceph osd unset norebalance
```
    
##Limit Scrub

```
ceph config set osd osd_scrub_begin_hour 2
ceph config set osd osd_scrub_end_hour 6
```
    
##Use K8s Storage

###RBD Provisioner

```
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rbd
provisioner: kubernetes.io/rbd
reclaimPolicy: Retain
volumeBindingMode: Immediate
parameters:
  monitors: 192.168.0.131:6789,192.168.0.132:6789,192.168.0.133:6789
  adminId: admin
  adminSecretName: ceph-secret
  adminSecretNamespace: kube-system
  pool: rbd
  userId: admin
  userSecretName: ceph-secret
  userSecretNamespace: kube-system
  fsType: xfs
  imageFormat: "2"
  imageFeatures: "layering,exclusive-lock"
```

`userSecretName` The name of Ceph Secret for userId to map RBD image. It must exist in the same namespace as PVCs. This parameter is required. The provided secret must have type "kubernetes.io/rbd", for example created in this way:

kubectl create secret generic ceph-secret --type="kubernetes.io/rbd" \
  --from-literal=key='xxx' \
  --namespace=kube-system
  
###Mount by PVC

```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test01
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 8TG
  storageClassName: rbd
```

Deployment

```
volumes:
      - name: data
        persistentVolumeClaim:
          claimName: test01
```

The name of a PersistentVolumeClaim object must be a valid DNS subdomain name.

###Mount By Volumes

```
volumes:
    - name: rbdpd
      rbd:
        monitors:
        - '192.168.0.131:6789'
        - '192.168.0.132:6789'
        - '192.168.0.133:6789'
        pool: rbd
        image: xxx
        fsType: xfs
        readOnly: true
        user: admin
        keyring: /etc/ceph/ceph.client.admin.keyring
```

###Delete Pool

ceph config set mon mon_allow_pool_delete true
    ceph osd pool delete nfs01 nfs01 --yes-i-really-really-mean-it
    
###Backup And Restore

```
#create snapshot
rbd snap create rbd/vscode@vscode.init.0922
rbd snap ls rbd/vscode
#export snapshot
rbd export-diff rbd/vscode@vscode.init.0922 vscodebk
rbd export-diff --from-snap vscode.init.0922 rbd/vscode@vscode.incr.0922.0922 vscode.incr.0922.0922
#list diff content
rbd diff --from-snap test.snap.t1 test_pool/test_image@test.snap.t2 --format plain
#restore backup
rbd create --size 10240 --image-format=2 --image-feature=layering --image-feature=exclusive-lock backup_image
rbd import-diff vscode.init.0922 rbd/backup_image
rbd import-diff vscode@vscode.incr.0922.0922 rbd/backup_image
```

##Preempt Mgr

ceph mgr fail <mgr name>

#NFS

#ceph orch apply nfs nfs01 nfs01 --placement "ceph01 ceph02 ceph03 ceph04"
    #ceph orch rm nfs.nfs01

#https://docs.ceph.com/en/octopus/cephfs/fs-nfs-exports/?
    ceph nfs cluster create cephfs nfs01 "ceph01,ceph02,ceph03,ceph04"
    ceph nfs cluster ls
    ceph nfs cluster info nfs01
    
    ceph nfs export create cephfs  cephfs nfs01 /nfs01 --path=/nfs01
    ceph nfs export ls nfs01 --detailed
    ceph nfs export get nfs01 /nfs01
    
    mount -t nfs -o port=<ganesha-port> <ganesha-host-name>:<ganesha-pseudo-path> <mount-point>
    mount -t nfs -o port=2049 ceph01:/nfs01 /mnt/nfs
    
    
## repair inconsistent

```
ceph health detail
ceph pg repair 2.18

#unfound
ceph pg 10.e mark_unfound_lost revert
```

## repair monitor

```
cephadm unit --name mon.ceph03 stop

cephadm shell --mount /var/lib/ceph/073f85b0-f375-11eb-b4f2-000c29e0bb80/mon.ceph03/:/var/lib/ceph/mon/ceph-ceph03

ceph-mon -i ceph03 --extract-monmap /tmp/monmap
monmaptool /tmp/monmap --rm ceph01
monmaptool /tmp/monmap --rm ceph02
ceph-mon -i ceph03 --inject-monmap /tmp/monmap
exit

cephadm unit --name mon.ceph03 start

ceph -s
```

## apply monitor

```
ceph orch apply prometheus --placement 'count:1'
ceph orch apply alertmanager --placement 'count:1'
ceph orch apply grafana --placement 'count:1'
```

## repair ceph config in osd

```
docker exec -it 9615c0c1334e bash
cd /var/lib/ceph/osd/ceph-0
cat config
```

## check service log

```
ceph log last cephadm
cephadm logs --name osd.0
```

Navigation

Recent Posts

Friend Links