CEPH
Aufbau
Storage-Node und Monitor Nodes
- omar
- nelson
- bay
Admin-Node
- kekistan
Vorbreitung
Installation von ssh und ntp
Auf allen Nodes
- sudo apt -y install ntp ssh
- wget -q -O- 'https://download.ceph.com/keys/release.asc' | apt-key add -
- echo deb http://download.ceph.com/debian-luminous/ $(lsb_release -sc) main | tee /etc/apt/sources.list.d/ceph.list
- apt-get update
ssh-key auf allen nodes
ssh-keygen
- root@kekistand:~# ssh-keygen
ssh-copy-id
- root@kekistand:~# ssh-copy-id nelson
- root@kekistand:~# ssh-copy-id bay
- root@kekistand:~# ssh-copy-id omar
Hilfskript
- cat /usr/local/sbin/root-all-nodes
#!/bin/bash echo localhost $* LIST="bay omar nelson" for k in $LIST do echo $k ssh $k $* done
Anlegen des Users cephadmin und keyhandling
- root-all-nodes useradd -s /bin/bash -m cephadmin
- sudo -u cephadmin ssh-keygen -q
- root-all-nodes mkdir /home/cephadmin/.ssh
- root-all-nodes chown cephadmin.cephadmin /home/cephadmin/.ssh
- root-all-nodes chmod 700 /home/cephadmin/.ssh
- root-all-nodes 'echo "cephadmin ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/ceph'
- scp /home/cephadmin/.ssh/id_rsa.pub /tmp
- scp /home/cephadmin/.ssh/id_rsa.pub bay:/tmp
- scp /home/cephadmin/.ssh/id_rsa.pub nelson:/tmp
- scp /home/cephadmin/.ssh/id_rsa.pub omar:/tmp
- root-all-nodes cat /tmp/id_rsa.pub > /home/cephadmin/.ssh/authorized_keys
Purge
Install
- apt install ceph-deploy
- su - cephadmin
- mkdir my-first-ceph-cluster
- cd my-first-ceph-cluster
Purge
- ceph-deploy purge kekistan bay nelson omar
- ceph-deploy purgedata kekistan bay nelson omar
- ceph-deploy forgetkeys
Monitor Nodes
- ceph-deploy new bay nelson omar
- echo "public network = 10.0.0.0/8" >> ceph.conf
- echo "osd pool default size = 2" >> ceph.conf
Install Ceph
- ceph-deploy install kekistan bay nelson omar
Initialisierung der Monitore und sammeln der der Keys
- ceph-deploy mon create-initial
Vorbereiten der Storages
- ceph-deploy osd prepare bay:/dev/sdb nelson:/dev/sdb omar:/dev/sdb
Aktivieren der Storages
- ceph-deploy osd activate bay:/dev/sdb1 nelson:/dev/sdb1 omar:/dev/sdb1
Löschen eines Storages
Auf der Adminnode
- ceph osd out 1
Auf dem OSD
- sudo /etc/init.d/ceph stop osd.1
Auf der Adminnode
- ceph osd crush remove osd.1
- ceph auth del osd.1
- ceph osd rm 1
Kopieren der Admin-keyrings
- ceph-deploy admin kekistan bay nelson omar
- sudo chmod 644 /etc/ceph/ceph.client.admin.keyring
Status check
- ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -1 1.44955 root default -2 0.48318 host bay 0 0.48318 osd.0 up 1.00000 1.00000 -3 0.48318 host nelson 1 0.48318 osd.1 up 1.00000 1.00000 -4 0.48318 host omar 2 0.48318 osd.2 up 1.00000 1.00000
- ceph health
HEALTH_OK
Block Device Client
auf dem Client cephadmin einrichten und keys von admin node kopieren ...
- ceph-deploy install bajor
- ceph-deploy admin bajor
- rbd create bar --size 4096 --image-feature layering
- sudo rbd map bar --name client.admin
- sudo mkfs.ext4 /dev/rbd0
- sudo mkdir /mnt/ceph-block-device
- sudo mount /dev/rbd0 /mnt/ceph-block-device
Anlegen eines Metadata Server(Voraussetzung für das CephFS)
- ceph-deploy mds create bay nelson omar
- ceph osd pool create cephfs_data 1
- ceph osd pool create cephfs_metadata 1
- ceph fs new cephfs cephfs_metadata cephfs_data
- ceph mds stat
Client File System
- installieren von seph-fs-common
apt install ceph-fs-common
- auf dem Client das secret speichern
mkdir /etc/ceph/ vi /etc/ceph/admin.secret
- TIPP: secret auf dem client soll in der form sein
ÖLUJmb5rfHOH8GigijgGTfu==
- eintrag in der fstab anlegen
vi /etc/fstab 10.1.1.1,10.1.1.2,10.1.1.3:/ /mnt/cephfs ceph name=admin,secretfile=/etc/ceph/admin.secret 0 0
- mountpunkt erstellen & Files System mounten
mkdir /mnt/cephfs mount /mnt/cephfs
Links
- http://docs.ceph.com/docs/master/start/
- https://linoxide.com/ubuntu-how-to/create-ceph-cluster-ubuntu-16-04/
- https://www.howtoforge.com/tutorial/how-to-install-a-ceph-cluster-on-ubuntu-16-04/
- https://www.howtoforge.com/tutorial/how-to-mount-cephfs-on-centos-7/
- http://www.admin-magazin.de/Das-Heft/2014/04/Die-verteilten-Speichersysteme-GlusterFS-und-Ceph-im-Vergleich/(offset)/12
- https://www.sebastien-han.fr/blog/2012/07/06/nfs-over-rbd/