摘要：
注意ceph-deploy-xxxxx。noarch rpm仅由部署节点使用。其他节点不需要它。部署节点还需要安装其余的rpm。在部署节点上，安装ceph deploy。在root下，输入下载的rpm包目录并执行：yumlocainstall-y/*注意：如果报告错误如下：#安装依赖包python分发#删除源rdo版本yunwei。repo#安装
一、部署准备：
准备5台虚拟机（linux系统为centos7.6版本）
1台部署节点（配一块硬盘，运行ceph-depoly）
3台ceph节点（每台配置两块硬盘，第一块为系统盘并运行mon，第二块作为osd数据盘）
1台客户端（可以使用ceph提供的文件系统，块存储，对象存储）
（1）所有ceph集群节点（包括客户端）设置静态域名解析；
127.0.0.1localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.24.11dlp 192.168.24.8controller 192.168.24.9compute 192.168.24.10 storage
（2）所有集群节点（包括客户端）创建cent用户，并设置密码，后执行如下命令：
useradd cent && echo "123" | passwd --stdin cent
echo -e 'Defaults:cent !requiretty cent ALL = (root) NOPASSWD:ALL' | tee /etc/sudoers.d/ceph
chmod 440 /etc/sudoers.d/ceph
（3）在部署节点切换为cent用户，设置无密钥登陆各节点包括客户端节点
su – cent
ceph@dlp15:17:01~#ssh-keygen
ceph@dlp15:17:01~#ssh-copy-id controller
ceph@dlp15:17:01~#ssh-copy-id compute
ceph@dlp15:17:01~#ssh-copy-id storage
ceph@dlp15:17:01~#ssh-copy-id dlp
（4）在部署节点切换为cent用户，在cent用户家目录，设置如下文件：vi~/.ssh/config# create new ( define all nodes and users )
su – cent
cd .ssh
vim config
Host dlp Hostname dlp User cent Host controller Hostname controller User cent Host compute Hostname compute User cent Host storage Hostname storage User cent
chmod 600 ~/.ssh/config
二、所有节点配置国内ceph源：
（1）all-node（包括客户端）在/etc/yum.repos.d/创建 ceph-yunwei.repo
cd /etc/yum.repos.d
vim ceph-yunwei.repo
[ceph-yunwei] name=ceph-yunwei-install baseurl=https://mirrors.aliyun.com/centos/7.6.1810/storage/x86_64/ceph-jewel/ enable=1gpgcheck=0
（2）到国内ceph源中https://mirrors.aliyun.com/centos/7.6.1810/storage/x86_64/ceph-jewel/下载如下所需rpm包。注意：红色框中为ceph-deploy的rpm，只需要在部署节点安装，下载需要到https://mirrors.aliyun.com/ceph/rpm-jewel/el7/noarch/中找到最新对应的ceph-deploy-xxxxx.noarch.rpm 下载
ceph-10.2.11-0.el7.x86_64.rpm ceph-base-10.2.11-0.el7.x86_64.rpm ceph-common-10.2.11-0.el7.x86_64.rpm ceph-deploy-1.5.39-0.noarch.rpm ceph-devel-compat-10.2.11-0.el7.x86_64.rpm cephfs-java-10.2.11-0.el7.x86_64.rpm ceph-fuse-10.2.11-0.el7.x86_64.rpm ceph-libs-compat-10.2.11-0.el7.x86_64.rpm ceph-mds-10.2.11-0.el7.x86_64.rpm ceph-mon-10.2.11-0.el7.x86_64.rpm ceph-osd-10.2.11-0.el7.x86_64.rpm ceph-radosgw-10.2.11-0.el7.x86_64.rpm ceph-resource-agents-10.2.11-0.el7.x86_64.rpm ceph-selinux-10.2.11-0.el7.x86_64.rpm ceph-test-10.2.11-0.el7.x86_64.rpm libcephfs1-10.2.11-0.el7.x86_64.rpm libcephfs1-devel-10.2.11-0.el7.x86_64.rpm libcephfs_jni1-10.2.11-0.el7.x86_64.rpm libcephfs_jni1-devel-10.2.11-0.el7.x86_64.rpm librados2-10.2.11-0.el7.x86_64.rpm librados2-devel-10.2.11-0.el7.x86_64.rpm libradosstriper1-10.2.11-0.el7.x86_64.rpm libradosstriper1-devel-10.2.11-0.el7.x86_64.rpm librbd1-10.2.11-0.el7.x86_64.rpm librbd1-devel-10.2.11-0.el7.x86_64.rpm librgw2-10.2.11-0.el7.x86_64.rpm librgw2-devel-10.2.11-0.el7.x86_64.rpm python-ceph-compat-10.2.11-0.el7.x86_64.rpm python-cephfs-10.2.11-0.el7.x86_64.rpm python-rados-10.2.11-0.el7.x86_64.rpm python-rbd-10.2.11-0.el7.x86_64.rpm rbd-fuse-10.2.11-0.el7.x86_64.rpm rbd-mirror-10.2.11-0.el7.x86_64.rpm rbd-nbd-10.2.11-0.el7.x86_64.rpm
（3）将下载好的rpm拷贝到所有节点，并安装。注意ceph-deploy-xxxxx.noarch.rpm 只有部署节点用到，其他节点不需要，部署节点也需要安装其余的rpm
（4）在部署节点（cent用户下执行）：安装 ceph-deploy，在root用户下，进入下载好的rpm包目录，执行：
yum localinstall -y ./*注意：如遇到如下报错：
处理办法:
#安装依赖包python-distribute
#将这个源移走rdo-release-yunwei.repo
#在安装ceph-deploy-1.5.39-0.noarch.rpmyum localinstall ceph-deploy-1.5.39-0.noarch.rpm -y
#查看版本：ceph -v
（5）在部署节点（cent用户下执行）：配置新集群
ceph-deploy newcontroller compute storage vim ./ceph.conf #添加： osd_pool_default_size = 2
可选参数如下：public_network = 192.168.254.0/24cluster_network = 172.16.254.0/24osd_pool_default_size = 3osd_pool_default_min_size = 1osd_pool_default_pg_num = 8osd_pool_default_pgp_num = 8osd_crush_chooseleaf_type = 1 [mon] mon_clock_drift_allowed = 0.5 [osd] osd_mkfs_type =xfs osd_mkfs_options_xfs = -f filestore_max_sync_interval = 5filestore_min_sync_interval = 0.1filestore_fd_cache_size = 655350filestore_omap_header_cache_size = 655350filestore_fd_cache_random = trueosd op threads = 8osd disk threads = 4filestore op threads = 8max_open_files = 655350
（6）在部署节点执行（cent用户下执行）：所有节点安装ceph软件
所有节点有如下软件包：
root@rab116:13:59~/cephjrpm#ls ceph-10.2.11-0.el7.x86_64.rpm ceph-resource-agents-10.2.11-0.el7.x86_64.rpm librbd1-10.2.11-0.el7.x86_64.rpm ceph-base-10.2.11-0.el7.x86_64.rpm ceph-selinux-10.2.11-0.el7.x86_64.rpm librbd1-devel-10.2.11-0.el7.x86_64.rpm ceph-common-10.2.11-0.el7.x86_64.rpm ceph-test-10.2.11-0.el7.x86_64.rpm librgw2-10.2.11-0.el7.x86_64.rpm ceph-devel-compat-10.2.11-0.el7.x86_64.rpm libcephfs1-10.2.11-0.el7.x86_64.rpm librgw2-devel-10.2.11-0.el7.x86_64.rpm cephfs-java-10.2.11-0.el7.x86_64.rpm libcephfs1-devel-10.2.11-0.el7.x86_64.rpm python-ceph-compat-10.2.11-0.el7.x86_64.rpm ceph-fuse-10.2.11-0.el7.x86_64.rpm libcephfs_jni1-10.2.11-0.el7.x86_64.rpm python-cephfs-10.2.11-0.el7.x86_64.rpm ceph-libs-compat-10.2.11-0.el7.x86_64.rpm libcephfs_jni1-devel-10.2.11-0.el7.x86_64.rpm python-rados-10.2.11-0.el7.x86_64.rpm ceph-mds-10.2.11-0.el7.x86_64.rpm librados2-10.2.11-0.el7.x86_64.rpm python-rbd-10.2.11-0.el7.x86_64.rpm ceph-mon-10.2.11-0.el7.x86_64.rpm librados2-devel-10.2.11-0.el7.x86_64.rpm rbd-fuse-10.2.11-0.el7.x86_64.rpm ceph-osd-10.2.11-0.el7.x86_64.rpm libradosstriper1-10.2.11-0.el7.x86_64.rpm rbd-mirror-10.2.11-0.el7.x86_64.rpm ceph-radosgw-10.2.11-0.el7.x86_64.rpm libradosstriper1-devel-10.2.11-0.el7.x86_64.rpm rbd-nbd-10.2.11-0.el7.x86_64.rpm
所有节点安装上述软件包（包括客户端）：
yum localinstall ./*-y
（7）在部署节点执行，所有节点安装ceph软件
ceph-deploy install dlp controller compute storage
（8）在部署节点初始化集群（cent用户下执行）：
ceph-deploy mon create-initial
（9）每个节点将第二块硬盘做分区（注意，存储节点的磁盘是sdc）
fdisk /dev/sdb
列出节点磁盘
ceph-deploy disk list controller
擦净节点磁盘
ceph-deploy disk zap controller:/dev/sdb1
（10）准备Object Storage Daemon：
ceph-deploy osd prepare controller:/dev/sdb1 compute:/dev/sdb1 storage:/dev/sdb1
（11）激活Object Storage Daemon：
ceph-deploy osd activate controller:/dev/sdb1 compute:/dev/sdb1 storage:/dev/sdb1
（12）在部署节点transfer config files
ceph-deploy admin dlp controller compute storage
(每个节点做)sudo chmod 644 /etc/ceph/ceph.client.admin.keyring
（13）在ceph集群中任意节点检测：
ceph -s
三、客户端设置：
（1）客户端也要有cent用户：
useradd cent && echo "123" | passwd --stdin cent
echo-e 'Defaults:cent !requiretty cent ALL = (root) NOPASSWD:ALL' | tee /etc/sudoers.d/ceph
chmod440 /etc/sudoers.d/ceph
在部署节点执行，安装ceph客户端及设置：
ceph-deploy install controller
ceph-deploy admin controller（2）客户端执行
sudo chmod 644 /etc/ceph/ceph.client.admin.keyring（3）客户端执行，块设备rdb配置：
创建rbd：rbd create disk01（起的名字） --size 10G（指定rdb大小） --image-feature layering删除：rbd rm disk01（rdb名字）列示rbd：rbd ls –l现在只是创建好了一块10G的硬盘，要想用的话，还需要映射
映射rbd的image map：sudo rbd map disk01取消映射：sudo rbd unmap disk01显示map：rbd showmapped映射完就可以用lsblk查看到这块硬盘了，但还需要格式化挂载才可以用
格式化disk01文件系统xfs：sudo mkfs.xfs /dev/rbd0挂载硬盘：sudo mount /dev/rbd0 /mnt验证是否挂着成功：df -hT如果不想用这块硬盘了，怎么办？
umonut /dev/rbd0 /mnt
sudo rbd unmap disk01
lsblk
rbd rm disk01（4）File System配置：
在部署节点执行，选择一个node来创建MDS：
ceph-deploy mds create node1
以下操作在node1上执行：
sudo chmod 644 /etc/ceph/ceph.client.admin.keyring在MDS节点node1上创建 cephfs_data 和 cephfs_metadata 的 pool
ceph osd pool create cephfs_data 128 ceph osd pool create cephfs_metadata 128
开启pool：
ceph fs new cephfs cephfs_metadata cephfs_data显示ceph fs：ceph fs ls
ceph mds stat以下操作在客户端执行，安装ceph-fuse：
yum -y install ceph-fuse
获取admin key：
sshcent@node1"sudo ceph-authtool -p /etc/ceph/ceph.client.admin.keyring" >admin.key chmod600 admin.key
挂载ceph-fs：
mount-t ceph node1:6789:/ /mnt -o name=admin,secretfile=admin.key df-h
停止ceph-mds服务：
systemctl stop ceph-mds@node1
ceph mds fail 0
ceph fs rm cephfs --yes-i-really-mean-it
ceph osd lspools
显示结果：0 rbd,1 cephfs_data,2cephfs_metadata,
ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it
四、删除环境：
ceph-deploy purge dlp node1 node2 node3 controller
ceph-deploy purgedata dlp node1 node2 node3 controller
ceph-deploy forgetkeys
rm -rf ceph*