Ubuntu 22.04 on all nodes
Let’s prepare our network environment to deploy Ceph without the Internet
We need to create 6 nodes:
- 3 for ceph installation (ceph1, ceph2, ceph3)
- 1 for apt repository (aptly)
- 1 for docker registry (harbor)
- 1 for dns server (infra)
Also we need set up 3 networks:
Ceph Internal to connect only ceph nodes to each otherInternal to connect ceph, aptly and harbor nodesvLan to connect aptly and harbor nodes to the internet
And we need a dns server to resolve names connected with ip pools:
ceph.internal.lan : 172.16.1.0/24 ::: only for ceph nodesinternal.lan : 172.16.2.0/24 ::: for internal interfaces of all nodesvlan.lan : 192.168.137.0/24 ::: for external interfaces with the Internet access
Final structure in yaml format:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
|
nodes:
ceph1:
interfaces:
eth0:
name: ceph1.internal.lan
ip: 172.16.1.1
eth1:
name: ceph1.ceph.internal.lan
ip: 172.16.2.1
ceph2:
interfaces:
eth0:
name: ceph2.internal.lan
ip: 172.16.1.2
eth1:
name: ceph2.ceph.internal.lan
ip: 172.16.2.2
ceph3:
interfaces:
eth0:
name: ceph3.internal.lan
ip: 172.16.1.3
eth1:
name: ceph3.ceph.internal.lan
ip: 172.16.2.3
aptly:
interfaces:
eth0:
name: aptly.internal.lan
ip: 172.16.1.101
eth1:
name: aptly.vlan.lan
ip: 192.168.137.101
harbor:
interfaces:
eth0:
name: harbor.internal.lan
ip: 172.16.1.102
eth1:
name: harbor.vlan.lan
ip: 192.168.137.102
infra:
eth0:
name: bind.internal.lan
ip: 172.16.1.103
eth1:
name: bind.vlan.lan
ip: 192.168.137.103
|
Listing of netplan to configure inerfaces on nodes:
Apply 0600 to all files in /etc/netplan/
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
|
nodes:
ceph1:
/etc/netplan/ceph.yaml: |
network:
ethernets:
eth0:
addresses:
- 172.16.1.1/24
nameservers:
search:
- internal.lan
- ceph.internal.lan
addresses:
- 172.16.1.103
eth1:
addresses:
- 172.16.2.1/24
version: 2
ceph2:
/etc/netplan/ceph.yaml: |
network:
ethernets:
eth0:
addresses:
- 172.16.1.2/24
nameservers:
search:
- internal.lan
- ceph.internal.lan
addresses:
- 172.16.1.103
eth1:
addresses:
- 172.16.2.2/24
version: 2
ceph3:
/etc/netplan/ceph.yaml: |
network:
ethernets:
eth0:
addresses:
- 172.16.1.3/24
nameservers:
search:
- internal.lan
- ceph.internal.lan
addresses:
- 172.16.1.103
eth1:
addresses:
- 172.16.2.3/24
version: 2
aptly:
/etc/netplan/aptly.yaml: |
network:
ethernets:
eth0:
addresses:
- 172.16.1.101/24
nameservers:
search:
- internal.lan
- ceph.internal.lan
addresses:
- 172.16.1.103
eth1:
dhcp4: true
version: 2
harbor:
/etc/netplan/harbor.yaml: |
network:
ethernets:
eth0:
addresses:
- 172.16.1.102/24
nameservers:
search:
- internal.lan
- ceph.internal.lan
addresses:
- 172.16.1.103
eth1:
dhcp4: true
version: 2
infra:
/etc/netplan/infra.yaml: |
network:
ethernets:
eth0:
addresses:
- 172.16.1.103/24
nameservers:
search:
- internal.lan
- ceph.internal.lan
addresses:
- 172.16.1.103
eth1:
dhcp4: true
version: 2
|
Install on infra bind with two dns-zones:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
|
/var/lib/bind/internal.lan: |
$ORIGIN internal.lan.
$TTL 60
@ IN SOA infra.internal.lan. root.localhost. (
1 ; Serial
10800 ; Refresh
900 ; Retry
604800 ; Expire
86400 ) ; Negative Cache TTL
; Name Server Information
; @ = name of zone from named.conf.local
@ IN 3600 NS ns1.internal.lan.
@ IN 3600 NS ns2.internal.lan.
; IP address of Name Server
ns1 IN 3600 A 172.16.1.103
ns2 IN 3600 A 172.16.1.103
ceph1 IN A 172.16.1.1
ceph2 IN A 172.16.1.2
ceph3 IN A 172.16.1.3
ceph IN A 172.16.1.4
s3 IN A 172.16.1.5
aptly IN A 172.16.1.101
harbor IN A 172.16.1.102
infra IN A 172.16.1.103
/var/lib/bind/ceph.internal.lan : |
$ORIGIN ceph.internal.lan.
$TTL 60
@ IN SOA infra.ceph.internal.lan. root.localhost. (
1 ; Serial
10800 ; Refresh
900 ; Retry
604800 ; Expire
86400 ) ; Negative Cache TTL
; Name Server Information
; @ = name of zone from named.conf.local
@ IN 3600 NS ns1.ceph.internal.lan.
@ IN 3600 NS ns2.ceph.internal.lan.
; IP address of Name Server
ns1 IN 3600 A 172.16.1.103
ns2 IN 3600 A 172.16.1.103
ceph1 IN A 172.16.2.1
ceph2 IN A 172.16.2.2
ceph3 IN A 172.16.2.3
|
Generate on infra root CA and certificates on node infra
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
|
mkdr ~/tls; cd ~/tls
SUBJ="/CN=Lan trust issuer"
openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -key ca.key \
-sha256 -days 1024 -subj "$SUBJ" -out ca.crt
DOMAIN=harbor.internal.lan
openssl req -new -newkey rsa:4096 -sha256 -nodes \
-keyout "$DOMAIN.key" -subj "/CN=$DOMAIN" -out "$DOMAIN.csr"
cat <<- EOF | tee ./additional.info
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @dns_names
[dns_names]
DNS.1 = $DOMAIN
EOF
openssl x509 -req -in "$DOMAIN.csr" -CA ca.crt \
-CAkey ca.key -CAcreateserial -out "$DOMAIN.crt" \
-days 365 -sha256 -extfile ./additional.info
# Repeat for aptly.internal.lan and check
openssl verify -CAfile ca.crt harbor.internal.lan.crt
openssl verify -CAfile ca.crt aptly.internal.lan.crt
|
Start our own apt repo on node aptly
Install aptly:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
|
echo "deb [signed-by=/etc/apt/keyrings/aptly.asc] http://repo.aptly.info/ squeeze main" | sudo tee /etc/apt/sources.list.d/aptly.list
sudo mkdir -p /etc/apt/keyrings; sudo chmod 755 /etc/apt/keyrings
sudo wget -O /etc/apt/keyrings/aptly.asc https://www.aptly.info/pubkey.txt
sudo apt update && sudo apt install aptly
cat <<EOF | sudo tee /etc/aptly.conf
{
"rootDir": "/opt/aptly",
"downloadConcurrency": 4,
"downloadSpeedLimit": 0,
"architectures": ["amd64"],
"dependencyFollowSuggests": false,
"dependencyFollowRecommends": false,
"dependencyFollowAllVariants": false,
"dependencyFollowSource": false,
"dependencyVerboseResolve": false,
"gpgDisableSign": false,
"gpgDisableVerify": false,
"gpgProvider": "gpg",
"downloadSourcePackages": true,
"skipLegacyPool": true,
"ppaDistributorID": "ubuntu",
"ppaCodename": "",
"FileSystemPublishEndpoints": {
"on-premises-repos": {
"rootDir": "/var/www/aptly",
"linkMethod": "symlink",
"verifyMethod": "md5"
}
},
"enableMetricsEndpoint": false
}
EOF
sudo aptly repo create -comment="ceph repo" -component="main" -distribution="jammy" debian-squid
sudo aptly repo create -comment="shared repo" -component="main" -distribution="jammy" shared
sudo apt-get install -y rng-tools
sudo rngd -r /dev/urandom
sudo gpg --default-new-key-algo rsa8192 --gen-key --keyring pubring
sudo gpg --list-keys
sudo aptly publish repo shared filesystem:on-premises-repos:shared
sudo aptly publish repo debian-squid filesystem:on-premises-repos:debian-squid
sudo gpg --export --armor | sudo tee /var/www/aptly/shared/on-premises-repos.asc > /dev/null
|
Also add tar with preflight repository:
1
2
|
git clone https://github.com/ceph/cephadm-ansible
sudo tar zcf /var/www/aptly/cephadm-ansible.tar.gz cephadm-ansible
|
Start nginx to publish apt repository:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
|
sudo apt install nginx
sudo systemctl enable --now nginx
mkdir ~/tls
scp infra.internal.lan:~/tls/{ca.crt,aptly.internal.lan.key,aptly.internal.lan.crt} tls/
sudo cp ~/tls/{aptly.internal.lan.key,aptly.internal.lan.crt} /etc/nginx/
sudo cp ~/tls/ca.crt /var/www/aptly
cat <<EOF | sudo tee /etc/nginx/sites-enabled/default
server {
listen 80;
server_name aptly.internal.lan;
location / {
return 301 https://aptly.internal.lan$request_uri;
}
}
server {
listen 443 ssl default_server;
root /var/www/aptly;
index index.html index.htm index.nginx-debian.html;
server_name aptly.internal.lan;
ssl_certificate aptly.internal.lan.crt;
ssl_certificate_key aptly.internal.lan.key;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!aNULL:!MD5;
location / {
autoindex on;
}
}
EOF
sudo nginx -t
sudo nginx -s reload
|
Put necessary packages in shared repo:
1
2
3
4
5
6
|
sudo apt install apt-rdepends
sudo rm -f /var/cache/apt/archives/*.deb
PACKAGES_TO_INSTALL=("ansible" "jq" "chrony" "podman" "lvm2" "chrony" "librbd1" "python3-cephfs" "python3-rados" "python3-rbd" "python3-rgw" "libbabeltrace1" "libcephfs2" "liboath0" "librabbitmq4" "librados2" "libradosstriper1" "librdkafka1" "libsnappy1v5" "libthrift-0.16.0" "python3-prettytable" "libgoogle-perftools4" "liblua5.3-0" "liblttng-ust1" "liblttng-ust1" "liblua5.3-dev" "luarocks")
for pkg in `apt-rdepends ${PACKAGES_TO_INSTALL[@]} | grep -v "^ " | sed 's/debconf-2.0/debconf/g'`; do sudo apt install --reinstall --download-only --yes "$pkg"; done
sudo aptly repo add shared /var/cache/apt/archives/*.deb
sudo aptly publish update jammy filesystem:on-premises-repos:shared
|
Clone squid release of ceph repo and create a mirror:
1
2
3
4
5
6
7
8
9
10
11
12
13
|
sudo apt-get install apt-mirror
cat <<EOF | sudo tee /etc/apt/mirror.list
set base_path /media/debian-squid
set nthreads 20
set _tilde 0
deb https://download.ceph.com/debian-squid/ jammy main
EOF
sudo apt-mirror
sudo aptly repo add debian-squid /media/debian-squid/mirror/download.ceph.com/debian-squid/pool/main/c/ceph/*.deb
sudo aptly publish update jammy filesystem:on-premises-repos:debian-squid
|
Add ca.crt on each nodes
1
2
3
|
curl -LO https://aptly.internal.lan/ca.crt -k
sudo mv ca.crt /usr/local/share/ca-certificates
sudo update-ca-certificates
|
Run harbor on node harbor
Install docker
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
|
sudo useradd -s /bin/bash -G docker,sudo -m harbor
sudo -iu harbor
wget https://github.com/goharbor/harbor/releases/download/v2.11.2/harbor-online-installer-v2.11.2.tgz \
&& tar zxf harbor-online-installer-v2.11.2.tgz \
&& rm harbor-online-installer-v2.11.2.tgz \
&& cd harbor
scp infra.internal.lan:~/tls/{harbor.internal.lan.key,harbor.internal.lan.crt} ~harbor/harbor
cp harbor.yml.tmpl harbor.yml
cat << EOF > harbor.yml
hostname: harbor.internal.lan
http:
port: 80
https:
port: 443
certificate: /home/harbor/harbor/harbor.internal.lan.crt
private_key: /home/harbor/harbor/harbor.internal.lan.key
harbor_admin_password: Harbor12345
database:
password: root123
max_idle_conns: 100
max_open_conns: 900
conn_max_lifetime: 5m
conn_max_idle_time: 0
data_volume: /data
trivy:
ignore_unfixed: false
skip_update: false
skip_java_db_update: false
offline_scan: false
security_check: vuln
insecure: false
timeout: 5m0s
jobservice:
max_job_workers: 10
job_loggers:
- STD_OUTPUT
- FILE
logger_sweeper_duration: 1 #days
notification:
webhook_job_max_retry: 3
webhook_job_http_client_timeout: 3 #seconds
log:
level: info
local:
rotate_count: 50
rotate_size: 200M
location: /var/log/harbor
_version: 2.11.0
proxy:
http_proxy:
https_proxy:
no_proxy:
components:
- core
- jobservice
- trivy
upload_purging:
enabled: true
age: 168h
interval: 24h
dryrun: false
cache:
enabled: false
expire_hours: 24
EOF
sudo ./install.sh --with-trivy
|
Start communicating with internal.lan zone from another lan
Add into /etc/bind/named.conf.local on your dns-server:
1
2
3
4
5
6
|
...
zone "internal.lan" {
type slave;
notify no;
masters { 192.168.137.202; };
};
|
Check
Restart
1
|
sudo systemctl restart named
|
And get A-record:
Enable forwarding on infra:
1
2
3
|
sudo sysctl --write net.ipv4.ip_forward=1
echo "net.ipv4.ip_forward = 1" | sudo tee --append /etc/sysctl.conf
cat /proc/sys/net/ipv4/ip_forward
|
Add firewall rules on infra:
1
2
3
|
sudo iptables -A FORWARD -i eth0 -o eth1 -s 192.168.137.0/24 -m conntrack --ctstate NEW -j ACCEPT
sudo iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
sudo iptables -A POSTROUTING -t nat -j MASQUERADE
|
And add route on your client:
192.168.137.202 address of infra in your network
1
|
sudo ip route add 172.16.1.0/24 via 192.168.137.202
|
Add ca.crt
1
2
3
|
curl -LO https://aptly.internal.lan/ca.crt -k
sudo mv ca.crt /usr/local/share/ca-certificates/ca.internal.lan.crt
sudo update-ca-certificates
|
In linux we have to add ca.crt in browser using manual way!
- google chrome / brave: settings - privacy - security - manage certificates - authorities - import - chose ca.crt - restart brave
- firefox: Settings - Privacy & Security - View Certificates - Authorities - Import - chose ca.crt - restart firefox
Install ceph
On ceph1.internal.lan, ceph2.internal.lan and ceph3.internal.lan
Disable official repos:
1
2
|
# for ubuntu24 you need to delete file in /etc/apt/sources.list.d/
sudo sed -i 's/^deb/#> deb/g' /etc/apt/sources.list
|
Add gpg-key:
1
2
3
|
sudo mkdir -pv /etc/apt/keyrings
sudo chmod 755 /etc/apt/keyrings
curl https://aptly.internal.lan/on-premises-repos.asc | gpg --dearmor | sudo tee /etc/apt/keyrings/on-premises-repos.gpg > /dev/null
|
Add our repository
1
2
3
|
cat <<EOF | sudo tee /etc/apt/sources.list.d/on-premises-repos.list
deb [arch=amd64 signed-by=/etc/apt/keyrings/on-premises-repos.gpg] https://aptly.internal.lan/shared/ jammy main
EOF
|
Let’s install cluster from node ceph1.internal.lan
Clone ceph preflight and choose squid release:
1
2
3
4
5
6
7
|
ssh ceph1.internal.lan
curl -LO https://aptly.internal.lan/cephadm-ansible.tar.gz
tar -zxf cephadm-ansible.tar.gz
cd cephadm-ansible/
git checkout squid
sed -i 's/\$HOME\/ansible\/ansible\.log/\.\/ansible\.log/g' ansible.cfg
|
Install packages:
1
2
|
sudo apt update
sudo apt install ansible jq
|
Create inventory and gather facts:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
mkdir -pv inventory
cat <<EOF > inventory/hosts.yml
---
all:
hosts:
ceph1.internal.lan:
ceph2.internal.lan:
ceph3.internal.lan:
EOF
ansible -i inventory/hosts.yml all -m raw -a "hostname"
ansible -i inventory/hosts.yml all -m ansible.builtin.gather_facts --tree /tmp/facts
CEPH_RELEASE=squid
DISTRIBUTION_RELEASE=$(jq '.ansible_facts.ansible_distribution_release' /tmp/facts/ceph1.internal.lan | sed 's/"//g')
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/on-premises-repos.gpg] https://aptly.internal.lan/debian-$CEPH_RELEASE/ $DISTRIBUTION_RELEASE main"
|
Add this line on each ceph nodes:
1
|
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/on-premises-repos.gpg] https://aptly.internal.lan/debian-$CEPH_RELEASE/ $DISTRIBUTION_RELEASE main" | sudo tee --append /etc/apt/sources.list.d/on-premises-repos.list
|
Add vars.ini:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
|
cat <<EOF > vars.ini
---
ceph_origin: custom
custom_repo_gpgkey: https://aptly.internal.lan/on-premises-repos.asc
ceph_stable_key: https://aptly.internal.lan/on-premises-repos.asc
ceph_custom_repositories:
- name: ceph_custom
state: present
description: Ceph custom repo
gpgkey: https://aptly.internal.lan/on-premises-repos.asc
baseurl: https://aptly.internal.lan/debian-squid
file: debian-squid
components: main
priority: '2'
enabled: 1
- name: shared
state: present
description: Repo with dependencies
gpgkey: https://aptly.internal.lan/on-premises-repos.asc
baseurl: https://aptly.internal.lan/shared
file: shared
components: main
priority: '2'
enabled: 1
EOF
|
And run preflight playbook:
1
2
|
ansible-playbook -i inventory/hosts.yml cephadm-preflight.yml \
--extra-vars @vars.ini
|
Get images and put it into docker registry:
Create repo ceph in harbor before.
List of default images
Cephadm on github
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
|
docker pull quay.io/ceph/ceph:v19
docker pull quay.io/ceph/grafana:9.4.12
docker pull quay.io/prometheus/prometheus:v2.43.0
docker pull quay.io/prometheus/alertmanager:v0.25.0
docker pull quay.io/prometheus/node-exporter:v1.5.0
docker pull quay.io/ceph/keepalived:2.2.4
docker pull quay.io/ceph/haproxy:2.3
docker tag quay.io/ceph/ceph:v19 harbor.internal.lan/ceph/ceph:v19
docker tag quay.io/ceph/grafana:9.4.12 harbor.internal.lan/ceph/grafana:9.4.12
docker tag quay.io/prometheus/prometheus:v2.43.0 harbor.internal.lan/ceph/prometheus:v2.43.0
docker tag quay.io/prometheus/alertmanager:v0.25.0 harbor.internal.lan/ceph/alertmanager:v0.25.0
docker tag quay.io/prometheus/node-exporter:v1.5.0 harbor.internal.lan/ceph/node-exporter:v1.5.0
docker tag quay.io/ceph/keepalived:2.2.4 harbor.internal.lan/ceph/keepalived:2.2.4
docker tag quay.io/ceph/haproxy:2.3 harbor.internal.lan/ceph/haproxy:2.3
docker login https://harbor.internal.lan/ -u admin
docker push harbor.internal.lan/ceph/ceph:v19
docker push harbor.internal.lan/ceph/grafana:9.4.12
docker push harbor.internal.lan/ceph/prometheus:v2.43.0
docker push harbor.internal.lan/ceph/alertmanager:v0.25.0
docker push harbor.internal.lan/ceph/node-exporter:v1.5.0
docker push harbor.internal.lan/ceph/keepalived:2.2.4
docker push harbor.internal.lan/ceph/haproxy:2.3
|
Ensure that each node of ceph has fdqn:
1
2
3
|
sudo hostnamectl set-hostname ceph1.internal.lan
sudo hostnamectl set-hostname ceph2.internal.lan
sudo hostnamectl set-hostname ceph3.internal.lan
|
Create configs:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
|
mkdir ceph-squid
cd ceph-squid/
cat <<EOF > ceph-spec.yml
---
service_type: host
addr: 172.16.1.1
hostname: ceph1.internal.lan
---
service_type: host
addr: 172.16.1.2
hostname: ceph2.internal.lan
---
service_type: host
addr: 172.16.1.3
hostname: ceph3.internal.lan
---
service_type: mon
placement:
hosts:
- ceph1.internal.lan
- ceph2.internal.lan
- ceph3.internal.lan
---
service_type: rgw
service_id: rgw_zone
placement:
hosts:
- ceph1.internal.lan
- ceph2.internal.lan
- ceph3.internal.lan
---
service_type: mgr
placement:
hosts:
- ceph1.internal.lan
- ceph2.internal.lan
- ceph3.internal.lan
---
service_type: osd
service_id: default_drive_group
placement:
hosts:
- ceph1.internal.lan
- ceph2.internal.lan
- ceph3.internal.lan
data_devices:
paths:
- path: /dev/sdb
crush_device_class: ssd
EOF
# https://github.com/ceph/ceph/blob/main/src/sample.ceph.conf
cat <<EOF> ceph.conf
[global]
#All clusters have a front-side public network.
#If you have two network interfaces, you can configure a private / cluster
#network for RADOS object replication, heartbeats, backfill,
#recovery, etc.
public_network = 172.16.1.0/24
# cluster_network = {network}[, {network}]
cluster_network = 172.16.2.0/24
#Clusters require authentication by default.
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
#Choose reasonable number of replicas and placement groups.
osd_journal_size = 5120 # osd journal size = 2 * (expected throughput * filestore max sync interval. Example: expected throughput=1000 Mb/s, filestore max sync interval=5s)
osd_pool_default_size = 3 # Write an object n times.
osd_pool_default_min_size = 2 # Allow writing n copies in a degraded state.
osd_pool_default_pg_autoscale_mode = on # on, off, or warn
# Only used if autoscaling is off or warn:
osd_pool_default_pg_num = 128
#Choose a reasonable crush leaf type.
# 0 for a 1-node cluster.
# 1 for a multi node cluster in a single rack
# 2 for a multi node, multi chassis cluster with multiple hosts in a chassis
# 3 for a multi node cluster with hosts across racks, etc.
osd_crush_chooseleaf_type = 1
[mgr]
mgr/cephadm/container_image_prometheus = 'harbor.internal.lan/ceph/prometheus:v2.43.0'
mgr/cephadm/container_image_node_exporter = 'harbor.internal.lan/ceph/node-exporter:v1.5.0'
mgr/cephadm/container_image_grafana = 'harbor.internal.lan/ceph/grafana:9.4.12'
mgr/cephadm/container_image_alertmanager = 'harbor.internal.lan/ceph/alertmanager:v0.25.0'
EOF
|
And deploy ceph cluster:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
export MON_IP=172.16.1.1
export CLUSTER_NETWORK=172.16.2.0/24
export DASHBOARD_USER=admin
export DASHBOARD_PASS=yourstrongpass
export SSH_USER=goto
sudo cephadm --image harbor.internal.lan/ceph/ceph:v19 \
bootstrap --mon-ip "$MON_IP" \
--registry-url harbor.internal.lan/ceph \
--registry-username admin \
--registry-password Harbor12345 \
--ssh-user "$SSH_USER" \
--cluster-network "$CLUSTER_NETWORK" \
--initial-dashboard-user "$DASHBOARD_USER" \
--initial-dashboard-password "$DASHBOARD_PASS" \
--dashboard-password-noupdate \
--apply-spec ceph-spec.yml \
--allow-fqdn-hostname \
--config ceph.conf
|