Sergey Chulanov's site
Posts Tags Categories Series About me
Sergey Chulanov's site

Ceph on isolated environment

Sergey Chulanov included in Ceph Devops
   2078 words   10 minutes 
Contents

Ubuntu 22.04 on all nodes

Let’s prepare our network environment to deploy Ceph without the Internet

We need to create 6 nodes:

  • 3 for ceph installation (ceph1, ceph2, ceph3)
  • 1 for apt repository (aptly)
  • 1 for docker registry (harbor)
  • 1 for dns server (infra)

Also we need set up 3 networks:

  • Ceph Internal to connect only ceph nodes to each other
  • Internal to connect ceph, aptly and harbor nodes
  • vLan to connect aptly and harbor nodes to the internet

And we need a dns server to resolve names connected with ip pools:

  • ceph.internal.lan : 172.16.1.0/24 ::: only for ceph nodes
  • internal.lan : 172.16.2.0/24 ::: for internal interfaces of all nodes
  • vlan.lan : 192.168.137.0/24 ::: for external interfaces with the Internet access

Final structure in yaml format:

nodes:
  ceph1:
    interfaces:
      eth0:
        name: ceph1.internal.lan
        ip: 172.16.1.1
      eth1:
        name: ceph1.ceph.internal.lan
        ip: 172.16.2.1
  ceph2:
    interfaces:
      eth0:
        name: ceph2.internal.lan
        ip: 172.16.1.2
      eth1:
        name: ceph2.ceph.internal.lan
        ip: 172.16.2.2
  ceph3:
    interfaces:
      eth0:
        name: ceph3.internal.lan
        ip: 172.16.1.3
      eth1:
        name: ceph3.ceph.internal.lan
        ip: 172.16.2.3
  aptly:
    interfaces:
      eth0:
        name: aptly.internal.lan
        ip: 172.16.1.101
      eth1:
        name: aptly.vlan.lan
        ip: 192.168.137.101
  harbor:
    interfaces:
      eth0:
        name: harbor.internal.lan
        ip: 172.16.1.102
      eth1:
        name: harbor.vlan.lan
        ip: 192.168.137.102
  infra:
      eth0:
        name: bind.internal.lan
        ip: 172.16.1.103
      eth1:
        name: bind.vlan.lan
        ip: 192.168.137.103

Listing of netplan to configure inerfaces on nodes:

Apply 0600 to all files in /etc/netplan/

nodes:
  ceph1:
    /etc/netplan/ceph.yaml: |
      network:
        ethernets:
          eth0:
            addresses:
              - 172.16.1.1/24
            nameservers:
              search:
                - internal.lan
                - ceph.internal.lan
              addresses:
                - 172.16.1.103
          eth1:
            addresses:
            - 172.16.2.1/24
        version: 2
  ceph2:
    /etc/netplan/ceph.yaml: |
      network:
        ethernets:
          eth0:
            addresses:
              - 172.16.1.2/24
            nameservers:
              search:
                - internal.lan
                - ceph.internal.lan
              addresses:
                - 172.16.1.103
          eth1:
            addresses:
            - 172.16.2.2/24
        version: 2
  ceph3:
    /etc/netplan/ceph.yaml: |
      network:
        ethernets:
          eth0:
            addresses:
              - 172.16.1.3/24
            nameservers:
              search:
                - internal.lan
                - ceph.internal.lan
              addresses:
                - 172.16.1.103
          eth1:
            addresses:
            - 172.16.2.3/24
        version: 2
  aptly:
    /etc/netplan/aptly.yaml: |
      network:
        ethernets:
          eth0:
            addresses:
              - 172.16.1.101/24
            nameservers:
              search:
                - internal.lan
                - ceph.internal.lan
              addresses:
                - 172.16.1.103
          eth1:
            dhcp4: true
        version: 2
  harbor:
    /etc/netplan/harbor.yaml: |
      network:
        ethernets:
          eth0:
            addresses:
              - 172.16.1.102/24
            nameservers:
              search:
                - internal.lan
                - ceph.internal.lan
              addresses:
                - 172.16.1.103
          eth1:
            dhcp4: true
        version: 2
  infra:
    /etc/netplan/infra.yaml: |
      network:
        ethernets:
          eth0:
            addresses:
              - 172.16.1.103/24
            nameservers:
              search:
                - internal.lan
                - ceph.internal.lan
              addresses:
                - 172.16.1.103
          eth1:
            dhcp4: true
        version: 2

Install on infra bind with two dns-zones:

/var/lib/bind/internal.lan: |
  $ORIGIN internal.lan.
  $TTL    60
  @       IN      SOA     infra.internal.lan. root.localhost. (
                                1         ; Serial
                            10800         ; Refresh
                              900         ; Retry
                           604800         ; Expire
                            86400 )       ; Negative Cache TTL
  
  ; Name Server Information
  ; @ = name of zone from named.conf.local
  
  @       IN 3600 NS  ns1.internal.lan.
  @       IN 3600 NS  ns2.internal.lan.
  
  ; IP address of Name Server
  
  ns1     IN 3600 A 172.16.1.103
  ns2     IN 3600 A 172.16.1.103
  
  ceph1   IN      A 172.16.1.1
  ceph2   IN      A 172.16.1.2
  ceph3   IN      A 172.16.1.3
  ceph    IN      A 172.16.1.4
  s3      IN      A 172.16.1.5
  aptly   IN      A 172.16.1.101
  harbor  IN      A 172.16.1.102
  infra   IN      A 172.16.1.103
/var/lib/bind/ceph.internal.lan : |
  $ORIGIN ceph.internal.lan.
  $TTL    60
  @       IN      SOA     infra.ceph.internal.lan. root.localhost. (
                                1         ; Serial
                            10800         ; Refresh
                              900         ; Retry
                           604800         ; Expire
                            86400 )       ; Negative Cache TTL
  
  ; Name Server Information
  ; @ = name of zone from named.conf.local
  
  @        IN 3600 NS ns1.ceph.internal.lan.
  @        IN 3600 NS ns2.ceph.internal.lan.
  
  ; IP address of Name Server
  
  ns1     IN 3600 A 172.16.1.103
  ns2     IN 3600 A 172.16.1.103
  
  ceph1 IN  A 172.16.2.1
  ceph2 IN  A 172.16.2.2
  ceph3 IN  A 172.16.2.3

Generate on infra root CA and certificates on node infra

mkdr ~/tls; cd ~/tls

SUBJ="/CN=Lan trust issuer"
openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -key ca.key \
  -sha256 -days 1024 -subj "$SUBJ" -out ca.crt

DOMAIN=harbor.internal.lan
openssl req -new -newkey rsa:4096 -sha256 -nodes \
  -keyout "$DOMAIN.key" -subj "/CN=$DOMAIN" -out "$DOMAIN.csr"

cat <<- EOF | tee ./additional.info
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @dns_names

[dns_names]
DNS.1 = $DOMAIN
EOF

openssl x509 -req -in "$DOMAIN.csr" -CA ca.crt \
  -CAkey ca.key -CAcreateserial -out "$DOMAIN.crt" \
  -days 365 -sha256 -extfile ./additional.info

# Repeat for aptly.internal.lan and check
openssl verify -CAfile ca.crt harbor.internal.lan.crt
openssl verify -CAfile ca.crt aptly.internal.lan.crt

Start our own apt repo on node aptly

Install aptly:

echo "deb [signed-by=/etc/apt/keyrings/aptly.asc] http://repo.aptly.info/ squeeze main" | sudo tee /etc/apt/sources.list.d/aptly.list
sudo mkdir -p /etc/apt/keyrings; sudo chmod 755 /etc/apt/keyrings
sudo wget -O /etc/apt/keyrings/aptly.asc https://www.aptly.info/pubkey.txt
sudo apt update && sudo apt install aptly

cat <<EOF | sudo tee /etc/aptly.conf
{
  "rootDir": "/opt/aptly",
  "downloadConcurrency": 4,
  "downloadSpeedLimit": 0,
  "architectures": ["amd64"],
  "dependencyFollowSuggests": false,
  "dependencyFollowRecommends": false,
  "dependencyFollowAllVariants": false,
  "dependencyFollowSource": false,
  "dependencyVerboseResolve": false,
  "gpgDisableSign": false,
  "gpgDisableVerify": false,
  "gpgProvider": "gpg",
  "downloadSourcePackages": true,
  "skipLegacyPool": true,
  "ppaDistributorID": "ubuntu",
  "ppaCodename": "",
  "FileSystemPublishEndpoints": {
    "on-premises-repos": {
      "rootDir": "/var/www/aptly",
      "linkMethod": "symlink",
      "verifyMethod": "md5"
    }
  },
  "enableMetricsEndpoint": false
}
EOF

sudo aptly repo create -comment="ceph repo" -component="main" -distribution="jammy" debian-squid
sudo aptly repo create -comment="shared repo" -component="main" -distribution="jammy" shared

sudo apt-get install -y rng-tools
sudo rngd -r /dev/urandom
sudo gpg --default-new-key-algo rsa8192 --gen-key --keyring pubring
sudo gpg --list-keys

sudo aptly publish repo shared filesystem:on-premises-repos:shared
sudo aptly publish repo debian-squid filesystem:on-premises-repos:debian-squid

sudo gpg --export --armor | sudo tee /var/www/aptly/shared/on-premises-repos.asc > /dev/null

Also add tar with preflight repository:

git clone https://github.com/ceph/cephadm-ansible
sudo tar zcf /var/www/aptly/cephadm-ansible.tar.gz cephadm-ansible

Start nginx to publish apt repository:

sudo apt install nginx
sudo systemctl enable --now nginx

mkdir ~/tls
scp infra.internal.lan:~/tls/{ca.crt,aptly.internal.lan.key,aptly.internal.lan.crt} tls/
sudo cp ~/tls/{aptly.internal.lan.key,aptly.internal.lan.crt} /etc/nginx/
sudo cp ~/tls/ca.crt /var/www/aptly

cat <<EOF | sudo tee /etc/nginx/sites-enabled/default
server {
  listen 80;
  server_name aptly.internal.lan;

  location / {
    return 301 https://aptly.internal.lan$request_uri;
  }
}

server {
  listen 443 ssl default_server;
  root /var/www/aptly;
  index index.html index.htm index.nginx-debian.html;
  server_name aptly.internal.lan;

    ssl_certificate   aptly.internal.lan.crt;
    ssl_certificate_key aptly.internal.lan.key;
    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
    ssl_ciphers         HIGH:!aNULL:!MD5;

  location / {
    autoindex on;
  }
}
EOF

sudo nginx -t
sudo nginx -s reload

Put necessary packages in shared repo:

sudo apt install apt-rdepends
sudo rm -f /var/cache/apt/archives/*.deb
PACKAGES_TO_INSTALL=("ansible" "jq" "chrony" "podman" "lvm2" "chrony" "librbd1" "python3-cephfs" "python3-rados" "python3-rbd" "python3-rgw" "libbabeltrace1" "libcephfs2" "liboath0" "librabbitmq4" "librados2" "libradosstriper1" "librdkafka1" "libsnappy1v5" "libthrift-0.16.0" "python3-prettytable" "libgoogle-perftools4" "liblua5.3-0" "liblttng-ust1" "liblttng-ust1" "liblua5.3-dev" "luarocks")
for pkg in `apt-rdepends ${PACKAGES_TO_INSTALL[@]} | grep -v "^ " | sed 's/debconf-2.0/debconf/g'`; do sudo apt install --reinstall --download-only --yes "$pkg"; done
sudo aptly repo add shared /var/cache/apt/archives/*.deb
sudo aptly publish update jammy filesystem:on-premises-repos:shared

Clone squid release of ceph repo and create a mirror:

sudo apt-get install apt-mirror

cat <<EOF | sudo tee /etc/apt/mirror.list
set base_path    /media/debian-squid
set nthreads     20
set _tilde 0
deb https://download.ceph.com/debian-squid/ jammy main
EOF

sudo apt-mirror

sudo aptly repo add debian-squid /media/debian-squid/mirror/download.ceph.com/debian-squid/pool/main/c/ceph/*.deb
sudo aptly publish update jammy filesystem:on-premises-repos:debian-squid

Add ca.crt on each nodes

curl -LO https://aptly.internal.lan/ca.crt -k
sudo mv ca.crt /usr/local/share/ca-certificates
sudo update-ca-certificates

Run harbor on node harbor

Install docker

sudo useradd -s /bin/bash -G docker,sudo -m harbor
sudo -iu harbor

wget https://github.com/goharbor/harbor/releases/download/v2.11.2/harbor-online-installer-v2.11.2.tgz \
  && tar zxf harbor-online-installer-v2.11.2.tgz \
  && rm harbor-online-installer-v2.11.2.tgz \
  && cd harbor

scp infra.internal.lan:~/tls/{harbor.internal.lan.key,harbor.internal.lan.crt} ~harbor/harbor
cp harbor.yml.tmpl harbor.yml

cat << EOF > harbor.yml
hostname: harbor.internal.lan
http:
  port: 80
https:
  port: 443
  certificate: /home/harbor/harbor/harbor.internal.lan.crt
  private_key: /home/harbor/harbor/harbor.internal.lan.key
harbor_admin_password: Harbor12345
database:
  password: root123
  max_idle_conns: 100
  max_open_conns: 900
  conn_max_lifetime: 5m
  conn_max_idle_time: 0
data_volume: /data
trivy:
  ignore_unfixed: false
  skip_update: false
  skip_java_db_update: false
  offline_scan: false
  security_check: vuln
  insecure: false
  timeout: 5m0s
jobservice:
  max_job_workers: 10
  job_loggers:
    - STD_OUTPUT
    - FILE
  logger_sweeper_duration: 1 #days
notification:
  webhook_job_max_retry: 3
  webhook_job_http_client_timeout: 3 #seconds
log:
  level: info
  local:
    rotate_count: 50
    rotate_size: 200M
    location: /var/log/harbor
_version: 2.11.0
proxy:
  http_proxy:
  https_proxy:
  no_proxy:
  components:
    - core
    - jobservice
    - trivy
upload_purging:
  enabled: true
  age: 168h
  interval: 24h
  dryrun: false
cache:
  enabled: false
  expire_hours: 24
EOF

sudo ./install.sh --with-trivy

Start communicating with internal.lan zone from another lan

Add into /etc/bind/named.conf.local on your dns-server:

...
zone "internal.lan" {
  type slave;
  notify no;
  masters { 192.168.137.202; };
};

Check

sudo named-checkconf

Restart

sudo systemctl restart named

And get A-record:

dig ceph1.internal.lan

Enable forwarding on infra:

sudo sysctl --write net.ipv4.ip_forward=1
echo "net.ipv4.ip_forward = 1" | sudo tee --append /etc/sysctl.conf
cat /proc/sys/net/ipv4/ip_forward

Add firewall rules on infra:

sudo iptables -A FORWARD -i eth0 -o eth1 -s 192.168.137.0/24 -m conntrack --ctstate NEW -j ACCEPT
sudo iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
sudo iptables -A POSTROUTING -t nat -j MASQUERADE

And add route on your client:

192.168.137.202 address of infra in your network

sudo ip route add 172.16.1.0/24 via 192.168.137.202

Add ca.crt

curl -LO https://aptly.internal.lan/ca.crt -k
sudo mv ca.crt /usr/local/share/ca-certificates/ca.internal.lan.crt
sudo update-ca-certificates

In linux we have to add ca.crt in browser using manual way!

  • google chrome / brave: settings - privacy - security - manage certificates - authorities - import - chose ca.crt - restart brave
  • firefox: Settings - Privacy & Security - View Certificates - Authorities - Import - chose ca.crt - restart firefox

Install ceph

On ceph1.internal.lan, ceph2.internal.lan and ceph3.internal.lan

Disable official repos:

# for ubuntu24 you need to delete file in /etc/apt/sources.list.d/
sudo sed -i 's/^deb/#> deb/g' /etc/apt/sources.list

Add gpg-key:

sudo mkdir -pv /etc/apt/keyrings
sudo chmod 755 /etc/apt/keyrings
curl https://aptly.internal.lan/on-premises-repos.asc | gpg --dearmor | sudo tee /etc/apt/keyrings/on-premises-repos.gpg > /dev/null

Add our repository

cat <<EOF | sudo tee /etc/apt/sources.list.d/on-premises-repos.list
deb [arch=amd64 signed-by=/etc/apt/keyrings/on-premises-repos.gpg] https://aptly.internal.lan/shared/ jammy main
EOF

Let’s install cluster from node ceph1.internal.lan

Clone ceph preflight and choose squid release:

ssh ceph1.internal.lan
curl -LO https://aptly.internal.lan/cephadm-ansible.tar.gz
tar -zxf cephadm-ansible.tar.gz
cd cephadm-ansible/
git checkout squid

sed -i 's/\$HOME\/ansible\/ansible\.log/\.\/ansible\.log/g' ansible.cfg

Install packages:

sudo apt update
sudo apt install ansible jq

Create inventory and gather facts:

mkdir -pv inventory
cat <<EOF > inventory/hosts.yml
---
all:
  hosts:
    ceph1.internal.lan:
    ceph2.internal.lan:
    ceph3.internal.lan:
EOF
ansible -i inventory/hosts.yml all -m raw -a "hostname"

ansible -i inventory/hosts.yml all -m ansible.builtin.gather_facts --tree /tmp/facts
CEPH_RELEASE=squid
DISTRIBUTION_RELEASE=$(jq '.ansible_facts.ansible_distribution_release' /tmp/facts/ceph1.internal.lan | sed 's/"//g')
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/on-premises-repos.gpg] https://aptly.internal.lan/debian-$CEPH_RELEASE/ $DISTRIBUTION_RELEASE main"

Add this line on each ceph nodes:

echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/on-premises-repos.gpg] https://aptly.internal.lan/debian-$CEPH_RELEASE/ $DISTRIBUTION_RELEASE main" | sudo tee --append /etc/apt/sources.list.d/on-premises-repos.list

Add vars.ini:

cat <<EOF > vars.ini
---
ceph_origin: custom
custom_repo_gpgkey: https://aptly.internal.lan/on-premises-repos.asc
ceph_stable_key: https://aptly.internal.lan/on-premises-repos.asc
ceph_custom_repositories:
  - name: ceph_custom
    state: present
    description: Ceph custom repo
    gpgkey: https://aptly.internal.lan/on-premises-repos.asc
    baseurl: https://aptly.internal.lan/debian-squid
    file: debian-squid
    components: main
    priority: '2'
    enabled: 1
  - name: shared
    state: present
    description: Repo with dependencies
    gpgkey: https://aptly.internal.lan/on-premises-repos.asc
    baseurl: https://aptly.internal.lan/shared
    file: shared
    components: main
    priority: '2'
    enabled: 1
EOF

And run preflight playbook:

ansible-playbook -i inventory/hosts.yml cephadm-preflight.yml \
  --extra-vars @vars.ini

Get images and put it into docker registry:

Create repo ceph in harbor before.
List of default images Cephadm on github

docker pull quay.io/ceph/ceph:v19
docker pull quay.io/ceph/grafana:9.4.12
docker pull quay.io/prometheus/prometheus:v2.43.0
docker pull quay.io/prometheus/alertmanager:v0.25.0
docker pull quay.io/prometheus/node-exporter:v1.5.0
docker pull quay.io/ceph/keepalived:2.2.4
docker pull quay.io/ceph/haproxy:2.3

docker tag quay.io/ceph/ceph:v19 harbor.internal.lan/ceph/ceph:v19
docker tag quay.io/ceph/grafana:9.4.12 harbor.internal.lan/ceph/grafana:9.4.12
docker tag quay.io/prometheus/prometheus:v2.43.0 harbor.internal.lan/ceph/prometheus:v2.43.0
docker tag quay.io/prometheus/alertmanager:v0.25.0 harbor.internal.lan/ceph/alertmanager:v0.25.0
docker tag quay.io/prometheus/node-exporter:v1.5.0 harbor.internal.lan/ceph/node-exporter:v1.5.0
docker tag quay.io/ceph/keepalived:2.2.4 harbor.internal.lan/ceph/keepalived:2.2.4
docker tag quay.io/ceph/haproxy:2.3 harbor.internal.lan/ceph/haproxy:2.3

docker login https://harbor.internal.lan/ -u admin

docker push harbor.internal.lan/ceph/ceph:v19
docker push harbor.internal.lan/ceph/grafana:9.4.12
docker push harbor.internal.lan/ceph/prometheus:v2.43.0
docker push harbor.internal.lan/ceph/alertmanager:v0.25.0
docker push harbor.internal.lan/ceph/node-exporter:v1.5.0
docker push harbor.internal.lan/ceph/keepalived:2.2.4
docker push harbor.internal.lan/ceph/haproxy:2.3

Ensure that each node of ceph has fdqn:

sudo hostnamectl set-hostname ceph1.internal.lan
sudo hostnamectl set-hostname ceph2.internal.lan
sudo hostnamectl set-hostname ceph3.internal.lan

Create configs:

mkdir ceph-squid
cd ceph-squid/

cat <<EOF > ceph-spec.yml
---
service_type: host
addr: 172.16.1.1
hostname: ceph1.internal.lan
---
service_type: host
addr:  172.16.1.2
hostname: ceph2.internal.lan
---
service_type: host
addr:  172.16.1.3
hostname: ceph3.internal.lan
---
service_type: mon
placement:
  hosts:
    - ceph1.internal.lan
    - ceph2.internal.lan
    - ceph3.internal.lan
---
service_type: rgw
service_id: rgw_zone
placement:
  hosts:
    - ceph1.internal.lan
    - ceph2.internal.lan
    - ceph3.internal.lan
---
service_type: mgr
placement:
  hosts:
    - ceph1.internal.lan
    - ceph2.internal.lan
    - ceph3.internal.lan
---
service_type: osd
service_id: default_drive_group
placement:
  hosts:
    - ceph1.internal.lan
    - ceph2.internal.lan
    - ceph3.internal.lan
data_devices:
  paths:
    - path: /dev/sdb
      crush_device_class: ssd
EOF

# https://github.com/ceph/ceph/blob/main/src/sample.ceph.conf
cat <<EOF> ceph.conf
[global]
  #All clusters have a front-side public network.
  #If you have two network interfaces, you can configure a private / cluster 
  #network for RADOS object replication, heartbeats, backfill,
  #recovery, etc.
  public_network = 172.16.1.0/24
  # cluster_network = {network}[, {network}] 
  cluster_network = 172.16.2.0/24
  
  #Clusters require authentication by default.
  auth_cluster_required = cephx
  auth_service_required = cephx
  auth_client_required = cephx
  
  #Choose reasonable number of replicas and placement groups.
  osd_journal_size = 5120 # osd journal size = 2 * (expected throughput * filestore max sync interval. Example: expected throughput=1000 Mb/s, filestore max sync interval=5s)
  osd_pool_default_size = 3  # Write an object n times.
  osd_pool_default_min_size = 2 # Allow writing n copies in a degraded state.
  osd_pool_default_pg_autoscale_mode = on # on, off, or warn
  # Only used if autoscaling is off or warn:
  osd_pool_default_pg_num = 128
  
  #Choose a reasonable crush leaf type.
  # 0 for a 1-node cluster.
  # 1 for a multi node cluster in a single rack
  # 2 for a multi node, multi chassis cluster with multiple hosts in a chassis
  # 3 for a multi node cluster with hosts across racks, etc.
  osd_crush_chooseleaf_type = 1

[mgr]
  mgr/cephadm/container_image_prometheus = 'harbor.internal.lan/ceph/prometheus:v2.43.0'
  mgr/cephadm/container_image_node_exporter = 'harbor.internal.lan/ceph/node-exporter:v1.5.0'
  mgr/cephadm/container_image_grafana = 'harbor.internal.lan/ceph/grafana:9.4.12'
  mgr/cephadm/container_image_alertmanager = 'harbor.internal.lan/ceph/alertmanager:v0.25.0'
EOF

And deploy ceph cluster:

export MON_IP=172.16.1.1
export CLUSTER_NETWORK=172.16.2.0/24
export DASHBOARD_USER=admin
export DASHBOARD_PASS=yourstrongpass
export SSH_USER=goto

sudo cephadm --image harbor.internal.lan/ceph/ceph:v19 \
  bootstrap --mon-ip "$MON_IP" \
  --registry-url harbor.internal.lan/ceph \
  --registry-username admin \
  --registry-password Harbor12345 \
  --ssh-user "$SSH_USER" \
  --cluster-network "$CLUSTER_NETWORK" \
  --initial-dashboard-user "$DASHBOARD_USER" \
  --initial-dashboard-password "$DASHBOARD_PASS" \
  --dashboard-password-noupdate \
  --apply-spec ceph-spec.yml \
  --allow-fqdn-hostname \
  --config ceph.conf
Updated on 2026-09-16
Read Markdown
 Ceph
Back | Home