Longhorn V2 Data Engine を試してみる
takeharak
K3s で利用している分散ブロックストレージシステム Longhorn の V2 Data Engine (Preview Feature) が v1.6.0 リリースで ARM64 に対応してたので Raspberry Pi 5 で試してみる
今回の環境
TL;DR
１． 環境設定
# /boot/firmware/cmdline.txt
- console=serial0,115200 multipath=off dwc_otg.lpm_enable=0 console=tty1 root=LABEL=writable rootfstype=ext4 rootwait fixrtc
+ console=serial0,115200 multipath=off dwc_otg.lpm_enable=0 console=tty1 root=LABEL=writable rootfstype=ext4 rootwait fixrtc cgroup_memory=1 cgroup_enable=memory
sudo apt update
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
70 packages can be upgraded. Run 'apt list --upgradable' to see them.
sudo apt full-upgrade -y
...
Restarting services...
/etc/needrestart/restart.d/systemd-manager
systemctl restart avahi-daemon.service cron.service polkit.service ssh.(out)service systemd-journald.service systemd-networkd.service systemd-resolved.(out)service systemd-timesyncd.service systemd-udevd.service udisks2.service
Service restarts being deferred:
systemctl restart ModemManager.service
/etc/needrestart/restart.d/dbus.service
systemctl restart systemd-logind.service
systemctl restart unattended-upgrades.service
systemctl restart wpa_supplicant.service
No containers need to be restarted.
User sessions running outdated binaries:
ubuntu @ session #1: apt[2184], sshd[1751]
ubuntu @ user manager service: systemd[1756]
No VM guests are running outdated hypervisor (qemu) binaries on this host.
sudo reboot
Broadcast message from root@k3s on pts/1 (Thu 2024-09-19 15:12:37 UTC):
The system will reboot now!
２． K3s のインストール
curl -sfL https://get.k3s.io | sh -
[INFO] Finding release for channel stable
[INFO] Using v1.30.4+k3s1 as release
[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.30.4+k3s1/sha256sum-arm64.txt
[INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.30.4+k3s1/k3s-arm64
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Skipping installation of SELinux RPM
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
k3s kubectl get node
NAME STATUS ROLES AGE VERSION
k3s Ready control-plane,master 10s v1.30.4+k3s1
３．Longhorn 依存関係の設定
for Longhorn
sudo apt install -y nfs-common open-iscsi
...
Processing triggers for man-db (2.12.0-4build2) ...
Processing triggers for initramfs-tools (0.142ubuntu25.2) ...
update-initramfs: Generating /boot/initrd.img-6.8.0-1011-raspi
Using DTB: bcm2712-rpi-5-b.dtb
Installing /lib/firmware/6.8.0-1011-raspi/device-tree/broadcom/bcm2712-rpi-5-b.dtb into /boot/dtbs/6.8.0-1011-raspi/./bcm2712-rpi-5-b.dtb
Scanning processes..
Scanning processor microcode...
Scanning linux images...
Running kernel seems to be up-to-date.
The processor microcode seems to be up-to-date.
No services need to be restarted.
No containers need to be restarted.
No user sessions are running outdated binaries.
No VM guests are running outdated hypervisor (qemu) binaries on this host.
curl -sSfL https://raw.githubusercontent.com/longhorn/longhorn/v1.7.1/scripts/environment_check.sh | bash
[INFO] Required dependencies 'kubectl jq mktemp sort printf' are installed.
[INFO] All nodes have unique hostnames.
[INFO] Waiting for longhorn-environment-check pods to become ready (0/1)...
[INFO] All longhorn-environment-check pods are ready (1/1).
[INFO] MountPropagation is enabled
[INFO] Checking kernel release...
[INFO] Checking iscsid...
[INFO] Checking multipathd...
[INFO] Checking packages...
[INFO] Checking nfs client...
[INFO] Cleaning up longhorn-environment-check pods...
[INFO] Cleanup completed.
for V2 Data Engine
sudo apt install -y linux-modules-extra-`uname -r`
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'linux-modules-6.8.0-1011-raspi' instead of 'linux-modules-extra-6.8.0-1011-raspi'
linux-modules-6.8.0-1011-raspi is already the newest version (6.8.0-1011.12).
linux-modules-6.8.0-1011-raspi set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 11 not upgraded.
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.7.1/deploy/prerequisite/longhorn-spdk-setup.yaml
daemonset.apps/longhorn-spdk-setup created
echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
echo "vm.nr_hugepages=1024" >> /etc/sysctl.conf
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.7.1/deploy/prerequisite/longhorn-nvme-cli-installation.yaml
daemonset.apps/longhorn-nvme-cli-installation created
/etc/modules-load.d/
bash -c "$(curl -sfL https://raw.githubusercontent.com/longhorn/longhorn/v1.7.1/scripts/environment_check.sh)" -s -s
[INFO] Required dependencies 'kubectl jq mktemp sort printf' are installed.
[INFO] All nodes have unique hostnames.
[INFO] Waiting for longhorn-environment-check pods to become ready (0/1)...
[INFO] All longhorn-environment-check pods are ready (1/1).
[INFO] MountPropagation is enabled
[INFO] Checking kernel release...
[INFO] Checking iscsid...
[INFO] Checking multipathd...
[INFO] Checking packages...
[INFO] Checking nfs client...
[INFO] Checking x86-64 SSE4.2 instruction set...
[WARN] Skip SSE4.2 instruction set check on node k3s because it is not x86_64
[INFO] Checking kernel module nvme_tcp...
[INFO] Checking kernel module uio_pci_generic...
[INFO] Checking hugepage...
[INFO] Cleaning up longhorn-environment-check pods...
[INFO] Cleanup completed.
４． Longhorn のインストール
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.7.1/deploy/longhorn.yaml
namespace/longhorn-system created
priorityclass.scheduling.k8s.io/longhorn-critical created
serviceaccount/longhorn-service-account created
serviceaccount/longhorn-ui-service-account created
serviceaccount/longhorn-support-bundle created
configmap/longhorn-default-setting created
configmap/longhorn-storageclass created
customresourcedefinition.apiextensions.k8s.io/backingimagedatasources.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/backingimagemanagers.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/backingimages.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/backupbackingimages.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/backups.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/backuptargets.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/backupvolumes.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/engineimages.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/engines.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/instancemanagers.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/nodes.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/orphans.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/recurringjobs.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/replicas.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/settings.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/sharemanagers.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/snapshots.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/supportbundles.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/systembackups.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/systemrestores.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/volumeattachments.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/volumes.longhorn.io created
clusterrole.rbac.authorization.k8s.io/longhorn-role created
clusterrolebinding.rbac.authorization.k8s.io/longhorn-bind created
clusterrolebinding.rbac.authorization.k8s.io/longhorn-support-bundle created
service/longhorn-backend created
service/longhorn-frontend created
service/longhorn-conversion-webhook created
service/longhorn-admission-webhook created
service/longhorn-recovery-backend created
daemonset.apps/longhorn-manager created
deployment.apps/longhorn-driver-deployer created
deployment.apps/longhorn-ui created
kubectl get pods --namespace longhorn-system
NAME READY STATUS RESTARTS AGE
csi-attacher-644c7b7568-nwzv2 1/1 Running 0 13s
csi-attacher-644c7b7568-qjsng 1/1 Running 0 13s
csi-attacher-644c7b7568-wlft6 1/1 Running 0 13s
csi-provisioner-58cc84b487-8gmrf 1/1 Running 0 12s
csi-provisioner-58cc84b487-bsxlr 1/1 Running 0 12s
csi-provisioner-58cc84b487-nj6vw 1/1 Running 0 12s
csi-resizer-6d5c898684-kq8bg 1/1 Running 0 12s
csi-resizer-6d5c898684-tck5r 1/1 Running 0 12s
csi-resizer-6d5c898684-xfcl2 1/1 Running 0 12s
csi-snapshotter-68b686dc4-6q77s 1/1 Running 0 12s
csi-snapshotter-68b686dc4-m8xc2 1/1 Running 0 12s
csi-snapshotter-68b686dc4-pmrw6 1/1 Running 0 12s
engine-image-ei-f4f7aa25-wr7qw 1/1 Running 0 2m33s
instance-manager-f587e928da509409e711d4e2b1470b0d 1/1 Running 0 2m3s
longhorn-csi-plugin-2qmbz 3/3 Running 0 12s
longhorn-driver-deployer-658fb64689-f8f88 1/1 Running 0 3m21s
longhorn-manager-kb52f 2/2 Running 1 3m21s
longhorn-ui-69455786f4-gzccc 1/1 Running 0 3m21s
longhorn-ui-69455786f4-n8qtg 1/1 Running 0 3m21s
５． 確認
kubectl proxy
Starting to serve on 127.0.0.1:8001
open http://localhost:8001/api/v1/namespaces/longhorn-system/services/http:longhorn-frontend:http/proxy/#/dashboard
