Installing k8s storage with Ondat
5 min read
In this tutorial we will walk thought installation of persistent storage in kubernetes with OnDat (Storage OS) product.
Fist why Ondat? There are other free alternatives like rook or paid ones like Portworx
My main reason to use OnDat is because of a limitation in the VPS host. It can't provide separate storage disks. That is a requirement in solutions like rook and portworx.
Because of that limitation I almost switch of VPS provider from contabo to hetzner.
Using another provider like hetzner could be a better alternative because it has native integration with its cloud storage system, that way we don't need to worry about it and can be more performatic since communication between host and storage could be in a internal network.
Ondat is a good alternative for on promise cluster, and after some evaluation I decide to stick with it. It has a free license for 1TiB storage in cluster. For now this is plenty for my use case, but it can be expansive if you need more resources.
Take a look at Ondat https://docs.ondat.io/docs/prerequisites/ and https://docs.ondat.io/docs/best-practices/ and judge by yourself if is a good solution for you.
We will use the same master nodes as etcd nodes for OnDat. Your cluster has 3 master, etcd, and worker nodes, this could be a limiting point for a more performatic cluster, for instance you could separate the masters, etcd from the workers, if the cluster grows I think it should be a better solution this separation.
Now lets begin, all steps here are documented in https://docs.ondat.io/docs/prerequisites/ and https://docs.ondat.io/docs/install/
First we need the kernel modules loaded, we are using Ubuntu Server 20.04 LTS
For this you could use a ansible playbook for automation, make sure ansible is installed on your system (Linux or WSL Windows, Or Mac)
git clone firstname.lastname@example.org:gcsilva/ondat-playbook.git cd ondat-playbook
Edit the inventory/hosts.yml file and add your hosts ips or dns names.
Also make sure you can ssh without a password using just ssh keys in the systems. For this follow this tutorial https://phoenixnap.com/kb/ssh-with-key
Now run the playbook
ansible-playbook -i inventories/contabo/ playbooks/ondat.yml
If you get some error of a the role not being found, link the role folder in the playbook folder:
cd playbooks ln -s ../roles .
Run the playbook again.
We will use ansible to install etcd.
First clone the playbooks:
git clone https://github.com/storageos/deploy.git cd k8s/deploy-storageos/etcd-helpers/etcd-ansible-systemd
Edit the file hosts put the external ssh hosts and in the ip put the internal VPN ips. (if you follow my other post about k8s cluster in contabo we setup a private VPN)
The host should be something like
["10.8.0.1" fqdn="10.8.0.1" host2.srv.yourdomain.com ansible_user=root ip="10.8.0.2" fqdn="10.8.0.2" host3.srv.yourdomain.com ansible_user=root ip="10.8.0.3" fqdn="10.8.0.3"] host1.srv.yourdomain.com ansible_user=root ip=
If you client machine is on the same VPN as the servers you could use their private ips as the host
Now edit the installation configuration in group_vars/all
We need to change etcd_port_client and etcd_port_peers to something that do not colide with the etcd running on the masters for kubernetes, my is 2381 and 2382 respectively
Also change advertise_format to ip, and disable the tls tls: enabled: false. We are not using tls because it complicates a little bit the deployment of storage because you will need the certificates, and we are using an encrypted VPN anyway
Now install etcd
ansible-playbook -i hosts install.yaml
Install ondat cluster operator
kubectl create -f https://github.com/storageos/cluster-operator/releases/download/v2.4.4/storageos-operator.yaml
Create ta secret for the passwords
apiVersion: v1 kind: Secret metadata: name: "storageos-api" namespace: "storageos-operator" labels: app: "storageos" type: "kubernetes.io/storageos" data: # echo -n '<secret>' | base64 apiUsername: c3RvcmFnZW9z apiPassword: c3RvcmFnZW9z # CSI Credentials csiProvisionUsername: c3RvcmFnZW9z csiProvisionPassword: c3RvcmFnZW9z csiControllerPublishUsername: c3RvcmFnZW9z csiControllerPublishPassword: c3RvcmFnZW9z csiNodePublishUsername: c3RvcmFnZW9z csiNodePublishPassword: c3RvcmFnZW9z csiControllerExpandUsername: c3RvcmFnZW9z csiControllerExpandPassword: c3RvcmFnZW9z
Change the username and password with something you generate with the command:
echo -n 'user' | base64 echo -n 'secret' | base64
Now create a svc pointing to your external etcd cluster
kubectl create namespace storageos-etcd
Apply the yamls:
apiVersion: v1 kind: Endpoints metadata: name: storageos-etcd namespace: storageos-etcd labels: app: etcd cluster: storageos subsets: - addresses: - ip: 10.1.10.216 - ip: 10.1.10.217 - ip: 10.1.10.218 ports: - name: client port: 2381 protocol: TCP
apiVersion: v1 kind: Service metadata: name: storageos-etcd namespace: storageos-etcd labels: app: etcd cluster: storageos spec: clusterIP: None ports: - name: client port: 2381 targetPort: 2381 selector: null
Label all the nodes you want be part of the cluster:
kubectl label nodes node1 ondat/storage=true # repeat for all nodes
Now create the cluster apply the follow yaml:
apiVersion: "storageos.com/v1" kind: StorageOSCluster metadata: name: "ondat" namespace: "storageos-operator" spec: # Ondat Pods are in kube-system by default secretRefName: "storageos-api" # Reference from the Secret created in the previous step secretRefNamespace: "storageos-operator" # Namespace of the Secret k8sDistro: "upstream" images: nodeContainer: "storageos/node:v2.4.4" # Ondat version kvBackend: address: 'storageos-etcd.storageos-etcd:2381' # Example address, change for your etcd endpoint # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can set ETCD server ips resources: requests: memory: "512Mi" cpu: 1 nodeSelectorTerms: - matchExpressions: - key: "ondat/storage" operator: In values: - "true"
Check your cluster health:
kubectl -n kube-system get pods
The pods storageos-daemonset most be running
Lets login in the management ui
kubectl port-forward -n kube-system svc/storageos 5705
Browser to http://localhost:5705
Login with the username and password you choose on the secret.
Lets test the cluster:
git clone https://github.com/storageos/use-cases.git cd use-cases/00-basics kubectl apply -f pvc-basic.yaml kubectl apply -f pod.yaml kubectl apply -f pod.yaml kubectl exec -it d1 -- bash echo 'hello' > /mnt/helloworld.txt exit kubeclt delete pod d1 # kill the pod kubectl apply -f pod.yml kubectl exec -it d1 -- bash cat /mnt/helloworld.txt # content are preserved
You should see the volume in ondat ui:
Now you need to register your cluster, follow the steps in https://docs.ondat.io/docs/operations/licensing/ to do it.
Your cluster is up! You can create new storage classes with different replication factors or encrypted volumes as well.