Azure Kubernetes sFTP Solution
In this post, we’ll take a look at deploying a highly available sFTP solution to Azure Kubernetes with user files stored in an Azure NFSv4 File Share. The sFTP application reads user credentials from a file named users.conf, containing secrets from an Azure Key Vault.
Here is the link to my Github account where you can download the code mentioned in this article:
https://github.com/rnemeth90/kubernetes-sftp
We will work through the following steps in this article:
- Deploy the AzureFile CSI driver to the AKS cluster
- Create a configMap that our initContainer will
- Deploy a persistent volume claim that an Azure File share will back
- Deploy a replicaSet consisting of our initContainer and application container
- Deploy a service to serve traffic
First, you will need to deploy the Azure Files CSI driver to your AKS cluster. AKS uses this daemon set to dynamically provision/destroy Azure NFSv4 File Shares. The Azure Files CSI driver creates a storage account in the node pool resource group, in which it will then provision the file share.
Deploying the Azure Files CSI driver is a simple task. You will need to run this script:
curl -skSL https://raw.githubusercontent.com/kubernetes-sigs/azurefile-csi-driver/master/deploy/install-driver.sh | bash -s master --
You can use the following commands to verify that the daemon set has exists:
kubectl -n kube-system get pod -o wide --watch -l app=csi-azurefile-controller
kubectl -n kube-system get pod -o wide --watch -l app=csi-azurefile-node
kubectl create -f https://raw.githubusercontent.com/kubernetes-sigs/azurefile-csi-driver/master/deploy/example/storageclass-azurefile-nfs.yaml
Read more about the AzureFiles CSI driver at the following Github link:
https://github.com/kubernetes-sigs/azurefile-csi-driver/blob/master/docs/install-driver-on-aks.md
apiVersion: v1
kind: ConfigMap
metadata:
creationTimestamp: null
name: testcm
data:
init.sh: |-
#!/bin/bash
CONF_FILE="/etc/sftp/users.conf"
KEYVAULT="Insert name of Key Vault"
AZ_SPN_ID="Insert service principal Id"
AZ_SPN_PASSWORD="Insert service principal password"
AZ_SPN_TENANT_ID="Insert Az AAD Tenant Id"
AZ_SUBSCRIPTION_ID="Insert Az Subscription Id"
az login --service-principal --username "${AZ_SPN_ID}" --password "${AZ_SPN_PASSWORD}" --tenant "${AZ_SPN_TENANT_ID}"
az account set --subscription "${AZ_SUBSCRIPTION_ID}"
SECRETS+=($(az keyvault secret list --vault-name $KEYVAULT --query "[].id" -o tsv))
chmod 755 /home
if [[ -e $CONF_FILE ]]; then
rm -rf "${CONF_FILE}"
touch $CONF_FILE
else
touch $CONF_FILE
fi
for SECRET in "${SECRETS[@]}"; do
SECRETNAME=$(basename $SECRET | tr -d '\r')
SECRETVALUE=$(az keyvault secret show --vault-name $KEYVAULT --name $SECRETNAME --query 'value' | tr -d '"' | tr -d '\r')
echo "$SECRETNAME:$SECRETVALUE:::upload" >> $CONF_FILE
done
if [[ ! -s $CONF_FILE ]]; then
echo "** ERROR: user.conf is empty" 1>&2
exit 1
fi
The initContainer will use this configMap to read secrets from an Azure Key Vault and then write the secrets to the users.conf file. The users.conf file contains user account information. In the Key Vault, the secret name should be the username and the secret value should be the password. This Key Vault must exist and contain the secrets prior to deploying this solution.
When I deployed this solution in my environment, I used an Azure DevOps pipeline to create/manage the Key Vault with Terraform and then deploy the secrets using an inline Powershell script. However, this is beyond the scope of this article, and I will not be covering it here. You will also need to create a Service Principal in your AAD tenant and grant it access to the Key Vault. The Service Principal needs to have at least read access to the Key Vault.
Before deploying the configMap, you need to fill in values for these variables:
KEYVAULT="Insert name of Key Vault"
AZ_SPN_ID="Insert service principal Id"
AZ_SPN_PASSWORD="Insert service principal password"
AZ_SPN_TENANT_ID="Insert Az AAD Tenant Id"
AZ_SUBSCRIPTION_ID="Insert Az Subscription Id"
These values pertain to the service principal with read access to the Key Vault. After filling out these variables, you can deploy the configMap using kubectl.
kubectl create -f configmap.yaml
You can deploy the PVC using this yaml file:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-ftp-clientdirs
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 100Gi
storageClassName: azurefile-csi-nfs
When creating the persistent volume claim for the deployment, you specify azurefile-csi-nfs as the StorageClass. This will create an NFSv4 share in a premium storage account. The reclaim policy of the storage class ensures that the file share is deleted when the associated Persistent Volume is deleted. Change the reclaim policy to “Retain” if you want the file shares to persist after deleting the PV. Furthermore, the storage class enables the file share to be expandable by modifying the storage request size on the PVC.
After deploying the PVC, you can verify its existence by running:
kubectl get pvc
You should see output similar to the following (the manifest for the replicaSet is by far the most complicated of all the code referenced here):
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: sftp
name: sftp
spec:
replicas: 2
selector:
matchLabels:
app: sftp
strategy: {}
template:
metadata:
labels:
app: sftp
spec:
initContainer:
enabled: true
name: chmodder
image: mcr.microsoft.com/azure-cli
command:
- /bin/sh
- -c
- "/scripts/init.sh"
volumeMounts:
- name: init-shell-script
mountPath: /scripts
containers:
- name: sftp
image: docker.io/atmoz/sftp:alpine
imagePullPolicy: IfNotPresent
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
livenessProbe:
tcpSocket:
port: 22
initialDelaySeconds: 1
timeoutSeconds: 5
periodSeconds: 10
failureThreshold: 3
readinessProbe:
tcpSocket:
port: 22
initialDelaySeconds: 1
timeoutSeconds: 5
periodSeconds: 10
failureThreshold: 3
volumeMounts:
- name: pvc-ftp-clientdirs
mountPath: /home
- name: sftp
mountPath: /etc/sftp
dnsPolicy: ClusterFirst
restartPolicy: Never
volumes:
- name: pvc-ftp-clientdirs
persistentVolumeClaim:
claimName: pvc-ftp-clientdirs
- name: init-shell-script
configMap:
name: init-shell-script
- name: sftp
emptyDir: {}
You may need to change the CPU/memory requests and limits. These values worked for me, but your results may vary. You can also add/change the labels as you see fit.
This will handle routing traffic to our pods.
apiVersion: v1
kind: Service
metadata:
name: sftp-service
spec:
selector:
app: sftp
type: LoadBalancer
ports:
- protocol: TCP
port: 22
targetPort: 22
Users will be connected in a round-robin fashion to the pods. This manifest will create a service of type LoadBalancer. Which will in turn create a new public Azure Load Balancer in the resource group that contains your Azure Kubernetes Service cluster if one does not already exist. If you already have a public load balancer, a new frontend IP address will be added.
That’s all for now. Please feel free to contact me if you have any questions.