Ansible Host Configuration

This can be any laptop or computer that can reach the Gateway Device over the Home Network.

There are some assumptions:

The PXE Installer or Manual installation prepares the nodes to connect automatically to the cluster subnet.
SSH keys that Ansible will use are manually copied over to the Gateway Device and k8s nodes ahead of running the Ansible Playbooks.
Ubuntu 22.04 Server (minimal) has been tested with the playbooks. Other distros, even Debian based ones may not have all the pre-reqs.

More details:

Configuring the Gateway Device
Using the PXE Installer to build the Kubernetes Control-Plane Node and Worker Nodes.

Galaxy Packages

The Ansible Host (and Gateway Device) will also need some Ansible Galaxy packages installed. For the Ansible Host you can use the build-ansible-host.sh script to install the necessary packages.

This is the list of Ansible related packages needed to run all the Ansible Playbook Roles.

ansible-galaxy collection install community.general
ansible-galaxy collection install kubernetes.core
ansible-galaxy install nginxinc.nginx
pip install kubernetes

Ansible Hosts File

From the Ansible Host workstation it is possible to provision all hosts. Those within the cluster use the Gateway Device as a “jump host”.

In the example below, it is worth mentioning the Gateway Device is reached using the Home Network IP Address (eg: 192.168.1.xxx (Gateway Device)). Other nodes within the cluster are accessed using their hostnames (configured by the Gateway Device/Gateway Device).

The Docker Registry is co-located on the k8s control-node node. It can be deployed anywhere except the Gateway Device.

[control-nodes]
k8s-control-plane-node ansible_user=ubuntu ansible_ssh_private_key_file=~/.ssh/id_rsa

[workers]
k8s-node[2:7] ansible_user=ubuntu ansible_ssh_private_key_file=~/.ssh/id_rsa

[gateways]
gateway ansible_host=192.168.1.xxx (Gateway Device) ansible_user=ubuntu

[docker-registry-server]
registry ansible_host=k8s-control-plane-node ansible_user=ubuntu ansible_ssh_private_key_file=~/.ssh/id_rsa

[control-nodes:vars]
ansible_ssh_common_args='-o ProxyCommand="ssh -W %h:%p -q ubuntu@192.168.1.xxx (Gateway Device)"'

[workers:vars]
ansible_ssh_common_args='-o ProxyCommand="ssh -W %h:%p -q ubuntu@192.168.1.xxx (Gateway Device)"'

[docker-registry-server:vars]
ansible_ssh_common_args='-o ProxyCommand="ssh -W %h:%p -q ubuntu@192.168.1.xxx (Gateway Device)"'

gateway-install - main.yaml

There is one file that specifies what nodes to expect in the cluster. The gateway-install/defaults/main.yaml file contains a list of known nodes with their MAC addresses. This is used to build the dnsmasq allowed list of DHCP clients.

Known cluster nodes

# Known nodes on cluster network
nodes:
- name: "k8s-control-plane-node"
  mac: "b8:ae:ed:7e:57:54"
  static_ip: "192.168.57.31"  
- name: "k8s-node2"
  mac: "f4:4d:30:65:88:c0"
  static_ip: "192.168.57.32"
...

Gateway Device Network Interfaces

In the gateway-install/defaults/main.yaml file there are three variables that determine which interfaces in the Gateway Device (aka Gateway Device) are for WWW and cluster-subnet access. These must match the hardware names in the Gateway Device. This will depend on the type of devices and how Raspbian or Ubuntu interprets them.

Here is an example of two fixed-ethernet interfaces on the RaspberryPi…

gateway_www_interface: eth1
gateway_dnsmasq_interface: eth0

You will want to verify these mappings before running the Gateway Device Playbook. The username is whatever you chose when installing the Raspbian or Ubuntu OS manually on the Gateway Device. In this case, instead of pi, ubuntu was chosen to match all the other node usernames.

Cluster Node Network Interfaces

This is hard to predict. Sometimes the Ubuntu OS picks different names for the built-in NICs and there doesn’t appear to be a reason. The same Ubuntu OS on two different Intel NUCs gave different hardware names for the built in NIC.

To help combat this the gateway-install/defaults/main.yaml has a list of possible network interface names.

It maybe necessary to add hardware names to this list to provision the inbuilt NIC.

k8s_node_interfaces:
  - eno1    
  - enp0s25 
  - enp2s0

Roles

gateway-install

Purpose:: Deploys and configures the ecosystem.

When running this playbook on a RaspberryPi it will take quite a long time to complete because it is downloading and unpacking ISOs for PXE Ubuntu Installer.

ansible-playbook -i hosts build-gateway.yaml -K

To just update the mkdocs you can use the mkdocs tag…

ansible-playbook -i hosts build-gateway.yaml -K -t mkdocs

To just update new domain names, DHCP, WOL scripts with a new node MAC and IP address…

ansible-playbook -i hosts build-gateway.yaml -K -t dnsmasq_setup

Ensure that the hosts file will pick up the new node as the node count is bounded.

[workers]
k8s-node[2:7] ansible_user=ubuntu ansible_ssh_private_key_file=~/.ssh/id_rsa

docker-install

Purpose: Deploys Docker and a local self-signed Docker Registry.

Notice

Docker cannot be installed on the Gateway Device as it conflicts with the Gateway Device iptables for routing traffic.

This playbook will install Docker, create SSL certs and then build any Docker Containers need by the project. The first time this playbook runs it may take some time to complete.

This playbook must be run before building the cluster so that the self-signed docker registry cert can be incorporated into all the k8s nodes. If not, they will refuse to pull from an unknown registry.

This Playbook also bulds the Flamenco & Docker containers.

build-cluster

nfs-server-install

Purpose: Deploys an NFS Server on the Gateway Device.

k8s-install

Purpose: Deploys common k8s software on the Control Plane and Worker nodes.

k8s-control-node-install

Purpose: Deploys and configures the k8s Control Plane specific software.

k8s-worker-install

Purpose: Deploys and configures the k8s Worker specific software.

k8s-join-workers

Purpose: Attempts to contact the k8s Control Plane node and join a Worker node to the cluster.

k8s Playbooks

These Ansible Roles are responsible for deploying applications into a pre-built cluster.

cluster-foundation

Purpose: Installs calico, ingress, monitoring and Intel GPU drivers into the cluster.

Deploys in-cluster NFS Provisioner to allow applications to contact and external NFS Server.

deploy-flamenco

Purpose:: Deploys Flamenco Manager as a Pod in the cluster and either a DaemonSet or Deployment of Worker Pods across all nodes (except the Control Plane node).