Overview
This section of the documentation is based around installation and configuration. There are sevearl ways to approach building out a cluster and some of the steps mentioned here may not be optimal.
Learn the lingo
This documentation comes with a page devoted to Terminology to help demystify networking, kubenernetes and blender terminology.
Equipment
Kubernetes is not very fussy about what hardware you choose, however in this project having similar (ie: Intel amd64) machines makes the installation easier than having multi-architecture choices.
The only multi-architecture dependency in this project is the use of a [RaspberryPi](https://www.raspberrypi.org/) (arm64) Gateway Device serving amd64 ISO over Gateway for PXE Installer.
Ansible Host
This can be any laptop or computer that can reach the Gateway Device over the Home Network. There are some assumptions:
- The PXE Installer or Manual installation prepares the nodes to connect automatically to the cluster subnet. See Adding a New Node for details on either method.
- SSH keys that Ansible will use are manually copied over to the Gateway Device and k8s nodes ahead of running the Ansible Playbooks.
The Ansible Host will also need some Ansible Galaxy packages installed.
Hardware and Networking Gear
- Layer 2 Unmanaged Switch (eg: Netgear GS305 )
- 1 Desktop PC with two NICs (Gateway/Router) running Raspbian or Ubuntu 22.04
- 3 Desktop PCs capable of running Ubuntu 22.04
Nice to haves
- BIOS PXE legacy boot mode
- BIOS support for Wake-On-Lan*
- Intel i915 or NVIDIA CUDA compatible GPU
* Note some computers claim Wake-On-Lan support however there are two types of WOL support. One form by the BIOS and one supported by OS. The latter requires software to set the WOL flags on every boot and does not retain this setting during a power supply interruption.
Gateway Device
- RaspberryPi or low-end appliance/MiniPC.
- 2GB+ RAM, 16GB+ Storage.
- At least two network interfaces (Inbuilt NIC, wifi or USB NIC).
- Base OS installed (ie: Raspbian)
The Gateway Device is usually powered on 24/7 and provides network services to the cluster. A low powered device such as a RaspberryPi is a good choice.
You can connect to this gateway on the Home Network wifi. The fixed-ethernet is connected to the Layer 2 Switch hosting the cluster nodes.
The Docker Registry cannot be installed on the Gateway Device. The iptables from Docker and gateway routing interfere with each other.
Gateway Services
Gateway | DNS | Gateway and PXE Installer
Control Plane Node
- Manual or PXE installation of Ubuntu 22.04
- 2GB+ RAM for k8s.
- 6GB+ RAM for PXE Ubuntu installer.
The control plane can be a smaller footprint than the worker nodes as by default it will not schedule any compute work on itself. Other services, like Docker Registry, can also be installed on it so long as it has sufficient disk space.
Control Plane Node Installation
Worker Node
- Manual or PXE installation of Ubuntu 22.04
- 2GB+ RAM for k8s.
- 6GB+ RAM for PXE Ubuntu installer.
The worker node is the simplest of all nodes and requires various drivers to support NFS and GPU. Some of these are installed at the OS level, some installed as k8s charts from the device manufacturers.
Docker Registry
- Local Docker Installation within the cluster subnet
- Self-Signed (HTTPS) Registry
A local self-signed docker registry is used to host custom containers that include Blender and Flamenco.
The Docker Registry cannot be installed on the Gateway Device. The iptables from Docker and gateway routing interfere with each other.
Blender & Flamenco Docker Container
- Custom Dockerfile containing everything needed to run Flamenco and Blender
Flamenco Worker Tags
The Flamenco Worker Tag feature in Flamenco is used to abstractly group Workers. In this project, FWTs are used to help group types of Flamenco Workers in the Cluster. Specifically between those that have access to a GPU and those that do not.
Enhancements
These are optional lifestyle improvements to the cluster. They are not required to use the cluster but make it easier to manage.