Flamenco Workers

Flamenco Workers perform the blender Flamenco Jobs dispatched by the Flamenco Manager.

When they start up they check to see if they can meet the requested Flamenco Job types such as blender, ffmpeg etc.

They also register themselves with Flamenco Manager. The Flamenco Manager Console should show the workers as seen in the screenshot below.

Lifecycle of a Flamenco Worker

In this project, when a Flamenco Worker is finished working on a Blender Task it will remain online and waiting for new Tasks until the Flamenco Worker Deployment in the cluster is scaled down or it is destroyed.

When a Flamenco Worker is terminated, it will attempt to inform the Flamenco Manager using the Flamenco Manager API to de-register itself.

When the Worker Deployment scales down there is no specific order in which the workers are terminated. If a Worker is actively working on a Task then it may be interrupted and Flamenco Manager will re-schedule the Task(s) assigned to that Worker.

Worker DeRegistration

When a Worker gracefully shuts down it will make a couple of calls to the Flamenco Manager to de-register itself. This is done using the Flamenco Manager API.

This can be written in any language, but Python seemed to be an easy choice given Flamenco Manager publishes Python backed API library.

This script is called when the Flamenco Worker Pod is terminating using the k8s lifecycle hook preStop. The flamenco-api-delete-worker.py along with FLAMENCO_API_PORT and FLAMENCO_API_URL ENV vars are packaged into the Flamenco Docker Container (example).

Example preStop lifecycle hook in Flamenco Pod…

      apiVersion: apps/v1
      kind: DaemonSet
      metadata:
        name: flamenco-worker-intel-gpu
        namespace: flamenco-worker
        labels:
          app.kubernetes.io/name: flamenco-worker-intel-gpu
          app.kubernetes.io/namespace: flamenco-worker
      spec:
...
          spec:
...
            containers:            
              - name: flamenco
                image: "{{ docker_registry_server }}/flamenco:{{flamenco_version}}-blender-{{blender_version}}"
                command: ["/code/flamenco/flamenco-worker"]
                imagePullPolicy: Always
...
                lifecycle:
                    preStop:
                      exec:
                        command: ["/bin/sh", "-c", "python3 /code/flamenco_api_client/flamenco-api-client/flamenco-api-delete-worker.py"]

Example delete-worker Python script…

import os
import time
from flamenco.manager import ApiClient, Configuration, ApiException
from flamenco.manager.apis import WorkerMgtApi, WorkerApi
import yaml

FLAMENCO_API_URL = os.getenv(
    "FLAMENCO_API_URL", "http://flamenco-service.flamenco-manager.svc"
)
FLAMENCO_API_PORT = os.getenv("FLAMENCO_API_PORT", "8080")
HOSTNAME = os.getenv("HOSTNAME")


def delete_worker():
    """Delete this worker from Flamenco Manager."""

    worker_credentials_file = (
        "/root/.local/share/flamenco/flamenco-worker-credentials.yaml"
    )
    with open(worker_credentials_file, "r", encoding="UTF-8") as file:
        worker_credentials = yaml.safe_load(file)

    configuration = Configuration(
        host=f"{FLAMENCO_API_URL}:{FLAMENCO_API_PORT}",
        username=worker_credentials["worker_id"],
        password=worker_credentials["worker_secret"],
    )
    api_client = ApiClient(configuration)

    try:
        worker_api = WorkerApi(api_client)
        worker_api.sign_off()
        print(f"Signed-Off Worker {HOSTNAME}")
        worker_mgmt_api = WorkerMgtApi(api_client)
        worker_mgmt_api.delete_worker(worker_id=worker_credentials["worker_id"])
        print(f"Deleted Worker {HOSTNAME}")
    except ApiException as api_exception:
        print(f"FAILED to Delete Worker {HOSTNAME}: %s\n" % api_exception)


if __name__ == "__main__":
    delete_worker()

Troubleshooting

Example success-path log for Flamenco Worker Pod

 INF starting Flamenco Worker ARCH=amd64 OS=linux git=8a36cf1d pid=1 releaseCycle=release version=3.1
 INF Blender found on this system, it will be used unless Flamenco Manager specifies a path to a different Blender path=/usr/local/blender/blender version="Blender 3.4.0"
 INF FFmpeg found on this system path=/code/flamenco/tools/ffmpeg-linux-amd64 version=5.0.1-static
 INF loaded configuration config={"ConfiguredManager":"http://flamenco-service.flamenco-manager.svc:8080/","ManagerURL":"http://flamenco-service.flamenco-manager.svc:8080/","TaskTypes":["blender","ffmpeg","file-management","misc"]}
 INF registered at Manager code=200 resp={"address":"10.244.107.204","name":"flamenco-workers-wlxqv","platform":"linux","software":"","status":"","supported_task_types":["blender","ffmpeg","file-management","misc"],"uuid":"6cb96b42-87b8-469d-8aee-832b0a9e1336"}
 INF Saved configuration file filename=/root/.local/share/flamenco/flamenco-worker-credentials.yaml
 INF signing on at Manager manager=http://flamenco-service.flamenco-manager.svc:8080/ name=flamenco-workers-wlxqv softwareVersion=3.1 taskTypes=["blender","ffmpeg","file-management","misc"]
 INF manager accepted sign-on startup_state=awake
 INF opening database dsn=/root/.local/share/flamenco/flamenco-worker.sqlite
 INF output uploader: running
 INF state change curState=starting newState=awake