Flamenco Setup (MacOS)

This section goes through setting up Flamenco Manager and Workers on non-cluster nodes using the official quick start docs.

Copy the executable and configuration files into the right locations.
Allow MacOS to run Manager or Worker executables. Specify the network location of Flamenco Manager using Flamenco Worker configuration file.
Set up Flamenco to use the absolute path of Blender in the Flamenco Manager configuration file for MacOS platform.
Demonstrate the Flamenco Manager and Worker will running on the same AMD64 MacOS Laptop (known as the Flamenco darwin platform).
Collect a set of example success and failure logs (as of Flamenco v3.2) to help you check against what you may be seeing in the logs.

This setup is running the Flamenco Manager and Worker on the same MacOS Laptop (ie: darwin platform) because it is the most tricky of the three to get working due to the additional security hoops and where configuration files are placed.

Note

This guide does not use UPnP discovery of Manager. Instead, the flamenco-manager.yaml is used to specify executables and the flamenco-worker.yaml configuration file specifies where the Flamenco Manager is located on the network.

Copying Files into the Right Places

We are going to spend a moment on ensuring files are in the right locations. This guide is just one way and you can make variations. The important thing to note is that all workstations are running the same release for Add-On, Manager and Workers.

For MacOS, the working directory is the Users home directory… this is where the flamenco-manager.yaml or flamenco-worker.yaml files should be copied to. This will be explored in more detail later in this article.

Ensure you read the release notes to know what other steps are needed in the event of a version upgrade. See the Flamenco download page for upgrade steps.

The diagram below shows two workstations where one is a Manager and Worker, the other is just a Worker. Both are connected to a shared-storage (NAS) where the latest Flamenco release is stored.

Depending on the platform, you may be able to launch the Flamenco Manager or Worker directly from the shared storage. However it is more reliable to copy the executables and configuration files to the workstation.

The diagram above shows the shared-storage holding the current copies of Flamenco Add-On, Executables and configuration files. This makes it easier to ensure the same release is used across all workstations.

Step 1: copy the Add-On, Manager and Worker executables to the workstation.

Step 2: copy the Manage and/or Worker configuration files to the workstation. Place these in the users home directory.

Step 3: edit the configuration files is there are any per-workstation specific settings.

Step 4: Launch the Manager and Workers on the workstations.

That’s it!

The remainder of this article will dive deeper in to what is a working directory and what a successful Flamenco cluster looks like when using MacOS workstation.

Flamenco Manager Setup

The Flamenco Manager setup will expect to find a Blender executable or the setup process will fail. If a MacOS is running the Flamenco Manager, you can enter the full path for Blender.

/Applications/Blender.app/Contents/MacOS/Blender

Working Directory

When you launch the manager or worker there is a concept of a working directory, where the executable believes it is started. On MacOS this is the users home directory. On Windows and Linux it is where the exectuable is located.

This is important because using flamenco-manager.yaml or flamenco-worker.yaml files should be in the working directory of the executable or they will not be found.

Note

For MacOS the “working directory” is the users Home Directory.

Other files, such as databases and credential files are saved in other ‘default’ locations. These are not the working directory by default.

Add-On, Configuration and Executables can be saved to a central shared-storage location. This makes it easy for all workstation users to reference the same release and configuration. You can launch the flamenco manager and worker from the shared storage or copy them over to the workstation before launching them.

The configuration files may need to be copied over either to your home directory or working directory of the executable.

Platform	Working Directory
Linux	The folder the executable was launch from within.
Windows	The folder the exectuable was launched from in Explorer.
MacOS	Users Home Directory (eg: /Users/your-user/)

Running Flamenco Executables on MacOS

On MacOS, there are some security hoops to allow the downloaded flamenco-manager, flamenco-worker and ffmpeg to be run.

Initially, launching a downloaded executable from the web is not allowed. You may see something like this…

To remedy this you have to open the Security panel and allow this executable to run. Otherwise, it will just not be allowed and you wont be asked again.

Even after allowing it in the Security panel, you are asked again…

When you run Flamenco Manager or Worker it will launch but then fail to run ffmpeg. You have to then re-open Security panel and allow ffmpeg to run.

Blender Executable Location

When Manager and Worker start they will look for blender executable, run it if found and get the Blender version. (Note: Flamenco only supports one version of Blender at any given time.) The blender executable location is either specified in the flamenco-manager.yaml configuration file (for each platform) or specified during the Flamenco Setup Assistant.

In this guide, we are not using the Flamenco Setup Assistant.

For each supported platform, it is helpful to specify the typical location of blender executable.

Platform	Exec Location
Linux	/usr/bin/blender/blender
Windows	D:\Apps\Blender\blender.exe
MacOS	/Applications/Blender.app/Contents/MacOS/Blender

This is then saved in the flamenco-manager.yaml configuration…

variables:
  blender:
    values:
      - platform: blender
        value: blender
      - platform: windows
        value: D:\\Apps\\Blender\\blender.exe
      - platform: darwin
        value: /Applications/Blender.app/Contents/MacOS/Blender

This means, that all workers of type darwin (ie: MacOS) will have Blender installed and available at /Applications/Blender.app/Contents/MacOS/Blender. This is a fair assumption for Macs on the same release, same Blender installation.

For all workers of type windows (ie: Windows) will have Blender installed and available at D:\Apps\Blender\blender.exe.

For the linux type, it is using the systems environment (PATH) to figure out which Blender is installed as default and use that executable.

Manager Log - Success

This shows the Flamenco Manager using flamenco-manager.yaml configuration file with the following contents.

_meta:
  version: 3
manager_name: Flamenco Manager
database: flamenco-manager.sqlite
listen: :8080
autodiscoverable: true
local_manager_storage_path: ./flamenco-manager-storage
shared_storage_path: /User/your-user/Downloads/flamenco/
shaman:
  enabled: true
  garbageCollect:
    period: 24h0m0s
    maxAge: 744h0m0s
    extraCheckoutPaths: []
task_timeout: 10m0s
worker_timeout: 1m0s
blocklist_threshold: 3
task_fail_after_softfail_count: 3
variables:
  blender:
    values:
      - platform: linux
        value: blender
      - platform: windows
        value: D:\Apps\Blender\blender.exe
      - platform: darwin
        value: /Applications/Blender.app/Contents/MacOS/Blender 
  blenderArgs:
    values:
      - platform: all
        value: -b -y

Launching the flamenco-manager from the Terminal…

2023-03-19 22:18:05 [flamenco-3.2-macos-amd64] > ./flamenco-manager

Example output…

2023-03-19T22:18:15-04:00 INF starting Flamenco arch=amd64 git=v3.1-96-ga692444f os=darwin releaseCycle=release version=3.2
2023-03-19T22:18:15-04:00 INF listening port=8080
2023-03-19T22:18:15-04:00 WRN SSDP: net.ListenPacket error: listen udp4 0.0.0.0:1900: bind: address already in use
2023-03-19T22:18:15-04:00 ERR Unable to create UPnP/SSDP server. This means that Workers will not be able to automatically find this Manager. Run them with the `-manager URL` argument, picking a URL from this list. urls=["http://192.168.1.62:8080/","http://127.0.0.1:8080/","http://[::1]:8080/"]
2023-03-19T22:18:15-04:00 INF opening database dsn=flamenco-manager.sqlite
2023-03-19T22:18:15-04:00 INF shaman: opening file store storageDir=/Users/your-user/Downloads/flamenco-3.2-macos-amd64/storage/file-store
2023-03-19T22:18:15-04:00 INF opening checkout directory checkoutDir=/Users/your-user/Downloads/flamenco-3.2-macos-amd64/storage/jobs
2023-03-19T22:18:15-04:00 INF serving job-specific files directly from disk onDisk=/Users/your-user/Downloads/flamenco-3.2-macos-amd64/flamenco-manager-storage url=/job-files
2023-03-19T22:18:15-04:00 INF TimeoutChecker: starting up checkInterval=1m0s initialSleep=5m0s taskTimeout=10m0s workerTimeout=1m0s
2023-03-19T22:18:15-04:00 INF sleep scheduler starting checkInterval=1m0s
2023-03-19T22:18:15-04:00 INF possble URL at which to reach Flamenco Manager count=3
2023-03-19T22:18:15-04:00 INF - http://192.168.1.62:8080/
2023-03-19T22:18:15-04:00 INF - http://127.0.0.1:8080/
2023-03-19T22:18:15-04:00 INF - http://[::1]:8080/

2023-03-19T22:21:17-04:00 INF registering new worker name=MacBook-Pro.localdomain
2023-03-19T22:21:17-04:00 INF worker signing on initialStatus=awake wName=MacBook-Pro.localdomain wUUID=2e098d60-3ef8-4ffc-80b5-f6897f71f08e
2023-03-19T22:21:17-04:00 INF worker changed status currentStatus=starting newStatus=awake wName=MacBook-Pro.localdomain wUUID=2e098d60-3ef8-4ffc-80b5-f6897f71f08e

Note, the UPnP has failed to work because the port is already in use.

2023-03-19T22:18:15-04:00 WRN SSDP: net.ListenPacket error: listen udp4 0.0.0.0:1900: bind: address already in use

This means some other service is using port 1900 such as Spotify.

Worker Log - Blender Not Found Warning

It is quite likely that Blender executable may not be found by a worker when it starts up. You will see in the logs the following message:

...
2023-03-19T22:20:54-04:00 WRN Blender could not be found, Flamenco Manager will have to supply a full path
...

The worker is informing you that it will start up but it does not know/find Blender executable. It will use whatever the Flamenco Manager tells it to use. This information is sent when a Flamenco Job is dispatched to the worker. The blender executable variables in the previous section is how the Flamenco Manager knows what to send.

When a Flamenco Job is sent over, you will see something like the following…

2023-03-19T22:26:28-04:00 INF obtained task task={"commands":[{"name":"blender-render","parameters":{"args":["--render-output","/Users/your-user/Downloads/flamenco/test/2023-03-19_172627/######","--render-format","PNG","--render-frame","1"],"argsBefore":[],"blendfile":"/Users/your-user/Downloads/flamenco-3.2-macos-amd64/storage/jobs/test/test.flamenco.blend","exe":"/Applications/Blender.app/Contents/MacOS/Blender","exeArgs":"-b -y"}}],"job":"800fed97-c1dc-42a1-991e-28f6a3c8103e","job_priority":50,"job_type":"simple-blender-render","name":"render-1","priority":50,"status":"active","task_type":"blender","uuid":"a6110b49-4a9a-4e9c-a8f2-7e67e0651682"}

Which is fairly unreadable at first glance but has the exe field that specifies the blender executable /Applications/Blender.app/Contents/MacOS/Blender.

It is easier to read if parsed…

{
  "commands": [
    {
      "name": "blender-render",
      "parameters": {
        "args": [
          "--render-output",
          "/Users/your-user/Downloads/flamenco/test/2023-03-19_222627/######",
          "--render-format",
          "PNG",
          "--render-frame",
          "1"
        ],
        "argsBefore": [],
        "blendfile": "/Users/your-user/Downloads/flamenco-3.2-macos-amd64/storage/jobs/test/test.flamenco.blend",
        "exe": "/Applications/Blender.app/Contents/MacOS/Blender",
        "exeArgs": "-b -y"
      }
    }
  ],
  "job": "800fed97-c1dc-42a1-991e-28f6a3c8103e",
  "job_priority": 50,
  "job_type": "simple-blender-render",
  "name": "render-1",
  "priority": 50,
  "status": "active",
  "task_type": "blender",
  "uuid": "a6110b49-4a9a-4e9c-a8f2-7e67e0651682"
}

Worker Log - Cannot find Manager

This is an example of not using flamenco-worker.yaml configuration file that specifies where to find Flamenco Manager (http://localhost:8080) on the network.

2023-03-19 22:20:51 [flamenco-3.2-macos-amd64] > ./flamenco-worker

2023-03-19T22:20:54-04:00 INF starting Flamenco Worker ARCH=amd64 OS=darwin git=v3.1-96-ga692444f pid=16683 releaseCycle=release version=3.2
2023-03-19T22:20:54-04:00 WRN Blender could not be found, Flamenco Manager will have to supply a full path
2023-03-19T22:20:54-04:00 INF FFmpeg found on this system path=/Users/your-user/Downloads/flamenco-3.2-macos-amd64/tools/ffmpeg-darwin-amd64 version=5.0.1-tessus
2023-03-19T22:20:54-04:00 INF auto-discovering Manager via UPnP/SSDP timeout=10m0s

The worker is waiting to auto-discover the Manager via UPnP.

Worker Log - Success

The following log is based on using flamenco-worker.yaml with the following contents…

manager_url: http://localhost:8080/
task_types: [blender, ffmpeg, file-management, misc]

Log output…

2023-03-19T22:21:17-04:00 INF starting Flamenco Worker ARCH=amd64 OS=darwin git=v3.1-96-ga692444f pid=16707 releaseCycle=release version=3.2
2023-03-19T22:21:17-04:00 WRN Blender could not be found, Flamenco Manager will have to supply a full path
2023-03-19T22:21:17-04:00 INF FFmpeg found on this system path=/Users/your-user/Downloads/flamenco-3.2-macos-amd64/tools/ffmpeg-darwin-amd64 version=5.0.1-tessus
2023-03-19T22:21:17-04:00 INF loaded configuration config={"ConfiguredManager":"http://localhost:8080/","ManagerURL":"http://localhost:8080/","TaskTypes":["blender","ffmpeg","file-management","misc"]}
2023-03-19T22:21:17-04:00 INF registered at Manager code=200 resp={"address":"::1","name":"MacBook-Pro.localdomain","platform":"darwin","software":"","status":"","supported_task_types":["blender","ffmpeg","file-management","misc"],"uuid":"2e098d60-3ef8-4ffc-80b5-f6897f71f08e"}
2023-03-19T22:21:17-04:00 INF Saved configuration file filename="/Users/your-user/Library/Application Support/Flamenco/flamenco-worker-credentials.yaml"
2023-03-19T22:21:17-04:00 INF signing on at Manager manager=http://localhost:8080/ name=MacBook-Pro.localdomain softwareVersion=3.2 taskTypes=["blender","ffmpeg","file-management","misc"]
2023-03-19T22:21:17-04:00 INF manager accepted sign-on startup_state=awake
2023-03-19T22:21:17-04:00 INF opening database dsn="/Users/your-user/Library/Application Support/Flamenco/flamenco-worker.sqlite"
2023-03-19T22:21:17-04:00 INF output uploader: running
2023-03-19T22:21:17-04:00 INF state change curState=starting newState=awake

At this point the Worker has successfully registered with Flamenco Manager.

Worker Log - Successful Job

2023-03-19T22:26:28-04:00 INF obtained task task={"commands":[{"name":"blender-render","parameters":{"args":["--render-output","/Users/your-user/Downloads/flamenco/test/2023-03-19_172627/######","--render-format","PNG","--render-frame","1"],"argsBefore":[],"blendfile":"/Users/your-user/Downloads/flamenco-3.2-macos-amd64/storage/jobs/test/test.flamenco.blend","exe":"/Applications/Blender.app/Contents/MacOS/Blender","exeArgs":"-b -y"}}],"job":"800fed97-c1dc-42a1-991e-28f6a3c8103e","job_priority":50,"job_type":"simple-blender-render","name":"render-1","priority":50,"status":"active","task_type":"blender","uuid":"a6110b49-4a9a-4e9c-a8f2-7e67e0651682"}

2023-03-19T22:26:28-04:00 INF starting task job=800fed97-c1dc-42a1-991e-28f6a3c8103e task=a6110b49-4a9a-4e9c-a8f2-7e67e0651682 taskType=blender
2023-03-19T22:26:28-04:00 INF running Flamenco command command=blender-render parameters={"args":["--render-output","/Users/your-user/Downloads/flamenco/test/2023-03-19_172627/######","--render-format","PNG","--render-frame","1"],"argsBefore":[],"blendfile":"/Users/your-user/Downloads/flamenco-3.2-macos-amd64/storage/jobs/test/test.flamenco.blend","exe":"/Applications/Blender.app/Contents/MacOS/Blender","exeArgs":"-b -y"} task=a6110b49-4a9a-4e9c-a8f2-7e67e0651682
2023-03-19T22:26:28-04:00 INF going to execute Blender cmdName=blender-render command=blender-render execCmd="/Applications/Blender.app/Contents/MacOS/Blender -b -y /Users/your-user/Downloads/flamenco-3.2-macos-amd64/storage/jobs/test/test.flamenco.blend --render-output /Users/your-user/Downloads/flamenco/test/2023-03-19_172627/###### --render-format PNG --render-frame 1" task=a6110b49-4a9a-4e9c-a8f2-7e67e0651682
2023-03-19T22:26:37-04:00 INF output produced command=blender-render outputFile=/Users/your-user/Downloads/flamenco/test/2023-03-19_172627/000001.png task=a6110b49-4a9a-4e9c-a8f2-7e67e0651682
2023-03-19T22:26:37-04:00 INF output uploader: processing file before uploading to Manager image=/Users/your-user/Downloads/flamenco/test/2023-03-19_172627/000001.png task=a6110b49-4a9a-4e9c-a8f2-7e67e0651682
2023-03-19T22:26:37-04:00 INF command exited succesfully command=blender-render pid=16895 task=a6110b49-4a9a-4e9c-a8f2-7e67e0651682
2023-03-19T22:26:37-04:00 INF command exited succesfully command=blender-render task=a6110b49-4a9a-4e9c-a8f2-7e67e0651682
2023-03-19T22:26:37-04:00 INF output uploader: Manager accepted our image image=/Users/your-user/Downloads/flamenco/test/2023-03-19_172627/000001.png 

task=a6110b49-4a9a-4e9c-a8f2-7e67e0651682
2023-03-19T22:26:39-04:00 INF obtained task task={"commands":[{"name":"frames-to-video","parameters":{"args":["-c:v","h264","-crf","20","-g","18","-vf","pad=ceil(iw/2)*2:ceil(ih/2)*2","-pix_fmt","yuv420p","-r",24,"-y"],"exe":"ffmpeg","fps":24,"inputGlob":"/Users/your-user/Downloads/flamenco/test/2023-03-19_172627/*.png","outputFile":"/Users/your-user/Downloads/flamenco/test/2023-03-19_172627/test-1.mp4"}}],"job":"800fed97-c1dc-42a1-991e-28f6a3c8103e","job_priority":50,"job_type":"simple-blender-render","name":"preview-video","priority":50,"status":"active","task_type":"ffmpeg","uuid":"03605f11-a9c4-4680-9819-a96c7f2e9a62"}
2023-03-19T22:26:39-04:00 INF starting task job=800fed97-c1dc-42a1-991e-28f6a3c8103e task=03605f11-a9c4-4680-9819-a96c7f2e9a62 taskType=ffmpeg
2023-03-19T22:26:39-04:00 INF running Flamenco command command=frames-to-video parameters={"args":["-c:v","h264","-crf","20","-g","18","-vf","pad=ceil(iw/2)*2:ceil(ih/2)*2","-pix_fmt","yuv420p","-r",24,"-y"],"exe":"ffmpeg","fps":24,"inputGlob":"/Users/your-user/Downloads/flamenco/test/2023-03-19_172627/*.png","outputFile":"/Users/your-user/Downloads/flamenco/test/2023-03-19_172627/test-1.mp4"} task=03605f11-a9c4-4680-9819-a96c7f2e9a62
2023-03-19T22:26:39-04:00 INF going to execute FFmpeg command=frames-to-video execCmd="/Users/your-user/Downloads/flamenco-3.2-macos-amd64/tools/ffmpeg-darwin-amd64 -r 24 -pattern_type glob -i /Users/your-user/Downloads/flamenco/test/2023-03-19_172627/*.png -c:v h264 -crf 20 -g 18 -vf pad=ceil(iw/2)*2:ceil(ih/2)*2 -pix_fmt yuv420p -r 24 -y -r 24 /Users/your-user/Downloads/flamenco/test/2023-03-19_172627/test-1.mp4" task=03605f11-a9c4-4680-9819-a96c7f2e9a62
2023-03-19T22:26:40-04:00 INF command exited succesfully command=frames-to-video pid=16908 task=03605f11-a9c4-4680-9819-a96c7f2e9a62
2023-03-19T22:26:40-04:00 INF command exited succesfully command=frames-to-video task=03605f11-a9c4-4680-9819-a96c7f2e9a62

This includes rendering one frame (the example Job sent by the Add-On only asked for one frame) and then creating a small demo video of what was rendered.

From a deployment perspective, Flamenco allows for more configuration and it is worth considering some availability and reliability questions like:

Is the Flamenco Manager is always the same workstation? What happens if that workstation is busy, or offline?
Should the Flamenco Manager database be saved on a workstation or on shared-storage?
Does the Flamenco Manager have to be on a workstation or is it better hosted on a network server?

Fortunately, all these questions can be answered using Flamenco Manager configuration options.

Troubleshooting

Blender Error: command exited abnormally with code

You may see Flamenco soft-fail errors which are typically one or more problems with Blender (or ffmpeg) when it was given a Task to do by the Flamenco.

The Flamenco Worker will have something like “abnormal error” message.

You need to inspect the Flamenco Full Log of the failed Task.

2023-05-26T15:11:57Z Task assigned to worker think (6a7625a9-ee5a-4add-8124-ae795489f2d7)
2023-05-26T15:11:57Z task changed status queued -> active
going to run: D:\\Apps\\Blender3.4.1\\blender.exe ["-b" "-y" "F:\\flamenco\\jobs\\lego-sakura-camera-spin-i6ki\\lego-sakura-camera-spin.flamenco.blend" "--render-output" "F:\\flamenco/output/lego-sakura-camera-spin/2023-05-26_151155/######" "--render-format" "PNG" "--render-frame" "30"]
pid=2160 > Blender 3.4.1 (hash 55485cb379f7 built 2022-12-20 01:51:19)
pid=2160 > Error: File format is not supported in file 'F:\flamenco\jobs\lego-sakura-camera-spin-i6ki\lego-sakura-camera-spin.flamenco.blend'
pid=2160 > 
pid=2160 > Blender quit
Failed: command exited abnormally with code 1
2023-05-26T15:12:00Z Task failed by 1 worker, Manager will mark it as soft failure. 1 more failure will cause hard failure.
2023-05-26T15:51:17Z task changed status active -> soft-failed