Build
Pre-Reqs
- Docker Desktop with
buildxusing tonistiigi/binfmt for multi-platform support. - Using Flamenco Dev Container and build setup Docker Setup.
Summary
Copy Dev Dockerfile into working-copy parent folder and issue the following Docker buildx commands.
Create a development build
#> docker buildx build --load --target go_build -t flamenco_dev .
#> docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
flamenco_dev latest 162c466bf078 6 minutes ago 5.4GB
Create a multi-platform build
#> docker buildx create --use
#> docker buildx build --platform=linux/amd64,linux/arm64 --target export -t test . --output execs
#> ls -R execs/
execs/:
linux_amd64 linux_arm64
execs/linux_amd64:
flamenco-manager_amd64 flamenco-worker_amd64
execs/linux_arm64:
flamenco-manager_arm64 flamenco-worker_arm64
Details
Why buildx
The buildx tool is part of Docker Desktop with docker buildx installed. Install instructions from Docker with post-install setup for your development user account.
- One of the the buildx tooling benefits is to create well organised parallel builds using buildx cache.
- The other significant benefit is reduced Docker image sizes as you can pick key parts of each stage that you need in the final deployment image.
- The Docker buildx framework uses qemu to integrate cross-platform builds. This makes creating amd64, arm64 images straightforward from the same Dockerfile.
Platform Builds (amd64, arm64 etc)
This page is concerned with building various platform targets such as amd64 and arm64. This is done using the Docker buildx feature and a multi-stage Dockerfile. The buildx feature uses qemu to emulate platform targets.
The purpose of these platform builds is to generate machine-compatible executables for the Flamenco Manager and Worker.
Example build for two platforms…
docker buildx build -t flamenco_dev --platform=linux/amd64,linux/arm64 .
This multi-platform feature is employed by referencing buildx ENV variables in the Dockerfile. Specifically in this project: BUILDPLATFORM, TARGETOS, TARGETARCH.
Note
You must push or load the docker image from buildx cache otherwise it will not persist and be available to Docker.
Warning
You can only push an image to docker images that matches platform type. For example, you cannot push arm64 to docker running on amd64 machine.
Including Blender Executable
Blender is included because the Flamenco Setup Assistant will not allow the Manager to initialise without finding (and running) Blender. This may be addressed in future Flamenco releases - associated issue 100195.
There is only amd64 tar.xz file for Blender and this is included for any buildx platform in this example.
Host/Local Platform Build
Using this example Mutli-Stage buildx compatible Flamenco Development Dockerfile on an amd64 workstation building an amd64 flamenco build target.
Create a buildx context…
docker buildx create --use
If we are building on an amd64 workstation then the BUILDPLATFORM ENV variable is the workstation platform (ie: amd64) so we dont have to specify the --platform=linux/amd64 option.
Launching the build process for amd64 only…
docker buildx build --load --target go_build .
The --load option will push the image to the Docker images once successfully built.
[+] Building 270.2s (21/21) FINISHED
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 3.18kB 0.0s
=> [internal] load metadata for docker.io/library/ubuntu:jammy 0.2s
=> [internal] load build context 0.1s
=> => transferring context: 81.87kB 0.1s
=> [blender 1/5] FROM docker.io/library/ubuntu:jammy@sha256:dfd64a3b4296d8c9b62aa3309984f8620b98d87e47492599ee20739e8eb54fbf 0.0s
=> => resolve docker.io/library/ubuntu:jammy@sha256:dfd64a3b4296d8c9b62aa3309984f8620b98d87e47492599ee20739e8eb54fbf 0.0s
=> CACHED [build 2/3] RUN --mount=type=cache,target=/var/cache/apt,sharing=locked --mount=type=cache,target=/var/lib/apt,sharing=locked apt-get update && apt install -y curl gz 0.0s
=> CACHED [build 3/3] RUN --mount=type=cache,target=/var/cache/apt,sharing=locked --mount=type=cache,target=/var/lib/apt,sharing=locked apt-get update && apt install -y curl gz 0.0s
=> CACHED [go_build 2/9] COPY --from=build / / 0.0s
=> CACHED [blender 2/5] RUN apt-get update && apt install -y curl gzip xz-utils 0.0s
=> CACHED [blender 3/5] RUN curl -o /usr/local/blender-3.5.1-linux-x64.tar.xz https://download.blender.org/release/Blender3.5/blender-3.5.1-linux-x64.tar.xz 0.0s
=> CACHED [blender 4/5] RUN tar -xf /usr/local/blender-3.5.1-linux-x64.tar.xz -C /usr/local/ 0.0s
=> CACHED [blender 5/5] RUN mv /usr/local/blender-3.5.1-linux-x64 /usr/local/blender 0.0s
=> CACHED [go_build 3/9] COPY --from=blender /usr/local/blender /usr/local/blender 0.0s
=> CACHED [go_build 4/9] RUN mkdir -p /media/shared/flamenco/development && mkdir -p /code 0.0s
=> CACHED [go_build 5/9] WORKDIR /code 0.0s
=> CACHED [go_build 6/9] COPY go.mod go.sum . 0.0s
=> CACHED [go_build 7/9] RUN go mod download 0.0s
=> CACHED [go_build 8/9] COPY . /code 0.0s
=> CACHED [go_build 9/9] RUN --mount=type=cache,target=/root/.cache/go-build --mount=type=cache,target=/go/pkg GOOS=linux GOARCH=amd64 make with-deps 0.0s
=> exporting to docker image format 269.5s
=> => exporting layers 174.1s
=> => exporting manifest sha256:39f36aa192b91b64114149d785b101f9aff8abc30ab4b09cdbfd110f468885f3 0.0s
=> => exporting config sha256:162c466bf078c71df251bad74b07b4a1d50bc2438630d1ba4f9c93165a1bcec8 0.0s
=> => sending tarball 95.3s
=> importing to docker 63.6s
Multi-Platform Builds
Using this example Mutli-Stage buildx compatible Flamenco Development Dockerfile.
Docker has default buildx configuration that supports popular platform targets. You can also create your own custom buildx builder.
Create a buildx context…
docker buildx create --use
This context is given a random name…
clever_proskuriakova
docker buildx inspect
Example on MacOS…
Name: default
Driver: docker
Nodes:
Name: default
Endpoint: desktop-linux
Status: running
Buildkit: 20.10.21
Platforms: linux/amd64, linux/arm64, linux/riscv64,
linux/ppc64le, linux/s390x, linux/386, linux/arm/v7,
linux/arm/v6
Example on Ubuntu…
Name: clever_proskuriakova
Driver: docker-container
Last Activity: 2023-03-11 02:16:04 +0000 UTC
Nodes:
Name: clever_proskuriakova0
Endpoint: unix:///var/run/docker.sock
Status: running
Buildkit: v0.11.6
Platforms: linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64
If you dont see any emulators you can install them with tonistiigi/binfmt.
docker run --privileged --rm tonistiigi/binfmt --install all
Unable to find image 'tonistiigi/binfmt:latest' locally
latest: Pulling from tonistiigi/binfmt
8d4d64c318a5: Pull complete
e9c608ddc3cb: Pull complete
Digest: sha256:66e11bea77a5ea9d6f0fe79b57cd2b189b5d15b93a2bdb925be22949232e4e55
Status: Downloaded newer image for tonistiigi/binfmt:latest
installing: s390x OK
installing: ppc64le OK
installing: mips64le OK
installing: mips64 OK
installing: arm OK
installing: arm64 OK
installing: riscv64 OK
{
"supported": [
"linux/amd64",
"linux/arm64",
"linux/riscv64",
"linux/ppc64le",
"linux/s390x",
"linux/386",
"linux/mips64le",
"linux/mips64",
"linux/arm/v7",
"linux/arm/v6"
],
"emulators": [
"qemu-aarch64",
"qemu-arm",
"qemu-mips64",
"qemu-mips64el",
"qemu-ppc64le",
"qemu-riscv64",
"qemu-s390x"
]
}
You can list all the buildkits and their supported platforms…
docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS
clever_proskuriakova * docker-container
clever_proskuriakova0 unix:///var/run/docker.sock running v0.11.6 linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64
default docker
default default running 23.0.6 linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
Under the hood it is using qemu within a container to emulate the appropriate platform.
This Flamenco Development Container is based on using Docker buildkit and documentation from Tonis Tiigi at Docker - Faster Multi-Platform Builds: Dockerfile Cross-Compilation Guide.
Multi-platform Docker builds are currently only supported when using BuildKit with docker-container drivers. So you have to create a custom buildx context.
Then use buildx to build the arm64 target…
docker buildx build --platform=linux/arm64 .
Note, the --load option is not present because the host workstation is amd64 and you cannot push an arm64 image into the hosts Docker images because the platform doesn’t match.
This uses the buildx_buildkit_clever_proskuriakova context…
+] Building 96.9s (19/19) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 2.96kB 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/ubuntu:jammy 0.7s
=> [internal] load build context 0.1s
=> => transferring context: 81.52kB 0.1s
=> [blender 1/5] FROM docker.io/library/ubuntu:jammy@sha256:dfd64a3b4296d8c9b62aa3309984f8620b98d87e47492599ee20739e8eb54fbf 0.0s
=> => resolve docker.io/library/ubuntu:jammy@sha256:dfd64a3b4296d8c9b62aa3309984f8620b98d87e47492599ee20739e8eb54fbf 0.0s
=> CACHED [build 2/3] RUN --mount=type=cache,target=/var/cache/apt,sharing=locked --mount=type=cache,target=/var/lib/apt,sharing=locked apt-get upda 0.0s
=> CACHED [build 3/3] RUN --mount=type=cache,target=/var/cache/apt,sharing=locked --mount=type=cache,target=/var/lib/apt,sharing=locked apt-get upda 0.0s
=> CACHED [go_build 2/9] COPY --from=build / / 0.0s
=> CACHED [blender 2/5] RUN apt-get update && apt install -y curl gzip xz-utils 0.0s
=> CACHED [blender 3/5] RUN curl -o /usr/local/blender-3.5.1-linux-x64.tar.xz https://download.blender.org/release/Blender3.5/blender-3.5.1-linux-x64.tar.xz 0.0s
=> CACHED [blender 4/5] RUN tar -xf /usr/local/blender-3.5.1-linux-x64.tar.xz -C /usr/local/ 0.0s
=> CACHED [blender 5/5] RUN mv /usr/local/blender-3.5.1-linux-x64 /usr/local/blender 0.0s
=> CACHED [go_build 3/9] COPY --from=blender /usr/local/blender /usr/local/blender 0.0s
=> CACHED [go_build 4/9] RUN mkdir -p /media/shared/flamenco/development && mkdir -p /code 0.0s
=> CACHED [go_build 5/9] WORKDIR /code 0.0s
=> CACHED [go_build 6/9] COPY go.mod go.sum . 0.0s
=> CACHED [go_build 7/9] RUN go mod download 0.0s
=> CACHED [go_build 8/9] COPY . /code 0.0s
=> [go_build 9/9] RUN --mount=type=cache,target=/root/.cache/go-build --mount=type=cache,target=/go/pkg GOOS=${TARGETOS} GOARCH=${TARGETARCH} make 96.0s
You can run multiple platforms at once and Docker buildx will create the builds in parallel.
Build amd64 and arm64 targets…
docker buildx build --platform=linux/amd64,linux/arm64 .
Exporting Flamenco Execs
For the native workstation builds this is documented in Executables guide and you copy the execs out of the Docker flamenco_dev image.
For cross-platform executables, for example arm64, there is no image because you cannot push the buildx images into the Docker images if they are a different machine/platform type.
There is a stage in the example Dockerfile that allows you to extract these executables. It is at the end of the mutli-stage build and will create an execs folder in the working-copy directory.
docker buildx build --platform=linux/amd64,linux/arm64 --target export -t test . --output execs
Within the execs/ folder you will find platform specific executables.
#> ls execs/linux_arm64/
flamenco-manager_arm64 flamenco-worker_arm64
#> file execs/linux_arm64/flamenco-manager_arm64
execs/linux_arm64/flamenco-manager_arm64: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=Y2Stauo0uz0aoBZv6R_A/4yRY24jTniprnptY4cRL/e8IEkEKk2hA3DMr4Vk3Y/P-HFOLZcHCFq3ji28TL5, with debug_info, not stripped
Dockerfile RUN Mounts
This is a neat way of not poluting the docker images with temporary files such as those created during loadbuild.
RUN --mount=type=cache,target=/root/.cache/go-build \
--mount=type=cache,target=/go/pkg \
GOOS=${TARGETOS} GOARCH=${TARGETARCH} make with-deps
There are a few statements in the example Dockerfile that include mounts to the working-copy directory, caches and temporary directories to help cache and speed up the build.
Troubleshooting
addon-packer not found
This is an error pointing to wrong machine type of a tool.
#0 322.9 ./addon-packer -filename web/static/flamenco3-addon.zip
#0 322.9 ./addon-packer: 1: 5: not found
#0 322.9 ./addon-packer: 1: ./addon-packer: 1: �: not found
]ѓ: not founddon-packer: 1: ELF�P�@�@8@@@@�����ddP�
#0 322.9 : not found
#0 322.9 ./addon-packer: 2: Syntax error: word unexpected (expecting ")")
#0 322.9 make[2]: *** [Makefile:103: webapp-static] Error 2
#0 322.9 make[2]: Leaving directory '/code'
#0 322.9 make[1]: Leaving directory '/code'
#0 322.9 make[1]: *** [Makefile:70: flamenco-manager] Error 2
#0 322.9 make: *** [Makefile:65: with-deps] Error 2
------
Dockerfile:90
--------------------
89 | # for the appropriate target OS & Arch
90 | >>> RUN --mount=type=cache,target=/root/.cache/go-build \
91 | >>> --mount=type=cache,target=/go/pkg \
92 | >>> GOOS=${TARGETOS} GOARCH=${TARGETARCH} make with-deps
93 |
--------------------
ERROR: failed to solve: process "/bin/sh -c GOOS=${TARGETOS} GOARCH=${TARGETARCH} make with-deps" did not complete successfully: exit code: 2
Solution: Check that tonistiigi/binfmt emulation tool is installed.
docker run --privileged --rm tonistiigi/binfmt --install all
SYS_UTIMES
3.1-96-ga692444f -X git.blender.org/flamenco/internal/appinfo.ReleaseCycle=release" git.blender.org/flamenco/cmd/flamenco-manager
#0 58.91 git.blender.org/flamenco/internal/manager/config
#0 58.95 git.blender.org/flamenco/pkg/api
#0 59.05 git.blender.org/flamenco/pkg/shaman/touch
#0 59.08 # git.blender.org/flamenco/pkg/shaman/touch
#0 59.08 pkg/shaman/touch/touch_linux.go:38:38: undefined: syscall.SYS_UTIMES
#0 59.14 git.blender.org/flamenco/web
#0 59.15 git.blender.org/flamenco/internal/upnp_ssdp
#0 60.06 git.blender.org/flamenco/internal/manager/job_compilers
#0 60.06 git.blender.org/flamenco/internal/find_blender
#0 60.23 git.blender.org/flamenco/internal/manager/persistence
#0 60.48 git.blender.org/flamenco/internal/manager/webupdates
#0 60.59 git.blender.org/flamenco/internal/manager/sleep_scheduler
#0 60.59 git.blender.org/flamenco/internal/manager/task_logs
#0 60.73 git.blender.org/flamenco/internal/manager/task_state_machine
#0 60.86 git.blender.org/flamenco/internal/manager/timeout_checker
#0 60.96 make[1]: Leaving directory '/code'
#0 60.96 make[1]: *** [Makefile:61: flamenco-manager] Error 1
#0 60.96 make: *** [Makefile:55: with-deps] Error 2
Solution: This is addressed in Flamenco v3.3 or newer.
Resources
- Docker buildx multi-platform it is possible to build for different platform targets (eg: amd64, arm64, riscv64 etc).
- There is more background info from Faster Multi-Platform Builds: Dockerfile Cross-Compilation Guide blog post by Tonis Tiigi.
- https://www.docker.com/blog/multi-arch-images/ blog article on Docker Desktop and multi-platform builds using qemu.
- Extracting files from multi-stage Docker Builds using BuildKit