parent
98439f7f08
commit
a2f1de6459
@ -0,0 +1,156 @@
|
||||
---
|
||||
title: "Remote builder"
|
||||
description: "Connect buildx to an external buildkitd instance"
|
||||
keywords: build, buildx, buildkit
|
||||
---
|
||||
|
||||
The buildx remote driver allows for more complex custom build workloads that
|
||||
allow users to connect to external buildkit instances. This is useful for
|
||||
scenarios that require manual management of the buildkit daemon, or where a
|
||||
buildkit daemon is exposed from another source.
|
||||
|
||||
To connect to a running buildkitd instance:
|
||||
|
||||
```console
|
||||
$ docker buildx create \
|
||||
--name remote \
|
||||
--driver remote \
|
||||
tcp://localhost:1234
|
||||
```
|
||||
|
||||
## Remote Buildkit over Unix sockets
|
||||
|
||||
In this scenario, we'll create a setup with buildkitd listening on a unix
|
||||
socket, and have buildx connect through it.
|
||||
|
||||
Firstly, ensure that [buildkit](https://github.com/moby/buildkit) is installed.
|
||||
For example, you can launch an instance of buildkitd with:
|
||||
|
||||
```console
|
||||
$ sudo ./buildkitd --group $(id -gn) --addr unix://$HOME/buildkitd.sock
|
||||
```
|
||||
|
||||
Alternatively, [see here](https://github.com/moby/buildkit/blob/master/docs/rootless.md)
|
||||
for running buildkitd in rootless mode or [here](https://github.com/moby/buildkit/tree/master/examples/systemd)
|
||||
for examples of running it as a systemd service.
|
||||
|
||||
You should now have a unix socket accessible to your user, that is available to
|
||||
connect to:
|
||||
|
||||
```console
|
||||
$ ls -lh /home/user/buildkitd.sock
|
||||
srw-rw---- 1 root user 0 May 5 11:04 /home/user/buildkitd.sock
|
||||
```
|
||||
|
||||
You can then connect buildx to it with the remote driver:
|
||||
|
||||
```console
|
||||
$ buildx create \
|
||||
--name remote-unix \
|
||||
--driver remote \
|
||||
unix://$HOME/buildkitd.sock
|
||||
```
|
||||
|
||||
If you list available builders, you should then see `remote-unix` among them:
|
||||
|
||||
```console
|
||||
$ buildx ls
|
||||
NAME/NODE DRIVER/ENDPOINT STATUS PLATFORMS
|
||||
test remote
|
||||
test0 unix:///home/.../buildkitd.sock running linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
|
||||
default * docker
|
||||
default default running linux/amd64, linux/386
|
||||
```
|
||||
|
||||
We can switch to this new builder as the default using `docker buildx use remote-unix`,
|
||||
or specify it per build:
|
||||
|
||||
```console
|
||||
$ docker buildx build --builder=remote-unix -t test --load .
|
||||
```
|
||||
|
||||
(remember that `--load` is necessary when not using the default `docker`
|
||||
driver, to load the build result into the docker daemon)
|
||||
|
||||
## Remote Buildkit in Docker container
|
||||
|
||||
In this scenario, we'll create a similar setup to the `docker-container`
|
||||
driver, by manually booting a buildkit docker container and connecting to it
|
||||
using the buildx remote driver. In most cases you'd probably just use the
|
||||
`docker-container` driver that connects to buildkit through the Docker daemon,
|
||||
but in this case we manually create a container and access it via it's exposed
|
||||
port.
|
||||
|
||||
First, we need to generate certificates for buildkit - you can use the
|
||||
[create-certs.sh](https://github.com/moby/buildkit/v0.10.3/master/examples/kubernetes/create-certs.sh)
|
||||
script as a starting point. Note, that while it is *possible* to expose
|
||||
buildkit over TCP without using TLS, it is **not recommended**, since this will
|
||||
allow arbitrary access to buildkit without credentials.
|
||||
|
||||
With our certificates generated in `.certs/`, we startup the container:
|
||||
|
||||
```console
|
||||
$ docker run -d --rm \
|
||||
--name=remote-buildkitd \
|
||||
--privileged \
|
||||
-p 1234:1234 \
|
||||
-v $PWD/.certs:/etc/buildkit/certs \
|
||||
moby/buildkit:latest \
|
||||
--addr tcp://0.0.0.0:1234 \
|
||||
--tlscacert /etc/buildkit/certs/ca.pem \
|
||||
--tlscert /etc/buildkit/certs/daemon-cert.pem \
|
||||
--tlskey /etc/buildkit/certs/daemon-key.pem
|
||||
```
|
||||
|
||||
The above command starts a buildkit container and exposes the daemon's port
|
||||
1234 to localhost.
|
||||
|
||||
We can now connect to this running container using buildx:
|
||||
|
||||
```console
|
||||
$ docker buildx create \
|
||||
--name remote-container \
|
||||
--driver remote \
|
||||
--driver-opt cacert=.certs/ca.pem,cert=.certs/client-cert.pem,key=.certs/client-key.pem,servername=... \
|
||||
tcp://buildkitd.default.svc:1234
|
||||
```
|
||||
|
||||
## Remote Buildkit in Kubernetes
|
||||
|
||||
In this scenario, we'll create a similar setup to the `kubernetes` driver by
|
||||
manually creating a buildkit `Deployment`. While the `kubernetes` driver will
|
||||
do this under-the-hood, it might sometimes be desirable to scale buildkit
|
||||
manually. Additionally, when executing builds from inside Kubernetes pods,
|
||||
the buildx builder will need to be recreated from within each pod or copied
|
||||
between them.
|
||||
|
||||
Firstly, we can create a kubernetes deployment of buildkitd, as per the
|
||||
instructions [here](https://github.com/moby/buildkit/tree/master/examples/kubernetes).
|
||||
Following the guide, we setup certificates for the buildkit daemon and client
|
||||
(as above using [create-certs.sh](https://github.com/moby/buildkit/blob/v0.10.3/examples/kubernetes/create-certs.sh))
|
||||
and create a `Deployment` of buildkit pods with a service that connects to
|
||||
them.
|
||||
|
||||
Assuming that the service is called `buildkitd`, we can create a remote builder
|
||||
in buildx, ensuring that the listed certificate files are present:
|
||||
|
||||
```console
|
||||
$ docker buildx create \
|
||||
--name remote-kubernetes \
|
||||
--driver remote \
|
||||
--driver-opt cacert=.certs/ca.pem,cert=.certs/client-cert.pem,key=.certs/client-key.pem \
|
||||
tcp://buildkitd.default.svc:1234
|
||||
```
|
||||
|
||||
Note that the above will only work in-cluster (since the buildkit setup guide
|
||||
only creates a ClusterIP service). To configure the builder to be accessible
|
||||
remotely, you can use an appropriately configured Ingress, which is outside the
|
||||
scope of this guide.
|
||||
|
||||
Alternatively, to access remotely use the port forwarding mechanism in kubectl:
|
||||
|
||||
```console
|
||||
$ kubectl port-forward svc/buildkitd 1234:1234
|
||||
```
|
||||
|
||||
Then you can simply point the remote driver at `tcp://localhost:1234`.
|
Loading…
Reference in New Issue