diff --git a/README.md b/README.md index 0dd4b829..5a9af3ee 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,10 @@ # buildx -[![PkgGoDev](https://img.shields.io/badge/go.dev-docs-007d9c?logo=go&logoColor=white)](https://pkg.go.dev/github.com/docker/buildx) -[![Build Status](https://github.com/docker/buildx/workflows/build/badge.svg)](https://github.com/docker/buildx/actions?query=workflow%3Abuild) -[![Go Report Card](https://goreportcard.com/badge/github.com/docker/buildx)](https://goreportcard.com/report/github.com/docker/buildx) -[![codecov](https://codecov.io/gh/docker/buildx/branch/master/graph/badge.svg)](https://codecov.io/gh/docker/buildx) +[![GitHub release](https://img.shields.io/github/release/docker/buildx.svg?style=flat-square)](https://github.com/docker/buildx/releases/latest) +[![PkgGoDev](https://img.shields.io/badge/go.dev-docs-007d9c?style=flat-square&logo=go&logoColor=white)](https://pkg.go.dev/github.com/docker/buildx) +[![Build Status](https://img.shields.io/github/workflow/status/docker/buildx/build?label=build&logo=github&style=flat-square)](https://github.com/docker/buildx/actions?query=workflow%3Abuild) +[![Go Report Card](https://goreportcard.com/badge/github.com/docker/buildx?style=flat-square)](https://goreportcard.com/report/github.com/docker/buildx) +[![codecov](https://img.shields.io/codecov/c/github/docker/buildx?logo=codecov&style=flat-square)](https://codecov.io/gh/docker/buildx) `buildx` is a Docker CLI plugin for extended build capabilities with [BuildKit](https://github.com/moby/buildkit). @@ -32,18 +33,25 @@ Key features: - [Working with builder instances](#working-with-builder-instances) - [Building multi-platform images](#building-multi-platform-images) - [High-level build options](#high-level-build-options) -- [Documentation](docs/reference/buildx.md) - - [`buildx bake`](docs/reference/buildx_bake.md) +- [Guides](docs/guides) + - [CI/CD](docs/guides/cicd.md) + - [CNI networking](docs/guides/cni-networking.md) + - [Registry mirror](docs/guides/registry-mirror.md) + - [Resource limiting](docs/guides/resource-limiting.md) + - [Using a custom network](docs/guides/custom-network.md) + - [Using a custom registry configuration](docs/guides/custom-registry-config.md) +- [Reference](docs/reference/buildx.md) + - [`buildx bake`](docs/reference/buildx_bake.md) - [`buildx build`](docs/reference/buildx_build.md) - [`buildx create`](docs/reference/buildx_create.md) - [`buildx du`](docs/reference/buildx_du.md) - [`buildx imagetools`](docs/reference/buildx_imagetools.md) - [`buildx imagetools create`](docs/reference/buildx_imagetools_create.md) - - [`buildx imagetools inspect`](docs/reference/buildx_imagetools_inspect.md) + - [`buildx imagetools inspect`](docs/reference/buildx_imagetools_inspect.md) - [`buildx inspect`](docs/reference/buildx_inspect.md) - [`buildx install`](docs/reference/buildx_install.md) - - [`buildx ls`](docs/reference/buildx_ls.md) - - [`buildx prune`](docs/reference/buildx_prune.md) + - [`buildx ls`](docs/reference/buildx_ls.md) + - [`buildx prune`](docs/reference/buildx_prune.md) - [`buildx rm`](docs/reference/buildx_rm.md) - [`buildx stop`](docs/reference/buildx_stop.md) - [`buildx uninstall`](docs/reference/buildx_uninstall.md) diff --git a/commands/build.go b/commands/build.go index 5e8110d2..2b00c2c7 100644 --- a/commands/build.go +++ b/commands/build.go @@ -344,7 +344,6 @@ func buildCmd(dockerCli command.Cli, rootOpts *rootOptions) *cobra.Command { flags.StringSliceVar(&options.allow, "allow", []string{}, `Allow extra privileged entitlement (e.g., "network.host", "security.insecure")`) flags.StringArrayVar(&options.buildArgs, "build-arg", []string{}, "Set build-time variables") - flags.SetAnnotation("build-arg", annotation.ExternalURL, []string{"https://docs.docker.com/engine/reference/commandline/build/#set-build-time-variables---build-arg"}) flags.StringArrayVar(&options.cacheFrom, "cache-from", []string{}, `External cache sources (e.g., "user/app:cache", "type=local,src=path/to/dir")`) diff --git a/docs/guides/cicd.md b/docs/guides/cicd.md new file mode 100644 index 00000000..5119c091 --- /dev/null +++ b/docs/guides/cicd.md @@ -0,0 +1,48 @@ +# CI/CD + +## GitHub Actions + +Docker provides a [GitHub Action that will build and push your image](https://github.com/docker/build-push-action/#about) +using Buildx. Here is a simple workflow: + +```yaml +name: ci + +on: + push: + branches: + - 'main' + +jobs: + docker: + runs-on: ubuntu-latest + steps: + - + name: Set up QEMU + uses: docker/setup-qemu-action@v1 + - + name: Set up Docker Buildx + uses: docker/setup-buildx-action@v1 + - + name: Login to DockerHub + uses: docker/login-action@v1 + with: + username: ${{ secrets.DOCKERHUB_USERNAME }} + password: ${{ secrets.DOCKERHUB_TOKEN }} + - + name: Build and push + uses: docker/build-push-action@v2 + with: + push: true + tags: user/app:latest +``` + +In this example we are also using 3 other actions: + +* [`setup-buildx`](https://github.com/docker/setup-buildx-action) action will create and boot a builder using by +default the `docker-container` [builder driver](../reference/buildx_create.md#driver). +This is **not required but recommended** using it to be able to build multi-platform images, export cache, etc. +* [`setup-qemu`](https://github.com/docker/setup-qemu-action) action can be useful if you want +to add emulation support with QEMU to be able to build against more platforms. +* [`login`](https://github.com/docker/login-action) action will take care to log +in against a Docker registry. diff --git a/docs/guides/cni-networking.md b/docs/guides/cni-networking.md new file mode 100644 index 00000000..07aa4cb4 --- /dev/null +++ b/docs/guides/cni-networking.md @@ -0,0 +1,23 @@ +# CNI networking + +It can be useful to use a bridge network for your builder if for example you +encounter a network port contention during multiple builds. If you're using +the BuildKit image, CNI is not yet available in it, but you can create +[a custom BuildKit image with CNI support](https://github.com/moby/buildkit/blob/master/docs/cni-networking.md). + +Now build this image: + +```console +$ docker buildx build --tag buildkit-cni:local --load . +``` + +Then [create a `docker-container` builder](../reference/buildx_create.md) that +will use this image: + +```console +$ docker buildx create --use \ + --name mybuilder \ + --driver docker-container \ + --driver-opt "image=buildkit-cni:local" \ + --buildkitd-flags "--oci-worker-net=cni" +``` diff --git a/docs/guides/custom-network.md b/docs/guides/custom-network.md new file mode 100644 index 00000000..4377f290 --- /dev/null +++ b/docs/guides/custom-network.md @@ -0,0 +1,48 @@ +# Using a custom network + +[Create a network](https://docs.docker.com/engine/reference/commandline/network_create/) +named `foonet`: + +```console +$ docker network create foonet +``` + +[Create a `docker-container` builder](../reference/buildx_create.md) named +`mybuilder` that will use this network: + +```console +$ docker buildx create --use \ + --name mybuilder \ + --driver docker-container \ + --driver-opt "network=foonet" +``` + +Boot and [inspect `mybuilder`](../reference/buildx_inspect.md): + +```console +$ docker buildx inspect --bootstrap +``` + +[Inspect the builder container](https://docs.docker.com/engine/reference/commandline/inspect/) +and see what network is being used: + +```console +$ docker inspect buildx_buildkit_mybuilder0 --format={{.NetworkSettings.Networks}} +map[foonet:0xc00018c0c0] +``` + +## What's `buildx_buildkit_mybuilder0`? + +`buildx_buildkit_mybuilder0` is the container name. It can be broken down like this: + +* `buildx_buildkit_` is a hardcoded prefix +* `mybuilder0` is the name of the node (defaults to builder name + position in the list of nodes) + +```console +$ docker buildx ls +NAME/NODE DRIVER/ENDPOINT STATUS PLATFORMS +mybuilder * docker-container + mybuilder0 unix:///var/run/docker.sock running linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6 +default docker + default default running linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6 +``` diff --git a/docs/guides/custom-registry-config.md b/docs/guides/custom-registry-config.md new file mode 100644 index 00000000..4d702b51 --- /dev/null +++ b/docs/guides/custom-registry-config.md @@ -0,0 +1,63 @@ +# Using a custom registry configuration + +If you [create a `docker-container` or `kubernetes` builder](../reference/buildx_create.md) and +have specified certificates for registries in the [BuildKit daemon configuration](https://github.com/moby/buildkit/blob/master/docs/buildkitd.toml.md), +the files will be copied into the container under `/etc/buildkit/certs` and +configuration will be updated to reflect that. + +Take the following `buildkitd.toml` configuration that will be used for +pushing an image to this registry using self-signed certificates: + +```toml" +debug = true +[registry."myregistry.com"] + ca=["/etc/certs/myregistry.pem"] + [[registry."myregistry.com".keypair]] + key="/etc/certs/myregistry_key.pem" + cert="/etc/certs/myregistry_cert.pem" +``` +> `/etc/buildkitd.toml` + +Here we have configured a self-signed certificate for `myregistry.com` registry. + +Now [create a `docker-container` builder](../reference/buildx_create.md) +that will use this BuildKit configuration: + +```console +$ docker buildx create --use \ + --name mybuilder \ + --driver docker-container \ + --config /etc/buildkitd.toml +``` + +Inspecting the builder container, you can see that buildkitd configuration +has changed: + +```console +$ docker exec -it buildx_buildkit_mybuilder0 cat /etc/buildkit/buildkitd.toml +``` +```toml +debug = true + +[registry] + + [registry."myregistry.com"] + ca = ["/etc/buildkit/certs/myregistry.com/myregistry.pem"] + + [[registry."myregistry.com".keypair]] + cert = "/etc/buildkit/certs/myregistry.com/myregistry_cert.pem" + key = "/etc/buildkit/certs/myregistry.com/myregistry_key.pem" +``` + +And certificates copied inside the container: + +```console +$ docker exec -it buildx_buildkit_mybuilder0 ls /etc/buildkit/certs/myregistry.com/ +myregistry.pem myregistry_cert.pem myregistry_key.pem +``` + +Now you should be able to push to the registry with this builder: + +```console +$ docker buildx build --push --tag myregistry.com/myimage:latest . +``` diff --git a/docs/guides/opentelemetry.md b/docs/guides/opentelemetry.md new file mode 100644 index 00000000..0d67750a --- /dev/null +++ b/docs/guides/opentelemetry.md @@ -0,0 +1,31 @@ +# OpenTelemetry support + +To capture the trace to [Jaeger](https://github.com/jaegertracing/jaeger), set +`JAEGER_TRACE` environment variable to the collection address using a `driver-opt`. + +First create a Jaeger container: + +```console +$ docker run -d --name jaeger -p "6831:6831/udp" -p "16686:16686" jaegertracing/all-in-one +``` + +Then [create a `docker-container` builder](../reference/buildx_create.md) +that will use the Jaeger instance via the `JAEGER_TRACE` env var: + +```console +$ docker buildx create --use \ + --name mybuilder \ + --driver docker-container \ + --driver-opt "network=host" \ + --driver-opt "env.JAEGER_TRACE=localhost:6831" +``` + +Boot and [inspect `mybuilder`](../reference/buildx_inspect.md): + +```console +$ docker buildx inspect --bootstrap +``` + +Buildx commands should be traced at `http://127.0.0.1:16686/`: + +![](https://user-images.githubusercontent.com/1951866/124468052-ef085400-dd98-11eb-84ab-7ac8e261dd52.png) diff --git a/docs/guides/registry-mirror.md b/docs/guides/registry-mirror.md new file mode 100644 index 00000000..22f36133 --- /dev/null +++ b/docs/guides/registry-mirror.md @@ -0,0 +1,60 @@ +# Registry mirror + +You can define a registry mirror to use for your builds by providing a [BuildKit daemon configuration](https://github.com/moby/buildkit/blob/master/docs/buildkitd.toml.md) +while creating a builder with the [`--config` flags](../reference/buildx_create.md#config). + +```toml +debug = true +[registry."docker.io"] + mirrors = ["mirror.gcr.io"] +``` +> `/etc/buildkitd.toml` + +> :information_source: `debug = true` has been added to be able to debug requests +in the BuildKit daemon and see if the mirror is effectively used. + +Then [create a `docker-container` builder](../reference/buildx_create.md) +that will use this BuildKit configuration: + +```console +$ docker buildx create --use \ + --name mybuilder \ + --driver docker-container \ + --config /etc/buildkitd.toml +``` + +Boot and [inspect `mybuilder`](../reference/buildx_inspect.md): + +```console +$ docker buildx inspect --bootstrap +``` + +Build an image: + +```console +$ docker buildx build --load . -f-< `/etc/buildkitd.toml` + +Now you can [create a `docker-container` builder](../reference/buildx_create.md) +that will use this BuildKit configuration to limit parallelism. + +```console +$ docker buildx create --use \ + --name mybuilder \ + --driver docker-container \ + --config /etc/buildkitd.toml +``` + +## Limit on TCP connections + +We are also now limiting TCP connections to **4 per registry** with an additional +connection not used for layer pulls and pushes. This limitation will be able to +manage TCP connection per host to avoid your build being stuck while pulling +images. The additional connection is used for metadata requests +(image config retrieval) to enhance the overall build time. + +More info: [moby/buildkit#2259](https://github.com/moby/buildkit/pull/2259) diff --git a/docs/reference/buildx_bake.md b/docs/reference/buildx_bake.md index 8b3e7180..f99c3e17 100644 --- a/docs/reference/buildx_bake.md +++ b/docs/reference/buildx_bake.md @@ -264,7 +264,6 @@ $ docker buildx bake --progress=plain ... ``` - ### Always attempt to pull a newer version of the image (--pull) Same as `build --pull`. @@ -278,9 +277,6 @@ Same as `build --pull`. Override target configurations from command line. The pattern matching syntax is defined in https://golang.org/pkg/path/#Match. - -**Examples** - ```console $ docker buildx bake --set target.args.mybuildarg=value $ docker buildx bake --set target.platform=linux/arm64 @@ -313,8 +309,7 @@ groups to inherit from. Note: Design of bake command is work in progress, the user experience may change based on feedback. - -**Example HCL definition** +HCL definition example: ```hcl group "default" { @@ -870,7 +865,6 @@ target "app" { Please note that in most cases you should just use a single multi-stage Dockerfile with multiple targets for similar behavior. This case is recommended when you have multiple Dockerfiles that can't be easily merged into one. - ### Extension field with Compose [Special extension](https://github.com/compose-spec/compose-spec/blob/master/spec.md#extension) diff --git a/docs/reference/buildx_build.md b/docs/reference/buildx_build.md index 39ac9ac9..228a41d2 100644 --- a/docs/reference/buildx_build.md +++ b/docs/reference/buildx_build.md @@ -17,7 +17,7 @@ Start a build | --- | --- | | [`--add-host stringSlice`](https://docs.docker.com/engine/reference/commandline/build/#add-entries-to-container-hosts-file---add-host) | Add a custom host-to-IP mapping (format: `host:ip`) | | [`--allow stringSlice`](#allow) | Allow extra privileged entitlement (e.g., `network.host`, `security.insecure`) | -| [`--build-arg stringArray`](https://docs.docker.com/engine/reference/commandline/build/#set-build-time-variables---build-arg) | Set build-time variables | +| [`--build-arg stringArray`](#build-arg) | Set build-time variables | | [`--build-context stringArray`](#build-context) | Additional build contexts (e.g., name=path) | | [`--builder string`](#builder) | Override the configured builder instance | | [`--cache-from stringArray`](#cache-from) | External cache sources (e.g., `user/app:cache`, `type=local,src=path/to/dir`) | @@ -27,7 +27,7 @@ Start a build | `--iidfile string` | Write the image ID to the file | | `--label stringArray` | Set metadata for an image | | [`--load`](#load) | Shorthand for `--output=type=docker` | -| `--metadata-file string` | Write build result metadata to the file | +| [`--metadata-file string`](#metadata-file) | Write build result metadata to the file | | `--network string` | Set the networking mode for the `RUN` instructions during build | | `--no-cache` | Do not use cache when building the image | | `--no-cache-filter stringArray` | Do not cache specified stages | @@ -39,7 +39,7 @@ Start a build | `-q`, `--quiet` | Suppress the build output and print image ID on success | | [`--secret stringArray`](#secret) | Secret to expose to the build (format: `id=mysecret[,src=/local/secret]`) | | [`--shm-size bytes`](#shm-size) | Size of `/dev/shm` | -| `--ssh stringArray` | SSH agent socket or keys to expose to the build (format: `default\|[=\|[,]]`) | +| [`--ssh stringArray`](#ssh) | SSH agent socket or keys to expose to the build (format: `default\|[=\|[,]]`) | | [`-t`](https://docs.docker.com/engine/reference/commandline/build/#tag-an-image--t), [`--tag stringArray`](https://docs.docker.com/engine/reference/commandline/build/#tag-an-image--t) | Name and optionally a tag (format: `name:tag`) | | [`--target string`](https://docs.docker.com/engine/reference/commandline/build/#specifying-target-build-stage---target) | Set the target build stage to build | | [`--ulimit ulimit`](#ulimit) | Ulimit options | @@ -58,76 +58,195 @@ here we’ll document a subset of the new flags. ## Examples -### Override the configured builder instance (--builder) +### Allow extra privileged entitlement (--allow) -Same as [`buildx --builder`](buildx.md#builder). +``` +--allow=ENTITLEMENT +``` -### Set the target platforms for the build (--platform) +Allow extra privileged entitlement. List of entitlements: + +- `network.host` - Allows executions with host networking. +- `security.insecure` - Allows executions without sandbox. See + [related Dockerfile extensions](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md#run---securityinsecuresandbox). + +For entitlements to be enabled, the `buildkitd` daemon also needs to allow them +with `--allow-insecure-entitlement` (see [`create --buildkitd-flags`](buildx_create.md#buildkitd-flags)) + +**Examples** +```console +$ docker buildx create --use --name insecure-builder --buildkitd-flags '--allow-insecure-entitlement security.insecure' +$ docker buildx build --allow security.insecure . ``` ---platform=value[,value] + +### Set build-time variables (--build-arg) + +Same as [`docker build` command](https://docs.docker.com/engine/reference/commandline/build/#set-build-time-variables---build-arg). + +There are also useful built-in build args like: + +* `BUILDKIT_CONTEXT_KEEP_GIT_DIR=` trigger git context to keep the `.git` directory +* `BUILDKIT_INLINE_CACHE=` inline cache metadata to image config or not +* `BUILDKIT_MULTI_PLATFORM=` opt into determnistic output regardless of multi-platform output or not + +```console +$ docker buildx build --build-arg BUILDKIT_MULTI_PLATFORM=1 . ``` -Set the target platform for the build. All `FROM` commands inside the Dockerfile -without their own `--platform` flag will pull base images for this platform and -this value will also be the platform of the resulting image. The default value -will be the current platform of the buildkit daemon. +More built-in build args can be found in [dockerfile frontend docs](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/syntax.md#built-in-build-args). -When using `docker-container` driver with `buildx`, this flag can accept multiple -values as an input separated by a comma. With multiple values the result will be -built for all of the specified platforms and joined together into a single manifest -list. +### Additional build contexts (--build-context) -If the `Dockerfile` needs to invoke the `RUN` command, the builder needs runtime -support for the specified platform. In a clean setup, you can only execute `RUN` -commands for your system architecture. -If your kernel supports [`binfmt_misc`](https://en.wikipedia.org/wiki/Binfmt_misc) -launchers for secondary architectures, buildx will pick them up automatically. -Docker desktop releases come with `binfmt_misc` automatically configured for `arm64` -and `arm` architectures. You can see what runtime platforms your current builder -instance supports by running `docker buildx inspect --bootstrap`. +``` +--build-context=name=VALUE +``` -Inside a `Dockerfile`, you can access the current platform value through -`TARGETPLATFORM` build argument. Please refer to the [`docker build` -documentation](https://docs.docker.com/engine/reference/builder/#automatic-platform-args-in-the-global-scope) -for the full description of automatic platform argument variants . +Define additional build context with specified contents. In Dockerfile the context can be accessed when `FROM name` or `--from=name` is used. +When Dockerfile defines a stage with the same name it is overwritten. -The formatting for the platform specifier is defined in the [containerd source -code](https://github.com/containerd/containerd/blob/v1.4.3/platforms/platforms.go#L63). +The value can be a local source directory, container image (with docker-image:// prefix), Git or HTTP URL. -**Examples** +Replace `alpine:latest` with a pinned one: ```console -$ docker buildx build --platform=linux/arm64 . -$ docker buildx build --platform=linux/amd64,linux/arm64,linux/arm/v7 . -$ docker buildx build --platform=darwin . +$ docker buildx build --build-context alpine=docker-image://alpine@sha256:0123456789 . ``` -### Set type of progress output (--progress) +Expose a secondary local source directory: +```console +$ docker buildx build --build-context project=path/to/project/source . +# docker buildx build --build-context project=https://github.com/myuser/project.git . ``` ---progress=VALUE + +```Dockerfile +FROM alpine +COPY --from=project myfile / ``` -Set type of progress output (auto, plain, tty). Use plain to show container -output (default "auto"). +### Override the configured builder instance (--builder) -> You can also use the `BUILDKIT_PROGRESS` environment variable to set -> its value. +Same as [`buildx --builder`](buildx.md#builder). -The following example uses `plain` output during the build: +### Use an external cache source for a build (--cache-from) + +``` +--cache-from=[NAME|type=TYPE[,KEY=VALUE]] +``` + +Use an external cache source for a build. Supported types are `registry`, +`local` and `gha`. + +- [`registry` source](https://github.com/moby/buildkit#registry-push-image-and-cache-separately) + can import cache from a cache manifest or (special) image configuration on the + registry. +- [`local` source](https://github.com/moby/buildkit#local-directory-1) can + import cache from local files previously exported with `--cache-to`. +- [`gha` source](https://github.com/moby/buildkit#github-actions-cache-experimental) + can import cache from a previously exported cache with `--cache-to` in your + GitHub repository + +If no type is specified, `registry` exporter is used with a specified reference. + +`docker` driver currently only supports importing build cache from the registry. ```console -$ docker buildx build --load --progress=plain . +$ docker buildx build --cache-from=user/app:cache . +$ docker buildx build --cache-from=user/app . +$ docker buildx build --cache-from=type=registry,ref=user/app . +$ docker buildx build --cache-from=type=local,src=path/to/cache . +$ docker buildx build --cache-from=type=gha . +``` -#1 [internal] load build definition from Dockerfile -#1 transferring dockerfile: 227B 0.0s done -#1 DONE 0.1s +More info about cache exporters and available attributes: https://github.com/moby/buildkit#export-cache + +### Export build cache to an external cache destination (--cache-to) -#2 [internal] load .dockerignore -#2 transferring context: 129B 0.0s done -#2 DONE 0.0s -... +``` +--cache-to=[NAME|type=TYPE[,KEY=VALUE]] +``` + +Export build cache to an external cache destination. Supported types are +`registry`, `local`, `inline` and `gha`. + +- [`registry` type](https://github.com/moby/buildkit#registry-push-image-and-cache-separately) exports build cache to a cache manifest in the registry. +- [`local` type](https://github.com/moby/buildkit#local-directory-1) type + exports cache to a local directory on the client. +- [`inline` type](https://github.com/moby/buildkit#inline-push-image-and-cache-together) + type writes the cache metadata into the image configuration. +- [`gha` type](https://github.com/moby/buildkit#github-actions-cache-experimental) + type exports cache through the [Github Actions Cache service API](https://github.com/tonistiigi/go-actions-cache/blob/master/api.md#authentication). + +`docker` driver currently only supports exporting inline cache metadata to image +configuration. Alternatively, `--build-arg BUILDKIT_INLINE_CACHE=1` can be used +to trigger inline cache exporter. + +Attribute key: + +- `mode` - Specifies how many layers are exported with the cache. `min` on only + exports layers already in the final build stage, `max` exports layers for + all stages. Metadata is always exported for the whole build. + +```console +$ docker buildx build --cache-to=user/app:cache . +$ docker buildx build --cache-to=type=inline . +$ docker buildx build --cache-to=type=registry,ref=user/app . +$ docker buildx build --cache-to=type=local,dest=path/to/cache . +$ docker buildx build --cache-to=type=gha . +``` + +More info about cache exporters and available attributes: https://github.com/moby/buildkit#export-cache + +### Load the single-platform build result to `docker images` (--load) + +Shorthand for [`--output=type=docker`](#docker). Will automatically load the +single-platform build result to `docker images`. + +### Write build result metadata to the file (--metadata-file) + +To output build metadata such as the image digest, pass the `--metadata-file` flag. +The metadata will be written as a JSON object to the specified file. The +directory of the specified file must already exist and be writable. + +```console +$ docker buildx build --load --metadata-file metadata.json . +$ cat metadata.json +``` +```json +{ + "containerimage.buildinfo": { + "frontend": "dockerfile.v0", + "attrs": { + "context": "https://github.com/crazy-max/buildkit-buildsources-test.git#master", + "filename": "Dockerfile", + "source": "docker/dockerfile:master" + }, + "sources": [ + { + "type": "docker-image", + "ref": "docker.io/docker/buildx-bin:0.6.1@sha256:a652ced4a4141977c7daaed0a074dcd9844a78d7d2615465b12f433ae6dd29f0", + "pin": "sha256:a652ced4a4141977c7daaed0a074dcd9844a78d7d2615465b12f433ae6dd29f0" + }, + { + "type": "docker-image", + "ref": "docker.io/library/alpine:3.13", + "pin": "sha256:026f721af4cf2843e07bba648e158fb35ecc876d822130633cc49f707f0fc88c" + } + ] + }, + "containerimage.config.digest": "sha256:2937f66a9722f7f4a2df583de2f8cb97fc9196059a410e7f00072fc918930e66", + "containerimage.descriptor": { + "annotations": { + "config.digest": "sha256:2937f66a9722f7f4a2df583de2f8cb97fc9196059a410e7f00072fc918930e66", + "org.opencontainers.image.created": "2022-02-08T21:28:03Z" + }, + "digest": "sha256:19ffeab6f8bc9293ac2c3fdf94ebe28396254c993aea0b5a542cfb02e0883fa3", + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "size": 506 + }, + "containerimage.digest": "sha256:19ffeab6f8bc9293ac2c3fdf94ebe28396254c993aea0b5a542cfb02e0883fa3" +} ``` ### Set the export action for the build result (-o, --output) @@ -148,8 +267,6 @@ If just the path is specified as a value, `buildx` will use the local exporter with this path as the destination. If the value is "-", `buildx` will use `tar` exporter and write to `stdout`. -**Examples** - ```console $ docker buildx build -o . . $ docker buildx build -o outdir . @@ -203,7 +320,7 @@ The most common usecase for multi-platform images is to directly push to a regis Attribute keys: - `dest` - destination path where tarball will be written. If not specified the -tar will be loaded automatically to the current docker instance. + tar will be loaded automatically to the current docker instance. - `context` - name for the docker context where to import the result #### `image` @@ -221,146 +338,78 @@ Attribute keys: The `registry` exporter is a shortcut for `type=image,push=true`. - -### Push the build result to a registry (--push) - -Shorthand for [`--output=type=registry`](#registry). Will automatically push the -build result to registry. - -### Load the single-platform build result to `docker images` (--load) - -Shorthand for [`--output=type=docker`](#docker). Will automatically load the -single-platform build result to `docker images`. - -### Use an external cache source for a build (--cache-from) - -``` ---cache-from=[NAME|type=TYPE[,KEY=VALUE]] -``` - -Use an external cache source for a build. Supported types are `registry`, -`local` and `gha`. - -- [`registry` source](https://github.com/moby/buildkit#registry-push-image-and-cache-separately) -can import cache from a cache manifest or (special) image configuration on the -registry. -- [`local` source](https://github.com/moby/buildkit#local-directory-1) can -import cache from local files previously exported with `--cache-to`. -- [`gha` source](https://github.com/moby/buildkit#github-actions-cache-experimental) -can import cache from a previously exported cache with `--cache-to` in your -GitHub repository - -If no type is specified, `registry` exporter is used with a specified reference. - -`docker` driver currently only supports importing build cache from the registry. - -**Examples** - -```console -$ docker buildx build --cache-from=user/app:cache . -$ docker buildx build --cache-from=user/app . -$ docker buildx build --cache-from=type=registry,ref=user/app . -$ docker buildx build --cache-from=type=local,src=path/to/cache . -$ docker buildx build --cache-from=type=gha . -``` - -More info about cache exporters and available attributes: https://github.com/moby/buildkit#export-cache - -### Export build cache to an external cache destination (--cache-to) +### Set the target platforms for the build (--platform) ``` ---cache-to=[NAME|type=TYPE[,KEY=VALUE]] +--platform=value[,value] ``` -Export build cache to an external cache destination. Supported types are -`registry`, `local`, `inline` and `gha`. - -- [`registry` type](https://github.com/moby/buildkit#registry-push-image-and-cache-separately) exports build cache to a cache manifest in the registry. -- [`local` type](https://github.com/moby/buildkit#local-directory-1) type -exports cache to a local directory on the client. -- [`inline` type](https://github.com/moby/buildkit#inline-push-image-and-cache-together) -type writes the cache metadata into the image configuration. -- [`gha` type](https://github.com/moby/buildkit#github-actions-cache-experimental) -type exports cache through the [Github Actions Cache service API](https://github.com/tonistiigi/go-actions-cache/blob/master/api.md#authentication). +Set the target platform for the build. All `FROM` commands inside the Dockerfile +without their own `--platform` flag will pull base images for this platform and +this value will also be the platform of the resulting image. The default value +will be the current platform of the buildkit daemon. -`docker` driver currently only supports exporting inline cache metadata to image -configuration. Alternatively, `--build-arg BUILDKIT_INLINE_CACHE=1` can be used -to trigger inline cache exporter. +When using `docker-container` driver with `buildx`, this flag can accept multiple +values as an input separated by a comma. With multiple values the result will be +built for all of the specified platforms and joined together into a single manifest +list. -Attribute key: +If the `Dockerfile` needs to invoke the `RUN` command, the builder needs runtime +support for the specified platform. In a clean setup, you can only execute `RUN` +commands for your system architecture. +If your kernel supports [`binfmt_misc`](https://en.wikipedia.org/wiki/Binfmt_misc) +launchers for secondary architectures, buildx will pick them up automatically. +Docker desktop releases come with `binfmt_misc` automatically configured for `arm64` +and `arm` architectures. You can see what runtime platforms your current builder +instance supports by running `docker buildx inspect --bootstrap`. -- `mode` - Specifies how many layers are exported with the cache. `min` on only -exports layers already in the final build stage, `max` exports layers for -all stages. Metadata is always exported for the whole build. +Inside a `Dockerfile`, you can access the current platform value through +`TARGETPLATFORM` build argument. Please refer to the [`docker build` +documentation](https://docs.docker.com/engine/reference/builder/#automatic-platform-args-in-the-global-scope) +for the full description of automatic platform argument variants . -**Examples** +The formatting for the platform specifier is defined in the [containerd source +code](https://github.com/containerd/containerd/blob/v1.4.3/platforms/platforms.go#L63). ```console -$ docker buildx build --cache-to=user/app:cache . -$ docker buildx build --cache-to=type=inline . -$ docker buildx build --cache-to=type=registry,ref=user/app . -$ docker buildx build --cache-to=type=local,dest=path/to/cache . -$ docker buildx build --cache-to=type=gha . +$ docker buildx build --platform=linux/arm64 . +$ docker buildx build --platform=linux/amd64,linux/arm64,linux/arm/v7 . +$ docker buildx build --platform=darwin . ``` -More info about cache exporters and available attributes: https://github.com/moby/buildkit#export-cache - - -### Additional build contexts (--build-context) +### Set type of progress output (--progress) ``` ---build-context=name=VALUE +--progress=VALUE ``` -Define additional build context with specified contents. In Dockerfile the context can be accessed when `FROM name` or `--from=name` is used. -When Dockerfile defines a stage with the same name it is overwritten. - -The value can be a local source directory, container image (with docker-image:// prefix), Git or HTTP URL. - -**Examples** - -Replace `alpine:latest` with a pinned one: +Set type of progress output (auto, plain, tty). Use plain to show container +output (default "auto"). -```console -$ docker buildx build --build-context alpine=docker-image://alpine@sha256:0123456789 . -``` +> You can also use the `BUILDKIT_PROGRESS` environment variable to set +> its value. -Expose a secondary local source directory: +The following example uses `plain` output during the build: ```console -$ docker buildx build --build-context project=path/to/project/source . -# docker buildx build --build-context project=https://github.com/myuser/project.git . -``` - -```Dockerfile -FROM alpine -COPY --from=project myfile / -``` - +$ docker buildx build --load --progress=plain . -### Allow extra privileged entitlement (--allow) +#1 [internal] load build definition from Dockerfile +#1 transferring dockerfile: 227B 0.0s done +#1 DONE 0.1s +#2 [internal] load .dockerignore +#2 transferring context: 129B 0.0s done +#2 DONE 0.0s +... ``` ---allow=ENTITLEMENT -``` - -Allow extra privileged entitlement. List of entitlements: - -- `network.host` - Allows executions with host networking. -- `security.insecure` - Allows executions without sandbox. See -[related Dockerfile extensions](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md#run---securityinsecuresandbox). -For entitlements to be enabled, the `buildkitd` daemon also needs to allow them -with `--allow-insecure-entitlement` (see [`create --buildkitd-flags`](buildx_create.md#buildkitd-flags)) - -**Examples** +### Push the build result to a registry (--push) -```console -$ docker buildx create --use --name insecure-builder --buildkitd-flags '--allow-insecure-entitlement security.insecure' -$ docker buildx build --allow security.insecure . -``` +Shorthand for [`--output=type=registry`](#registry). Will automatically push the +build result to registry. -### Secret to expose to the build (`--secret`) +### Secret to expose to the build (--secret) ``` --secret=[type=TYPE[,KEY=VALUE] @@ -386,8 +435,8 @@ RUN --mount=type=secret,id=aws,target=/root/.aws/credentials \ aws s3 cp s3://... ... ``` -```shell -docker buildx build --secret id=aws,src=$HOME/.aws/credentials . +```console +$ docker buildx build --secret id=aws,src=$HOME/.aws/credentials . ``` #### `env` @@ -405,16 +454,47 @@ RUN --mount=type=bind,target=. \ SECRET_TOKEN=$(cat /run/secrets/SECRET_TOKEN) yarn run test ``` -```shell -SECRET_TOKEN=token docker buildx build --secret id=SECRET_TOKEN . +```console +$ SECRET_TOKEN=token docker buildx build --secret id=SECRET_TOKEN . ``` -### Size of `/dev/shm` (--shm-size) +### Size of /dev/shm (--shm-size) The format is ``. `number` must be greater than `0`. Unit is optional and can be `b` (bytes), `k` (kilobytes), `m` (megabytes), or `g` (gigabytes). If you omit the unit, the system uses bytes. +### SSH agent socket or keys to expose to the build (--ssh) + +``` +--ssh=default|[=|[,]] +``` + +This can be useful when some commands in your Dockerfile need specific SSH +authentication (e.g., cloning a private repository). + +`--ssh` exposes SSH agent socket or keys to the build and can be used with the +[`RUN --mount=type=ssh` mount](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/syntax.md#run---mounttypessh). + +Example to access Gitlab using an SSH agent socket: + +```dockerfile +# syntax=docker/dockerfile:1.3 +FROM alpine +RUN apk add --no-cache openssh-client +RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan gitlab.com >> ~/.ssh/known_hosts +RUN --mount=type=ssh ssh -q -T git@gitlab.com 2>&1 | tee /hello +# "Welcome to GitLab, @GITLAB_USERNAME_ASSOCIATED_WITH_SSHKEY" should be printed here +# with the type of build progress is defined as `plain`. +``` + +```console +$ eval $(ssh-agent) +$ ssh-add ~/.ssh/id_rsa +(Input your passphrase here) +$ docker buildx build --ssh default=$SSH_AUTH_SOCK . +``` + ### Set ulimits (--ulimit) `--ulimit` is specified with a soft and hard limit as such: diff --git a/docs/reference/buildx_create.md b/docs/reference/buildx_create.md index 8c47ef2d..4adece33 100644 --- a/docs/reference/buildx_create.md +++ b/docs/reference/buildx_create.md @@ -47,8 +47,6 @@ The `--append` flag changes the action of the command to append a new node to an existing builder specified by `--name`. Buildx will choose an appropriate node for a build based on the platforms it supports. -**Examples** - ```console $ docker buildx create mycontext1 eager_beaver @@ -67,8 +65,6 @@ Adds flags when starting the buildkitd daemon. They take precedence over the configuration file specified by [`--config`](#config). See `buildkitd --help` for the available flags. -**Example** - ``` --buildkitd-flags '--debug --debugaddr 0.0.0.0:6666' ``` @@ -132,46 +128,22 @@ Passes additional driver-specific options. Details for each driver: - `docker` - No driver options - `docker-container` - - `image=IMAGE` - Sets the container image to be used for running buildkit. - - `network=NETMODE` - Sets the network mode for running the buildkit container. - - `cgroup-parent=CGROUP` - Sets the cgroup parent of the buildkit container if docker is using the "cgroupfs" driver. Defaults to `/docker/buildx`. + - `image=IMAGE` - Sets the container image to be used for running buildkit. + - `network=NETMODE` - Sets the network mode for running the buildkit container. + - `cgroup-parent=CGROUP` - Sets the cgroup parent of the buildkit container if docker is using the "cgroupfs" driver. Defaults to `/docker/buildx`. - `kubernetes` - - `image=IMAGE` - Sets the container image to be used for running buildkit. - - `namespace=NS` - Sets the Kubernetes namespace. Defaults to the current namespace. - - `replicas=N` - Sets the number of `Pod` replicas. Defaults to 1. - - `requests.cpu` - Sets the request CPU value specified in units of Kubernetes CPU. Example `requests.cpu=100m`, `requests.cpu=2` - - `requests.memory` - Sets the request memory value specified in bytes or with a valid suffix. Example `requests.memory=500Mi`, `requests.memory=4G` - - `limits.cpu` - Sets the limit CPU value specified in units of Kubernetes CPU. Example `limits.cpu=100m`, `limits.cpu=2` - - `limits.memory` - Sets the limit memory value specified in bytes or with a valid suffix. Example `limits.memory=500Mi`, `limits.memory=4G` - - `nodeselector="label1=value1,label2=value2"` - Sets the kv of `Pod` nodeSelector. No Defaults. Example `nodeselector=kubernetes.io/arch=arm64` - - `rootless=(true|false)` - Run the container as a non-root user without `securityContext.privileged`. [Using Ubuntu host kernel is recommended](https://github.com/moby/buildkit/blob/master/docs/rootless.md). Defaults to false. - - `loadbalance=(sticky|random)` - Load-balancing strategy. If set to "sticky", the pod is chosen using the hash of the context path. Defaults to "sticky" - - `qemu.install=(true|false)` - Install QEMU emulation for multi platforms support. - - `qemu.image=IMAGE` - Sets the QEMU emulation image. Defaults to `tonistiigi/binfmt:latest` - -**Examples** - -#### Use a custom network - -```console -$ docker network create foonet -$ docker buildx create --name builder --driver docker-container --driver-opt network=foonet --use -$ docker buildx inspect --bootstrap -$ docker inspect buildx_buildkit_builder0 --format={{.NetworkSettings.Networks}} -map[foonet:0xc00018c0c0] -``` - -#### OpenTelemetry support - -To capture the trace to [Jaeger](https://github.com/jaegertracing/jaeger), set -`JAEGER_TRACE` environment variable to the collection address using the `driver-opt`: - -```console -$ docker run -d --name jaeger -p 6831:6831/udp -p 16686:16686 jaegertracing/all-in-one -$ docker buildx create --name builder --driver docker-container --driver-opt network=host --driver-opt env.JAEGER_TRACE=localhost:6831 --use -$ docker buildx inspect --bootstrap -# buildx command should be traced at http://127.0.0.1:16686/ -``` + - `image=IMAGE` - Sets the container image to be used for running buildkit. + - `namespace=NS` - Sets the Kubernetes namespace. Defaults to the current namespace. + - `replicas=N` - Sets the number of `Pod` replicas. Defaults to 1. + - `requests.cpu` - Sets the request CPU value specified in units of Kubernetes CPU. Example `requests.cpu=100m`, `requests.cpu=2` + - `requests.memory` - Sets the request memory value specified in bytes or with a valid suffix. Example `requests.memory=500Mi`, `requests.memory=4G` + - `limits.cpu` - Sets the limit CPU value specified in units of Kubernetes CPU. Example `limits.cpu=100m`, `limits.cpu=2` + - `limits.memory` - Sets the limit memory value specified in bytes or with a valid suffix. Example `limits.memory=500Mi`, `limits.memory=4G` + - `nodeselector="label1=value1,label2=value2"` - Sets the kv of `Pod` nodeSelector. No Defaults. Example `nodeselector=kubernetes.io/arch=arm64` + - `rootless=(true|false)` - Run the container as a non-root user without `securityContext.privileged`. [Using Ubuntu host kernel is recommended](https://github.com/moby/buildkit/blob/master/docs/rootless.md). Defaults to false. + - `loadbalance=(sticky|random)` - Load-balancing strategy. If set to "sticky", the pod is chosen using the hash of the context path. Defaults to "sticky" + - `qemu.install=(true|false)` - Install QEMU emulation for multi platforms support. + - `qemu.image=IMAGE` - Sets the QEMU emulation image. Defaults to `tonistiigi/binfmt:latest` ### Remove a node from a builder (--leave) @@ -179,8 +151,6 @@ The `--leave` flag changes the action of the command to remove a node from a builder. The builder needs to be specified with `--name` and node that is removed is set with `--node`. -**Examples** - ```console $ docker buildx create --name mybuilder --node mybuilder0 --leave ``` @@ -204,7 +174,7 @@ The `--node` flag specifies the name of the node to be created or modified. If none is specified, it is the name of the builder it belongs to, with an index number suffix. -### Set the platforms supported by the node +### Set the platforms supported by the node (--platform) ``` --platform PLATFORMS @@ -216,14 +186,12 @@ will also automatically detect the platforms it supports, but manual values take priority over the detected ones and can be used when multiple nodes support building for the same platform. -**Examples** - ```console $ docker buildx create --platform linux/amd64 $ docker buildx create --platform linux/arm64,linux/arm/v8 ``` -### Automatically switch to the newly created builder +### Automatically switch to the newly created builder (--use) The `--use` flag automatically switches the current builder to the newly created one. Equivalent to running `docker buildx use $(docker buildx create ...)`. diff --git a/docs/reference/buildx_imagetools_create.md b/docs/reference/buildx_imagetools_create.md index 4251ad5e..0b076eb5 100644 --- a/docs/reference/buildx_imagetools_create.md +++ b/docs/reference/buildx_imagetools_create.md @@ -57,16 +57,15 @@ or a JSON of OCI descriptor object. In order to define annotations or additional platform properties like `os.version` and `os.features` you need to add them in the OCI descriptor object encoded in JSON. -``` -docker buildx imagetools inspect --raw alpine | jq '.manifests[0] | .platform."os.version"="10.1"' > descr.json -docker buildx imagetools create -f descr.json myuser/image +```console +$ docker buildx imagetools inspect --raw alpine | jq '.manifests[0] | .platform."os.version"="10.1"' > descr.json +$ docker buildx imagetools create -f descr.json myuser/image ``` The descriptor in the file is merged with existing descriptor in the registry if it exists. The supported fields for the descriptor are defined in [OCI spec](https://github.com/opencontainers/image-spec/blob/master/descriptor.md#properties) . - ### Set reference for new image (-t, --tag) ``` @@ -75,10 +74,7 @@ The supported fields for the descriptor are defined in [OCI spec](https://github Use the `-t` or `--tag` flag to set the name of the image to be created. -**Examples** - ```console $ docker buildx imagetools create --dry-run alpine@sha256:5c40b3c27b9f13c873fefb2139765c56ce97fd50230f1f2d5c91e55dec171907 sha256:c4ba6347b0e4258ce6a6de2401619316f982b7bcc529f73d2a410d0097730204 - $ docker buildx imagetools create -t tonistiigi/myapp -f image1 -f image2 ``` diff --git a/docs/reference/buildx_imagetools_inspect.md b/docs/reference/buildx_imagetools_inspect.md index f6ea01ac..0b913029 100644 --- a/docs/reference/buildx_imagetools_inspect.md +++ b/docs/reference/buildx_imagetools_inspect.md @@ -21,8 +21,6 @@ Show details of image in the registry Show details of image in the registry. -Example: - ```console $ docker buildx imagetools inspect alpine diff --git a/docs/reference/buildx_ls.md b/docs/reference/buildx_ls.md index 75dbaf8b..c2123453 100644 --- a/docs/reference/buildx_ls.md +++ b/docs/reference/buildx_ls.md @@ -14,18 +14,17 @@ List builder instances Lists all builder instances and the nodes for each instance -**Example** - ```console $ docker buildx ls NAME/NODE DRIVER/ENDPOINT STATUS PLATFORMS elated_tesla * docker-container elated_tesla0 unix:///var/run/docker.sock running linux/amd64 - elated_tesla1 ssh://ubuntu@1.2.3.4 running linux/arm64, linux/arm/v7, linux/arm/v6 + elated_tesla1 ssh://ubuntu@1.2.3.4 running linux/arm64*, linux/arm/v7, linux/arm/v6 default docker default default running linux/amd64 ``` Each builder has one or more nodes associated with it. The current builder's -name is marked with a `*`. +name is marked with a `*` in `NAME/NODE` and explicit node to build against for +the target platform marked with a `*` in the `PLATFORMS` column. diff --git a/docs/reference/buildx_version.md b/docs/reference/buildx_version.md index ba756282..a3ab2d16 100644 --- a/docs/reference/buildx_version.md +++ b/docs/reference/buildx_version.md @@ -10,10 +10,9 @@ Show buildx version information -## Examples - -### View version information +## Description +View version information ```console $ docker buildx version