Create a Multi-Build CI Pipeline

Introduction

The Dagger SDKs makes it easy to build an application for multiple OS and architecture combinations. This guide provides a working example of a CI tool that performs this task.

Requirements

This guide assumes that:

• You have a Go, Python or Node.js development environment. If not, install Go, Python or Node.js.
• You have a Dagger SDK installed for one of the above languages. If not, follow the installation instructions for the Dagger Go, Python or Node.js SDK.
• You have Docker installed and running on the host system. If not, install Docker.
• You have an application that you wish to build. This guide assumes a Go application, but you can use an application of your choice.

tip

Dagger pipelines are executed as standard OCI containers. This portability enables you to do very powerful things. For example, if you're a Python developer, you can use the Python SDK to create a pipeline (written in Python) that builds an application written in a different language (Go) without needing to learn that language.

Example

Assume that the Go application to be built is stored in the current directory on the host. The following code listing demonstrates how to build this Go application for multiple OS and architecture combinations using the Dagger SDKs.

Go

// Create a multi-build pipeline for a Go application.
package main

import (
	"context"
	"fmt"
	"os"

	"dagger.io/dagger"
)

func main() {
	println("Building with Dagger")

	// define build matrix
	geese := []string{"linux", "darwin"}
	goarches := []string{"amd64", "arm64"}

	ctx := context.Background()
	// initialize dagger client
	c, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
	if err != nil {
		panic(err)
	}

	// get reference to the local project
	src := c.Host().Directory(".")

	// create empty directory to put build outputs
	outputs := c.Directory()

	golang := c.Container().
		// get golang image
		From("golang:latest").
		// mount source code into golang image
		WithDirectory("/src", src).
		WithWorkdir("/src")

	for _, goos := range geese {
		for _, goarch := range goarches {
			// create a directory for each OS and architecture
			path := fmt.Sprintf("build/%s/%s/", goos, goarch)

			build := golang.
				// set GOARCH and GOOS in the build environment
				WithEnvVariable("GOOS", goos).
				WithEnvVariable("GOARCH", goarch).
				WithExec([]string{"go", "build", "-o", path})

			// add build to outputs
			outputs = outputs.WithDirectory(path, build.Directory(path))
		}
	}

	// write build artifacts to host
	ok, err := outputs.Export(ctx, ".")
	if err != nil {
		panic(err)
	}

	if !ok {
		panic("did not export files")
	}
}

This code listing does the following:

• It defines the build matrix, consisting of two OSs (darwin and linux) and two architectures (amd64 and arm64).
• It creates a Dagger client with Connect().
• It uses the client's Host().Directory(".") method to obtain a reference to the current directory on the host. This reference is stored in the src variable.
• It uses the client's Container().From() method to initialize a new container from a base image. This base image contains all the tooling needed to build the application - in this case, the golang:latest image. This From() method returns a new Container class with the results.
• It uses the Container.Directory() method to mount the host directory into the container at the /src mount point.
• It uses the Container.WithWorkdir() method to set the working directory in the container.
• It iterates over the build matrix, creating a directory in the container for each OS/architecture combination and building the Go application for each such combination. The Go build process is instructed via the GOOS and GOARCH build variables, which are reset for each case via the Container.WithEnvVariable() method.
• It obtains a reference to the build output directory in the container with the WithDirectory() method, and then uses the Directory.Export() method to write the build directory from the container to the host.

Node.js

import { connect } from "@dagger.io/dagger"

// Create a multi-build pipeline for a Go application.

// define build matrix
const oses = ["linux", "darwin"]
const arches = ["amd64", "arm64"]

// initialize dagger client
connect(
  async (client) => {
    console.log("Building with Dagger")

    // get reference to the local project
    const src = client.host().directory(".")

    // create empty directory to put build outputs
    var outputs = client.directory()

    const golang = client
      .container()
      // get golang image
      .from("golang:latest")
      // mount source code into golang image
      .withDirectory("/src", src)
      .withWorkdir("/src")

    for (const os of oses) {
      for (const arch of arches) {
        // create a directory for each OS and architecture
        const path = `build/${os}/${arch}/`

        const build = golang
          // set GOARCH and GOOS in the build environment
          .withEnvVariable("GOOS", os)
          .withEnvVariable("GOARCH", arch)
          .withExec(["go", "build", "-o", path])

        // add build to outputs
        outputs = outputs.withDirectory(path, build.directory(path))
      }
    }

    // write build artifacts to host
    await outputs.export(".")
  },
  { LogOutput: process.stderr },
)

This code listing does the following:

• It defines the build matrix, consisting of two OSs (darwin and linux) and two architectures (amd64 and arm64).
• It creates a Dagger client with connect().
• It uses the client's host().directory(".") method to obtain a reference to the current directory on the host. This reference is stored in the src variable.
• It uses the client's container().from() method to initialize a new container from a base image. This base image contains all the tooling needed to build the application - in this case, the golang:latest image. This from() method returns a new Container class with the results.
• It uses the Container.withDirectory() method to return the container image with the host directory written at the /src path.
• It uses the Container.withWorkdir() method to set the working directory in the container.
• It iterates over the build matrix, creating a directory in the container for each OS/architecture combination and building the Go application for each such combination. The Go build process is instructed via the GOOS and GOARCH build variables, which are reset for each case via the Container.withEnvVariable() method.
• It obtains a reference to the build output directory in the container with the withDirectory() method, and then uses the Directory.export() method to write the build directory from the container to the host.

Python

"""Create a multi-build pipeline for a Go application."""

import itertools
import sys

import anyio

import dagger


async def main():
    print("Building with Dagger")

    # define build matrix
    oses = ["linux", "darwin"]
    arches = ["amd64", "arm64"]

    # initialize dagger client
    async with dagger.Connection(dagger.Config(log_output=sys.stderr)) as client:
        # get reference to the local project
        src = client.host().directory(".")

        # create empty directory to put build outputs
        outputs = client.directory()

        golang = (
            # get `golang` image
            client.container()
            .from_("golang:latest")
            # mount source code into `golang` image
            .with_directory("/src", src)
            .with_workdir("/src")
        )

        for goos, goarch in itertools.product(oses, arches):
            # create a directory for each OS and architecture
            path = f"build/{goos}/{goarch}/"

            build = (
                golang
                # set GOARCH and GOOS in the build environment
                .with_env_variable("GOOS", goos)
                .with_env_variable("GOARCH", goarch)
                .with_exec(["go", "build", "-o", path])
            )

            # add build to outputs
            outputs = outputs.with_directory(path, build.directory(path))

        # write build artifacts to host
        await outputs.export(".")


anyio.run(main)

This code listing does the following:

• It defines the build matrix, consisting of two OSs (darwin and linux) and two architectures (amd64 and arm64).
• It creates a Dagger client with dagger.Connection().
• It uses the client's host().directory(".") method to obtain a reference to the current directory on the host. This reference is stored in the src variable.
• It uses the client's container().from_() method to initialize a new container from a base image. This base image contains all the tooling needed to build the application - in this case, the golang:latest image. This from_() method returns a new Container class with the results.
• It uses the Container.with_directory() method to mount the host directory into the container at the /src mount point.
• It uses the Container.with_workdir() method to set the working directory in the container.
• It iterates over the build matrix, creating a directory in the container for each OS/architecture combination and building the Go application for each such combination. The Go build process is instructed via the GOOS and GOARCH build variables, which are reset for each case via the Container.with_env_variable() method.
• It obtains a reference to the build output directory in the container with the with_directory() method, and then uses the Directory.export() method to write the build directory from the container to the host.

Conclusion

This guide showed you how to build an application for multiple OS and architecture combinations with Dagger.

Use the API Key Concepts page and the Go, Node.js and Python SDK References to learn more about Dagger.