Skip to main content

Use Dagger with GitLab CI/CD and Google Cloud

Introduction​

This tutorial teaches you how to use a Dagger pipeline to continuously build and deploy a Go application with GitLab on Google Cloud Run. You will learn how to:

  • Configure a Google Cloud service account and assign it the correct roles
  • Create a Google Cloud Run service accessible at a public URL
  • Create a Dagger pipeline using the Dagger SDKs
  • Run the Dagger pipeline on your local host to manually build and deploy the application on Google Cloud Run
  • Use the same Dagger pipeline with GitLab CI/CD to automatically build and deploy the application on Google Cloud Run on every repository commit

Requirements​

This tutorial assumes that:

Step 1: Create a Google Cloud service account​

The Dagger pipeline demonstrated in this tutorial (re)builds a container image of an application every time a new commit is added to the application's repository. It then publishes the container image to Google Container registry and deploys it at a public URL using Google Cloud infrastructure.

This requires the following:

  • A Google Cloud service account with all necessary privileges
  • A Google Cloud Run service with a public URL and defined resource/capacity/access rules
  • Access to various Google Cloud APIs
info

This step discusses how to create a Google Cloud service account. If you already have a Google Cloud service account and key for your project, skip to Step 2.

Create a Google Cloud service account, as follows:

  1. Log in to the Google Cloud Console and select your project.

  2. From the navigation menu, click IAM & Admin -> Service Accounts.

  3. Click Create Service Account.

  4. In the Service account details section, enter a string in the Service account ID field. This string forms the prefix of the unique service account email address.

    Create Google Cloud service account

  5. Click Create and Continue.

  6. In the Grant this service account access to project section, select the Service Account Token Creator and Editor roles.

    Create Google Cloud service account roles

  7. Click Continue.

  8. Click Done.

Once the service account is created, the Google Cloud Console displays it in the service account list, as shown below. Note the service account email address, as you will need it in the next step.

List Google Cloud service accounts

Next, create a JSON key for the service account as follows:

  1. From the navigation menu, click IAM & Admin -> Service Accounts.
  2. Click the newly-created service account in the list of service accounts.
  3. Click the Keys tab on the service account detail page.
  4. Click Add Key -> Create new key.
  5. Select the JSON key type.
  6. Click Create.

The key is created and automatically downloaded to your local host through your browser as a JSON file.

Create Google Cloud service account key

warning

Store the JSON service account key file safely as it cannot be retrieved again.

Step 2: Configure Google Cloud APIs and a Google Cloud Run service​

The next step is to enable access to the required Google Cloud APIs:

  1. From the navigation menu, select the APIs & Services -> Enabled APIs & services option.

  2. Select the Enable APIs and Services option.

  3. On the API Library page, search for and select the Cloud Run API entry.

  4. On the API detail page, click Enable.

    Enable Google Cloud API

  5. Repeat the previous two steps for the IAM Service Account Credentials API.

Once the APIs are enabled, the Google Cloud Console displays the updated status of the APIs.

The final step is to create a Google Cloud Run service and corresponding public URL endpoint. This service will eventually host the container deployed by the Dagger pipeline.

  1. From the navigation menu, select the Serverless -> Cloud Run product.

  2. Select the Create Service option.

  3. Select the Deploy one revision from an existing container image option. Click Test with a sample container to have a container image URL pre-filled.

  4. Continue configuring the service with the following inputs:

    • Service name: myapp (modify as needed)
    • Region: us-central1 (modify as needed)
    • CPU allocation and pricing: CPU is only allocated during request processing
    • Minimum number of instances: 0 (modify as needed)
    • Maximum number of instances: 1 (modify as needed)
    • Ingress: Allow all traffic
    • Authentication: Allow unauthenticated invocations

    Create Google Cloud Run service

  5. Click Create to create the service.

The new service is created. The Google Cloud Console displays the service details, including its public URL, on the service detail page, as shown below.

View Google Cloud Run service details

Step 3: Create the Dagger pipeline​

The next step is to create a Dagger pipeline to do the heavy lifting: build a container image of the application, release it to Google Container Registry and deploy it on Google Cloud Run.

  1. In the application directory, install the Dagger SDK and the Google Cloud Run client library:

    go get dagger.io/dagger@latest
    go get cloud.google.com/go/run/apiv2

  2. Create a new sub-directory named ci. Within the ci directory, create a file named main.go and add the following code to it. Replace the PROJECT placeholder with your Google Cloud project identifier and adjust the region (us-central1) and service name (myapp) if you specified different values when creating the Google Cloud Run service in Step 2.

    package main

    import (
    	"context"
    	"fmt"
    	"os"

    	run "cloud.google.com/go/run/apiv2"
    	runpb "cloud.google.com/go/run/apiv2/runpb"
    	"dagger.io/dagger"
    )

    const GCR_SERVICE_URL = "projects/PROJECT/locations/us-central1/services/myapp"
    const GCR_PUBLISH_ADDRESS = "gcr.io/PROJECT/myapp"

    func main() {
    	// create Dagger client
    	ctx := context.Background()
    	daggerClient, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
    	if err != nil {
    		panic(err)
    	}
    	defer daggerClient.Close()

    	// get working directory on host
    	source := daggerClient.Host().Directory(".", dagger.HostDirectoryOpts{
    		Exclude: []string{"ci"},
    	})

    	// build application
    	builder := daggerClient.Container(dagger.ContainerOpts{Platform: "linux/amd64"}).
    		From("golang:1.20").
    		WithDirectory("/src", source).
    		WithWorkdir("/src").
    		WithEnvVariable("CGO_ENABLED", "0").
    		WithExec([]string{"go", "build", "-o", "myapp"})

    	// add binary to alpine base
    	prodImage := daggerClient.Container(dagger.ContainerOpts{Platform: "linux/amd64"}).
    		From("alpine").
    		WithFile("/bin/myapp", builder.File("/src/myapp")).
    		WithEntrypoint([]string{"/bin/myapp"})

    	// publish container to Google Container Registry
    	addr, err := prodImage.Publish(ctx, GCR_PUBLISH_ADDRESS)
    	if err != nil {
    		panic(err)
    	}

    	// print ref
    	fmt.Println("Published at:", addr)

    	// create Google Cloud Run client
    	gcrClient, err := run.NewServicesClient(ctx)
    	if err != nil {
    		panic(err)
    	}
    	defer gcrClient.Close()

    	// define service request
    	gcrRequest := &runpb.UpdateServiceRequest{
    		Service: &runpb.Service{
    			Name: GCR_SERVICE_URL,
    			Template: &runpb.RevisionTemplate{
    				Containers: []*runpb.Container{
    					{
    						Image: addr,
    						Ports: []*runpb.ContainerPort{
    							{
    								Name:          "http1",
    								ContainerPort: 1323,
    							},
    						},
    					},
    				},
    			},
    		},
    	}

    	// update service
    	gcrOperation, err := gcrClient.UpdateService(ctx, gcrRequest)
    	if err != nil {
    		panic(err)
    	}

    	// wait for service request completion
    	gcrResponse, err := gcrOperation.Wait(ctx)
    	if err != nil {
    		panic(err)
    	}

    	// print ref
    	fmt.Println("Deployment for image", addr, "now available at", gcrResponse.Uri)

    }

    This code listing performs the following operations:

    • It imports the Dagger and Google Cloud Run client libraries.
    • It creates a Dagger client with Connect(). This client provides an interface for executing commands against the Dagger engine.
    • It uses the client's Host().Directory() method to obtain a reference to the current directory on the host, excluding the ci directory. This reference is stored in the source variable.
    • In the first stage of the build, it uses the client's Container().From() method to initialize a new container from a base image. The additional Platform argument to the Container() method instructs Dagger to build for a specific architecture. In this example, the base image is the golang:1.20 image and the architecture is linux/amd64, which is one of the architectures supported by Google Cloud. This method returns a Container representing an OCI-compatible container image.
    • It uses the previous Container object's WithDirectory() method to return the container image with the host directory written at the /src path, and the WithWorkdir() method to set the working directory in the container.
    • It chains the WithEnvVariable() method to set the CGO_ENABLED variable in the container environment and the WithExec() method to compile the Go application with go build.
    • Once the application is built, it moves to the second stage of the build. It again uses the client's Container().From() method to initialize a new container from an alpine base image.
    • It uses the previous Container object's WithFile() method to transfer the compiled binary file from the first stage to the new container filesystem.
    • It sets the container entrypoint to the binary file using the WithEntrypoint() method.
    • It uses the container object's Publish() method to publish the container to Google Container Registry, and prints the SHA identifier of the published image.
    • It creates a Google Cloud Run client, creates a service request instructing the Google Cloud Run service to use the newly-published container image, and sends the requests to the Google Cloud Run API.

  3. Run the following command to update go.sum:

    go mod tidy
tip

Most Container object methods return a revised Container object representing the new state of the container. This makes it easy to chain methods together. Dagger evaluates pipelines "lazily", so the chained operations are only executed when required - in this case, when the container is published. Learn more about lazy evaluation in Dagger.

Step 4: Test the Dagger pipeline on the local host​

Configure credentials for the Google Cloud SDK on the local host, as follows:

  1. Configure Docker credentials for Google Container Registry on the local host using the following commands. Replace the SERVICE-ACCOUNT-ID placeholder with the service account email address created in Step 1, and the SERVICE-ACCOUNT-KEY-FILE placeholder with the location of the service account JSON key file downloaded in Step 1.

    gcloud auth activate-service-account SERVICE-ACCOUNT-ID --key-file=SERVICE-ACCOUNT-KEY-FILE
    gcloud auth configure-docker
    info

    This step is necessary because Dagger relies on the host's Docker credentials and authorizations when publishing to remote registries.

  2. Set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the location of the service account JSON key file, replacing the SERVICE-ACCOUNT-KEY-FILE placeholder in the following command. This variable is used by the Google Cloud Run client library during the client authentication process.

    export GOOGLE_APPLICATION_CREDENTIALS=SERVICE-ACCOUNT-KEY-FILE

Once credentials are configured, test the Dagger pipeline by running the command below:

dagger run go run ci/main.go

Dagger performs the operations defined in the pipeline script, logging each operation to the console. At the end of the process, the built container is deployed to Google Cloud Run and a message similar to the one below appears in the console output:

Deployment for image gcr.io/PROJECT/myapp@sha256:b1cf... now available at https://...run.app

Browse to the URL shown in the deployment message to see the running application.

If you deployed the example application from Appendix A, you see the output below:

Hello, Dagger!

Step 5: Create a GitLab CI/CD pipeline​

Dagger executes your pipelines entirely as standard OCI containers. This means that the same pipeline will run the same, whether on on your local machine or a remote server.

This also means that it's very easy to move your Dagger pipeline from your local host to GitLab - all that's needed is to transfer the pipeline script from your local clone to your GitLab repository, and then define a GitLab CI/CD pipeline to run it on every commit.

  1. Create a new GitLab CI/CD pipeline configuration file in your application directory at .gitlab-ci.yml with the following content:

    .docker:
      image: golang:alpine
      services:
        - docker:${DOCKER_VERSION}-dind
      variables:
        DOCKER_HOST: tcp://docker:2376
        DOCKER_TLS_VERIFY: '1'
        DOCKER_TLS_CERTDIR: '/certs'
        DOCKER_CERT_PATH: '/certs/client'
        DOCKER_DRIVER: overlay2
        DOCKER_VERSION: '20.10.16'
    .dagger:
      extends: [.docker]
      before_script:
        - apk add docker-cli curl
        - cd /usr/local && { curl -L https://dl.dagger.io/dagger/install.sh | sh; cd -; }
        - cat $GOOGLE_APPLICATION_CREDENTIALS | docker login -u _json_key --password-stdin https://gcr.io
    build-deploy:
      extends: [.dagger]
      script:
        - dagger run go run ci/main.go

    This GitLab CI/CD pipeline runs on every commit to the repository master branch. It consists of three jobs, as below:

    • The first job tells the GitLab runner to use the Docker executor with a Docker-in-Docker (dind) service. It also configures TLS and sets the location for Docker to generate its TLS certificates.
    • The second job adds the Docker CLI and authenticates to Google Container Registry from the GitLab runner. This is necessary because Dagger relies on the host's Docker credentials and authorizations when publishing to remote registries. For authentication, the job relies on the Google Cloud service account credentials, which are stored in the GOOGLE_APPLICATION_CREDENTIALS variable (more on this later).
    • The third and final job executes the Dagger pipeline code.

  2. This GitLab CI/CD pipeline looks for a Google Cloud service account key in the GOOGLE_APPLICATION_CREDENTIALS GitLab variable. Create this variable as follows:

    1. Navigate to the Settings -> CI/CD -> Variables page in the GitLab Web interface.
    2. Click Add variable to create a new variable.
    3. Configure the variable with the following inputs:
      • Name: GOOGLE_APPLICATION_CREDENTIALS
      • Value: The contents of the service account JSON key file downloaded in Step 1
      • Type: File
      • Flags: Protect variable
    4. Click Add variable to save the variable.

    Create GitLab variable

  3. Commit and push the changes to the GitLab repository:

    git add .
    git commit -a -m "Added pipeline and CI code"
    git push

Step 6: Test the Dagger pipeline on GitLab​

info

This step requires a properly-configured GitLab Runner. Refer to Appendix B for instructions on how to configure a self-hosted GitLab Runner for use with Dagger.

Test the Dagger pipeline by committing a change to the GitLab repository.

If you are using the example application described in Appendix A, the following commands modify and commit a simple change to the application's index page:

git pull
sed -i -e 's/Dagger/Dagger on GitLab/g' server.go
git commit -a -m "Update welcome message"
git push

The commit triggers the GitLab CI/CD pipeline defined in Step 6. The pipeline runs the various jobs, including the Dagger pipeline.

At the end of the process, a new version of the built container image is released to Google Container Registry and deployed on Google Cloud Run. A message similar to the one below appears in the GitHub Actions log:

Deployment for image gcr.io/PROJECT/myapp@sha256:h4si... now available at https://...run.app

Browse to the URL shown in the deployment message to see the running application. If you deployed the example application with the additional modification above, you see the following output:

Hello, Dagger on GitLab!

Conclusion​

This tutorial walked you through the process of creating a Dagger pipeline to continuously build and deploy a Go application on Google Cloud Run. It explained key concepts, objects and methods available in the Dagger SDKs to construct a Dagger pipeline.

Dagger executes your pipelines entirely as standard OCI containers. This means that pipelines can be tested and debugged locally, and that the same pipeline will run consistently on your local machine, a CI runner, a dedicated server, or any container hosting service. This portability is one of Dagger's key advantages, and this tutorial demonstrated it in action by using the same pipeline on the local host and on GitLab.

Use the API Key Concepts page and the Go, Node.js and Python SDK References to learn more about Dagger.

Appendix A: Create a GitLab repository with an example Go application​

This tutorial assumes that you have a GitLab repository with a application. If not, follow the steps below to create a GitLab repository and commit a simple Go web application to it.

  1. Log in to GitLab using the GitLab CLI:

    glab auth login -h gitlab.com

  2. Create a directory and module for the Go application:

    mkdir myapp
    cd myapp
    go mod init main

  3. Install the Echo web framework:

    go get github.com/labstack/echo/v4

  4. Create a file named server.go and add the following code to it to create a skeleton application:

    package main

    import (
      "net/http"

      "github.com/labstack/echo/v4"
    )

    func main() {
      e := echo.New()
      e.GET("/", func(c echo.Context) error {
        return c.String(http.StatusOK, "Hello, Dagger!")
      })
      e.Logger.Fatal(e.Start(":1323"))
    }

  5. Create a private repository in your GitLab account:

    glab repo create myapp

  6. Commit and push the application code:

    git add .
    git commit -a -m "Initial commit"
    git push --set-upstream origin master

Appendix B: Configure a self-hosted GitLab Runner for use with Dagger​

This tutorial assumes that you have a GitLab Runner application to run your GitLab CI/CD pipelines. This could be either a GitLab-managed runner or a self-hosted runner. Learn about GitLab Runner.

To use GitLab's managed runners, you must associate a valid credit card with your GitLab account. Alternatively, you can configure a self-hosted runner on your local host by following the steps below.

  1. Install GitLab Runner for your host's operating system.

  2. Navigate to the Settings -> CI/CD -> Runners page in the GitLab Web interface.

  3. Disable shared runners by unchecking the Enable shared runners for this project option.

    Disable shared runners

  4. Copy the project-specific registration token, as shown below:

    Runner registration token

  5. On your local host, register the runner using the command below. Replace the TOKEN placeholder with the registration token.

    sudo gitlab-runner register -n \
      --name dagger \
      --url https://gitlab.com/ \
      --executor docker \
      --docker-privileged \
      --docker-volumes /cache \
      --docker-volumes /certs/client \
      --docker-image docker:20.10.16 \
      --registration-token TOKEN

  6. Navigate to the Settings -> CI/CD -> Runners page in the GitLab Web interface. Confirm that the newly-registered runner is active for the project, as shown below:

    Runner registration