How to use GoReleaser with Cloud Native Storage - Goreleaser

In this tutorial, I want to describe, how quickly we can deploy our release artifacts to a cloud native storage when using GoReleaser. It’s just a few additional lines in your .goreleaser.yaml.

To better show this, I created a little demo and use the storage services of the big three cloud providers: Azure Blob Storage, AWS S3 and Google Cloud Storage.

You can use any S3 compatible storage provider too. GoReleaser support this too! The most prominent (self-hosted) solution is MinIO.

The infrastructure code

I created a very simple Terraform deployment to provision on all three cloud provider their appropriate cloud storage service. It’s a demo, why not?

You don’t need to use Terraform for this, you could use any other means like Pulumi, CLI or even the UI.

`main.tf`

terraform

terraform {
  required_providers {
    google  = {
      source  = "hashicorp/google"
      version = "4.9.0"
    }
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "2.94.0"
    }
    aws     = {
      source  = "hashicorp/aws"
      version = "3.74.0"
    }
  }
}

provider "azurerm" {
  features {}
}

provider "google" {
  credentials = file(var.gcp_auth_file)
  project     = var.gcp_project
  region      = var.gcp_region
}

provider "aws" {
  region = var.aws_region
}

`variables.tf`

terraform

variable "gcp_project" {
  type = string
}

variable "gcp_region" {
  default = "europe-west6"
}

variable "gcp_zone" {
  default = "europe-west6-a"
}

variable "gcp_bucket_location" {
  default = "EU"
}

variable "gcp_auth_file" {
  default = "./auth.json"
  description = "Path to the GCP auth file"
}

variable "aws_region" {
  default = "eu-central-1"
}

variable "azure_location" {
  default = "West Europe"
}

variable "name" {
  default = "goreleaser-quickbites"
}

`blob.tf`

terraform

resource "google_storage_bucket" "goreleaser-gcp-storage-bucket" {
  name                        = var.name
  location                    = var.gcp_bucket_location
  force_destroy               = true
  uniform_bucket_level_access = false
}
resource "google_storage_bucket_access_control" "goreleaser-gcp-storage-bucket-access-control" {
  bucket = google_storage_bucket.goreleaser-gcp-storage-bucket.name
  role   = "READER"
  entity = "allUsers"
}

resource "azurerm_resource_group" "goreleaser-azure-resource-group" {
  name     = var.name
  location = var.azure_location
}

resource "azurerm_storage_account" "goreleaser-azure-storage-account" {
  name                     = "gorleaserquickbites"
  resource_group_name      = azurerm_resource_group.goreleaser-azure-resource-group.name
  location                 = azurerm_resource_group.goreleaser-azure-resource-group.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
  allow_blob_public_access = true
  network_rules {
    default_action = "Allow"
  }
}

resource "azurerm_storage_container" "goreleaser-storage-container" {
  name                  = var.name
  storage_account_name  = azurerm_storage_account.goreleaser-azure-storage-account.name
  container_access_type = "container"
}

resource "aws_s3_bucket" "goreleaser-s3-bucket" {
  bucket = var.name
  acl    = "public-read"
}

Apply the Terraform script:

bash

terraform apply  -var  "gcp_project=xxx"

...
azurerm_storage_container.goreleaser-storage-container: Creation complete after 0s [id=https://goreleaserquickbites.blob.core.windows.net/goreleaser-quickbites]

Apply complete! Resources: 6 added, 0 changed, 0 destroyed.

Outputs:

aws-s3-bucket-name = "goreleaser-quickbites"
azure-storage-account-key = <sensitive>
azure-storage-account-name = "export AZURE_STORAGE_ACCOUNT=goreleaserquickbites"
gcp-bucket-url = "gs://goreleaser-quickbites"

Run this command

bash

terraform output azure-storage-account-key

to get the Azure Storage Account Key, as it is a output field with sensitive data in it.

bash

export AZURE_STORAGE_KEY=xxxx

Now we can add in our .goreleaser.yaml the new blobs field. Important is here to set the right provider: gs (for Google Cloud Storage), azblob (for Azure Blob) and s3 (for AWS S3 or compatible provider)!

yaml

# This is an example .goreleaser.yml file with some sensible defaults.
# Make sure to check the documentation at https://goreleaser.com
before:
  hooks:
    - go mod tidy
builds:
  - env:
      - CGO_ENABLED=0
    goos:
      - linux
      - darwin

release:
  disable: true
---
blobs:
  - provider: gs
    bucket: goreleaser-quickbites
  - provider: azblob
    bucket: goreleaser-quickbites
  - provider: s3
    bucket: goreleaser-quickbites
    region: eu-central-1

In this demo, I disabled the **release **section, as I don’t want to upload to GitHub.

Authentication

In terms of authentication the GoReleaser’s blob pipe authentication varies depending upon the blob provider as mentioned below:

S3 Provider

S3 provider support AWS default credential provider chain in the following order:

Environment variables.
Shared credentials file.
If your application is running on an Amazon EC2 instance, IAM role for Amazon EC2.

Azure Blob Provider Currently it supports authentication only

with environment variables:

AZURE_STORAGE_ACCOUNT
AZURE_STORAGE_KEY or AZURE_STORAGE_SAS_TOKEN

GCS

Provider GCS provider uses Application Default Credentials in the following order:

Environment Variable (GOOGLE_APPLICATION_CREDENTIALS)
Default Service Account from the compute instance (Compute Engine, Kubernetes Engine, Cloud function etc).

Run GoReleaser

After configuring we can finally execute GoReleaser, in your pipeline code via the command:

bash

goreleaser release --rm-dist

If everything went smooth, you should see a similar output, showing the upload of your artifacts.

  ...
   • publishing
   • blobs
   • uploading path=quick-bites/0.1/quick-bites_0.1_checksums.txt
   • uploading path=quick-bites/0.1/quick-bites_0.1_darwin_amd64.tar.gz
   • uploading path=quick-bites/0.1/quick-bites_0.1_linux_arm64.tar.gz
   • uploading path=quick-bites/0.1/quick-bites_0.1_darwin_arm64.tar.gz
   • uploading path=quick-bites/0.1/quick-bites_0.1_linux_amd64.tar.gz
   • uploading path=quick-bites/0.1/quick-bites_0.1_linux_386.tar.gz
   • uploading path=quick-bites/0.1/quick-bites_0.1_checksums.txt
   • uploading path=quick-bites/0.1/quick-bites_0.1_checksums.txt
   • uploading path=quick-bites/0.1/quick-bites_0.1_linux_386.tar.gz
   • uploading path=quick-bites/0.1/quick-bites_0.1_linux_amd64.tar.gz
   • uploading path=quick-bites/0.1/quick-bites_0.1_linux_arm64.tar.gz
   • uploading path=quick-bites/0.1/quick-bites_0.1_linux_amd64.tar.gz
   • uploading path=quick-bites/0.1/quick-bites_0.1_darwin_amd64.tar.gz
   • uploading path=quick-bites/0.1/quick-bites_0.1_darwin_arm64.tar.gz
   • uploading path=quick-bites/0.1/quick-bites_0.1_linux_386.tar.gz
   • uploading path=quick-bites/0.1/quick-bites_0.1_linux_arm64.tar.gz
   • uploading path=quick-bites/0.1/quick-bites_0.1_darwin_arm64.tar.gz
   • uploading path=quick-bites/0.1/quick-bites_0.1_darwin_amd64.tar.gz
   • release succeeded after 22.63s
  ...

One note: The provider fails silently, if your credentials are wrong. You would still see uploading and release succeeded. Keep this in mind, if the files are not appearing in the UI. I wasted some time on this. The culprit is the underlying library GoReleaser is using.

Let’s check in the consoles of the cloud provider too, If the files are present.

Google Cloud Storage:

Azure Blob Storage

AWS S3

Looks very good! Now you can share the URLs of the files for further use!

Want more Informations?

If you want to know more about some advanced options, feel free to check out the official documentation about the blob support in GoReleaser

And here is the example code: dirien/quick-bytes