Using Private Go Modules

We’ve started to write some microservices in Go. In our PHP applications we use Satis hosted on S3 to share common libraries. As long as you can access the S3 bucket, you can install the packages. In Go this was a little more complex.

We use a self-hosted GitLab instance, so I’ve tailored these steps for that. But the same thing should work for private GitHub repositories as well.

Setup

The first thing you’ll need to do is to create an access token. This token should have both the read_repository scope. If you are using this on a local machine, you may need to also add the write_repository scope if you’re using HTTP to access your repository.

GitLab Token

There’s a couple options for fetching Go modules from private repositories. The most common way I’ve seen recommended is using git config url."".insteadOf "". This works, but more elegant solution is using a ~/.netrc file. This file defines automatic logins for ftp, but other programs have also adopted it, including Git.

machine gitlab.company.com login USERNAME password TOKEN

Replace gitlab.company.com, USERNAME, and TOKEN with your values. Now when you push or pull via HTTPS in Git, it will use that information to login you in. This is why you need the write_repository scope for local purposes.

This will also work with GitHub, Bitbucket, or GitLab.com.

machine github.com login USERNAME password TOKEN

One final thing that you’ll have to do is set a GOPRIVATE environment variable. This contains a comma-separated list of module prefixes. Save this value to your ~/.bashrc or ~/.zshrc.

export GOPRIVATE=gitlab.company.com

Now you can run go get gitlab.company.com/go/pkg and start pulling private modules.

Docker

To allow for installing private modules in a Docker image, we use build arguments and create the ~/.netrc file mentioned above.

Here is what our Dockerfile looks like.

FROM golang:1.13-alpine AS builder

ARG GITLAB_LOGIN
ARG GITLAB_TOKEN

WORKDIR /app

# Allow pulling private modules
RUN apk add --no-cache git
RUN echo "machine git.petfinder.com login ${GITLAB_LOGIN} password ${GITLAB_TOKEN}" > ~/.netrc

# prevent the reinstallation of vendors at every change in the source code
COPY go.mod go.sum ./
RUN go mod download && go mod verify

# Copy and build the app
COPY . .
RUN go build -o bin/app .

FROM alpine:latest

COPY --from=builder /app/bin/app /app
RUN chown nobody: /app
USER nobody

CMD [ "/app" ]

Then when we build the Docker image we pass the arguments. Make sure you’re using a token with only the read_repository scope otherwise changes could be pushed from the Docker container.

docker build --build-arg GITLAB_LOGIN=${GITLAB_LOGIN} --build-arg GITLAB_TOKEN=${GITLAB_TOKEN} company/app .

If you’re using Docker Compose, you can pass the args in the build option.

version: '3.7'
services:
  app:
    build:
      context: .
      target: builder
      args:
        GITLAB_LOGIN: ${GITLAB_LOGIN}
        GITLAB_TOKEN: ${GITLAB_TOKEN}
    image: company/app
    ports:
      - 8080:8080
    volumes:
      - ./:/app:cached
    command: go run ./main.go

Continuous Integration

The same methodology applies to Continuous Integration. Again, make sure the token you’re using has only the read_repository scope. You don’t want to accidentally push or give someone else access to your repositories.

We use GitLab CI, but the same should apply to other CI platforms like Circle or Jenkins. In our before_script, we’re crafting our ~/.netrc file, pulling values from CI/CD Variable configuration.

.golang:
  image: golang:1.13
  before_script:
    - echo "machine git.petfinder.com login ${GITLAB_LOGIN} password ${GITLAB_TOKEN}" > ~/.netrc

test:unit:
  extends: .golang
  script: make test

Setup

Docker

Continuous Integration

Matt Loberg