I make apps for other people

How I’m vendoring in Go

Posted by Chris Jones
On April 21st, 2017 at 10:47

Permalink | Trackback | Links In |

No Comments |
Posted in Work

I’ve gotten questions about this at work when presenting and teaching people about Go development and there may still be some confusion about transitive dependencies and how to mix multiple Open Source projects with their own vendored dependencies. This may not be the best choice, but it works for an company with several different projects that need “always working” builds.

Create a vendor repository
We’re running GitHub Enterprise at work, and we have one major organization where most of our repositories live (and all of our Go repositories). This made it easy for me to squat on the vendor repository and start vendoring Open Source projects there.

Check out your new vendor repository. For the sake of clarity, I’ll assume it’s checked out in your $GOPATH at ~/go/src/gitrepo/vendor.

Pull down the dependencies you need with go get or git clone, then perform a git-adectomy by removing the .git directories. The project is now your own and is cut off from it’s head and master. (If you need to update a library, you can backport patches or clone a new head.)

find . -name .git -exec rm -r {} \;

Move the dependency directory (deplibrary) to your vendor repository directory.

mv deplibrary ~/go/src/gitrepo/vendor/

Add the dependency to your Git repo and check it in.

cd ~/go/src/gitrepo/vendor
git add --all deplibrary
git commit -m "Vendored deplibrary"

What about transitive dependencies?
A great example of managing transitive dependencies is the AWS SDK for Go. In this case, I had to move the vendored dependencies from the AWS SDK and put them in my vendor directory. In this way, the AWS SDK could still access those dependencies, while our Go libraries would also have access to them.

In the case of two programs or libraries requiring different versions:

  • Most Go programs maintain stable APIs
  • We would choose the latest version of the library, and fix broken consumers (if any). If the broken consumer was in a third party library, we’d fix it locally and send a pull request to the library maintainer. (We haven’t run into this scenario yet.)

A big Go program we have is split into four repositories:

  • a main program (web service, handler registration, configuration and logging setup)
  • an authenticated front-end web app
  • a non-authenticated front-end web app
  • a shared utilities library

And, consumed by the shared utilities library and the main program, we also have database drivers and wrappers, a third party HTTP routing library, LDAP and RADIUS libraries, etc. Because of the nature of the transitive dependencies, we could not vendor in each library (conflicting names, ugly import paths). Instead, every library is at a top level, including the vendored dependencies. When building our binary, our script needs to do something like:

# starts in the ${WORKSPACE} directory
rm -rf src/github* src/gopkg*
mkdir -p ${WORKSPACE}/src/github.company.com/org/
cd ${WORKSPACE}/src/github.company.com/org/
git clone git@github.company.com:org/vendor.git
git clone git@github.company.com:org/app-utils.git
git clone git@github.company.com:org/app-web.git
git clone git@github.company.com:org/app-auth.git
git clone git@github.company.com:org/app-server.git
git clone git@github.company.com:org/db-wrapper.git
cd app-server
go build
tar zcvf ${WORKSPACE}/app-server.${BUILD_ID}.tar.gz app-server properties/ static/ templates/ dbmap/

Using this pattern, we automatically get approved, reviewed, and known good versions of all the vendored dependencies, and vendoring a library immediately makes it available without changing our build automation scripts.

Leave a Reply

Comments will be closed on 6/5/2017.