Switching from private Github repos to npm private modules

Following software engineering best practice, the team strives to write code that is modular and self-contained, and thus reusable across multiple projects. We have tended to move away from writing monolithic applications, towards building services that do one thing really well and 'library' repositories with code shared between these services. As a result our work is spread across multiple repositories: one repo per service, and multiple versioned library repositories in use by those services.

One challenge with this approach has been installing private repositories, which are only accessible by users that have Github collaborator or owner access to these projects. Because npm did not have a private registry to publish to (without buying npm enterprise or some other private registry), we fell back to a make-shift approach:

  • Specify the package as a dependency from Github rather than the npm package registry.
  • Check in the package to source control so npm won't refetch it, and our deploy scripts and Travis CI don’t need our git credentials.
  • Use the git commit hash, not a tag or branch, to specify the package. Later versions of npm always refetch any packages specified by git tags or git branches, but not those specified by git commits.

This has caused headache for the team because for clarity we much prefer to use tags to specify versions, and we've been forced to pin our npm version because git support has changed in recent versions.

So we were very excited when npm finally released support for private modules. Now we can specify the exact package version we want to use in package.json instead of using the git hash, so:

"@spanishdict/lib": "1.0.0"

instead of

"lib": "git+ssh://github.com/spanishdict/lib#longmeaninglesshash"

Overall the process to switch from this approach to using private modules was simple, but some of the details weren't so obvious. Here's how we did it and some things we learned along the way.

1. Sign up for private modules

The first step is to sign up for npm private modules. If you're working under a github organization, npm recommends creating a separate user for your organization which can later be upgraded to an organization when support for it is ready.

We created a spanishdict npm user for this purpose.

2. Upgrade npm

npm private modules depends on scoped packages, which came into being sometime during npm 2 so any version earlier than that will result in a 404 when npm can’t find your scoped private modules. I recommend upgrading to the latest npm if you can, which is 2.9.0 at the time of writing this.

$ npm install -g npm@2.9.0

In fact we ran into an issue with npm 1.4.x, which interprets scoped packages as github shorthand. This means that if your scoped package name is “@spanishdict/pkg” and it is in a github repo called github.com/spanishdict/pkg, then someone with npm 1.4.x who has access to the github repo will successfully install it from github. This caused us some confusion and we reached out to npm about it, and they are considering a patch release of 1.4.x to prevent this behavior.

3. Publish a private module

Log in to npm on the command line using npm login or npm adduser, with the account that you used to sign up for private modules. npm whoami is useful for verifying that you've switched users correctly.

Navigate to the root of the repository you plan to publish to npm private registry. Update the name of the project package.json by prefixing it with the name of the private modules account. For example, we updated the name field to:

"name": "@spanishdict/module-name",

If you have specified private: true, remove this line before publishing as npm will give a warning about this. Scoped modules are published privately by default, but you can explicitly specify the access level when you publish to the npm registry:

$ npm publish --access restricted

Navigate to e.g. https://www.npmjs.com/~spanishdict to see the newly published private module.

4. Install your new private module

Navigate to the repository that will start consuming the new private module you just published. We had to remove the old checked-in module from source control; you won't have to do this if you're using this module for the first time.

$ npm uninstall module-name --save

But git caches committed files, so in addition to removing module-name from .gitignore, you have to delete the files from git's cache:

$ git rm --cached -r node_modules/module-name

Now install your module from the npm private registry. You must be logged in as a user with access to the private module (either as the owner that published the package, or as a collaborator that's been added to the package).

$ npm install @spanishdict/module-name --save

You'll see the new line in package.json with slightly different syntax:

"@spanishdict/module-name": "1.0.0",

5. Use your new private module

Now you can use the new module in code. Note that the require syntax changes slightly from require('module-name') to require('@spanishdict/module-name').

And that's it! Make a pull request to switch this repo to using private modules and feel free to link to this article so your reviewer can follow along.

Why are we checking these modules in to source control?

There's a lot of discussion about best practice surrounding checking in node_modules, but for us it comes down to how we handle deployments:

  • We don't need to put access credentials on external machines because the install step doesn't need to fetch private repositories.
  • Removes npm or any other registry as a point of failure and dependency for deploying code. For instance if npm goes down for whatever reason, we can still deploy our code to users. This has been less of a concern lately since npm has become much more stable, but we try to cut down external dependencies wherever possible.

That last point is the deciding factor for us, so for now we're sticking to checking in node_modules.

What do you think about npm private modules? Do you have a better way to handle private dependencies? Let us know in the comments below!