nodejs

Switching from private Github repos to npm private modules

Following software engineering best practice, the team strives to write code that is modular and self-contained, and thus reusable across multiple projects. We have tended to move away from writing monolithic applications, towards building services that do one thing really well and 'library' repositories with code shared between these services. As a result our work is spread across multiple repositories: one repo per service, and multiple versioned library repositories in use by those services.

One challenge with this approach has been installing private repositories, which are only accessible by users that have Github collaborator or owner access to these projects. Because npm did not have a private registry to publish to (without buying npm enterprise or some other private registry), we fell back to a make-shift approach:

  • Specify the package as a dependency from Github rather than the npm package registry.
  • Check in the package to source control so npm won't refetch it, and our deploy scripts and Travis CI don’t need our git credentials.
  • Use the git commit hash, not a tag or branch, to specify the package. Later versions of npm always refetch any packages specified by git tags or git branches, but not those specified by git commits.

This has caused headache for the team because for clarity we much prefer to use tags to specify versions, and we've been forced to pin our npm version because git support has changed in recent versions.

So we were very excited when npm finally released support for private modules. Now we can specify the exact package version we want to use in package.json instead of using the git hash, so:

"@spanishdict/lib": "1.0.0"

instead of

"lib": "git+ssh://github.com/spanishdict/lib#longmeaninglesshash"

Overall the process to switch from this approach to using private modules was simple, but some of the details weren't so obvious. Here's how we did it and some things we learned along the way.

1. Sign up for private modules

The first step is to sign up for npm private modules. If you're working under a github organization, npm recommends creating a separate user for your organization which can later be upgraded to an organization when support for it is ready.

We created a spanishdict npm user for this purpose.

2. Upgrade npm

npm private modules depends on scoped packages, which came into being sometime during npm 2 so any version earlier than that will result in a 404 when npm can’t find your scoped private modules. I recommend upgrading to the latest npm if you can, which is 2.9.0 at the time of writing this.

$ npm install -g npm@2.9.0

In fact we ran into an issue with npm 1.4.x, which interprets scoped packages as github shorthand. This means that if your scoped package name is “@spanishdict/pkg” and it is in a github repo called github.com/spanishdict/pkg, then someone with npm 1.4.x who has access to the github repo will successfully install it from github. This caused us some confusion and we reached out to npm about it, and they are considering a patch release of 1.4.x to prevent this behavior.

3. Publish a private module

Log in to npm on the command line using npm login or npm adduser, with the account that you used to sign up for private modules. npm whoami is useful for verifying that you've switched users correctly.

Navigate to the root of the repository you plan to publish to npm private registry. Update the name of the project package.json by prefixing it with the name of the private modules account. For example, we updated the name field to:

"name": "@spanishdict/module-name",

If you have specified private: true, remove this line before publishing as npm will give a warning about this. Scoped modules are published privately by default, but you can explicitly specify the access level when you publish to the npm registry:

$ npm publish --access restricted

Navigate to e.g. https://www.npmjs.com/~spanishdict to see the newly published private module.

4. Install your new private module

Navigate to the repository that will start consuming the new private module you just published. We had to remove the old checked-in module from source control; you won't have to do this if you're using this module for the first time.

$ npm uninstall module-name --save

But git caches committed files, so in addition to removing module-name from .gitignore, you have to delete the files from git's cache:

$ git rm --cached -r node_modules/module-name

Now install your module from the npm private registry. You must be logged in as a user with access to the private module (either as the owner that published the package, or as a collaborator that's been added to the package).

$ npm install @spanishdict/module-name --save

You'll see the new line in package.json with slightly different syntax:

"@spanishdict/module-name": "1.0.0",

5. Use your new private module

Now you can use the new module in code. Note that the require syntax changes slightly from require('module-name') to require('@spanishdict/module-name').

And that's it! Make a pull request to switch this repo to using private modules and feel free to link to this article so your reviewer can follow along.

Why are we checking these modules in to source control?

There's a lot of discussion about best practice surrounding checking in node_modules, but for us it comes down to how we handle deployments:

  • We don't need to put access credentials on external machines because the install step doesn't need to fetch private repositories.
  • Removes npm or any other registry as a point of failure and dependency for deploying code. For instance if npm goes down for whatever reason, we can still deploy our code to users. This has been less of a concern lately since npm has become much more stable, but we try to cut down external dependencies wherever possible.

That last point is the deciding factor for us, so for now we're sticking to checking in node_modules.

What do you think about npm private modules? Do you have a better way to handle private dependencies? Let us know in the comments below!

Javascript Testing with Mocha, Sinon, and Chai

Javascript Testing with Mocha, Sinon, and Chai

Recently, April 24th, I presented a short talk on some of the testing tools we use here at Fluencia. Primarily, the discussion is focused on Mocha, Chai, and Sinon, which are all npm modules that can be easily incorporated into a node project. In this post I'll give a somewhat abridged discussion of the topics covered in the talk.

Here at Fluencia, we use the tools Mocha, Sinon, and Chai to test our code. The testing framework developed from these tools and the synergy between them has been easy to build and maintain, and we wanted to share that with anyone who isn't doing testing or who would like to improve their testing setup. The audience for the talk was the awesome DC jQuery meetup group, but like I note in the slides, these tools can be used for testing back-end and front-end javascript.  I'll start with a brief intro to testing in general; if you are already familiar with the concepts, you can skip the intro and dive into learning the tools.

You can see a short clip of the presentation here.  The slides can be found here and the audio can be found here.

The 4 Keys to 100% Uptime with Node.js

On Tuesday, I presented at Nova Node on achieving 100% uptime with Node.js. [Update: video of the talk is now available.] This topic is near and dear to our hearts: giving our users the best experience means keeping Fluencia and SpanishDict running all the time, every day. This takes work, and over the past few months we've had opportunities to dive deep into error handling/recovery and zero-downtime deployment techniques.

We've found that the four keys to (nearly) 100% uptime with Node.js are:

  1. Sensibly handle uncaught exceptions.
  2. Use domains to catch and contain errors.
  3. Manage processes with cluster.
  4. Gracefully terminate connections.

1. Sensibly handle uncaught exceptions.

Uncaught exceptions happen when an Error is thrown outsisde of a try/catch block, or an error event is emitted and nothing is listening for it. Node's default action following an uncaught exception is to exit (crash) the process. If the process is a server, this leads to downtime, so we want to avoid uncaught exceptions as much as possible, and sensibly handle them when they do happen.

Avoid them can be accomplished by writing solid code and testing it thoroughly, and by using domains.

2. Use domains to catch and contain errors.

Domains can catch asynchronous errors—for example, error events—which try/catch cannot do.

var d = require('domain').create();

d.on('error', function (err) {
  console.log("domain caught", err);
});

var f = d.bind(function() {
  console.log(domain.active === d); // <-- data-preserve-html-node="true" true
  throw new Error("uh-oh");
});

setTimeout(f, 100);

The above example would crash the Node process if function f (which throws an error) were not called in the context of a domain. Instead, the domain's error handler is called, giving you a chance to take some action: retry, abort, ignore, or even rethrow. What to do is dependent on context, but the point is that you can choose, rather than having the process crash.

When code is running in the context of a domain—that is, a domain is "active"—then domain.active contains a reference to it. New EventEmitters will automatically bind to the active domain, saving you the hassle of manually listening to their error events (though there may still be good reason to do so—but if you miss one, it will automatically go to the domain error handler).

domain.add will add an existing EventEmitter to a domain. Domains have more methods than add and bind; the full list of is available in the docs.

3. Manage processes with cluster.

Domains alone are not enough to achieve continuous uptime. Sometimes an individual process will need to go down whether due to a rethrown error or to deployment. Using node's cluster module, we can spawn a master process that manages one or more worker processes, distributing work (for example, incoming connections from clients) among them. The master process need never stop running, meaning the socket always accepts connections, and when any particular worker stops working, the master can replace it.

We can send a signal to the master process when we want to initiate a reloading of workers. For example, to deploy without downtime, we might give our master a symlink that points to worker code, then later update that symlink to point to a new version of the code, and send a signal to the master process that causes it to shut down existing workers and fork new workers. The new workers will come up running the new code, and the master process keeps running the entire time!

Coordination between master and workers is important. Workers must communicate their state, and master must know what action to take based on that. We've found recluster to be a useful wrapper around Node's native cluster module that provides worker state management.

4. Gracefully terminate connections.

When workers/servers need to shutdown, they must do so gracefully, as they may be serving any number of in-flight requests, and also have open TCP connections to many more clients. Any call to process.exit should occur inside the callback to the server.close method, which only fires once a server has closed all its connections.

To close existing connections, we can add middleware that checks whether an app is in a shutting down state, and if so, causes connections to be closed ASAP:

var afterErrorHook = function(err) { // <-- data-preserve-html-node="true" called after an unrecoverable error
  app.set("isShuttingDown", true); // <-- data-preserve-html-node="true" set state
  server.close(function() {
    process.exit(1);  // <-- data-preserve-html-node="true" all clear to exit
  });
}

var shutdownMiddle = function(req, res, next) {
  if(app.get("isShuttingDown") {  // <-- data-preserve-html-node="true" check state
    req.connection.setTimeout(1);  // <-- data-preserve-html-node="true" kill keep-alive
  }
  next();
}

To ensure all servers shutdown eventually, we can also call process.exit inside of a timer.

Back to handling uncaught exceptions.

The Node docs say not to keep running after an uncaught exception, for good reason:

An unhandled exception means your application — and by extension node.js itself — is in an undefined state. Blindly resuming means anything could happen. You have been warned.

http://nodejs.org/api/process.html#processeventuncaughtexception

Now we have the pieces in place to sensibly handle uncaught exceptions:

  • Log error.
  • Server stops accepting new connections.
  • Worker tells cluster master it's done.
  • Master forks a replacement worker.
  • Worker exits gracefully when all connections are closed, or after timeout.

If the exception was triggered by a request, it is not possible to respond to that request because the uncaught exception lacks the context to determine which request it was. But we should be able to handle any other in-flight requests, minimizing effective downtime.

Remember that while parts of Node are very stable, and it is in production use with greate success by many, many companies (including us!), other parts such as domains and the cluster module are still classified as unstable or experimental, and may change in the future.

This was a quick summary of what I covered in the talk. The slides below contain additional example code and various tips and other considerations. Hopefully they will help you to achieve (nearly) 100% uptime with your Node application!