Carbon Silk

This article has a simple aim to improve the process of developing and deploying node.js web applications. The Continuous Integration folks will refer to what I am covering as an aptly named Automated Deployment. As you explore this guide you will learn:

• How to run a node.js script using Upstart.
• How to manage Jenkins (was Hudson) with Git.
• How to make and use Makefiles.

Lets crack on…

I have detailed an environment scheme below which works well for me. You are free to assign your machine names what you please but following this guide will be easiest if you follow my conventions.

Hostname	Description	Operating System
localhost	Your machine you develop from	Mac / Windows / Linux
ci.app	Jenkins and Git Server	Ubuntu
int.app	node.js application server	Ubuntu

Getting a working node.js app and environment

You will need to have an environment to run your app in. To save on resources I prefer using virtual machines hosted on my localhost. This also means you are not tied to a network in order to develop (great for coding on the train/in a coffee shop).

You can run node.js apps on your localhost but for this article we are just focusing the deployment process to an integration server (int.app).

The correct development flow for a node.js app should be that you test on localhost, then checkin when you are happy with your changes. The checkin will then trigger a build and automatically get built to int.app. We will be in this sweet spot by the end of this article.

Setting up your virtual machines

It is fairly straightforward (and free) to get a Linux virtual machine up and running these days. Following the steps on https://help.ubuntu.com/community/VirtualBox should be a good starting point.
I suggest you set up one machine (int.app) first with a barebones configuration and clone it to create ci.app, changing the hostname (sudo hostname newhostname) and getting a new IP address sudo dhclient on the cloned machine.

Configuring int.app

Our initial development centric dependencies are Git, node.js, Express and NPM.

Login to int.app

ssh int.app

Install Git

apt-get install git git-core

Install node.js

cd /tmp/
git clone https://github.com/joyent/node.git
cd node
./configure
make
make install

Install NPM

apt-get install curl
curl http://npmjs.org/install.sh | sudo sh

Install Express

npm install express

Now you have everything you need to get a node.js app up and running. Let’s create our demo app.

mkdir -p /data/apps/hello_world

Add these contents to /data/apps/hello_world/app.js.

var express = require('express')
  , app = express.createServer()

app.get('/', function(req, res){
  res.send([
      '<h1>Hello World</h1>'
  ].join('\n'));
});

app.listen(3000);
console.log('Express app started on port 3000');

Run your node.js app

node /data/apps/hello_world/app.js

Getting and using the IP address of int.app

SSH into int.app and run:

ifconfig -a

1. Grab the IP address from ‘inet addr’. This should look something like: 192.188.0.23.
2. Add this to your /etc/hosts file on your localhost like: 192.188.0.23 int.app

Testing the app

Now let’s test if everything is running ok. Open up http://int.app:3000/. With any luck you should be seeing the text “Hello World”.

Getting a build process working

We will be using Git as our source control management tool (VCS). Have a read of Joel Spolsky’s distributed VCS post to understand why we are using Git over something like SVN.

Setting up a Git server on ci.app

Git works in a de-centralised way but we need to have a central Git repository that we and other developers can push their changes to and other tools like Jenkins can pull updates from.

On ci.app run:

useradd git
su git
sudo mkdir -p /data/git/hello_world.git
sudo chown git -R /data/git/hello_world.git
cd /data/git/hello_world.git
git --bare init
curl http://utsl.gen.nz/git/post-update > /data/git/hello_world.git/hooks/post-update
exit

On localhost you can now run:

git clone ssh://ci.app/data/git/hello_world.git ~/hello_world
cd hello_world
git remote add origin ssh://git@ci.app/data/git/hello_world.git

Now you have an empty directory on your local machine where you can develop from. Let’s add app.js we made earlier to this repository.

Add these contents to ~/hello_world/app.js

var express = require('express')
  , app = express.createServer()

app.get('/', function(req, res){
  res.send([
      '<h1>Hello World</h1>'
  ].join('\n'));
});

app.listen(3000);
console.log('Express app started on port 3000');

Now lets check it in.

git add --all
git commit -m 'Initial checkin' app.js

Let’s push it to our remote Git server

git push origin master

Moving the app automatically

We are going to use make as a tool to bundle commands. We will use Make to move the application from the Git repository to the /data/apps/hello_world directory.

On localhost, add these contents to ~/hello_world/Makefile

install :
	sudo mkdir -p /data/apps/hello_world
	sudo cp ./app.js /data/apps/hello_world/

Check this file in and push changes:

git add --all
git commit -m 'adding build steps'
git push origin master

Now you could run this command and have the app put in the correct location for you:

make install

Automating the build

Jenkins is a very useful tool in the process of Continuous Integration. At the most basic level Jenkins is a tool to allow you to schedule things to happen based on timings or events such as a code checkin.

Installing Jenkins

Let’s install Jenkins on ci.app.

apt-get install jenkins

Once installed you should be able to load Jenkins in your browser: at http://ci.app:8080/.

Adding a new job to Jenkins

Here you can add a ‘job’ by:

1. Selecting ‘New Job‘.
2. Call the Job name “Hello World”.
3. Select the option ‘Build a free-style software project’.
4. Save

Once created you will be met with a page with a bunch of inputs. We will come back to that in a moment but first we need to add Git support to Jenkins.

Adding Git support to Jenkins

1. Go to http://ci.app:8080/pluginManager/available.
2. Click on the enable Git Plugin checkbox.
3. Click install at the bottom of the page.
4. Return back to your job configuration page (http://ci.app:8080/job/Hello World/configure)
5. Select the Git option under the Source Code Management section.
6. Add this url into the url field: ssh://git@localhost/data/git/hello_world.git
7. Select the Poll SCM checkbox under Build Triggers and enter: * * * * * (This will make Jenkins check every minute for changes to source files.)
8. Save the configuration
9. Click ‘build now’ to check everything is working ok.

As it stands, the Jenkins job is not being all that useful but we will discover it’s power later on.

Deploying your app using Upstart

Node.js scripts need to be run as daemons for hosting websites, which can make the deployment process a bit fiddly. We can use tools like Upstart to run and manage node.js scripts.

Install Upstart

apt-get install upstart

Adding Upstart app configuration

On localhost, add these contents to ~/hello_world/hello_world.conf

#!upstart
description "node.js hello world app server"
author      "James Broad"

start on startup
stop on shutdown

script
    export HOME="/root"

    exec /usr/local/bin/node /data/apps/hello_world/app.js 2>&1 >> /var/log/node.log
end script

We will be putting this file into /etc/event.d/hello_world on int.app later on. Once in that location you can run sudo start hello_world

Continuously deploying your app

Now that Jenkins is set up to keep an eye on our files, we can issue it some commands to deploy and run the app.js application.

Let’s return to localhost and add a deployment step to Makefile. This will copy the files needed to install and run our application.
Add these contents to ~/hello_world/Makefile

app_dir = /data/apps/hello_world
temp_install_dir = /tmp/hello_world
deployment_hostname = int.app

install :
	sudo mkdir -p $(app_dir)
	sudo cp ./app.js $(app_dir)/app.js
	sudo cp ./hello_world.conf /etc/event.d/hello_world

deploy :
	# Get rid of old temp installs
	ssh $(deployment_hostname) sudo rm -rf $(temp_install_dir)
	# Copy files over to remote machine
	rsync -r . $(deployment_hostname):$(temp_install_dir)
	# Install our app to the right location
	ssh $(deployment_hostname) cd $(temp_install_dir)\; make install
	ssh $(deployment_hostname) cd $(temp_install_dir)\; make start_app

start_app :
	sudo start  --no-wait -q hello_world

Check in and push your changes.

git add --all
git commit -m 'adding deployment steps'
git push origin master

Handling environment permissions

On int.app you will NEED to set up your SSH Keys to allow passwordless ssh access and add paswordless sudo rights for the user jenkins so it can carry out tasks that require sudo such as service restarting.

Passwordless SSH access

To set up passwordless SSH access for Jenkins, log into ci.app as the user jenkins ssh jenkins@ci.app. Run and complete the prompts:

ssh-keygen

Copy the output of ~/.ssh/id_dsa.pub or ~/.ssh/id_rsa.pub (depending on what option you chose) to the file ~/.ssh/authorized_keys on int.app.

Now SSH to int.app and add the jenkins user adduser jenkins, this will be needed for when Jenkins is installing stuff onto int.app.

Now we need to log into ci.app as user jenkins to be able to accept the SSH Keys:

ssh jenkins@ci.app
ssh jenkins@int.app
exit
exit

Passwordless sudo rights

For enabling passwordless sudo rights to the jenkins user you will need to run: sudo visudo and append the file with the contents:

# Allow Jenkins to run certain commands as sudo without password
jenkins ALL = NOPASSWD: /bin/mkdir, /bin/rm, /bin/cp, /sbin/start, /sbin/stop

Adding shell script stages to Jenkins

Go to your hello world configuration panel (http://ci.app:8080/job/Hello World) and choose ‘add build step’ under the Build section.

1. Select ‘Execute shell’ and add this line: make deploy.
2. Save the settings.
3. Run a build.

If everything went according to plan you should be able to load http://int.app:3000 and see the text ‘Hello World’.

Verifying deployments

Instead of us having to check the app is running every time a build happens, we can get Jenkins to do that for us by checking a url returns 200 response code.

Lets add a new Freestyle job in Jenkins and call it “Hello World Monitor”. Leave everything blank but click the checkbox for Monitor Site and add the url http://int.app:3000/.

We can now return back to our Hello World job and check the Build other projects checkbox. Then add Hello World Monitor (it will auto-suggest). Save this configuration and run a build.

Finally

If you have made it this far, thanks for reading, hopefully everything went well?

If everything went well, you are in a good place to start adding more Continuous Integration elements such as unit tests, JSLinting and many other automated steps.

There are a number of things I didn’t go into much depth on such as the node.js app itself, there are plenty of articles out there about how to do this. A good resource worth checking out is http://howtonode.org.

Feel free to get in touch if you need any help but be warned I have a busy day job so I may take time to respond.

Image courtesy of Me