This article has a simple aim to improve the process of developing and deploying node.js web applications. The Continuous Integration folks will refer to what I am covering as an aptly named Automated Deployment. As you explore this guide you will learn:

• How to run a node.js script using Upstart.
• How to manage Jenkins (was Hudson) with Git.
• How to make and use Makefiles.

Lets crack on…

I have detailed an environment scheme below which works well for me. You are free to assign your machine names what you please but following this guide will be easiest if you follow my conventions.

Getting a working node.js app and environment

You will need to have an environment to run your app in. To save on resources I prefer using virtual machines hosted on my localhost. This also means you are not tied to a network in order to develop (great for coding on the train/in a coffee shop).

You can run node.js apps on your localhost but for this article we are just focusing the deployment process to an integration server (int.app).

The correct development flow for a node.js app should be that you test on localhost, then checkin when you are happy with your changes. The checkin will then trigger a build and automatically get built to int.app. We will be in this sweet spot by the end of this article.

Setting up your virtual machines

It is fairly straightforward (and free) to get a Linux virtual machine up and running these days. Following the steps on https://help.ubuntu.com/community/VirtualBox should be a good starting point.
I suggest you set up one machine (int.app) first with a barebones configuration and clone it to create ci.app, changing the hostname (sudo hostname newhostname) and getting a new IP address sudo dhclient on the cloned machine.

Configuring int.app

Our initial development centric dependencies are Git, node.js, Express and NPM.

Login to int.app

ssh int.app

Install Git

apt-get install git git-core

Install node.js

cd /tmp/
git clone https://github.com/joyent/node.git
cd node
./configure
make
make install

Install NPM

apt-get install curl
curl http://npmjs.org/install.sh | sudo sh

Install Express

npm install express

Now you have everything you need to get a node.js app up and running. Let’s create our demo app.

mkdir -p /data/apps/hello_world

Add these contents to /data/apps/hello_world/app.js.

var express = require('express')
  , app = express.createServer()

app.get('/', function(req, res){
  res.send([
      '<h1>Hello World</h1>'
  ].join('\n'));
});

app.listen(3000);
console.log('Express app started on port 3000');

Run your node.js app

node /data/apps/hello_world/app.js

Getting and using the IP address of int.app

SSH into int.app and run:

ifconfig -a

1. Grab the IP address from ‘inet addr’. This should look something like: 192.188.0.23.
2. Add this to your /etc/hosts file on your localhost like: 192.188.0.23 int.app

Testing the app

Now let’s test if everything is running ok. Open up http://int.app:3000/. With any luck you should be seeing the text “Hello World”.

Getting a build process working

We will be using Git as our source control management tool (VCS). Have a read of Joel Spolsky’s distributed VCS post to understand why we are using Git over something like SVN.

Setting up a Git server on ci.app

Git works in a de-centralised way but we need to have a central Git repository that we and other developers can push their changes to and other tools like Jenkins can pull updates from.

On ci.app run:

useradd git
su git
sudo mkdir -p /data/git/hello_world.git
sudo chown git -R /data/git/hello_world.git
cd /data/git/hello_world.git
git --bare init
curl http://utsl.gen.nz/git/post-update > /data/git/hello_world.git/hooks/post-update
exit

On localhost you can now run:

git clone ssh://ci.app/data/git/hello_world.git ~/hello_world
cd hello_world
git remote add origin ssh://git@ci.app/data/git/hello_world.git

Now you have an empty directory on your local machine where you can develop from. Let’s add app.js we made earlier to this repository.

Add these contents to ~/hello_world/app.js

var express = require('express')
  , app = express.createServer()

app.get('/', function(req, res){
  res.send([
      '<h1>Hello World</h1>'
  ].join('\n'));
});

app.listen(3000);
console.log('Express app started on port 3000');

Now lets check it in.

git add --all
git commit -m 'Initial checkin' app.js

Let’s push it to our remote Git server

git push origin master

Moving the app automatically

We are going to use make as a tool to bundle commands. We will use Make to move the application from the Git repository to the /data/apps/hello_world directory.

On localhost, add these contents to ~/hello_world/Makefile

install :
	sudo mkdir -p /data/apps/hello_world
	sudo cp ./app.js /data/apps/hello_world/

Check this file in and push changes:

git add --all
git commit -m 'adding build steps'
git push origin master

Now you could run this command and have the app put in the correct location for you:

make install

Automating the build

Jenkins is a very useful tool in the process of Continuous Integration. At the most basic level Jenkins is a tool to allow you to schedule things to happen based on timings or events such as a code checkin.

Installing Jenkins

Let’s install Jenkins on ci.app.

apt-get install jenkins

Once installed you should be able to load Jenkins in your browser: at http://ci.app:8080/.

Adding a new job to Jenkins

Here you can add a ‘job’ by:

1. Selecting ‘New Job‘.
2. Call the Job name “Hello World”.
3. Select the option ‘Build a free-style software project’.
4. Save

Once created you will be met with a page with a bunch of inputs. We will come back to that in a moment but first we need to add Git support to Jenkins.

Adding Git support to Jenkins

1. Go to http://ci.app:8080/pluginManager/available.
2. Click on the enable Git Plugin checkbox.
3. Click install at the bottom of the page.
4. Return back to your job configuration page (http://ci.app:8080/job/Hello World/configure)
5. Select the Git option under the Source Code Management section.
6. Add this url into the url field: ssh://git@localhost/data/git/hello_world.git
7. Select the Poll SCM checkbox under Build Triggers and enter: * * * * * (This will make Jenkins check every minute for changes to source files.)
8. Save the configuration
9. Click ‘build now’ to check everything is working ok.

As it stands, the Jenkins job is not being all that useful but we will discover it’s power later on.

Deploying your app using Upstart

Node.js scripts need to be run as daemons for hosting websites, which can make the deployment process a bit fiddly. We can use tools like Upstart to run and manage node.js scripts.

Install Upstart

apt-get install upstart

Adding Upstart app configuration

On localhost, add these contents to ~/hello_world/hello_world.conf

#!upstart
description "node.js hello world app server"
author      "James Broad"

start on startup
stop on shutdown

script
    export HOME="/root"

    exec /usr/local/bin/node /data/apps/hello_world/app.js 2>&1 >> /var/log/node.log
end script

We will be putting this file into /etc/event.d/hello_world on int.app later on. Once in that location you can run sudo start hello_world

Continuously deploying your app

Now that Jenkins is set up to keep an eye on our files, we can issue it some commands to deploy and run the app.js application.

Let’s return to localhost and add a deployment step to Makefile. This will copy the files needed to install and run our application.
Add these contents to ~/hello_world/Makefile

app_dir = /data/apps/hello_world
temp_install_dir = /tmp/hello_world
deployment_hostname = int.app

install :
	sudo mkdir -p $(app_dir)
	sudo cp ./app.js $(app_dir)/app.js
	sudo cp ./hello_world.conf /etc/event.d/hello_world

deploy :
	# Get rid of old temp installs
	ssh $(deployment_hostname) sudo rm -rf $(temp_install_dir)
	# Copy files over to remote machine
	rsync -r . $(deployment_hostname):$(temp_install_dir)
	# Install our app to the right location
	ssh $(deployment_hostname) cd $(temp_install_dir)\; make install
	ssh $(deployment_hostname) cd $(temp_install_dir)\; make start_app

start_app :
	sudo start  --no-wait -q hello_world

Check in and push your changes.

git add --all
git commit -m 'adding deployment steps'
git push origin master

Handling environment permissions

On int.app you will NEED to set up your SSH Keys to allow passwordless ssh access and add paswordless sudo rights for the user jenkins so it can carry out tasks that require sudo such as service restarting.

Passwordless SSH access

To set up passwordless SSH access for Jenkins, log into ci.app as the user jenkins ssh jenkins@ci.app. Run and complete the prompts:

ssh-keygen

Copy the output of ~/.ssh/id_dsa.pub or ~/.ssh/id_rsa.pub (depending on what option you chose) to the file ~/.ssh/authorized_keys on int.app.

Now SSH to int.app and add the jenkins user adduser jenkins, this will be needed for when Jenkins is installing stuff onto int.app.

Now we need to log into ci.app as user jenkins to be able to accept the SSH Keys:

ssh jenkins@ci.app
ssh jenkins@int.app
exit
exit

Passwordless sudo rights

For enabling passwordless sudo rights to the jenkins user you will need to run: sudo visudo and append the file with the contents:

# Allow Jenkins to run certain commands as sudo without password
jenkins ALL = NOPASSWD: /bin/mkdir, /bin/rm, /bin/cp, /sbin/start, /sbin/stop

Adding shell script stages to Jenkins

Go to your hello world configuration panel (http://ci.app:8080/job/Hello World) and choose ‘add build step’ under the Build section.

1. Select ‘Execute shell’ and add this line: make deploy.
2. Save the settings.
3. Run a build.

If everything went according to plan you should be able to load http://int.app:3000 and see the text ‘Hello World’.

Verifying deployments

Instead of us having to check the app is running every time a build happens, we can get Jenkins to do that for us by checking a url returns 200 response code.

Lets add a new Freestyle job in Jenkins and call it “Hello World Monitor”. Leave everything blank but click the checkbox for Monitor Site and add the url http://int.app:3000/.

We can now return back to our Hello World job and check the Build other projects checkbox. Then add Hello World Monitor (it will auto-suggest). Save this configuration and run a build.

Finally

If you have made it this far, thanks for reading, hopefully everything went well?

If everything went well, you are in a good place to start adding more Continuous Integration elements such as unit tests, JSLinting and many other automated steps.

There are a number of things I didn’t go into much depth on such as the node.js app itself, there are plenty of articles out there about how to do this. A good resource worth checking out is http://howtonode.org.

Feel free to get in touch if you need any help but be warned I have a busy day job so I may take time to respond.

Image courtesy of Me

I created a terminal theme with a retro look of soft amber tones. I find this really nice to work with and have been using for a couple of months now. Download or fork it on Github.

To use: download the amber.terminal file to your computer, open up Terminal.app, go to the preferences, settings tab, then under the list of themes, click on the settings button and click import, now choose the amber.terminal file.

Is YQL relevant to you? Well if you do any work with web services then definitely, it allows you to pull in multiple data sources, sort, transform, query your data points, to name but a few selling points. It will still be relevant to you if you are familiar with the SQL syntax, as it is actually really fun to play with using the console, especially if you take the provided example queries as a jump off point to tinker with. To demonstrate how easy we can make things, take a look at this example of a search query using this code.

YQL is now becoming even more powerful with a feature called ‘Open Tables’ which in essence instructs YQL on how to construct a web service URL. The power of this is that it enables you to interject values from other web service calls via sub selects. Some examples of Open Table definitions and example queries are being constructed at github.com/spullara/yql-tables.

During my talk I demonstrated a lot of the PHP code used to consume the results of the YQL calls which can be found over on my GitHub repository or you can browse my example scripts at http://carbonsilk.com/yql_php/.

To make development in PHP straightforward I used CodeIgniter for it’s MVC strengths and ease of installation. The beauty of using the framework with YQL and OAuth libraries doing the hard work behind the scenes is that you can write a few simple lines like the following to retrieve instant results:

// Controller method to allow us to access http://someurl.tld/yql_php/search_wikipedia/
function search_wikipedia($term = 'open')
{
    echo '<pre>';
    $yql_query = 'select * from xml where
            url="http://en.wikipedia.org/w/api.php?action=opensearch&search=' . $term . '&format=xml"
            and itemPath = "SearchSuggestion.Section.Item"';
    print_r($this->yql_lib->query($yql_query));
}

The above code is the same query as could be found if you view in the provided console or view in my final project Search Wikipedia through YQL.

All I can say from doing this talk and looking deep into YQL is that there is a huge potential at our disposal here.

View the template

It’s been a while since releasing an open source web template, the last one being Mighty, added back in January of 2007. Mighty was a template created to allow others to take a basic design and allow them to modify most aspects of the template with relative ease (given some CSS understanding).

Time moved on and the itch arose again to create another open source web template. This time its called Panoramic. Panoramic is the name of the family business and this template design stemmed from a mock for the company I had created some time ago that never got released.

GitHub seemed the best hosting location for an open source web template, owing to its fantastic interface and functionality and thus Panoramic lives there: http://github.com/kulor/panoramic/tree/master.

Having the template hosted on a Revision Control System such as GitHub, has many advantages, the main one being that others can contribute to the code and or fork to a version that suits them. Additionally, users downloading the code can always rest assured they are getting the latest code and can see what has changed over time.

If you do use the Panoramic template, please add a comment to this page stating where it is in use, so others can gain inspiration for it’s uses.

I have produced a simple script that will let you check your sites are up and running.

The PHP script is hosted on GitHub at http://github.com/kulor/siteup/tree/master. You can see the output at http://carbonsilk.com/sandbox/siteup/

To get this script running, you will need to change some of the defaults found in index.php and place both the files in a web accessible directory on your server

Automating the script

The best application of this script would be to deploy on your server (not on the server you are checking is up) and run on a crontab at a fairly liberal interval (2 hours) to save server overload. To do this:

On your command line:
$ crontab -e

This will take you into an editor window. Enter the line below replacing {$path} with the location of your script
0 */2 * * * php -q /{$path}/siteup/index.php

Now you will receive an email if any of your programmed domains are down.