This article has a simple aim to improve the process of developing and deploying node.js web applications. The Continuous Integration folks will refer to what I am covering as an aptly named Automated Deployment. As you explore this guide you will learn:

• How to run a node.js script using Upstart.
• How to manage Jenkins (was Hudson) with Git.
• How to make and use Makefiles.

Lets crack on…

I have detailed an environment scheme below which works well for me. You are free to assign your machine names what you please but following this guide will be easiest if you follow my conventions.

Getting a working node.js app and environment

You will need to have an environment to run your app in. To save on resources I prefer using virtual machines hosted on my localhost. This also means you are not tied to a network in order to develop (great for coding on the train/in a coffee shop).

You can run node.js apps on your localhost but for this article we are just focusing the deployment process to an integration server (int.app).

The correct development flow for a node.js app should be that you test on localhost, then checkin when you are happy with your changes. The checkin will then trigger a build and automatically get built to int.app. We will be in this sweet spot by the end of this article.

Setting up your virtual machines

It is fairly straightforward (and free) to get a Linux virtual machine up and running these days. Following the steps on https://help.ubuntu.com/community/VirtualBox should be a good starting point.
I suggest you set up one machine (int.app) first with a barebones configuration and clone it to create ci.app, changing the hostname (sudo hostname newhostname) and getting a new IP address sudo dhclient on the cloned machine.

Configuring int.app

Our initial development centric dependencies are Git, node.js, Express and NPM.

Login to int.app

ssh int.app

Install Git

apt-get install git git-core

Install node.js

cd /tmp/
git clone https://github.com/joyent/node.git
cd node
./configure
make
make install

Install NPM

apt-get install curl
curl http://npmjs.org/install.sh | sudo sh

Install Express

npm install express

Now you have everything you need to get a node.js app up and running. Let’s create our demo app.

mkdir -p /data/apps/hello_world

Add these contents to /data/apps/hello_world/app.js.

var express = require('express')
  , app = express.createServer()

app.get('/', function(req, res){
  res.send([
      '<h1>Hello World</h1>'
  ].join('\n'));
});

app.listen(3000);
console.log('Express app started on port 3000');

Run your node.js app

node /data/apps/hello_world/app.js

Getting and using the IP address of int.app

SSH into int.app and run:

ifconfig -a

1. Grab the IP address from ‘inet addr’. This should look something like: 192.188.0.23.
2. Add this to your /etc/hosts file on your localhost like: 192.188.0.23 int.app

Testing the app

Now let’s test if everything is running ok. Open up http://int.app:3000/. With any luck you should be seeing the text “Hello World”.

Getting a build process working

We will be using Git as our source control management tool (VCS). Have a read of Joel Spolsky’s distributed VCS post to understand why we are using Git over something like SVN.

Setting up a Git server on ci.app

Git works in a de-centralised way but we need to have a central Git repository that we and other developers can push their changes to and other tools like Jenkins can pull updates from.

On ci.app run:

useradd git
su git
sudo mkdir -p /data/git/hello_world.git
sudo chown git -R /data/git/hello_world.git
cd /data/git/hello_world.git
git --bare init
curl http://utsl.gen.nz/git/post-update > /data/git/hello_world.git/hooks/post-update
exit

On localhost you can now run:

git clone ssh://ci.app/data/git/hello_world.git ~/hello_world
cd hello_world
git remote add origin ssh://git@ci.app/data/git/hello_world.git

Now you have an empty directory on your local machine where you can develop from. Let’s add app.js we made earlier to this repository.

Add these contents to ~/hello_world/app.js

var express = require('express')
  , app = express.createServer()

app.get('/', function(req, res){
  res.send([
      '<h1>Hello World</h1>'
  ].join('\n'));
});

app.listen(3000);
console.log('Express app started on port 3000');

Now lets check it in.

git add --all
git commit -m 'Initial checkin' app.js

Let’s push it to our remote Git server

git push origin master

Moving the app automatically

We are going to use make as a tool to bundle commands. We will use Make to move the application from the Git repository to the /data/apps/hello_world directory.

On localhost, add these contents to ~/hello_world/Makefile

install :
	sudo mkdir -p /data/apps/hello_world
	sudo cp ./app.js /data/apps/hello_world/

Check this file in and push changes:

git add --all
git commit -m 'adding build steps'
git push origin master

Now you could run this command and have the app put in the correct location for you:

make install

Automating the build

Jenkins is a very useful tool in the process of Continuous Integration. At the most basic level Jenkins is a tool to allow you to schedule things to happen based on timings or events such as a code checkin.

Installing Jenkins

Let’s install Jenkins on ci.app.

apt-get install jenkins

Once installed you should be able to load Jenkins in your browser: at http://ci.app:8080/.

Adding a new job to Jenkins

Here you can add a ‘job’ by:

1. Selecting ‘New Job‘.
2. Call the Job name “Hello World”.
3. Select the option ‘Build a free-style software project’.
4. Save

Once created you will be met with a page with a bunch of inputs. We will come back to that in a moment but first we need to add Git support to Jenkins.

Adding Git support to Jenkins

1. Go to http://ci.app:8080/pluginManager/available.
2. Click on the enable Git Plugin checkbox.
3. Click install at the bottom of the page.
4. Return back to your job configuration page (http://ci.app:8080/job/Hello World/configure)
5. Select the Git option under the Source Code Management section.
6. Add this url into the url field: ssh://git@localhost/data/git/hello_world.git
7. Select the Poll SCM checkbox under Build Triggers and enter: * * * * * (This will make Jenkins check every minute for changes to source files.)
8. Save the configuration
9. Click ‘build now’ to check everything is working ok.

As it stands, the Jenkins job is not being all that useful but we will discover it’s power later on.

Deploying your app using Upstart

Node.js scripts need to be run as daemons for hosting websites, which can make the deployment process a bit fiddly. We can use tools like Upstart to run and manage node.js scripts.

Install Upstart

apt-get install upstart

Adding Upstart app configuration

On localhost, add these contents to ~/hello_world/hello_world.conf

#!upstart
description "node.js hello world app server"
author      "James Broad"

start on startup
stop on shutdown

script
    export HOME="/root"

    exec /usr/local/bin/node /data/apps/hello_world/app.js 2>&1 >> /var/log/node.log
end script

We will be putting this file into /etc/event.d/hello_world on int.app later on. Once in that location you can run sudo start hello_world

Continuously deploying your app

Now that Jenkins is set up to keep an eye on our files, we can issue it some commands to deploy and run the app.js application.

Let’s return to localhost and add a deployment step to Makefile. This will copy the files needed to install and run our application.
Add these contents to ~/hello_world/Makefile

app_dir = /data/apps/hello_world
temp_install_dir = /tmp/hello_world
deployment_hostname = int.app

install :
	sudo mkdir -p $(app_dir)
	sudo cp ./app.js $(app_dir)/app.js
	sudo cp ./hello_world.conf /etc/event.d/hello_world

deploy :
	# Get rid of old temp installs
	ssh $(deployment_hostname) sudo rm -rf $(temp_install_dir)
	# Copy files over to remote machine
	rsync -r . $(deployment_hostname):$(temp_install_dir)
	# Install our app to the right location
	ssh $(deployment_hostname) cd $(temp_install_dir)\; make install
	ssh $(deployment_hostname) cd $(temp_install_dir)\; make start_app

start_app :
	sudo start  --no-wait -q hello_world

Check in and push your changes.

git add --all
git commit -m 'adding deployment steps'
git push origin master

Handling environment permissions

On int.app you will NEED to set up your SSH Keys to allow passwordless ssh access and add paswordless sudo rights for the user jenkins so it can carry out tasks that require sudo such as service restarting.

Passwordless SSH access

To set up passwordless SSH access for Jenkins, log into ci.app as the user jenkins ssh jenkins@ci.app. Run and complete the prompts:

ssh-keygen

Copy the output of ~/.ssh/id_dsa.pub or ~/.ssh/id_rsa.pub (depending on what option you chose) to the file ~/.ssh/authorized_keys on int.app.

Now SSH to int.app and add the jenkins user adduser jenkins, this will be needed for when Jenkins is installing stuff onto int.app.

Now we need to log into ci.app as user jenkins to be able to accept the SSH Keys:

ssh jenkins@ci.app
ssh jenkins@int.app
exit
exit

Passwordless sudo rights

For enabling passwordless sudo rights to the jenkins user you will need to run: sudo visudo and append the file with the contents:

# Allow Jenkins to run certain commands as sudo without password
jenkins ALL = NOPASSWD: /bin/mkdir, /bin/rm, /bin/cp, /sbin/start, /sbin/stop

Adding shell script stages to Jenkins

Go to your hello world configuration panel (http://ci.app:8080/job/Hello World) and choose ‘add build step’ under the Build section.

1. Select ‘Execute shell’ and add this line: make deploy.
2. Save the settings.
3. Run a build.

If everything went according to plan you should be able to load http://int.app:3000 and see the text ‘Hello World’.

Verifying deployments

Instead of us having to check the app is running every time a build happens, we can get Jenkins to do that for us by checking a url returns 200 response code.

Lets add a new Freestyle job in Jenkins and call it “Hello World Monitor”. Leave everything blank but click the checkbox for Monitor Site and add the url http://int.app:3000/.

We can now return back to our Hello World job and check the Build other projects checkbox. Then add Hello World Monitor (it will auto-suggest). Save this configuration and run a build.

Finally

If you have made it this far, thanks for reading, hopefully everything went well?

If everything went well, you are in a good place to start adding more Continuous Integration elements such as unit tests, JSLinting and many other automated steps.

There are a number of things I didn’t go into much depth on such as the node.js app itself, there are plenty of articles out there about how to do this. A good resource worth checking out is http://howtonode.org.

Feel free to get in touch if you need any help but be warned I have a busy day job so I may take time to respond.

Image courtesy of Me

After a quick scour around the internet and checking every option in the Gmail preferences it seemed like there was no way of hiding the chat (a feature I don’t use) and invite friend panels on the left bar of the web interface to Google Mail.

My solution was to use a custom CSS rule for Gmail which consists of the following CSS:

.nH.pp.T0 .nH.s,
.nH.pp.T0 .nH.pY
{
    display:none;
}

Minor Issues

Now this is not ideal CSS but it seems Gmail have no id’s or specific hooks to target the panels, so I went with the best specificity I could get to not affect other plugins you may have in the left rail (I also have a calendar panel). The downside to this CSS rule is I am left with some container borders as you can see in this snapshot.

Alternative solution

The alternative approach I could have taken would have been to add some custom JavaScript but I figured out CSS would be the fastest and most reliable way of hiding the panels. I may explore this as the anal side of me is not happy with the stray borders.

What would be nice is if Google allowed us to disable these items in the first place!

Making custom CSS rules work in Chrome

It seems Chrome (my browsing browser of choice) has no built in custom/user CSS functionality so I installed the Personalized Web extension which is nice as you can specify a regex pattern to match a url so I know I am only targeting Gmail using the pattern ^http[s]*://mail.google.com

I have recently released domdash.com an application to help those that have a growing domain portfolio keep on top of their status. This was a personal itch that I needed to scratch with a domain acquisition almost every couple of months and a new project being released almost every quarter, things were getting a little hard to keep atop of.

One of my issues was that I have domains that sit on a mixture of hosts and domain name providers, so there was not one central tool to let me get information about my portfolio.

I paid particular attention to simplicity and usability when creating this application, constantly asking “does this feature add value”? With that mentality I dropped a couple of features which was hard discarding the time spent working on them but worth it for streamlining the end result. I intend on adding or refining features as I feel they fit the purpose of the application.

The most exciting aspect of this project was working with CodeIgniter 2, an ORM system and moving to a dedicated and affordable slice host Linode who are cheaper than the better known Slicehost and seemed to have a pretty good reputation. So far Linode have been great, they have comprehensive documentation on getting applications and services configured on your distribution and this was how easy it was to get up and running:

http://twitter.com/kulor/status/12903252621

just set up an Ubuntu LAMP stack from scratch on my lunch break on linode.com (including signup)

Anyway, I hope you enjoy this application, if you have feedback, feel free to click on the feedback link on domdash.com.

If you are looking for a free solution to slow down 60 frames per second (FPS) AKA 60p footage to get a slow motion effect on your Mac, you can download and install the freeware video de-interlacer JSE Deinterlacer. This will allow you to export your video at a slower framerate like 24FPS through stretching the film to be double length and use the additional frames shot to fill the time, kind of like stretching out  a coiled spring. Fast frame rates are now available from the Canon 7d and 550d cameras with signs that many more cameras follow suit by offering this feature, thus the reason to offer a no-cost option of how to handle the footage.

Step one: Import

Specify the film you wish to process, thats it, use the default values for everything else.

Step Two: Project settings

This is the main part we need to care about. Ensure you have Deinterlace selected from the options and Both fields and Double movie duration checked. Normal height and Adaptive will be selected by default, this is what we want.

Step 3: Output

We specify your desired output video location and set the settings for the movie that is exported. Select the export option and choose Quicktime Movie.

Step 4: Movie Settings

If you are explicitly looking for slowed speed sound select the sound checkbox and your audio channel will be exported, else just unselect this. Click settings to choose more specific output options.

Step 5: Video Settings

Choose MPEG-4 Video as your compression, choose a frame rate of 24 – 30 using keyframes every 24 frames. The other settings are just for how high you want quality to be.

Step 6: Finishing up

Now you can return back to the output screen and select Ok, this will start the conversion process off that could take some time depending on the size of your source video. Once you have converted your video, you can take it into your choice of video editing suites such as iMovie.

I have talked in the past about how to get a fully working time lapse system on a Mac using a webcam and some command line scripting. That was great but could do with some tweaks to get a more elegant solution which I attempted to reach here.

I wanted to get more use out of my digital SLR camera so I set my camera up on a tripod, connected it up to an intervalometer (my mac) and gained a bunch of files that needed to be processed and turned into a movie. I created the following script to turn my images into a movie.

This script will run on any unix like environment with a bash shell and uses ffmpeg so make sure you have it installed on your *nix machine.

1. Download images2movie
2. Change the permissions to allow the script to be executed chmod a+x images2movie
3. Copy file to /usr/bin/ sudo cp images2movie /usr/bin/
4. Run by images2movie <directory> 30

If it all goes well, you should see the following output:

Movie generated: timelapse.mov

Some more work could be done to resize images in advance to ensure that ffmpeg doesn’t choke on the image sizes but I will save that for another time.

I created a terminal theme with a retro look of soft amber tones. I find this really nice to work with and have been using for a couple of months now. Download or fork it on Github.

To use: download the amber.terminal file to your computer, open up Terminal.app, go to the preferences, settings tab, then under the list of themes, click on the settings button and click import, now choose the amber.terminal file.

Shell scripts are a simple way of completing (repetitive) tasks on the command line, much like many programming languages will be used. You will see ~ something — ignore the ‘~’ this is to signify that it is a command entered onto the command line itself.

Writing the code

~ vi helloworld.sh

#!/bin/bash
# My first script
MESSAGE="Hello World!"
echo $MESSAGE

Here we have just entered a really simple example of using variables and returning the value.

Setting the permissions

We need to set permission to allow the script to be executed, this is needed to be done on all unix environments for something to be run

~ chmod +x helloworld.sh

Running the script

~ sh helloworld.sh

Hello World!

Now you should have the response “Hello World!” from running this script.

Going more advanced

Lets explore some of the basic options we have to play with in the shell by example.

Loop test

~ vi looptest.sh

#!/bin/bash
# My first loop
FILETYPES="xml html css js"
for TYPE in $FILETYPES
    do
        echo 'Looking for' $TYPE 'files'
        find . -name '*'$TYPE
    done

Now set the correct permissions and run the script

~ chmod +x looptest.sh
~ sh looptest.sh

Looking for xml files
./somefile.xml
Looking for html files
./somefile.html
Looking for css files
./somefile.css
Looking for js files
./somefile.js

If conditional test

~ vi iftest.sh

#!/bin/bash
# If the file does not exist, make it
FILE='test'
if [ ! -f $FILE ] ; then
    mkdir -p $FILE
fi
ls -las $FILE

Now set the correct permissions and run the script

~ chmod +x iftest.sh
~ sh iftest.sh

Hopefully this is enough to help you on your way to making more use of the powers of Unix…

Further reading

Bash Manual
Useful bash one liner scripts

Not too long ago dealing with my personal and work emails was turning into a full time job. Having countless subscriptions to mailing lists, trying to handle critical projects, trying to keep on top of communication regarding relocating to another country, it was all getting rather stupid. This is my solution to the hassle of email communication — project:inbox zero.

• Set up an archive folder
• Move all emails in your inbox older than a month to the archive folder
• Decide on the remaining emails in your inbox which need actioning, keep these ones there, move the others to the archive folder
• Now you have reached an inbox state that is maintainable and you will continue to maintain
• Remove all automatic redirection filters to folders. Any email should enter your inbox and you decide if it gets deleted or filed in that folder
• For new incoming messages be quick to archive messages that may be useful later but not actionable or delete messages with no later action/reference
• Your goal now is to action off all the items in your inbox at all times to get your inbox message count to read the majestic ’0′ figure
• Close you email client when you need to get on with work, else it will act as a distraction. Read your emails in the morning, lunch and 30 minutes before you go home, if anyone has something urgent they are likely to phone or instant message you.

I recently attended my old university — the University of Greenwich to view the final year Multimedia & Internet Technology students’ presentations of  what they had been up to over the course of their studies. I had been in this same position 4 years ago, however standing observing it all came rushing back like it was yesterday; the sea of emotions experienced at that time, some of excitement to be finished, some of uncertainty and panic as to what I would go on and do with my life.

Anyway, the work was ok, the presentations ok, the engagement for the most part was ok but everyone seemed to be playing it safe. This was their pitch, the chance for them to be hired or to make the right impression to people that could make a difference. Some tips I would and did put forward to students looking to secure a job in the web industry.

1. Have a website
   1. You want to work in the digital world then you need a digital presence, surprising how many students don’t have one.
   2. Have your best work shown and presented well, definitely quality over quantity here.
   3. Make sure you have a working email address. If you have just set up an account on a new site, check it works.
   4. Use LinkedIn or a similar service to list your qualifications
2. Have a career direction
   1. If you like film, animation and web then consider a role that encompasses them all – Advertising, online gaming, etc.
   2. It helps when speaking to people to tell them the job title your hoping to secure, an employer would be able to think if they need that position filled in their company a lot easier than having to correlate your skill set and make a job title themselves.
3. Have a speciality/selling point
   1. If you find handling web services or making websites accessible a doddle and others seem to struggle you are likely to have a specialist skill which bodes well in securing a job at the CV and interview stage. Be prepared to demonstrate/discuss in detail these attributes of yours.
4. Be proud and believe in yourself
   1. If you do not believe in yourself then no one else will. You are well educated, created work to be proud of, so believe in yourself. Dress smart, hold your head high, engage in conversation as opposed to taking the back seat, its all about confidence you portray.
   2. It’s likely you will not pass interviews or CV picking processes. For this have a thick skin, being safe in the knowledge that you will be fine, and that you will find better opportunities out there.
5. Give me your business card
   1. It’s great if you have one but make sure you hand it to me, not to say force but offer at least.

Me on a time lapse from James Broad on Vimeo.

This video is what you can create (that or something similar) by following these steps on your Mac.

Step 1

Download & install a command line webcam capture utility, we will be using isightcapture
http://www.macupdate.com/info.php/id/18598
Open the DMG and extract the binary to /usr/bin/
sudo cp /Volumes/isightcapture1_1/isightcapture /usr/bin/

Step 2

Set up new directory to work in.
Lets use ~/captures/
cd ~
mkdir captures
mkdir captures/scripts
mkdir captures/series

Step 3

Make script to take photo and save it
cd ~/captures/scripts
vi captureme.sh

Insert the contents into the file:

CAPTURE="isightcapture -t jpg"
cd $HOME/captures
D1=`date +%y%m%d/%H`
D2=`date +%y%m%d.%H%M%S`

# If the date directory does not exist, create it
if [ ! -d $D1 ] ; then
mkdir -p $D1
fi

# Construct the filename and path and capture a pic
FN="$D1/$D2.jpg"
$CAPTURE $FN

# Make a symlinked image of the last photo taken
if [ -h 'last.jpg' ] ; then
rm last.jpg
fi
ln -s $FN last.jpg

Save and quit (:wq)

To be able to run this script we need to allow execution permission on this file.

chmod a+x captureme.sh

Step 4

Put your capture script on cron
crontab -e

Write the contents:

* * * * * ~/captures/scripts/captureme.sh

Step 5

Install a utility to convert a series of images to a movie. We will be using ffmpeg
cd /tmp/
svn checkout svn://svn.ffmpeg.org/ffmpeg/trunk ffmpeg
cd ffmpeg

We need to install ffmpeg to our machine, this process is going to basically compile the code. Hopefully nothing will go wrong for you here, if it does though, take a look on http://stephenjungels.com/jungels.net/articles/ffmpeg-howto.html

./configure --enable-shared --disable-mmx
sudo make
sudo make install

Step 5

Make script to grab photos and run them through ffmpeg
mkdir ~/captures/videos/
cd ~/captures/scripts
vi make_complete_sequence.sh

Write the contents to the file:

COUNTER=0;
rm ~/captures/series/*.jpg
for i in `find ~/captures -name '*.jpg'` ;
do
#Write the filename to be friendly with ffmpeg's odd filename input
FILENAME=`printf '%03d.jpg' $COUNTER`
cp $i ~/captures/series/$FILENAME
let COUNTER=COUNTER+1;
done
nice ffmpeg -r 20 -vcodec copy  -i ~/captures/series/%3d.jpg ~/captures/videos/timelapse_complete.mov

Again setting the permissions to be able to run the script and then run it to generate a movie based on the images taken so far.

chmod a+x make_complete_sequence.sh
./make_complete_sequence.sh

Step 6

View the finished result
Browse in finder to your home directory > captures/videos and watch the result.