Z-Scale Controller part VI: the server

Welcome back to our Z-Scale controller series! We left off in our previous article at an interesting milestone: the controller hardware is now finalised, and is only waiting for input from a higher level process for things to start getting interesting!

I designed the first version of the controller as a standard Arduino shield, compatible with an Arduino Uno or other similar boards. This Arduino therefore needs to be connected to a computer where the “train server” will be running. The overall architecture looks like this:

Node server

The diagram above gives you a quick view of what we are going to achieve. The technology pieces used on the server are be the following:

  • Node.js : my goal was to use as few programming languages across the whole project as possible. The application is spread between a browser, a backend server, and backend hardware. The use of Node.js enabled me to use Javascript both on the browser and the server, which is something already. No Javascript on Arduino yet, but that would be an interesting project…
  • MongoDB: MongoDB is extremely easy to use with javascript. It is a lightweight database system that was designed to handle “humongous” amounts of data – hence the name. But conversely, it also works fine on small systems where a SQL database would simply be overkill. And don’t forget, the whole exercise is also a learning experience, and I didn’t know Mongo when I started – nor Node either.

Role of the server

In this project, most of the high level code will be running in the browser. The role of the server is as limited as possible, but is still is key:

  • Proxy access to the controller: since our controller talks in JSON, the server will mostly provide a direct serial socket to the controller so that the browser talks to it directly.
  • Provide a RESTful adapter to the backend storage (MongoDB) for the web application.
  • And of course, serve all static assets: HTML code, javascript and graphics.

The database

As mentioned above the database is MongoDB. There is actually very little to say about it: the database daemon (mongod) needs to be running, and that’s it. You can install it after downloading it from www.mongodb.org, there are binaries for about every architecture. No need to create a database, this will be done on the fly. Easy, right ?

Structure of the server application – choice of frameworks

When you install node.js – again, very easy -, you will be presented with what is not much more than a server-side javascript engine. Unless you have a lot of time on your hands, and are willing to write everything from scratch, it is best to dig into the young and vibrant Node ecosystem and look for ready-made libraries and frameworks.

While doing this sort of search is not such a challenging task for more mature languages, Node is a very different story: there are literally dozens of libraries and frameworks with overlapping and competing functions on the scene, and picking one is not easy! Given the three server features described above, here are the additional Node.js extensions/components/plugins that I leveraged – and it took me a lot of reading first to learn, then to select the right ones:

Proxy serial port to web application socket.io is exactly what we need to do this. It provides a browser-agnostic API to WebSocket-style communications between an application running in a browser and a server. I also used the “serialport” library on node to provide access to serial ports.
RESTful interface to database: I used two separate libraries to solve this one: Mongoose which facilitates talking to MongoDB from a Node application, as well as the ExpressJS web application framework.
Serve static content: again, ExpressJS is what I used here.

A quick word on Express: the main role of this framework, is to provide all the basic features required to build web applications. It is by itself fairly barebones, and it can be used in a number of ways – web applications are usually split between code executing on the server and code executing on the browser, with no cast-in-stone best practice on what this split should look like. I took the decision to have as much code as possible running on the browser, and keep the server very simple, so using ExpressJS was logical – there are more sophisticated web app frameworks around, sometimes built on top of Express, but they would have been overkill in this case.

Basic structure of the server

At the base of every Node application, you will find a file called “package.json”. This file, written in JSON format, describes a few basics and in particular lists the external requirements for the app. In our case, and given the dependencies described above, package.json looks like this:

{
    "name": "train-controller",
    "description": "A Z-Scale model train controller",
    "version": "0.0.1",
    "private": true,
    "dependencies": {
        "express": "3.x",
        "mongoose" : "3.x",
        "socket.io": "0.9.10",
        "serialport": "1.0.x"
    },
    "engines": {
        "node": "0.8.4",
        "npm": "1.1.49"
    }
}

With this file in the root directory of the server app, we just need to issue a “npm install” command to automatically download and install all the node modules (and their dependencies) in a few seconds. Very cool.

Organising the code

The next step is to organise the code for legibility. Again, neither Node nor Express require a particular code structure, but a best practice is to split the code in logical blocks. In our case, the main file (“server.js”) is in charge of defining the URL structure of the app – the ‘router’ in technical terms. For each “route” that is defined, we will create a corresponding file in a “routes” subdirectory.

All HTML and static Javascript content will be stored under a “public” subdirectory. I will describe how that directory is organised when I move on to the web app part.

server.js

The role of server.js, as described above, is primarily to define the URL structure of our app. For each type of data structure that is defined in the database, the corresponding REST interface is defined in a few steps as shown below for the “locomotive” interface:

...
/**
 * Interface for managing the locomotives
 */
app.get('/locos', locos.findAll);
app.get('/locos/:id', locos.findById);
app.post('/locos', locos.addLoco);
app.post('/locos/:id/picture', locos.uploadPic);
app.put('/locos/:id', locos.updateLoco);
app.delete('/locos/:id', locos.deleteLoco);
...

This is also where we define the location of static content and how it is served:

// Our static resources are in 'public'
app.use(express.static(__dirname + '/public'));

server.js is also where we define all high-level settings of the app itself. Express makes this very simple:

var app = express(),
    server = require('http').createServer(app),
    io = require('socket.io').listen(server);

app.configure(function () {
    app.use(express.logger('dev'));     /* 'default', 'short', 'tiny', 'dev' */
    app.use(express.favicon());
    app.use(express.bodyParser({ keepExtensions: true, uploadDir: __dirname + "/public/pics/tmp" }));
});

server.listen(8000);

One thing you can notice in the code above, is the presence of socket.io. This leads us to the second role of server.js: make socket.io available to the web app and provide proxying to serial ports.

I won’t describe the whole code here – the source is full of comments – but I will emphasise a few important points: At the core, we simply listen for clients that connect on the socket.io channel. When a client connects, we implement a simple protocol that lets this client:

– Connect to a serial port (using the name of the serial port as the argument)
– Disconnect from it
– Query the state of the port
– Write data to the open port
– Receive data from the port

Both the socket.io and serialport libraries use the standard Javascript event model, so hooking up serial events to socket events and the other way around is very straightforward. The server manages the following socket events:

socket.on('disconnect', ...
socket.on('openport', ...
socket.on('closeport', ...
socket.on('portstatus', ...
socket.on('controllerCommand', ...

And the serial port instance created upon the “openport” event above implements the following:

myPort.on("open", ...
myPort.on('data', ...
myPort.on("close", ...

Reading the source code, you will notice a few additional bit and pieces that are here to make the code more resilient. In particular, when we open a serial port, chances are the controller is already in the middle of sending something on its serial port, so the first line will probably be impossible to parse. The code waits until it is synchronised before sending a “port open” event to the web app.

db.js

db.js is where the MongoDB schema is defined: we define data models there, and those data models will be used in the routes for communicating with the Mongo database. Each model is defined in a similar fashion. For instance, the loco schema looks like this:

var LocoSchema = new Schema({
        name: String,
        year: String,           // Year the model was produced/bought
        reference: String,      // Manufacturer reference
        description: String,    // Up to the user
        picture: String,        // Filename in public/pics/locos
        documentation: String,  // PDF doc, filename in public/pics/locodocs/
        runtime: Number,        // Runtime of the loco in seconds
});

// Compile the schema by issuing the below:
mongoose.model('Loco', LocoSchema );

One downside of my approach, is that this schema data has to be kept in synch with the models of the web application that I will describe in the next articles. There might be other ways of handling this, but since we do not have many schemas in our application, this is not such a major deal.

Routes

Last, in the “routes” directory, we define the implementation of the REST interface to all those database objects, as described in the server.js configuration. Taking the locomotives used above as an example again, this looks as follows:

exports.findById = function(req, res) {
...
};

exports.findAll = function(req, res) {
...
};

exports.addLoco = function(req, res) {
...
};

exports.updateLoco = function(req, res) {
...
};

exports.deleteLoco = function(req, res) {
..
};

exports.uploadPic = function(req,res) {
...
};

Mongoose makes it very easy to interact with the Mongo database, by implementing database calls directly at the model level. For instance, for the REST URL defined as “app.get(‘/locos/:id’, locos.findById)”, the corresponding function is barely a few lines long:

exports.findById = function(req, res) {
    var id = req.params.id;
    console.log('Retrieving loco: ' + id);
    Loco.findById(id, function(err,item) {
        res.send(item);
    });
};

Conclusion

This is it for a description of the server: as you saw in the article, the idea was to keep it as simple and lightweight as possible, while still leveraging the right frameworks to make our life easy and minimize the amount of code to write.

Further enhancements to the server will include adding an API to give a list of serial ports to choose from to the web application, but that’s about it at this point…

Our next article in the series will move on to the web application itself.

Leave a Reply

Your email address will not be published. Required fields are marked *