A quick delve into Docker

The problem

When you run an application such as tomcat, you need to make sure you have the correct version of Java installed and configured, and then download the version of tomcat that is compatible with that version of Java. If you upgrade the version of Java, you’ve then got to setup a new JDK and potentially setup a new version of tomcat.

This is quite annoying as you’re going to have multiple versions of Java and tomcat installed on your machine, and at some point you’re going to get confused and have the wrong version running, or an environment variable not set correctly.

When you have an entire dev team doing the same thing, you’ll end up with people on different versions, different environment configurations, and ultimately you’ll get those “but it works on my environment” bugs at some point.

The solution

Docker allows you to run applications in a container. It’s a bit like a VM, but without the OS. That doesn’t make much sense, and it didn’t to me either to start with. When you run a VM you’re running a full blown OS, and the hypervisor layer is bridging the kernel of the guest and the host.

If you’re just running apache on that VM it’s a bit of an overkill.

You can think of a container as being a stripped down linux OS. Theres barely anything there, just the bare bones; a filesystem and networking. Theres no gui, no pre-installed packages like apache, theres literally just enough to start the container. This makes them very lightweight and fast. It’s then up to you to install your own applications in those containers.

Why is this good?

Don’t need a particular application anymore? Fine, delete the container. No need to hunt around your system manually uninstalling packages that you have scattered around.

Need to ship it to another machine? Publish the container and let others download it as an image ready to run.

When you start a new project, get your DevOps guy (or a dev) to build some containers for all of the dependencies of your project, you’ll probably need things like tomcat & mysql which are easy because theres already official docker containers for those, but you may also need to build your own custom containers to stub your integration points, or to install integration point software in a stub mode. Then, when your project kicks off and the devs are ready to get started, all they need to do is pull the images and run them, and they’ve got a full stack dev environment ready to use. Marvellous.

As I’m new to Docker, perhaps I’m not the best at explaining it. I’d highly recommend you watch this:

Lets have a look at a few containers, tomcat and mysql.


You could build your own tomcat container, but theres an official image that you can use, start it up using:

docker run -it --rm -p 8888:8080 tomcat:8.0

The docker run command is going to start a container from an image, as you likely won’t have that image, it will realise this and then pull it. You could pull it separately using a docker pull command, but the run figures this out for us.

The it flag is for interactive mode, so you can see the output, in this case, the output of catalina being executed. An alternative would be to use the d flag which runs it as a daemon (background task).

The rm flag is for automatically removing the container if it exits, you don’t strictly need this, but the tomcat official image page suggests you include it.

The -p flag tells docker to forward port 8080 on the container to 8888 on the host, so we can access tomcat outside of the container, it’d be a bit pointless without this. An alternative flag would be -P which forwards all ports.

The tomcat:8.0 is the image name along with its tag.

Run the docker run command and you should see the output of the catalina start process. You can open another tab and run docker ps to see its process state.

Now lets try and access the tomcat manager page, in order to do so you need to get the IP of the boot2docker instance. Remember, that boot2docker is the docker host, not the laptop, so you need to access containers via boot2dockers vm. It took me a little while to realise this, I was running a docker inspect on the container, finding the network settings/IP and trying to access that, not realising that its actually the boot2docker vm you need to access.

You can easily do this by obtaining the ip using  boot2docker ip. Then you can access:


Thats it, you should be on the tomcat page now. I’ll leave it up to you to make use of it, perhaps extend the tomcat image and deploy your own applications?


As with tomcat, there is an officially supported MySql container, download it using “docker pull” like so.

docker pull mysql:5.7
Pulling repository mysql
463d9ebad128: Download complete

Run up that mysql image using

docker run -d -p 3306:3306 mysql

The -d runs the container as a daemon (background task) and returns you the container id.

Next have a look at docker ps to confirm its running

docker ps
2eac3ea6b64e mysql:latest "/usr/bin/mysqld_saf 5 seconds ago Up 2 seconds;3306/tcp pensive_heisenberg

Now lets try to connect to it

docker run -it --link myfirstmysql:mysql --rm mysql sh -c 'exec mysql -h"$MYSQL_PORT_3306_TCP_ADDR" -P"$MYSQL_PORT_3306_TCP_PORT" -uroot -p"$MYSQL_ENV_MYSQL_ROOT_PASSWORD"'

This is spawning another container which will run the mysql client. At first, I was trying to use the mysql client on my local machine to connect to it, but then I realised that I was missing the point of docker, why install mysql locally, just to use the client, when really I could be doing that via docker?

see https://registry.hub.docker.com/_/mysql/

Round up

I’m yet to use this in anger, but I can already think of applications on previous projects that would have benefitted from this. I’ve been using virtualisation with vagrant and chef for a while, so I’m interested to see how different things will work out by using docker.

Backing up wordpress automatically

I’ve had some difficulties getting the BackWPup plugin to work, it seems that you can’t backup everything in one job as the script takes too long to run and the server will terminate it, causing a failed job.

The 2 errors I was seeing are

  1. WARNING: Job restart due to inactivity for more than 5 minutes.
  2. ERROR: Uploaded file size and local file size don’t match.

Which led me to this post, which recommends a better way to structure your backup jobs, basically just split them out into content, plugins, install etc so they finish within the timeout threshold.

Pig Latin parsing CSV files with quoted commas

In the not too distant past, I was working on a BigData engagement using Apache Pig. I took CSV parsing for granted and expected it to just work, however if you have quoted strings with commas, it won’t behave as you’d expect.


1,"This is a sample sentence, same sentence, just happens to include a few commas" 

When you use:

load 'input/oneLiner.txt' using PigStorage(',') 

It delimits based on the comma, regardless of it being in a quoted string, so you end up with 4 fields;

This is a sample sentence
same sentence
just happens to include a few commas

The solution to this is to use a custom loader, such as org.apache.pig.piggybank.storage.CSVExcelStorage().

To get started with this, I had to clone the piggybank repository (collection of user defined functions, why this didn’t make it to the base release I’m not entirely sure) and build from source, unfortunately I didn’t keep any notes for this, but its relatively straightforward, see the Apache Pig wiki page here

Getting your head around the Couchbase SyncGateway

I like Couchbase. One of the things that really appeals to me is the sync gateway. As a mobile developer I often find that the apps I’m developing are just interfaces into some backend service. Somewhere out there in the cloud I’ll have a web application that sits on top of some database (nodejs/mongoDB is a combo I’ve been using recently). Then there comes the mobile app, which will be consuming these services, which would be fine if 4G/wifi was everywhere (I can’t even get a cellular signal at my place, let alone dream of 4G).

We’re into the realms of apps working offline, you then have the pain of syncing data and dealing with conflicts. You can make your life easier by using a SyncAdapter on Android, or perhaps a framework like Restkit if you’re developing on iOS, heck, you can even implement the syncing yourself (don’t do that, that road leads to madness..speaking from experience)…OR…you can just use Couchbase and the SyncGateway.

In short, the SyncGateway is an application that sits between your Couchbase server, and your Couchbase Lite enabled mobile apps. This means you can access your data on your local CBLite database, and not have to worry (too much) about syncing this to the Couchbase server.

Getting setup

I have to admit, the documentation is a little confusing when it comes to explaining how the components hang together, but after attending Couchbase Live in London a month or so back I was able to track down those who are in the know, and put the missing piece into my puzzle of confusion; bucket syncing.

For the purpose of explaining how this works, I’ll use my “Coin Collector” android app as the example. The app needs to get its data on coins from a couchbase server. It should be able to work offline and sync periodically. I’m using bucket syncing so I can have a web page to administer coins such as adding new coins to altering market values.
The documentation is really missing a diagram like the following


Let me cover the 4 points in blue numbers:

  1. Regardless of which mobile platform you’re using, it’ll be connecting to the sync gateway via the REST apis, this is where “json over the wire” comes into play.
  2. As the mobile apps use their own bucket, you need to configure the gateway to tell it where to put documents. If you check my config below; then this is done by the “aussie-coins-syncgw” configuration element, you can see that the bucket is set to “aussie-coins-bucket-sync-db” on the localhost couchbase server (sync and db are running on my local vm)
  3. This is where the magic happens. Bucket shadowing in the later releases of the Sync Gateway allow it to sync changes between your “mobile” bucket, and your “backend” bucket. You can see this configured by the “shadow” element in my config.json
  4. Your backend server apps can just connect to the “aussie-coins-bucket” and be totally oblivious to what is happening in the mobile side of your architecture.
    "interface": ":4984",
    "adminInterface": ":4985",
    "log": ["CRUD", "CRUD+", "HTTP", "HTTP+", "Access", "Cache", "Shadow", "Shadow+", "Changes", "Changes+"],
    "databases": {
        "aussie-coins-syncgw": {
            "server": "http://localhost:8091",
            "bucket": "aussie-coins-bucket-sync-db",
            "sync": `function(doc) {channel(doc.channels);}`,
            "users": {
                "GUEST": {
                    "disabled": false,
                    "admin_channels": ["*"]
            "shadow": {
                 "server": "http://localhost:8091",
                 "bucket": "aussie-coins-bucket"

Some other points to notice in the configuration:

  • The interface port is the port the apps will connect on, the adminInterface is for administering the sync gateway, such as dynamically adding new databases, or altering channels.
  • Logs, I’ve chosen to log everything, you can restrict these if you need, check the Couchbase documentation for further info.
  • I’ve enabled the guest user access on all channels for the purpose of evaluating this, ideally we’d need to restrict the channels that users can use to stop any potential abuse.

Testing it out

As I mentioned above, since the mobile apps will be connecting to the Sync Gateway via a REST api, we can take the mobile app out of the picture and test using a rest client (I’m using Postman for Google Chrome). Lets cover 2 scenarios.

Server Producing

This scenario involves a new document being created on the server, and it being synced to the mobile bucket and available to view on the mobile apps.

Firstly, let me show you what I have in the “aussie-coins-bucket”.


Next, lets create a new document with an ID of 5, for the Ten Cent coin. We should then see it listed in our “aussie-coins-bucket” like so:


Now lets have a look at the log output from the Sync Gateway.

22:49:33.826838 Shadow+: Pulling "5", CAS=1e2dd7153a ... have UpstreamRev="", UpstreamCAS=0
22:49:33.826894 Shadow: Pulling "5", CAS=1e2dd7153a --> rev "1-1d7a1a352c0abb293fdd16883ef6985b"
22:49:33.826909 CRUD+: Invoking sync on doc "5" rev 1-1d7a1a352c0abb293fdd16883ef6985b
22:49:33.903707 Cache: SAVING #8
22:49:33.903984 CRUD: Stored doc "5" / "1-1d7a1a352c0abb293fdd16883ef6985b"
22:49:34.768280 Cache: Received #8 after 864ms ("5" / "1-1d7a1a352c0abb293fdd16883ef6985b")
22:49:34.768305 Cache:     #8 ==> channel "*"
22:49:34.768322 Changes+: Notifying that "aussie-coins-bucket-sync-db" changed (keys="{*}") count=3
22:49:59.849578 Shadow+: Pulling "5", CAS=2423d4b93a ... have UpstreamRev="1-1d7a1a352c0abb293fdd16883ef6985b", UpstreamCAS=c21019dd68
22:49:59.849623 Shadow: Pulling "5", CAS=2423d4b93a --> rev "2-971b4b3009127da5ed2a4770cb45cfe7"
22:49:59.849637 CRUD+: Invoking sync on doc "5" rev 2-971b4b3009127da5ed2a4770cb45cfe7
22:49:59.849749 CRUD+: Saving old revision "5" / "1-1d7a1a352c0abb293fdd16883ef6985b" (68 bytes)
22:49:59.849891 CRUD+: Backed up obsolete rev "5"/"1-1d7a1a352c0abb293fdd16883ef6985b"
22:49:59.850068 Cache: SAVING #9
22:49:59.850207 CRUD: Stored doc "5" / "2-971b4b3009127da5ed2a4770cb45cfe7"
22:50:00.790818 Cache: Received #9 after 940ms ("5" / "2-971b4b3009127da5ed2a4770cb45cfe7")
22:50:00.790838 Cache:     #9 ==> channel "*"
22:50:00.790868 Changes+: Notifying that "aussie-coins-bucket-sync-db" changed (keys="{*}") count=4

As we can see, the Sync Gateway has detected that there is a new document and that it needs to shadow it across, which is does successfully.

On the couchbase server, we can view that document in the mobile bucket, “aussie-coins-sync-db” like so:


Finally, just to prove the mobile clients can see that document via the API, do a GET on http://localhost:4984/aussie-coins-syncgw/5 and you’ll see the following:

    "_id": "5",
    "_rev": "2-971b4b3009127da5ed2a4770cb45cfe7",
    "coin": "Ten Cent"

Mobile Producer

Now we’ll try the opposite, producing documents from the mobile clients and seeing them synced across to the Couchbase server. From a REST client, do a PUT to http://localhost:4984/aussie-coins-syncgw/6 with a json body of:

  "coin":"Twenty Cent"

You should see a response of

    "id": "6",
    "ok": true,
    "rev": "1-e9c16d3887a3958314adff1e3cbd6097"

What we’ve done is to create a document with the ID of 6, for “Twenty Cent”.

Lets have a look at the Sync Gateway logs:

23:19:56.860618 HTTP:  #003: PUT /aussie-coins-syncgw/6
23:19:56.971056 CRUD+: Invoking sync on doc "6" rev 1-e9c16d3887a3958314adff1e3cbd6097
23:19:57.023839 Cache: SAVING #10
23:19:57.024110 CRUD: Stored doc "6" / "1-e9c16d3887a3958314adff1e3cbd6097"
23:19:57.024161 HTTP+: #003:     --> 201   (0.0 ms)
23:19:57.616316 Cache: Received #10 after 592ms ("6" / "1-e9c16d3887a3958314adff1e3cbd6097")
23:19:57.616340 Cache:     #10 ==> channel "*"
23:19:57.616353 Shadow: Pushing "6", rev "1-e9c16d3887a3958314adff1e3cbd6097"
23:19:57.616367 Changes+: Notifying that "aussie-coins-bucket-sync-db" changed (keys="{*}") count=6
23:19:57.852304 Shadow+: Pulling "6", CAS=1c6f07c3ce2 ... have UpstreamRev="", UpstreamCAS=0
23:19:57.852327 Shadow+: Not pulling "6", CAS=1c6f07c3ce2 (echo of rev "1-e9c16d3887a3958314adff1e3cbd6097")
23:19:57.852337 CRUD+: Invoking sync on doc "6" rev 1-e9c16d3887a3958314adff1e3cbd6097
23:19:57.865669 CRUD+: updateDoc("6"): Rev "1-e9c16d3887a3958314adff1e3cbd6097" leaves "1-e9c16d3887a3958314adff1e3cbd6097" still current
23:19:57.865751 Cache: SAVING #11
23:19:57.866050 CRUD: Stored doc "6" / "1-e9c16d3887a3958314adff1e3cbd6097"
23:19:58.617446 Cache: Received #11 after 751ms ("6" / "1-e9c16d3887a3958314adff1e3cbd6097")
23:19:58.617463 Cache:     #11 ==> channel "*"
23:19:58.617482 Changes+: Notifying that "aussie-coins-bucket-sync-db" changed (keys="{*}") count=7

We can then see the document in the “aussie-coins-bucket-sync-db”:


…and then in the “aussie-coins-bucket”:

The Sync Gateway is a useful application, and really does make the Couchbase offering even more appealing. Once you can get your head around the bucket shadowing (which you should if you’ve made it this far) then it can be easy to work with.

Comment or find me on Twitter (@jameselsey1986) if you have any questions!

Why the duplication?

Having 2 buckets for the same data had me raise an eyebrow initially, but after asking on Google Groups, it does make sense. You can’t expect the backend app servers to maintain sync meta data on new documents it creates. Perhaps Couchbase will alter this in the future.


Service first using JAX-WS

There are two ways for developing services using JAX-WS, service first, and contract first. Service first means you would typically write the implementation first and generate the WSDL afterwards, whereas contract first you would define the WSDL first, then write the implementation afterwards. There are pros and cons for each approach, but I won’t dwell on those now.

There are 2 parts to a JAX-WS service, the Service Endpoint Interface (SEI) and the Service Implementation Bean (SIB). The SEI is an interface where you abstractly declare the methods (or operations) that your service will provide, along with the inputs and outputs. The SIB is a concrete implementation of the SEI, where you actually implement the code for the SEI. Let me show you a basic example


import javax.jws.WebMethod;
import javax.jws.WebService;
import javax.jws.soap.SOAPBinding;

@SOAPBinding(style = SOAPBinding.Style.RPC)
public interface TeamService {

    public String getTeam();


import javax.jws.WebService;

@WebService(endpointInterface = "com.jameselsey.webservices.basicimplementationfirst.service.TeamService")
public class TeamServiceImpl implements TeamService{

    public String getTeam(){
        return "Geelong Cats";

That is pretty much it, now we just need to deploy the service. Lets avoid using a servlet container for now, as it complicates this example somewhat, we can run the service by implementing the following and taking advantage of the publish method on Endpoint.

import com.jameselsey.webservices.service.sib.TeamServiceImpl;
import javax.xml.ws.Endpoint;

public class Runner {

    public static void main(String[] args) {
        String address = "http://localhost:9876/footy";
        Endpoint.publish(address, new TeamServiceImpl());

        System.out.println("Server up and running on " + address);

If you then open http://localhost:9876/footy on your browser you’ll see a page listing the services and their WSDLs. The generated WSDL should look something like this.

<?xml version="1.0" encoding="UTF-8"?>
<definitions xmlns="http://schemas.xmlsoap.org/wsdl/" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:tns="http://service.basicimplementationfirst.webservices.jameselsey.com/" xmlns:wsam="http://www.w3.org/2007/05/addressing/metadata" xmlns:wsp="http://www.w3.org/ns/ws-policy" xmlns:wsp1_2="http://schemas.xmlsoap.org/ws/2004/09/policy" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" xmlns:xsd="http://www.w3.org/2001/XMLSchema" targetNamespace="http://service.basicimplementationfirst.webservices.jameselsey.com/" name="TeamServiceImplService">
   <types />
   <message name="getTeam" />
   <message name="getTeamResponse">
      <part name="return" type="xsd:string" />
   <portType name="TeamService">
      <operation name="getTeam">
         <input wsam:Action="http://service.basicimplementationfirst.webservices.jameselsey.com/TeamService/getTeamRequest" message="tns:getTeam" />
         <output wsam:Action="http://service.basicimplementationfirst.webservices.jameselsey.com/TeamService/getTeamResponse" message="tns:getTeamResponse" />
   <binding name="TeamServiceImplPortBinding" type="tns:TeamService">
      <soap:binding transport="http://schemas.xmlsoap.org/soap/http" style="rpc" />
      <operation name="getTeam">
         <soap:operation soapAction="" />
            <soap:body use="literal" namespace="http://service.basicimplementationfirst.webservices.jameselsey.com/" />
            <soap:body use="literal" namespace="http://service.basicimplementationfirst.webservices.jameselsey.com/" />
   <service name="TeamServiceImplService">
      <port name="TeamServiceImplPort" binding="tns:TeamServiceImplPortBinding">
         <soap:address location="http://localhost:9876/footy" />

The important parts in the WSDL that I should mention are:

The port section describes the operations that the service exposes, you can think of this a little bit like a Java interface, where it declares the inputs and outputs abstractly, but mentions nothing about the actual implementation. You will notice that the operations match onto the methods marked as @WebMethod. Notice that the name on the port matches the interface and not the implementation.
The binding section is where you mention implementation details for an implementation of the interface (port), such as transport, style, and usages. Transport is typically http (although smtp can be used), style matches onto the SOAPBinding annotation that we set. The default style is document however I’ve overridden this to be rpc as it makes the examples easier to follow as its less complex.
The service section maps interfaces onto implementations, or in WSDL terms, it lists the ports and their bindings. In the above example it mentions that there is an implementation of TeamServiceImplPort as described in TeamServiceImplPortBinding, available on http://localhost:9876/footy.

You can actually see this working, by creating a project from the WSDL using SOAPUI, it will generate the following request for you

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:sei="http://sei.basicimplementationfirst.webservices.jameselsey.com/" xmlns:sib="http://sib.service.webservices.jameselsey.com/">

As you can see the body contains a tag which is requesting the getTeam port, ultimately bound onto the getTeam Java interface method, implemented to return a pre defined string. If you execute the request, you’ll get the following response

<S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/">
      <ns2:getTeamResponse xmlns:ns2="http://sei.basicimplementationfirst.webservices.jameselsey.com/">
         <return>Geelong Cats</return>

This was a quick brain dump of a simple JAX-WS service (I’m currently studying for the Oracle web services certification, trying to write my notes up). I’ve covered how to implement a SEI/SIB, how to publish a service, covered in brief the elements of the WSDL we’re interested in, and showed how the service can be invoked. Please check back soon as I’ll continue this example and post more of my study notes.

Grails based survey system, the android app

Some time back I wrote an article describing the roosearch system I developed using grails. This is the second part, the android client, please checkout the previous article otherwise this might not make much sense!

After completing the grails component, I had a RESTful API available to me, and I just needed to build an app that could consume those services.

Customer lookup and QR codes

The app needs to be simple and quick to use, one of the things I remember from a UX discussion at DroidCon UK is “Don’t annoy your users, they control your app ratings and your income!”. In order to lookup the surveys quickly, I’ve added the ability to scan QR codes. Actually I didn’t have to do a great deal as there is already an app called ZXing by Google that scans QR codes, so I just needed to make Roosearch delegate to ZXing and handle the result.

Of course, we don’t want to exclude users that don’t have ZXing, or even a camera on their device, so I’ve also provided a text field where they can enter the customer Id manually if required.

When the user clicks on the “scan barcode” button, I first check if ZXing is installed using the following

    public void scanBarCode(View v) {
        final boolean scanAvailable = isIntentAvailable(this,
        if (!scanAvailable){
            Toast.makeText(this, "You need to install the ZXing barcode app to use this feature", Toast.LENGTH_SHORT).show();

        Intent intent = new Intent("com.google.zxing.client.android.SCAN");
        intent.putExtra("SCAN_MODE", "QR_CODE_MODE");
        startActivityForResult(intent, 0);

If the user does have ZXing installed on their device, and choose to use it, we can get the result back from the bar code scan using:

public void onActivityResult(int requestCode, int resultCode, Intent intent) {
        if (requestCode == 0) {
            if (resultCode == RESULT_OK) {
                String contents = intent.getStringExtra("SCAN_RESULT");
            } else if (resultCode == RESULT_CANCELED) {
                // Handle cancel

    private void performRooLookup(String rooId) {
        if (StringUtils.isBlank(rooId)) {
            Toast.makeText(this, "Please enter a valid customer id", Toast.LENGTH_SHORT).show();

        Integer customerId;
        try {
            customerId = Integer.parseInt(rooId);
        } catch (NumberFormatException e) {
            Toast.makeText(this, "Customer id needs to be numeric", Toast.LENGTH_SHORT).show();
        new FindRooTask(this, new FindRooTaskCompleteListener()).execute(customerId);

I then have the following buried in a service call, invoked by an AsyncTask, which handles finding Customer details:

    public Customer getCustomerDetails(int customerId) {

        try {
            final String url = "http://roosearchdev.jameselsey.cloudbees.net/api/customer/{query}";

            HttpHeaders requestHeaders = new HttpHeaders();

            // Create a new RestTemplate instance
            RestTemplate restTemplate = new RestTemplate();
            restTemplate.getMessageConverters().add(new MappingJacksonHttpMessageConverter());

            // Perform the HTTP GET request
            ResponseEntity<Customer> response = restTemplate.exchange(url, HttpMethod.GET,
                    new HttpEntity<Object>(requestHeaders), Customer.class, customerId);

            return response.getBody();
        } catch (Exception e) {
            System.out.println("Oops, got an error retrieving from server.. + e");
         return null;

A Customer looks like this:

public class Customer implements Parcelable {

    private String companyName;
    private String twitter;
    private String facebook;
    private List<SurveySummary> surveys = new ArrayList<SurveySummary>();
    //Accessors omitted

The SurveySummary just has a title and Id. The reason for just returning summaries is because a customer may have many surveys, and there is no need to obtain them all, we just obtain the title to display to the user, if selected, we’ll retrieve the survey by its id.

To recap, here are 2 screenshots that show the above; the landing screen, and then the customer display screen

Landing screen for Roosearch, where the user can enter a customer Id or scan a QR code

Landing screen for Roosearch, where the user can enter a customer Id or scan a QR code

Customer screen, display social media links, name, photo, and list of surveys that the customer has

Customer screen, display social media links, name, photo, and list of surveys that the customer has


The survey engine

This is where the magic happens. I have a single activity and single view that handles presenting the survey to the user. As the surveys can change number of questions, and number of responses, I needed a way of dynamically traversing the survey object and allowing user to move between the questions whilst retaining state of what they have selected so far.

I’ve created the following method that will redraw the layout for a given question id:

    public void drawQuestionOnScreen(int id) {
        TextView question = (TextView) findViewById(R.id.question);
        question.setText(s.getQuestion(id - 1).getText());   // subtract 1 as lists are indexed from 0

        LinearLayout linLay = (LinearLayout) findViewById(R.id.answers);
        RadioGroup rg = new RadioGroup(this);
        for (int aIndex = 0; aIndex < s.getQuestion(id - 1).getResponses().size(); aIndex++) {
            Answer a = s.getQuestion(id - 1).getAvailableOption(aIndex);
            RadioButton button = new RadioButton(this);

        TextView status = (TextView) findViewById(R.id.status);
        status.setText(format("%d of %d", id, s.getQuestionCount()));

As you can see, it will retrieve the question by Id, then iterate over the responses and generate RadioButtons. Moving to the next question is reasonably easy, firstly I work out if an option has been selected, and prevent moving on if not. After that, I mark the selected response in the survey object, and then work out if there is another question in the sequence to display, if not we can progress to the finish.

One of the questions in the given survey

One of the questions in the given survey

    public void next(View v) {
        RadioGroup rg = (RadioGroup) findViewById(1);

        int selectedRadioId = rg.getCheckedRadioButtonId();
        if(selectedRadioId == -1){
            Toast.makeText(this, "Please select a response", Toast.LENGTH_SHORT).show();

        s.getQuestion(questionIndex - 1).getResponses().get(selectedRadioId).setSelected(true);
        // work out if there is another question, then move to it
        if (s.getQuestionCount() > 1 && questionIndex < s.getQuestionCount()) {
        } else {
            // if there are no other questions, show dialog saying submit or not
            Toast.makeText(this, "Reached the end of the survey", Toast.LENGTH_SHORT).show();
            // HERE we should process the entire survey, crunch data and post off (maybe async)

            Intent i = new Intent(this, SurveyComplete.class);
            i.putExtra("com.roosearch.domain.Survey", s);

A similar approach is needed for moving back to previous questions, determine if there is a previous question to move to then redraw the screen, like so:

    public void previous(View v) {
        // work out if there is a previous question, and if so move to it
        if (s.getQuestionCount() > 1 && questionIndex > 1) {
        } else {
            //if there are no other questions, move back to home screen, finish() this and scrap any progress

Once the user completes all questions, the SurveyComplete activity is invoked.

Completing a survey

When the user has completed all questions, the survey object is passed into the SurveyComplete activity, which handles sending the responses back to the grails web application.

    protected void onResume()
        TextView tv = (TextView) findViewById(R.id.completeMessage);
        tv.setText("Thank you for taking the time to complete the survey");

        Survey s = getIntent().getExtras().getParcelable("com.roosearch.domain.Survey");

        if (s != null)
            StringBuffer sb = new StringBuffer();
            sb.append("\n" + s.getTitle() + "\n");
            for (Question q : s.getQuestions())
                sb.append("\nQ: " + q.getText());
                sb.append("\nA: " + q.getSelectedAnswer() + "\n");
            tv.append("\n\n" + sb.toString());

        new SurveyUploadTask(this, new SurveyUploadTaskCompleteListener()).execute(s);

    public class SurveyUploadTaskCompleteListener implements AsyncTaskCompleteListener<Void> {
        public void onTaskComplete(Void voidz) {
            Toast.makeText(SurveyComplete.this, "Survey uploaded", Toast.LENGTH_SHORT).show();

The activity uses an AsyncTask to post the data back to the grails API controller, and displays a toast when successful.

Survey completed, results uploaded, and summary presented to user

Survey completed, results uploaded, and summary presented to user


Wrapping it up

Overall quite a simple app, I spent probably around 2 or 3 weekends putting together, most of that time was spent getting to grips with some automated testing for android. The code is admittedly a little rough around the edges, but I was aiming for an MVP (most viable product) to get working, feel free to contribute or suggest improvements!

I chose to use maven, but would use gradle if I were to pick this up again. Be sure to check out the code on github and try running it against Roosearch web, it does work!

Click here for the source code on Github

Changing the hostname on your raspberry pi

By default, the hostname on a raspberry pi installation will be “raspberrypi”, which is great if you just have the one pi.

Two raspberry pis, one with a BrickPi attached for controlling lego mindstorms

Two raspberry pis, one with a BrickPi attached for controlling lego mindstorms

If you’ve got more than one, then you’re going to get hostname conflicts when you attach both to your network. Fortunately its easy to correct this.

Plug the pi that you want to change hostname onto the network (leave the other unattached). That way when you ssh onto raspberrypi, you know which one it is.

Next, edit the hosts file.

sudo nano /etc/hosts

You’ll need to change the last line to whatever you want to name the pi, in my case I called it robopi       localhost
::1             localhost ip6-localhost ip6-loopback
fe00::0         ip6-localnet
ff00::0         ip6-mcastprefix
ff02::1         ip6-allnodes
ff02::2         ip6-allrouters       robopi

Exit that file and then change the hostname file

sudo nano /etc/hostname

Change it to the same name you put in the hosts file


Thats the configuration changes done, next we need to restart the hostname service, but executing:

sudo /etc/init.d/hostname.sh

Then restart the pi

sudo reboot

After that, you should be able to ping and connect to robopi:

Jamess-MacBook-Pro:pi Elsey$ ping robopi
PING robopi.home ( 56 data bytes
64 bytes from icmp_seq=0 ttl=64 time=1.802 ms
64 bytes from icmp_seq=1 ttl=64 time=4.141 ms
--- robopi.home ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.802/2.971/4.141/1.170 ms
Jamess-MacBook-Pro:pi Elsey$ ssh pi@robopi
pi@robopi's password: 
Linux robopi 3.6.11+ #456 PREEMPT Mon May 20 17:42:15 BST 2013 armv6l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sun Dec 29 15:59:40 2013 from unknown
-bash: /etc/profile: is a directory
pi@robopi ~ $ hostname

Thats it, you can connect the original “raspberrypi” to the network, or change the hostname of that too

Getting to grips with Grails, building a survey management system

Sometime in late 2012 I was discussing dissertation project ideas with my girlfriend, as she was coming up to her final year of a computing bachelors. The usual option chosen by many graduates would be to just build a website or an app, or do some form of market research. We decided to encompass all 3 to produce something that works, but ultimately something that could be of value. If I had the time, energy, and funds I’d pursue this as it has potential for a startup, but I don’t, so the important thing that I’ve taken away is the experience working with groovy, grails, and android.

The Idea…

There are 2 main business drivers behind this project. Firstly we wanted to provide a service whereby restaurant owners can register, create surveys, and make them accessible to staff such as printing QR codes onto the back of their menus. Secondly, we wanted to approach this from the end users point of view, whereby customers sitting in the restaurant could download an app for free off the public market places, scan the said QR code, and be presented with the survey the restaurant owner had created. They would fill it in via the app and submit, the restaurant owner then has immediate access to the results in the form of statistics and graphs.

Landing page for Roosearch

Landing page for Roosearch

The outcomes that we’re after:

  • Better visibility to restaurant owners on how their customers feel
  • Easy and seamless access to surveys for the customers
  • A scalable application which can handle increasing users as demand grows
  • A platform for advertising new products and features

There are 3 components to this solution:

  1. Grails web application
  2. Rest API (built into the grails application)
  3. Android app

Why Grails?

  • Develop in groovy, so very accessible to java developers.
  • Quick to prototype with “convention over configuration”
  • Views auto generated if using scaffolding.
  • Easily deployable into the cloud, package as a war and deploy to cloudbees

The Prototype…

If you’re starting out with grails, I’d highly recommend that you get a copy of IntelliJ ultimate edition (and a copy of Grails in action), the support for grails is fantastic and I found it far easier than using eclipse. Whilst there are some excellent tutorials on grails out there (the official documentation is also very good) I’ll hold off and just jump right into how the application works.

One of the awesome features of grails is that it follows the “convention over configuration”, which simply means that if you follow the convention implied by the framework, you don’t have to be concerned about configuration. You can’t escape configuration entirely, but boilerplate plumbing can be inferred by convention. An example of that is if you name your controllers like “SurveyController”, grails automatically knows its a controller for the survey class, based on naming conventions. A similar convention applies for views.

Domain model

Roosearch entity relationship diagram

Roosearch entity relationship diagram

Our data model is quite simple. We have a user, the user has some surveys, each survey has a number of questions, and each questions has a number of predefined responses. The domain classes are self explanatory, but it’s probably worth mentioning a few tweaks I made.

class User {
    String firstName
    String lastName
    String emailAddress
    String companyName
    String facebookPageLink
    String twitterHandle

    static hasMany = [surveys: Survey]
    static constraints = {
        facebookPageLink nullable: true
        twitterHandle nullable: true

By default, all fields are mandatory, however in the above example of the User class we can override these constraints to set them as nullable. There are various other constraints that you can set, have a look at the documentation.

class Survey {

    Integer id
    String title

    static hasMany = [questions: Question]

    static mapping = {
        questions lazy: false

    static constraints = {

    String toString(){
        return title

The relationships between the classes are defined by the “static hasMany”. This basically says that one Survey has relationships to many Questions, and this relationship is identified by “questions”.

The mapping block instructs the questions to be eagerly loaded, so once a survey is loaded into memory, so are all of its questions, opposed to just the Ids which would then be loaded lazily.

It’s also useful to override the toString method on your domain objects, particularly if you have relationships as the scaffolding will create drop down lists in your views. If you don’t override toString with something sensible, you’ll just see the object hash codes instead, which isn’t very useful to the user.


It’s the responsibility of the controllers to manipulate the underlying data model (via services for example), and respond with views to the user. You can read more about the MVC pattern here.

To get started, you could simply enable scaffolding like so.

class LoginController {

    static scaffold = true

    def index = {
        render(view: "login.gsp")

Scaffolding is an excellent feature of grails to get you started. Grails knows the structure of your domain object, therefore it is able to dynamically create controller CRUD operations, and views to manipulate your objects. That one small line of code and you can create, updated, delete, and view your objects! Fantastic eh?!

The bad news…

Whilst scaffolding is great to get you started, the moment you want to do something out of the ordinary, or customisation on views, scaffolding becomes a bit useless, and you’ll have to implement your own controllers (and possibly views). Fortunately, grails is quite flexible so you can leave scaffolding on and just override the methods that you want to customise. As with views, you’re best off getting grails to generate them for you and then customise them, to save you having to write the entire controller/view from scratch.

The methods you can override, and there general uses are:

  • index – default action, usually just redirects to list
  • list – list all of the objects, handle pagination, filtering etc
  • create – render view to create new object
  • save – handle creation of new object, validation etc.
  • edit – render view to edit an object
  • update – handle update of object
  • delete – delete an object

You can read more about the controller actions on the grails documentation.

You can see in the show() method on the SurveyController that I’ve customised it to to add some charts into the response model. You can see how I generate the chart data by looking at the source code in github. The view can then render these as javascript charts (which I’ll come onto in a moment)

    def show(Long id) {
        def surveyInstance = Survey.get(id)
        if (!surveyInstance) {
            flash.message = message(code: 'default.not.found.message', args: [message(code: 'survey.label', default: 'Survey'), id])
            redirect(action: "list")

        def charts = getCharts(surveyInstance)

        [surveyInstance: surveyInstance, charts: charts]


Being quite fond of the default views that grails generates, and not wanting to invest a great deal of time with customisation for this prototype, I chose to generate the views and then just tweak as I needed. In reality, the only customisation I needed to do was to place a “generate QR code” link, and to insert some javacript charts for displaying survey statistics.

Having assessed HighCharts, D3, and the Google visualisation API, I opted for the latter as I felt it was far simpler to use and I didn’t have any need for the advanced features that HighCharts and D3 come with, and there was a plugin for gvisualisation.

Displaying charts was straightforward, after installing the visualisation plugin, add this snippet of code to iterate over the charts that were added to the model and display a barCoreChart.

<g:each in="${charts.values()}" var="item">
                    width="${400}" height="${240}"

            <div id="chart-${item.question_id}" align="center"></div>

This would then display something like the following, you can change various elements of the charts such as the chart type, axis labels, sizes and titles, please refer to the documentation.

Charts using Google Visualisation

Charts using Google Visualisation

QR codes

QR codes make it incredibly easy to share data to android devices, my intention was to embed a user ID in a QR code, when scanned the app can request all surveys pertinent to the user ID.

Generating QR codes is easy with the qrcode plugin. I have provided a link on the users view to generate a QR code:

<span class="property-value" aria-labelledby="qr-label">
                <g:link controller="user" action="generateQrCode" id="${userInstance.id}">Generate QR Code</g:link>

This is bound to the generateQrCode action on the user controller, which will create a QR code from a user id and display it

    def generateQrCode(Long id){
        println "Generate QR code here..."
        String data = "$id"
        int qrSize = 500

        QRCodeRenderer qrcodeRenderer = new QRCodeRenderer()
        qrcodeRenderer.renderPng(data, qrSize, response.outputStream)

As you can see, it is as simple as providing the data to be encoded, the size (x==y), and the output stream, in this case the response. When you click the link, you should see the following:

QR code generated by the qrcode plugin

QR code generated by the qrcode plugin


The website element is designed for the restaurant owners, the end users will be using an android app to complete surveys. Whilst I could have developed a mobile responsive page, I felt that an android app would bring a better overall experience to the user.

I have created a controller, ApiController that enables users to request surveys, and post responses.

Firstly, I created the URL mappings for this new controller

	static mappings = {
        "/api/customer/$customerid"(controller: "api", action: 'getCustomer')
        "/api/survey/$surveyid"(controller: "api", action: [GET: 'getSurvey'])
        "/api/survey"(controller: "api", action: [POST: 'surveyComplete'])

            constraints {
                // apply constraints here

		"/"(controller: "home")

Requests on /api/customer/$customerid, such as /api/customer/123 are routed to the getCustomer method on the api controller. The same is true for the second mapping, however the action is a GET on getSurvey (in hindsight, the first mapping should be restricted to the GET method too). The third mapping is a POST on /api/survey which will be invoked when the user has completed a survey on their device.

    def getCustomer(){
        User u = User.get(params.customerid)

        def surveysToPresent = [:]

        u.surveys.each {
            surveysToPresent << [title: it.title, id: it.id]
        render(contentType: 'text/json') {[
                'company_name': u.companyName,
                'twitter' : u.twitterHandle,
                'facebook' : u.facebookPageLink,
                'surveys' : [surveysToPresent]
        ]} as JSON

The getCustomer method finds the user from the customerid on the request path, retrieves the surveys and transforms them to a map containing the title and id (we don’t need the entire survey object when the user is presented with a list of surveys to select). The render statement enables us to return a json response very easily, we just return a map and grails (jackson) takes care of the json marshalling.

    def getSurvey(){
        Survey s = Survey.get(params.surveyid)

        def questionsToPresent = [:]

        questionsToPresent = s.questions.collect {
                    id: it.id,
                    text: it.text,
                    responses : it.responses.collect{ resp ->
                        [id: resp.id, text: resp.text]

        render(contentType: 'text/json') {[
                'id' : s.id,
                'title': s.title,
                'questions' : questionsToPresent
        ]} as JSON

The getSurvey method behaves in a similar manner to getCustomer, it builds a map and renders as json.

    def surveyComplete(){
        def jsonObject = request.JSON

        Survey theSurvey = Survey.findById(jsonObject.id)

        jsonObject.responses.each{ response ->
            theSurvey.questions.find {it.id == response.question_id}.responses.find {it.id == response.response_id}.numberOfPeopleSelected++
        theSurvey.save(flush: true, failOnError: true)

        render(status: 204)

The surveyComplete will retrieve a survey by id, find the responses the user has provided, and increment a count. The survey is then saved and a “204 No Content” is returned.

I’ll cover how the android app consumes these services in my next post.


As this project is just a prototype, I decided to host it on a free Cloudbees instance. The application doesn’t have any persistence layer, and all data is held in memory (which is fine for its current purpose), so when Cloudbees hibernates the instance after a period of inactivity, all user data will be lost. Deploying is simple, build the war using

grails war

Then upload the war file from the target directory to your cloud bees account, or use the command line cloud bees SDK.

View source code on Github

View live demo

(if the live demo link doesn’t work, try again in 10 minutes as the instance will be waking from hibernation)


Project Euler, problem 4, palindromes

Problem 4:

A palindromic number reads the same both ways. The largest palindrome made from the product of two 2-digit numbers is 9009 = 91 × 99.

Find the largest palindrome made from the product of two 3-digit numbers.

Firstly I decided to try and solve the problem using two 2-digit numbers so I can understand how the creator of this problem got to 9009.

In about 15 minutes I was able to come up with this brute force hack:

public class Problem4 {

    public static void main(String[] args) {

    private static void solve() {
        int highestPalindrome = 0;

        for (int left = 99; left &gt; 2; left--) {
            for (int right = 99; right &gt; 2; right--) {
                int candidate = left * right;
                if (isPalindrome(candidate)) {
                    System.out.println(format(&quot;Palindrome found! Using %d * %d = %d &quot;, left, right, candidate));
                    if (candidate &gt; highestPalindrome) {
                        highestPalindrome = candidate;
        System.out.println(&quot;Highest palindrome is &quot; + highestPalindrome);

    private static boolean isPalindrome(int palindrome) {
        String palindromeString = &quot;&quot; + palindrome;
        String reversed = new StringBuilder(palindromeString).reverse().toString();
        return palindromeString.equals(reversed);

Which gives the output of

Highest palindrome is 9009

Excellent, so we’re on the right track. The next thing I did was alter the loop statements to start at 999, which then prints the following

Highest palindrome is 906609

Which is the correct answer!

Any suggestions on improvements? Please comment!

Project Euler, problem 3, largest prime factor

Problem 3 :

The prime factors of 13195 are 5, 7, 13 and 29.

What is the largest prime factor of the number 600851475143 ?

To solve this, I needed to understand the process of prime factorisation (surprisingly, it is something which I was never taught at school/uni).

What we need to do, is to take the number 13195, then try and divide it by the smallest prime number; 2.

If it doesn’t divide equally, we try it with the next prime number; 3, and so on, until it divides evenly with no remainder.

So 13195 divides into 5 evenly, and we get a result of 2639.

We can retain that number 5, but since we’re only interested in the largest prime factor, we reset our divisor count to 2 and start the process again, trying to divide 2639 until we find a number that it divides evenly by, which happens to be 7.

  • 13195 / 5 = 2639
  • 2639 / 7 = 377
  • 377 / 13 = 29

On the last stage of prime factorisation, we’re trying to find a number that fits into 29 evenly, but there isn’t one, so we’re left with the prime number of 29.

We’re left with the prime factors of 5, 7, 13 and 29.

I found that the MathsIsFun.com had a good article on prime factorisation, with several examples and explanations.

Here is my solution using Java

 * The prime factors of 13195 are 5, 7, 13 and 29.
 * <p/>
 * What is the largest prime factor of the number 600851475143 ?
public class Problem3 {

    public static void main(String[] args) {

    public static int solve(long number) {
        int i;

        // start dividing by 2, as that is the smallest prime number, keep going until we're trying to divide the number by itself,
        // this indicates we've finished.
        for (i = 2; i <= number; i++) {
            //We're only interested in "i" if it divides evenly by the number
            if (number % i == 0) {
                // divide number by i and re-assign, so we can start counting up from 2 again
                number /= i;
                // the for loop will increment i, however if number is divisible by i with no remainder, we want to try again.
                // for example, if we divide number 2000 by 2, we want to decrement i so we can try to divide the resulting 1000 by 2 again

        return i;

Don’t forget the “L” on the end of the number, we need to use a long as its too large to fit in an int.

The answer is 6857