Getting your head around the Couchbase SyncGateway

I like Couchbase. One of the things that really appeals to me is the sync gateway. As a mobile developer I often find that the apps I’m developing are just interfaces into some backend service. Somewhere out there in the cloud I’ll have a web application that sits on top of some database (nodejs/mongoDB is a combo I’ve been using recently). Then there comes the mobile app, which will be consuming these services, which would be fine if 4G/wifi was everywhere (I can’t even get a cellular signal at my place, let alone dream of 4G).

We’re into the realms of apps working offline, you then have the pain of syncing data and dealing with conflicts. You can make your life easier by using a SyncAdapter on Android, or perhaps a framework like Restkit if you’re developing on iOS, heck, you can even implement the syncing yourself (don’t do that, that road leads to madness..speaking from experience)…OR…you can just use Couchbase and the SyncGateway.

In short, the SyncGateway is an application that sits between your Couchbase server, and your Couchbase Lite enabled mobile apps. This means you can access your data on your local CBLite database, and not have to worry (too much) about syncing this to the Couchbase server.

Getting setup

I have to admit, the documentation is a little confusing when it comes to explaining how the components hang together, but after attending Couchbase Live in London a month or so back I was able to track down those who are in the know, and put the missing piece into my puzzle of confusion; bucket syncing.

For the purpose of explaining how this works, I’ll use my “Coin Collector” android app as the example. The app needs to get its data on coins from a couchbase server. It should be able to work offline and sync periodically. I’m using bucket syncing so I can have a web page to administer coins such as adding new coins to altering market values.
The documentation is really missing a diagram like the following

couchbase-sync

Let me cover the 4 points in blue numbers:

  1. Regardless of which mobile platform you’re using, it’ll be connecting to the sync gateway via the REST apis, this is where “json over the wire” comes into play.
  2. As the mobile apps use their own bucket, you need to configure the gateway to tell it where to put documents. If you check my config below; then this is done by the “aussie-coins-syncgw” configuration element, you can see that the bucket is set to “aussie-coins-bucket-sync-db” on the localhost couchbase server (sync and db are running on my local vm)
  3. This is where the magic happens. Bucket shadowing in the later releases of the Sync Gateway allow it to sync changes between your “mobile” bucket, and your “backend” bucket. You can see this configured by the “shadow” element in my config.json
  4. Your backend server apps can just connect to the “aussie-coins-bucket” and be totally oblivious to what is happening in the mobile side of your architecture.
{
    "interface": ":4984",
    "adminInterface": ":4985",
    "log": ["CRUD", "CRUD+", "HTTP", "HTTP+", "Access", "Cache", "Shadow", "Shadow+", "Changes", "Changes+"],
    "databases": {
        "aussie-coins-syncgw": {
            "server": "http://localhost:8091",
            "bucket": "aussie-coins-bucket-sync-db",
            "sync": `function(doc) {channel(doc.channels);}`,
            "users": {
                "GUEST": {
                    "disabled": false,
                    "admin_channels": ["*"]
                }
            },
            "shadow": {
                 "server": "http://localhost:8091",
                 "bucket": "aussie-coins-bucket"
            }
        }
    }
}

Some other points to notice in the configuration:

  • The interface port is the port the apps will connect on, the adminInterface is for administering the sync gateway, such as dynamically adding new databases, or altering channels.
  • Logs, I’ve chosen to log everything, you can restrict these if you need, check the Couchbase documentation for further info.
  • I’ve enabled the guest user access on all channels for the purpose of evaluating this, ideally we’d need to restrict the channels that users can use to stop any potential abuse.

Testing it out

As I mentioned above, since the mobile apps will be connecting to the Sync Gateway via a REST api, we can take the mobile app out of the picture and test using a rest client (I’m using Postman for Google Chrome). Lets cover 2 scenarios.

Server Producing

This scenario involves a new document being created on the server, and it being synced to the mobile bucket and available to view on the mobile apps.

Firstly, let me show you what I have in the “aussie-coins-bucket”.

1-couchbase-coins_bucket

Next, lets create a new document with an ID of 5, for the Ten Cent coin. We should then see it listed in our “aussie-coins-bucket” like so:

2-couchbase-coins_bucket_new_coin

Now lets have a look at the log output from the Sync Gateway.

22:49:33.826838 Shadow+: Pulling "5", CAS=1e2dd7153a ... have UpstreamRev="", UpstreamCAS=0
22:49:33.826894 Shadow: Pulling "5", CAS=1e2dd7153a --> rev "1-1d7a1a352c0abb293fdd16883ef6985b"
22:49:33.826909 CRUD+: Invoking sync on doc "5" rev 1-1d7a1a352c0abb293fdd16883ef6985b
22:49:33.903707 Cache: SAVING #8
22:49:33.903984 CRUD: Stored doc "5" / "1-1d7a1a352c0abb293fdd16883ef6985b"
22:49:34.768280 Cache: Received #8 after 864ms ("5" / "1-1d7a1a352c0abb293fdd16883ef6985b")
22:49:34.768305 Cache:     #8 ==> channel "*"
22:49:34.768322 Changes+: Notifying that "aussie-coins-bucket-sync-db" changed (keys="{*}") count=3
22:49:59.849578 Shadow+: Pulling "5", CAS=2423d4b93a ... have UpstreamRev="1-1d7a1a352c0abb293fdd16883ef6985b", UpstreamCAS=c21019dd68
22:49:59.849623 Shadow: Pulling "5", CAS=2423d4b93a --> rev "2-971b4b3009127da5ed2a4770cb45cfe7"
22:49:59.849637 CRUD+: Invoking sync on doc "5" rev 2-971b4b3009127da5ed2a4770cb45cfe7
22:49:59.849749 CRUD+: Saving old revision "5" / "1-1d7a1a352c0abb293fdd16883ef6985b" (68 bytes)
22:49:59.849891 CRUD+: Backed up obsolete rev "5"/"1-1d7a1a352c0abb293fdd16883ef6985b"
22:49:59.850068 Cache: SAVING #9
22:49:59.850207 CRUD: Stored doc "5" / "2-971b4b3009127da5ed2a4770cb45cfe7"
22:50:00.790818 Cache: Received #9 after 940ms ("5" / "2-971b4b3009127da5ed2a4770cb45cfe7")
22:50:00.790838 Cache:     #9 ==> channel "*"
22:50:00.790868 Changes+: Notifying that "aussie-coins-bucket-sync-db" changed (keys="{*}") count=4

As we can see, the Sync Gateway has detected that there is a new document and that it needs to shadow it across, which is does successfully.

On the couchbase server, we can view that document in the mobile bucket, “aussie-coins-sync-db” like so:

3-couchbase_synced_coin

Finally, just to prove the mobile clients can see that document via the API, do a GET on http://localhost:4984/aussie-coins-syncgw/5 and you’ll see the following:

{
    "_id": "5",
    "_rev": "2-971b4b3009127da5ed2a4770cb45cfe7",
    "coin": "Ten Cent"
}

Mobile Producer

Now we’ll try the opposite, producing documents from the mobile clients and seeing them synced across to the Couchbase server. From a REST client, do a PUT to http://localhost:4984/aussie-coins-syncgw/6 with a json body of:

{
  "coin":"Twenty Cent"
}

You should see a response of

{
    "id": "6",
    "ok": true,
    "rev": "1-e9c16d3887a3958314adff1e3cbd6097"
}

What we’ve done is to create a document with the ID of 6, for “Twenty Cent”.

Lets have a look at the Sync Gateway logs:

23:19:56.860618 HTTP:  #003: PUT /aussie-coins-syncgw/6
23:19:56.971056 CRUD+: Invoking sync on doc "6" rev 1-e9c16d3887a3958314adff1e3cbd6097
23:19:57.023839 Cache: SAVING #10
23:19:57.024110 CRUD: Stored doc "6" / "1-e9c16d3887a3958314adff1e3cbd6097"
23:19:57.024161 HTTP+: #003:     --> 201   (0.0 ms)
23:19:57.616316 Cache: Received #10 after 592ms ("6" / "1-e9c16d3887a3958314adff1e3cbd6097")
23:19:57.616340 Cache:     #10 ==> channel "*"
23:19:57.616353 Shadow: Pushing "6", rev "1-e9c16d3887a3958314adff1e3cbd6097"
23:19:57.616367 Changes+: Notifying that "aussie-coins-bucket-sync-db" changed (keys="{*}") count=6
23:19:57.852304 Shadow+: Pulling "6", CAS=1c6f07c3ce2 ... have UpstreamRev="", UpstreamCAS=0
23:19:57.852327 Shadow+: Not pulling "6", CAS=1c6f07c3ce2 (echo of rev "1-e9c16d3887a3958314adff1e3cbd6097")
23:19:57.852337 CRUD+: Invoking sync on doc "6" rev 1-e9c16d3887a3958314adff1e3cbd6097
23:19:57.865669 CRUD+: updateDoc("6"): Rev "1-e9c16d3887a3958314adff1e3cbd6097" leaves "1-e9c16d3887a3958314adff1e3cbd6097" still current
23:19:57.865751 Cache: SAVING #11
23:19:57.866050 CRUD: Stored doc "6" / "1-e9c16d3887a3958314adff1e3cbd6097"
23:19:58.617446 Cache: Received #11 after 751ms ("6" / "1-e9c16d3887a3958314adff1e3cbd6097")
23:19:58.617463 Cache:     #11 ==> channel "*"
23:19:58.617482 Changes+: Notifying that "aussie-coins-bucket-sync-db" changed (keys="{*}") count=7

We can then see the document in the “aussie-coins-bucket-sync-db”:

4-couchbase_syncdb

…and then in the “aussie-coins-bucket”:

The Sync Gateway is a useful application, and really does make the Couchbase offering even more appealing. Once you can get your head around the bucket shadowing (which you should if you’ve made it this far) then it can be easy to work with.

Comment or find me on Twitter (@jameselsey1986) if you have any questions!

Why the duplication?

Having 2 buckets for the same data had me raise an eyebrow initially, but after asking on Google Groups, it does make sense. You can’t expect the backend app servers to maintain sync meta data on new documents it creates. Perhaps Couchbase will alter this in the future.

Resources

Grails based survey system, the android app

Some time back I wrote an article describing the roosearch system I developed using grails. This is the second part, the android client, please checkout the previous article otherwise this might not make much sense!

After completing the grails component, I had a RESTful API available to me, and I just needed to build an app that could consume those services.

Customer lookup and QR codes

The app needs to be simple and quick to use, one of the things I remember from a UX discussion at DroidCon UK is “Don’t annoy your users, they control your app ratings and your income!”. In order to lookup the surveys quickly, I’ve added the ability to scan QR codes. Actually I didn’t have to do a great deal as there is already an app called ZXing by Google that scans QR codes, so I just needed to make Roosearch delegate to ZXing and handle the result.

Of course, we don’t want to exclude users that don’t have ZXing, or even a camera on their device, so I’ve also provided a text field where they can enter the customer Id manually if required.

When the user clicks on the “scan barcode” button, I first check if ZXing is installed using the following

    public void scanBarCode(View v) {
        final boolean scanAvailable = isIntentAvailable(this,
                "com.google.zxing.client.android.SCAN");
        if (!scanAvailable){
            Toast.makeText(this, "You need to install the ZXing barcode app to use this feature", Toast.LENGTH_SHORT).show();
            return;
        }

        Intent intent = new Intent("com.google.zxing.client.android.SCAN");
        intent.putExtra("SCAN_MODE", "QR_CODE_MODE");
        startActivityForResult(intent, 0);
    }

If the user does have ZXing installed on their device, and choose to use it, we can get the result back from the bar code scan using:

public void onActivityResult(int requestCode, int resultCode, Intent intent) {
        if (requestCode == 0) {
            if (resultCode == RESULT_OK) {
                String contents = intent.getStringExtra("SCAN_RESULT");
                performRooLookup(contents);
            } else if (resultCode == RESULT_CANCELED) {
                // Handle cancel
            }
        }
    }

    private void performRooLookup(String rooId) {
        if (StringUtils.isBlank(rooId)) {
            Toast.makeText(this, "Please enter a valid customer id", Toast.LENGTH_SHORT).show();
            return;
        }

        Integer customerId;
        try {
            customerId = Integer.parseInt(rooId);
        } catch (NumberFormatException e) {
            Toast.makeText(this, "Customer id needs to be numeric", Toast.LENGTH_SHORT).show();
            return;
        }
        new FindRooTask(this, new FindRooTaskCompleteListener()).execute(customerId);
    }

I then have the following buried in a service call, invoked by an AsyncTask, which handles finding Customer details:

    public Customer getCustomerDetails(int customerId) {

        try {
            final String url = "http://roosearchdev.jameselsey.cloudbees.net/api/customer/{query}";

            HttpHeaders requestHeaders = new HttpHeaders();

            // Create a new RestTemplate instance
            RestTemplate restTemplate = new RestTemplate();
            restTemplate.getMessageConverters().add(new MappingJacksonHttpMessageConverter());

            // Perform the HTTP GET request
            ResponseEntity<Customer> response = restTemplate.exchange(url, HttpMethod.GET,
                    new HttpEntity<Object>(requestHeaders), Customer.class, customerId);

            return response.getBody();
        } catch (Exception e) {
            System.out.println("Oops, got an error retrieving from server.. + e");
        }
         return null;
    }

A Customer looks like this:

public class Customer implements Parcelable {

    @JsonProperty("company_name")
    private String companyName;
    private String twitter;
    private String facebook;
    private List<SurveySummary> surveys = new ArrayList<SurveySummary>();
    //Accessors omitted
}

The SurveySummary just has a title and Id. The reason for just returning summaries is because a customer may have many surveys, and there is no need to obtain them all, we just obtain the title to display to the user, if selected, we’ll retrieve the survey by its id.

To recap, here are 2 screenshots that show the above; the landing screen, and then the customer display screen

Landing screen for Roosearch, where the user can enter a customer Id or scan a QR code

Landing screen for Roosearch, where the user can enter a customer Id or scan a QR code

Customer screen, display social media links, name, photo, and list of surveys that the customer has

Customer screen, display social media links, name, photo, and list of surveys that the customer has

 

The survey engine

This is where the magic happens. I have a single activity and single view that handles presenting the survey to the user. As the surveys can change number of questions, and number of responses, I needed a way of dynamically traversing the survey object and allowing user to move between the questions whilst retaining state of what they have selected so far.

I’ve created the following method that will redraw the layout for a given question id:

    public void drawQuestionOnScreen(int id) {
        TextView question = (TextView) findViewById(R.id.question);
        question.setText(s.getQuestion(id - 1).getText());   // subtract 1 as lists are indexed from 0

        LinearLayout linLay = (LinearLayout) findViewById(R.id.answers);
        linLay.removeAllViews();
        RadioGroup rg = new RadioGroup(this);
        rg.setId(1);
        for (int aIndex = 0; aIndex < s.getQuestion(id - 1).getResponses().size(); aIndex++) {
            Answer a = s.getQuestion(id - 1).getAvailableOption(aIndex);
            RadioButton button = new RadioButton(this);
            button.setText(a.getText());
            button.setTextColor(R.color.dark_text_color);
            button.setId(aIndex);
            rg.addView(button);
        }
        linLay.addView(rg);

        TextView status = (TextView) findViewById(R.id.status);
        status.setText(format("%d of %d", id, s.getQuestionCount()));
    }

As you can see, it will retrieve the question by Id, then iterate over the responses and generate RadioButtons. Moving to the next question is reasonably easy, firstly I work out if an option has been selected, and prevent moving on if not. After that, I mark the selected response in the survey object, and then work out if there is another question in the sequence to display, if not we can progress to the finish.

One of the questions in the given survey

One of the questions in the given survey

    public void next(View v) {
        RadioGroup rg = (RadioGroup) findViewById(1);

        int selectedRadioId = rg.getCheckedRadioButtonId();
        if(selectedRadioId == -1){
            Toast.makeText(this, "Please select a response", Toast.LENGTH_SHORT).show();
            return;
        }

        s.getQuestion(questionIndex - 1).getResponses().get(selectedRadioId).setSelected(true);
        // work out if there is another question, then move to it
        if (s.getQuestionCount() > 1 && questionIndex < s.getQuestionCount()) {
            questionIndex++;
            drawQuestionOnScreen(questionIndex);
        } else {
            // if there are no other questions, show dialog saying submit or not
            Toast.makeText(this, "Reached the end of the survey", Toast.LENGTH_SHORT).show();
            // HERE we should process the entire survey, crunch data and post off (maybe async)

            Intent i = new Intent(this, SurveyComplete.class);
            i.putExtra("com.roosearch.domain.Survey", s);
            startActivity(i);
        }
    }

A similar approach is needed for moving back to previous questions, determine if there is a previous question to move to then redraw the screen, like so:

    public void previous(View v) {
        // work out if there is a previous question, and if so move to it
        if (s.getQuestionCount() > 1 && questionIndex > 1) {
            questionIndex--;
            drawQuestionOnScreen(questionIndex);
        } else {
            //if there are no other questions, move back to home screen, finish() this and scrap any progress
            finish();
        }
    }

Once the user completes all questions, the SurveyComplete activity is invoked.

Completing a survey

When the user has completed all questions, the survey object is passed into the SurveyComplete activity, which handles sending the responses back to the grails web application.

@Override
    protected void onResume()
    {
        super.onResume();
        TextView tv = (TextView) findViewById(R.id.completeMessage);
        tv.setText("Thank you for taking the time to complete the survey");
        tv.setTextColor(R.color.dark_text_color);

        Survey s = getIntent().getExtras().getParcelable("com.roosearch.domain.Survey");

        if (s != null)
        {
            StringBuffer sb = new StringBuffer();
            sb.append("\n" + s.getTitle() + "\n");
            for (Question q : s.getQuestions())
            {
                sb.append("\nQ: " + q.getText());
                sb.append("\nA: " + q.getSelectedAnswer() + "\n");
            }
            tv.append("\n\n" + sb.toString());
        }

        new SurveyUploadTask(this, new SurveyUploadTaskCompleteListener()).execute(s);
    }

    public class SurveyUploadTaskCompleteListener implements AsyncTaskCompleteListener<Void> {
        @Override
        public void onTaskComplete(Void voidz) {
            Toast.makeText(SurveyComplete.this, "Survey uploaded", Toast.LENGTH_SHORT).show();
        }
    }

The activity uses an AsyncTask to post the data back to the grails API controller, and displays a toast when successful.

Survey completed, results uploaded, and summary presented to user

Survey completed, results uploaded, and summary presented to user

 

Wrapping it up

Overall quite a simple app, I spent probably around 2 or 3 weekends putting together, most of that time was spent getting to grips with some automated testing for android. The code is admittedly a little rough around the edges, but I was aiming for an MVP (most viable product) to get working, feel free to contribute or suggest improvements!

I chose to use maven, but would use gradle if I were to pick this up again. Be sure to check out the code on github and try running it against Roosearch web, it does work!

Click here for the source code on Github

Robots! Part 2, the android client

Continuing on from my previous post, I’ve created an android client that I can use to send commands to my python server.

Ultimately I want to be able to control the robot remotely, the best way to do this would be to control the robot from a tablet or a phone which communicates wirelessly with the pi via bluetooth or wifi. In my previous post I described setting up a python application that will run on the raspberry pi and listen for commands. All I needed to do was to create a very basic android interface that can send commands to the raspberry pi.

The robot I intend to build will be based on tracks instead of wheels, there are many benefits to this but the most significant is that from an engineering perspective is that its much easier to build. A car needs forward and backwards drive, but also sideways drive for the front axle. In my opinion, it is far simpler to have a tracked vehicle with a motor controlling each side. When both motors are turned in the same direction the vehicle moves forward or backwards, and when the motors run in opposite directions the vehicle will turn on the spot.

My app interface mimics the layout of the vehicle itself, with an up and down button on the left and right hand side of the screen, as shown below.

Arrow buttons for controlling robots tracks

Arrow buttons for controlling robots tracks

You can see the code for this layout on the github repo here.

I want the user to be able to hold a button and the motor will run until they take their finger off. For this I’ve attached listeners on the buttons that will listen for the key up and key down events. It will send separate events for starting and stopping the motors, like so:

        Button leftForward = (Button) findViewById(R.id.leftForward);

        leftForward.setOnTouchListener(new View.OnTouchListener() {
            @Override
            public boolean onTouch(View view, MotionEvent motionEvent) {
                switch (motionEvent.getAction()) {
                    case MotionEvent.ACTION_DOWN: {
                        sendCommand(leftForwardCommand + "-Start");
                        break;
                    }
                    case MotionEvent.ACTION_UP: {
                        sendCommand(leftForwardCommand + "-Stop");
                        break;
                    }
                }
                return false;
            }
        });

As the sendCommand needs send a message over the network, I need to take this off the main UI thread otherwise I’d get an exception such as:

android.os.NetworkOnMainThreadException

To take this off the UI thread, I simply move the sending of the command into an AsyncTask, like so:

private void sendCommand(String command) {
        new SendCommandTask().execute(command);
    }

    class SendCommandTask extends AsyncTask<String, Void, Void> {

        @Override
        protected Void doInBackground(String... commands) {
            String command = commands[0];
            try {
                //TODO: make this configurable inside the app
                Socket socket = new Socket("192.168.0.6", 3033);
                PrintWriter out = new PrintWriter(new BufferedWriter(new OutputStreamWriter(socket.getOutputStream())), true);
                out.println(command);

                Log.d(TAG, "Successfully sent " + command);
            } catch (IOException ioe) {
                Log.d(TAG, "Unable to send command", ioe);
            }
            return null;
        }
    }

Now if I run the python server and then send press the buttons in the app, I see this output from the python application.

Server listening on port 3033...
...L-FORWARD-Start…..L-FORWARD-Stop….

Thats all for now, you can access this code on my github repository. Next post will either be in relation to building the robot, or using the BrickPi APIs.

Robots! (part 1)

Inspired by the android controlled lego robots I saw at DroidCon UK this year, and with difficulty finding a use for my raspberry pi, I’ve decided to have a go at building a robot that I can control via an android app. Having a 24 hour flight home from Australia at the weekend, I’ve had plenty of time to think about how I might approach this task (or challenge as I refer to it as I’ve no prior experience with robotics / socket programming).

Lego robot powered by the BrickPi

My plan is to have a python socket server running on the pi. This will provide a socket that an android client can invoke commands on. This python server will also interact with the python scripts that the BrickPi uses to control lego motors.

Whilst most of my experience revolves around Java, I’ve opted for python for the following reasons

  • Python is supported on the pi out of the box, no need to mess around with installing Java
  • The BrickPi has support for Python (and C)
  • I feel like learning something new

Baby steps…

Starting simple, I thought it best to create a simple script that listens on a socket, and then create a client that sends it some data to print out to the console. Once I have this working I can expand on it and make the client more sophisticated (an android app for example) and also enhance the server so it can handle different types of commands.

Server
There are plenty of example python scripts online, I found a good one here and tweaked it slightly to remove the parts I don’t want. (full credit to pythonadventures!)

#!/usr/bin/env python
# server.py

import socket
import select
import Queue
from threading import Thread
import sys

class ProcessThread(Thread):
    def __init__(self):
        super(ProcessThread, self).__init__()
        self.running = True
        self.q = Queue.Queue()

    def add(self, data):
        self.q.put(data)

    def stop(self):
        self.running = False

    def run(self):
        q = self.q
        while self.running:
            try:
                # block for 1 second only:
                value = q.get(block=True, timeout=1)
                process(value)
            except Queue.Empty:
                sys.stdout.write('.')
                sys.stdout.flush()
        #
        if not q.empty():
            print "Elements left in the queue:"
            while not q.empty():
                print q.get()

t = ProcessThread()
t.start()

def process(value):
    print value

def main():
    s = socket.socket()
    host = socket.gethostname()
    port = 3033
    s.bind((host, port))

    print "Server listening on port {p}...".format(p=port)

    s.listen(5)                 # Now wait for client connection.

    while True:
        try:
            client, addr = s.accept()
            ready = select.select([client,],[], [],2)
            if ready[0]:
                data = client.recv(4096)
                t.add(data)
        except KeyboardInterrupt:
            print
            print "Stopping server."
            break
        except socket.error, msg:
            print "Socket error %s" % msg
            break

    cleanup()

def cleanup():
    t.stop()
    t.join()

if __name__ == "__main__":
    main()

The socket is bound to a port, and then continually listens for incoming data. Once some data is received, it is added onto a queue, which is then sequentially executed.

Client
The client is is fairly straightforward, it involves opening a connection on a socket and writing data to it, a few lines of Java code.

public static void main(String[] args) throws IOException {

        InetAddress address = InetAddress.getLocalHost();
        Socket socket = new Socket(address, 3033);

        BufferedOutputStream bos = new BufferedOutputStream(socket.getOutputStream());

        OutputStreamWriter osw = new OutputStreamWriter(bos, "US-ASCII");

        System.out.println("Sending message...");

        osw.write("Hello!");
        osw.flush();
    }

Thats it for now, you can checkout my code on github (or just copy/paste the above) and run the client and server and see it in action.

I’ll start on the client next..

What I thought of “Instant Android Fragmentation Management How-to”


I was recently contacted by the marketing team from Packt Publishing and asked if I could write a review for one of their new ebooks, so here it is!

This book gives a brief yet comprehensive overview of some of the processes you can follow to improve your applications chance of working on the varying versions of android that are currently in existence.

Having developed apps that work perfectly fine on my Galaxy S2 2.3.3, only to find they don’t work on a hardware identical S2 with 4.0, I certainly feel the pain of OS fragmentation!

The book discusses some of the approaches you can take using the android compatibility library, fragments, loaders and action bars that enable you to write against the latest APIs but still have a good chance of having your app work on previous versions.

Things I liked about this book

  • A short and focused read, I got through it in around an hour.
  • Lots of code examples and references to external reading material.
  • The reference to ActionBarSherlock I felt was very much needed, as always if there are open source alternatives that work better, we should adopt them rather than struggle!

Things I didn’t like

  • I did feel that having some more screenshots would have helped emphasised the topic in discussion, whilst there were some screenshots throughout I did constantly find myself thinking “what should this actually look like?”
  • The chapters could have done with a more expanded overview, particularly the loaders section as it jumped right into creating one before explaining what it is, and how it differs from other non-UI asynchronous mechanisms like async tasks.

All in all I rate this as 4/5. The ebook is very reasonably priced and for the price of a few beers it’d certainly worth it, even for the discussions on ActionBarSherlock and Fragments. If your struggling to support multiple android versions, this book is a good starting point for you.

A few little tweaks to automate Android instrumentation testing via Robotium & Maven

Having recently revived an android project I haven’t opened in close to 6 months, I was left scratching my head as to why I couldn’t run any of my integration tests.

Thinking back, I remembered having problems getting robotium to instrument the clicking of a button, as simple as it sounds, theres a few little gotchas involved. Firstly, I needed to modify the test code to click on text, rather than a button, as it has been suggested by various users on StackOverflow that there does tend to be odd side effects when clicking buttons:

public void testClickGetSurveyTakesToSurveyRunner() {
 solo.clickOnText("Get Survey");
 solo.assertCurrentActivity("Current activity was not correct", SurveyRunner.class);
 }

Next, I needed to add this into the test projects AndroidManifest

<uses-sdk android:targetSdkVersion="10" />

And then to add this into the main projects AndroidManifest:

<supports-screens android:anyDensity="true"/> 

Both of these tweaks are mentioned on the Robotium FAQ, so it might be worth glancing there to see if any enhancements get made in the future

After making those changes, I stumbled into another error that reared its ugly head in the logcat:

ERROR/AndroidRuntime(1185): FATAL EXCEPTION: main
 java.lang.IllegalAccessError: Class ref in pre-verified class resolved to unexpected implementation
 at com.roosearch.android.activity.SurveyRunner.onResume(SurveyRunner.java:38)
 at android.app.Instrumentation.callActivityOnResume(Instrumentation.java:1150)
 at android.app.Activity.performResume(Activity.java:3832)
 at android.app.ActivityThread.performResumeActivity(ActivityThread.java:2110)
 at android.app.ActivityThread.handleResumeActivity(ActivityThread.java:2135)
 at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:1668)
 at android.app.ActivityThread.access$1500(ActivityThread.java:117)
 at android.app.ActivityThread$H.handleMessage(ActivityThread.java:931)
 at android.os.Handler.dispatchMessage(Handler.java:99)
 at android.os.Looper.loop(Looper.java:130)
 at android.app.ActivityThread.main(ActivityThread.java:3683)
 at java.lang.reflect.Method.invokeNative(Native Method)
 at java.lang.reflect.Method.invoke(Method.java:507)
 at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:839)
 at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:597)
 at dalvik.system.NativeStart.main(Native Method)

There seems to be a rather peculiar requirement, that any dependencies your using on your main application, also need to be in the test projects pom.xml, but marked with a scope of provided. After making that change as advised here and rebuilding, both the app and instrumentation app build and test successfully.

I was also pleasantly surprised that I could run the instrumentation tests via IntelliJ IDEA 12 Ultimate, last time I tried that (on an older version albeit), it wasn’t so simple…

Deploying the android libraries into your maven repository

Maven is a fantastic build tool, and a great addition to anyone developing on the android platform, however one of the first hurdles that people often stumble upon, is when their project involves one of the SDK libraries, such as Google Maps.

You’ll most likely see something like this when you attempt to first compile the project :

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6.597s
[INFO] Finished at: Thu Jul 19 10:28:12 BST 2012
[INFO] Final Memory: 7M/81M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project WifiSpotter: Could not resolve dependencies for project com.jameselsey.android.apps.wifispotter:WifiSpotter:apk:0.1-SNAPSHOT: Failed to collect dependencies for [com.google.android:android:jar:2.1.2 (provided), com.google.android:android-test:jar:2.2.1 (provided), com.pivotallabs:robolectric:jar:1.0 (test), junit:junit:jar:4.8.2 (test), com.google.android.maps:maps:jar:8_r1 (provided)]: Failed to read artifact descriptor for com.google.android.maps:maps:jar:8_r1: Could not transfer artifact com.google.android.maps:maps:pom:8_r1 from/to cloudbees-private-release-repository (https://repository-
[ERROR] jameselsey
[ERROR] .forge.cloudbees.com/release): IllegalArgumentException: Illegal character in authority at index 8: https://repository-
[ERROR] jameselsey
[ERROR] .forge.cloudbees.com/release/com/google/android/maps/maps/8_r1/maps-8_r1.pom

As we can see, maven is unable to find the maps artefact, and rightly so. This is because the Google jars are not available (at least not at the time of writing this) on the maven central repositories. Fortunately for us, its relatively easy to resolve this, we just need to obtain the artefacts from our android home area, and then install them to our maven repositories so our projects have access to them.

You could manually install all of them, but this would be quite a lengthy task, and relatively unnecessary with the help of the maven-android-sdk-deployer, this nifty little tool will extract the libraries from your local android SDK installation, and install them as maven dependencies, to your local repository. From there, you can just depend on them as any other maven artefact, such as :

<dependency>
  <groupId>android</groupId>
  <artifactId>android</artifactId>
  <version>4.1_r2</version>
  <scope>provided</scope>
</dependency>

There is no need for me to cover how to use this tool, since it is pretty well documented, however I would like to refer you to a previous post of mine regarding deploying maven artefacts to a CloudBees repository, combined with this post, you can quite easily and reliably maven-ise the android dependencies, and deploy them into the cloud. Then you can build your android projects using maven from anywhere, and also take advantage of the free Jenkins service they provide.

Remember to drop in the repository into the pom.xml, and then you can simply run :

mvn clean install deploy:deploy

Hope this helps, if anything is unclear, please let me know!

Extracting out your AsyncTasks into separate classes makes your code cleaner

If you’re an android developer, chances are you’ve used an Async task more than once. As your apps develop, grow, and become more complex, theres a high chance you’re going to have multiple Async tasks. In an application I’ve recently been developing, I had around 5 Async tasks for a single activity, which made the actual activity quite difficult to read.

Adhering to KIS and DRY, I wanted to devise a mechanism for extracting the logic of these Async tasks out into separate external classes, thus reducing the clutter of inner classes in my activities, and also meaning that I could re-use these Async tasks elsewhere, and also externalising them makes them easier to mock and unit test.

The easiest mechanism I’ve found, is to use generics and provide a callback, which is somewhat similar to the delegate/protocol pattern in iOS programming.

Firstly, create a class with a generic method, like follows :

/**
 * This is a useful callback mechanism so we can abstract our AsyncTasks out into separate, re-usable
 * and testable classes yet still retain a hook back into the calling activity. Basically, it'll make classes
 * cleaner and easier to unit test.
 *
 * @param <T>
 */
public interface AsyncTaskCompleteListener<T>
{
    /**
     * Invoked when the AsyncTask has completed its execution.
     * @param result The resulting object from the AsyncTask.
     */
    public void onTaskComplete(T result);
}

Then, in your activity you can reduce the inner class to just the following :

public class FetchMyDataTaskCompleteListener implements AsyncTaskCompleteListener<MyPojo>
    {

        @Override
        public void onTaskComplete(MyPojo result)
        {
            // do something with the result
        }
    }

This means you can now create a separate class for the task, below is an example. Don’t forget to assign the callback listener in the constructor, so when the onPostExecute() happens you can invoke the callback to the onTaskComplete in your activity:

public class FetchMyDataTask extends AsyncTask<String, Integer, MyPojo>
{
    private static final String TAG = "FetchMyDataTask";

    private Context context;
    private AsyncTaskCompleteListener<MyPojo> listener;

    public FetchVehicleTask(Context ctx, AsyncTaskCompleteListener<MyPojo> listener)
    {
        this.context = ctx;
        this.listener = listener;
    }

    protected void onPreExecute()
    {
        super.onPreExecute();
    }

    @Override
    protected MyPojo doInBackground(String... strings)
    {
        MyPojo myPojo = null;
        // do something with myPojo
        return myPojo;
    }

    @Override
    protected void onPostExecute(MyPojo myPojo)
    {
        super.onPostExecute(myPojo);
        listener.onTaskComplete(myPojo);
    }

All thats left to do, is to now “new up” the async task in your activity and start it. Don’t forget to initialise them with the callback listener, so use this to create the tasks in your activity:

new FetchMyDataTask(this, new FetchMyDataTaskCompleteListener()).execute("InputString");

Thats it, it takes a little while to understand how this fits together, but its an incredibly flexible technique for tidying up your Async tasks, I’ve seen it used in a number of applications.

Thanks!

Tesseract OCR on Android is easier if you Maven-ise it, works on Windows too…

I’ve spent the past few months working on an android application that involves an element of OCR capability, its been quite a painful journey so this is my attempt to reflect on these experiences and hopefully help others who follow in my path.

From Wikipedia : In geometry, the tesseract, also called an 8-cell or regular octachoron or cubic prism, is the four-dimensional analog of the cube.

First off, lets just cover the basics. OCR stands for Optical Character Recognition, which is the process of taking an image, and being able to interpret the image and obtain textual data from it. For example, you take a photograph of a road sign, this would be an image file such as a JPEG. You can clearly see that the road sign says “Slow down”, however to a computer program, its just an image file. OCR enables the program to literally scan the image and find text, which your program can then use elsewhere in its workings.

After spending the best part of several days attempting to create a simple OCR application on android, and after suffering much frustration due to the lack of general resources available on the subject, I was able to find some good material by Gautam and rmthetis.

Whilst those guys have made a fantastic effort in their tutorials, I still needed to do further tasks to get an OCR demo off the ground, so hopefully this post will be a good supplement to their work.

The first misconception I wanted to clean up on is “this will only work on linux“, whilst the tutorials from the aforementioned authors are targetted against linux development environments, the above statement is not entirely true. Following their tutorials I was able to get it working first time on my Windows 7 development laptop. I also got this working on my new replacement Windows 7 laptop a week or two later without any troubles (and mc Macbook Pro too), all I did was follow these instructions which are on the github page for tess-two:

git clone git://github.com/rmtheis/tess-two tess
cd tess/tess-two
ndk-build
android update project --path . 
ant release

The above will clone the tess-two project by rmtheis, build the shared objects, update the project, then build the APK. I didn’t have any trouble doing this on a Windows environment. It is worth noting however, that the ndk-build can take time, I think on my first laptop it took around 20-30 minutes to complete, but it was a low spec machine.

Maven is Awesome with a Capital A

Most of the pain I experienced was integrating the libraries into my existing project. My project was already using maven so it made sense to attempt to package up the tess projects as libraries and depend on them as I would any other library, such as commons-lang (I’ve recently discovered that Google Guava is far better, but thats another topic).

I had quite a lot of trouble doing this at first, but it probably wasn’t aided by the fact I was moving my development environment from Windows 7 to Mac OS X Lion, attempting to (for the 3rd time) migrate from IntelliJ Idea to Eclipse (I gave up, Idea is still far superior and easier to use IMHO) and I was trying to ensure the IDE was happy with the project structure, which is no mean feat. I had a lot of problems trying to get m2eclipse to recognise the APKLIB packaging concept, even with the m2eclipse android connector I still struggled with plugin compatibility issues and general mis-understanding between the eclipse plugins and maven. It always compiled directly from the command line however.

Eventually, all it boiled down to, was including a pom.xml in both the tess-two and eyes-two root folders, which instructs maven to package them up as apklib files. As detailed on the android-maven-plugin website, the plugin is smart enough to know that when you request an apklib type project, it knows exactly where to find all the artefacts to include.

After following the above instructions that I’ve pulled from the tess-two github page, drop in this pom.xml into the root folder, tess/tess-two :

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>tess-two</groupId>
  <artifactId>tess-two</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>apklib</packaging>
  <dependencies>
        <dependency>
            <groupId>com.google.android</groupId>
            <artifactId>android</artifactId>
            <scope>provided</scope>
            <version>2.3.3</version>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <scope>test</scope>
            <version>4.8.2</version>
        </dependency>
    </dependencies>

    <build> 
        <sourceDirectory>src</sourceDirectory>
        
        <plugins>
            <plugin>
                <groupId>com.jayway.maven.plugins.android.generation2</groupId>
                <artifactId>android-maven-plugin</artifactId>
                <extensions>true</extensions>
                <version>3.2.0</version>
                <configuration>
                    <sdk>
                        <platform>10</platform>
                    </sdk>
                    <undeployBeforeDeploy>true</undeployBeforeDeploy>
                    <attachSources>true</attachSources>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

Drop into the command line, and run :

mvn clean install

This will then package up the tess-two application into an APKLIB file, and install it to your local maven repository. You can then depend on tess-two for any project you like, using the following dependency :

        <dependency>
            <groupId>tess-two</groupId>
            <artifactId>tess-two</artifactId>
            <version>0.0.1-SNAPSHOT</version>
            <type>apklib</type>
        </dependency>

Of course, feel free to change the group ID or version numbers as you see fit, just make sure they match up to the tess-two pom. You can also use the above approach for the eyes-two project, but remember that eyes-two is dependent on tess-two, so don’t forget the dependency!

You should then be able to start using the TessBaseAPI as Gautam covers in his post, this post here merely gets you going with maven. I’ve been in contact with Robert, chances are we’ll be getting this properly mavenised soon, so you can then depend on it from the central repositories rather than doing the installation mentioned here, we’ll keep you posted. (Robert, if you’re reading, get back to me mate!)

Hope this is of help, any questions then please ask, I’ll also be covering the iOS tesseract soon too, that was a little easier to get started.

Thanks

How to install any application on the emulator

I’ve been developing an app recently that relies on being able to navigate and browse for images on the phone, which is all well and good when testing on a device as chances are you’ll have something like Astro installed.

This isn’t quite the case when developing on the emulator, since you’ll have just the stock version of android, which by default doesn’t come with a file browser (or at least not one you can easily interrogate).

The easiest thing to do, is just install a file browser (like Astro) onto the emulator….but wait…there is no market on the emulator so how do you do that?

Easy.

  1. You’ll need the applications APK file, the easiest way to do this is to just install it onto your device from the marketplace
  2. Connect your device to your development machine so that it is accessible via ddms.
  3. In ddms, go Device -> File Explorer
  4. Look under /data/app and search for the application you want to pull from the phone, then click the “pull from device” at the top left corner
  5. Save the apk somewhere
  6. Unplug your device, and start up the emulator
  7. Type adb devices and you should see just the emulator attached
  8. Now type adb install
  9. Voila! Your APK is now installed on the emulator, rinse and repeat for other applications