Getting your head around the Couchbase SyncGateway

I like Couchbase. One of the things that really appeals to me is the sync gateway. As a mobile developer I often find that the apps I’m developing are just interfaces into some backend service. Somewhere out there in the cloud I’ll have a web application that sits on top of some database (nodejs/mongoDB is a combo I’ve been using recently). Then there comes the mobile app, which will be consuming these services, which would be fine if 4G/wifi was everywhere (I can’t even get a cellular signal at my place, let alone dream of 4G).

We’re into the realms of apps working offline, you then have the pain of syncing data and dealing with conflicts. You can make your life easier by using a SyncAdapter on Android, or perhaps a framework like Restkit if you’re developing on iOS, heck, you can even implement the syncing yourself (don’t do that, that road leads to madness..speaking from experience)…OR…you can just use Couchbase and the SyncGateway.

In short, the SyncGateway is an application that sits between your Couchbase server, and your Couchbase Lite enabled mobile apps. This means you can access your data on your local CBLite database, and not have to worry (too much) about syncing this to the Couchbase server.

Getting setup

I have to admit, the documentation is a little confusing when it comes to explaining how the components hang together, but after attending Couchbase Live in London a month or so back I was able to track down those who are in the know, and put the missing piece into my puzzle of confusion; bucket syncing.

For the purpose of explaining how this works, I’ll use my “Coin Collector” android app as the example. The app needs to get its data on coins from a couchbase server. It should be able to work offline and sync periodically. I’m using bucket syncing so I can have a web page to administer coins such as adding new coins to altering market values.
The documentation is really missing a diagram like the following

couchbase-sync

Let me cover the 4 points in blue numbers:

  1. Regardless of which mobile platform you’re using, it’ll be connecting to the sync gateway via the REST apis, this is where “json over the wire” comes into play.
  2. As the mobile apps use their own bucket, you need to configure the gateway to tell it where to put documents. If you check my config below; then this is done by the “aussie-coins-syncgw” configuration element, you can see that the bucket is set to “aussie-coins-bucket-sync-db” on the localhost couchbase server (sync and db are running on my local vm)
  3. This is where the magic happens. Bucket shadowing in the later releases of the Sync Gateway allow it to sync changes between your “mobile” bucket, and your “backend” bucket. You can see this configured by the “shadow” element in my config.json
  4. Your backend server apps can just connect to the “aussie-coins-bucket” and be totally oblivious to what is happening in the mobile side of your architecture.
{
    "interface": ":4984",
    "adminInterface": ":4985",
    "log": ["CRUD", "CRUD+", "HTTP", "HTTP+", "Access", "Cache", "Shadow", "Shadow+", "Changes", "Changes+"],
    "databases": {
        "aussie-coins-syncgw": {
            "server": "http://localhost:8091",
            "bucket": "aussie-coins-bucket-sync-db",
            "sync": `function(doc) {channel(doc.channels);}`,
            "users": {
                "GUEST": {
                    "disabled": false,
                    "admin_channels": ["*"]
                }
            },
            "shadow": {
                 "server": "http://localhost:8091",
                 "bucket": "aussie-coins-bucket"
            }
        }
    }
}

Some other points to notice in the configuration:

  • The interface port is the port the apps will connect on, the adminInterface is for administering the sync gateway, such as dynamically adding new databases, or altering channels.
  • Logs, I’ve chosen to log everything, you can restrict these if you need, check the Couchbase documentation for further info.
  • I’ve enabled the guest user access on all channels for the purpose of evaluating this, ideally we’d need to restrict the channels that users can use to stop any potential abuse.

Testing it out

As I mentioned above, since the mobile apps will be connecting to the Sync Gateway via a REST api, we can take the mobile app out of the picture and test using a rest client (I’m using Postman for Google Chrome). Lets cover 2 scenarios.

Server Producing

This scenario involves a new document being created on the server, and it being synced to the mobile bucket and available to view on the mobile apps.

Firstly, let me show you what I have in the “aussie-coins-bucket”.

1-couchbase-coins_bucket

Next, lets create a new document with an ID of 5, for the Ten Cent coin. We should then see it listed in our “aussie-coins-bucket” like so:

2-couchbase-coins_bucket_new_coin

Now lets have a look at the log output from the Sync Gateway.

22:49:33.826838 Shadow+: Pulling "5", CAS=1e2dd7153a ... have UpstreamRev="", UpstreamCAS=0
22:49:33.826894 Shadow: Pulling "5", CAS=1e2dd7153a --> rev "1-1d7a1a352c0abb293fdd16883ef6985b"
22:49:33.826909 CRUD+: Invoking sync on doc "5" rev 1-1d7a1a352c0abb293fdd16883ef6985b
22:49:33.903707 Cache: SAVING #8
22:49:33.903984 CRUD: Stored doc "5" / "1-1d7a1a352c0abb293fdd16883ef6985b"
22:49:34.768280 Cache: Received #8 after 864ms ("5" / "1-1d7a1a352c0abb293fdd16883ef6985b")
22:49:34.768305 Cache:     #8 ==> channel "*"
22:49:34.768322 Changes+: Notifying that "aussie-coins-bucket-sync-db" changed (keys="{*}") count=3
22:49:59.849578 Shadow+: Pulling "5", CAS=2423d4b93a ... have UpstreamRev="1-1d7a1a352c0abb293fdd16883ef6985b", UpstreamCAS=c21019dd68
22:49:59.849623 Shadow: Pulling "5", CAS=2423d4b93a --> rev "2-971b4b3009127da5ed2a4770cb45cfe7"
22:49:59.849637 CRUD+: Invoking sync on doc "5" rev 2-971b4b3009127da5ed2a4770cb45cfe7
22:49:59.849749 CRUD+: Saving old revision "5" / "1-1d7a1a352c0abb293fdd16883ef6985b" (68 bytes)
22:49:59.849891 CRUD+: Backed up obsolete rev "5"/"1-1d7a1a352c0abb293fdd16883ef6985b"
22:49:59.850068 Cache: SAVING #9
22:49:59.850207 CRUD: Stored doc "5" / "2-971b4b3009127da5ed2a4770cb45cfe7"
22:50:00.790818 Cache: Received #9 after 940ms ("5" / "2-971b4b3009127da5ed2a4770cb45cfe7")
22:50:00.790838 Cache:     #9 ==> channel "*"
22:50:00.790868 Changes+: Notifying that "aussie-coins-bucket-sync-db" changed (keys="{*}") count=4

As we can see, the Sync Gateway has detected that there is a new document and that it needs to shadow it across, which is does successfully.

On the couchbase server, we can view that document in the mobile bucket, “aussie-coins-sync-db” like so:

3-couchbase_synced_coin

Finally, just to prove the mobile clients can see that document via the API, do a GET on http://localhost:4984/aussie-coins-syncgw/5 and you’ll see the following:

{
    "_id": "5",
    "_rev": "2-971b4b3009127da5ed2a4770cb45cfe7",
    "coin": "Ten Cent"
}

Mobile Producer

Now we’ll try the opposite, producing documents from the mobile clients and seeing them synced across to the Couchbase server. From a REST client, do a PUT to http://localhost:4984/aussie-coins-syncgw/6 with a json body of:

{
  "coin":"Twenty Cent"
}

You should see a response of

{
    "id": "6",
    "ok": true,
    "rev": "1-e9c16d3887a3958314adff1e3cbd6097"
}

What we’ve done is to create a document with the ID of 6, for “Twenty Cent”.

Lets have a look at the Sync Gateway logs:

23:19:56.860618 HTTP:  #003: PUT /aussie-coins-syncgw/6
23:19:56.971056 CRUD+: Invoking sync on doc "6" rev 1-e9c16d3887a3958314adff1e3cbd6097
23:19:57.023839 Cache: SAVING #10
23:19:57.024110 CRUD: Stored doc "6" / "1-e9c16d3887a3958314adff1e3cbd6097"
23:19:57.024161 HTTP+: #003:     --> 201   (0.0 ms)
23:19:57.616316 Cache: Received #10 after 592ms ("6" / "1-e9c16d3887a3958314adff1e3cbd6097")
23:19:57.616340 Cache:     #10 ==> channel "*"
23:19:57.616353 Shadow: Pushing "6", rev "1-e9c16d3887a3958314adff1e3cbd6097"
23:19:57.616367 Changes+: Notifying that "aussie-coins-bucket-sync-db" changed (keys="{*}") count=6
23:19:57.852304 Shadow+: Pulling "6", CAS=1c6f07c3ce2 ... have UpstreamRev="", UpstreamCAS=0
23:19:57.852327 Shadow+: Not pulling "6", CAS=1c6f07c3ce2 (echo of rev "1-e9c16d3887a3958314adff1e3cbd6097")
23:19:57.852337 CRUD+: Invoking sync on doc "6" rev 1-e9c16d3887a3958314adff1e3cbd6097
23:19:57.865669 CRUD+: updateDoc("6"): Rev "1-e9c16d3887a3958314adff1e3cbd6097" leaves "1-e9c16d3887a3958314adff1e3cbd6097" still current
23:19:57.865751 Cache: SAVING #11
23:19:57.866050 CRUD: Stored doc "6" / "1-e9c16d3887a3958314adff1e3cbd6097"
23:19:58.617446 Cache: Received #11 after 751ms ("6" / "1-e9c16d3887a3958314adff1e3cbd6097")
23:19:58.617463 Cache:     #11 ==> channel "*"
23:19:58.617482 Changes+: Notifying that "aussie-coins-bucket-sync-db" changed (keys="{*}") count=7

We can then see the document in the “aussie-coins-bucket-sync-db”:

4-couchbase_syncdb

…and then in the “aussie-coins-bucket”:

The Sync Gateway is a useful application, and really does make the Couchbase offering even more appealing. Once you can get your head around the bucket shadowing (which you should if you’ve made it this far) then it can be easy to work with.

Comment or find me on Twitter (@jameselsey1986) if you have any questions!

Why the duplication?

Having 2 buckets for the same data had me raise an eyebrow initially, but after asking on Google Groups, it does make sense. You can’t expect the backend app servers to maintain sync meta data on new documents it creates. Perhaps Couchbase will alter this in the future.

Resources

Grails based survey system, the android app

Some time back I wrote an article describing the roosearch system I developed using grails. This is the second part, the android client, please checkout the previous article otherwise this might not make much sense!

After completing the grails component, I had a RESTful API available to me, and I just needed to build an app that could consume those services.

Customer lookup and QR codes

The app needs to be simple and quick to use, one of the things I remember from a UX discussion at DroidCon UK is “Don’t annoy your users, they control your app ratings and your income!”. In order to lookup the surveys quickly, I’ve added the ability to scan QR codes. Actually I didn’t have to do a great deal as there is already an app called ZXing by Google that scans QR codes, so I just needed to make Roosearch delegate to ZXing and handle the result.

Of course, we don’t want to exclude users that don’t have ZXing, or even a camera on their device, so I’ve also provided a text field where they can enter the customer Id manually if required.

When the user clicks on the “scan barcode” button, I first check if ZXing is installed using the following

    public void scanBarCode(View v) {
        final boolean scanAvailable = isIntentAvailable(this,
                "com.google.zxing.client.android.SCAN");
        if (!scanAvailable){
            Toast.makeText(this, "You need to install the ZXing barcode app to use this feature", Toast.LENGTH_SHORT).show();
            return;
        }

        Intent intent = new Intent("com.google.zxing.client.android.SCAN");
        intent.putExtra("SCAN_MODE", "QR_CODE_MODE");
        startActivityForResult(intent, 0);
    }

If the user does have ZXing installed on their device, and choose to use it, we can get the result back from the bar code scan using:

public void onActivityResult(int requestCode, int resultCode, Intent intent) {
        if (requestCode == 0) {
            if (resultCode == RESULT_OK) {
                String contents = intent.getStringExtra("SCAN_RESULT");
                performRooLookup(contents);
            } else if (resultCode == RESULT_CANCELED) {
                // Handle cancel
            }
        }
    }

    private void performRooLookup(String rooId) {
        if (StringUtils.isBlank(rooId)) {
            Toast.makeText(this, "Please enter a valid customer id", Toast.LENGTH_SHORT).show();
            return;
        }

        Integer customerId;
        try {
            customerId = Integer.parseInt(rooId);
        } catch (NumberFormatException e) {
            Toast.makeText(this, "Customer id needs to be numeric", Toast.LENGTH_SHORT).show();
            return;
        }
        new FindRooTask(this, new FindRooTaskCompleteListener()).execute(customerId);
    }

I then have the following buried in a service call, invoked by an AsyncTask, which handles finding Customer details:

    public Customer getCustomerDetails(int customerId) {

        try {
            final String url = "http://roosearchdev.jameselsey.cloudbees.net/api/customer/{query}";

            HttpHeaders requestHeaders = new HttpHeaders();

            // Create a new RestTemplate instance
            RestTemplate restTemplate = new RestTemplate();
            restTemplate.getMessageConverters().add(new MappingJacksonHttpMessageConverter());

            // Perform the HTTP GET request
            ResponseEntity<Customer> response = restTemplate.exchange(url, HttpMethod.GET,
                    new HttpEntity<Object>(requestHeaders), Customer.class, customerId);

            return response.getBody();
        } catch (Exception e) {
            System.out.println("Oops, got an error retrieving from server.. + e");
        }
         return null;
    }

A Customer looks like this:

public class Customer implements Parcelable {

    @JsonProperty("company_name")
    private String companyName;
    private String twitter;
    private String facebook;
    private List<SurveySummary> surveys = new ArrayList<SurveySummary>();
    //Accessors omitted
}

The SurveySummary just has a title and Id. The reason for just returning summaries is because a customer may have many surveys, and there is no need to obtain them all, we just obtain the title to display to the user, if selected, we’ll retrieve the survey by its id.

To recap, here are 2 screenshots that show the above; the landing screen, and then the customer display screen

Landing screen for Roosearch, where the user can enter a customer Id or scan a QR code

Landing screen for Roosearch, where the user can enter a customer Id or scan a QR code

Customer screen, display social media links, name, photo, and list of surveys that the customer has

Customer screen, display social media links, name, photo, and list of surveys that the customer has

 

The survey engine

This is where the magic happens. I have a single activity and single view that handles presenting the survey to the user. As the surveys can change number of questions, and number of responses, I needed a way of dynamically traversing the survey object and allowing user to move between the questions whilst retaining state of what they have selected so far.

I’ve created the following method that will redraw the layout for a given question id:

    public void drawQuestionOnScreen(int id) {
        TextView question = (TextView) findViewById(R.id.question);
        question.setText(s.getQuestion(id - 1).getText());   // subtract 1 as lists are indexed from 0

        LinearLayout linLay = (LinearLayout) findViewById(R.id.answers);
        linLay.removeAllViews();
        RadioGroup rg = new RadioGroup(this);
        rg.setId(1);
        for (int aIndex = 0; aIndex < s.getQuestion(id - 1).getResponses().size(); aIndex++) {
            Answer a = s.getQuestion(id - 1).getAvailableOption(aIndex);
            RadioButton button = new RadioButton(this);
            button.setText(a.getText());
            button.setTextColor(R.color.dark_text_color);
            button.setId(aIndex);
            rg.addView(button);
        }
        linLay.addView(rg);

        TextView status = (TextView) findViewById(R.id.status);
        status.setText(format("%d of %d", id, s.getQuestionCount()));
    }

As you can see, it will retrieve the question by Id, then iterate over the responses and generate RadioButtons. Moving to the next question is reasonably easy, firstly I work out if an option has been selected, and prevent moving on if not. After that, I mark the selected response in the survey object, and then work out if there is another question in the sequence to display, if not we can progress to the finish.

One of the questions in the given survey

One of the questions in the given survey

    public void next(View v) {
        RadioGroup rg = (RadioGroup) findViewById(1);

        int selectedRadioId = rg.getCheckedRadioButtonId();
        if(selectedRadioId == -1){
            Toast.makeText(this, "Please select a response", Toast.LENGTH_SHORT).show();
            return;
        }

        s.getQuestion(questionIndex - 1).getResponses().get(selectedRadioId).setSelected(true);
        // work out if there is another question, then move to it
        if (s.getQuestionCount() > 1 && questionIndex < s.getQuestionCount()) {
            questionIndex++;
            drawQuestionOnScreen(questionIndex);
        } else {
            // if there are no other questions, show dialog saying submit or not
            Toast.makeText(this, "Reached the end of the survey", Toast.LENGTH_SHORT).show();
            // HERE we should process the entire survey, crunch data and post off (maybe async)

            Intent i = new Intent(this, SurveyComplete.class);
            i.putExtra("com.roosearch.domain.Survey", s);
            startActivity(i);
        }
    }

A similar approach is needed for moving back to previous questions, determine if there is a previous question to move to then redraw the screen, like so:

    public void previous(View v) {
        // work out if there is a previous question, and if so move to it
        if (s.getQuestionCount() > 1 && questionIndex > 1) {
            questionIndex--;
            drawQuestionOnScreen(questionIndex);
        } else {
            //if there are no other questions, move back to home screen, finish() this and scrap any progress
            finish();
        }
    }

Once the user completes all questions, the SurveyComplete activity is invoked.

Completing a survey

When the user has completed all questions, the survey object is passed into the SurveyComplete activity, which handles sending the responses back to the grails web application.

@Override
    protected void onResume()
    {
        super.onResume();
        TextView tv = (TextView) findViewById(R.id.completeMessage);
        tv.setText("Thank you for taking the time to complete the survey");
        tv.setTextColor(R.color.dark_text_color);

        Survey s = getIntent().getExtras().getParcelable("com.roosearch.domain.Survey");

        if (s != null)
        {
            StringBuffer sb = new StringBuffer();
            sb.append("\n" + s.getTitle() + "\n");
            for (Question q : s.getQuestions())
            {
                sb.append("\nQ: " + q.getText());
                sb.append("\nA: " + q.getSelectedAnswer() + "\n");
            }
            tv.append("\n\n" + sb.toString());
        }

        new SurveyUploadTask(this, new SurveyUploadTaskCompleteListener()).execute(s);
    }

    public class SurveyUploadTaskCompleteListener implements AsyncTaskCompleteListener<Void> {
        @Override
        public void onTaskComplete(Void voidz) {
            Toast.makeText(SurveyComplete.this, "Survey uploaded", Toast.LENGTH_SHORT).show();
        }
    }

The activity uses an AsyncTask to post the data back to the grails API controller, and displays a toast when successful.

Survey completed, results uploaded, and summary presented to user

Survey completed, results uploaded, and summary presented to user

 

Wrapping it up

Overall quite a simple app, I spent probably around 2 or 3 weekends putting together, most of that time was spent getting to grips with some automated testing for android. The code is admittedly a little rough around the edges, but I was aiming for an MVP (most viable product) to get working, feel free to contribute or suggest improvements!

I chose to use maven, but would use gradle if I were to pick this up again. Be sure to check out the code on github and try running it against Roosearch web, it does work!

Click here for the source code on Github