Saturday, October 30, 2010

From homegrown JSON protocol to JSON-RPC

As I discussed in the previous post, JSON now starts to become fashionable, mostly because of its more compact encoding compared to XML. If we are commited to communicate over JSON with the server, the need for the most common client-server communication pattern, Remote Procedure Call (RPC) arise.

JSON has such a solution, called JSON-RPC. JSON-RPC is a lightweight client-server protocol. Beside the usual stuff (remote method identification, parameter and result encoding, exceptions) it has some remarkable properties that make it particularly suitable for mobile communication.

There are two JSON-RPC specifications out there, JSON-RPC 1.0 and 2.0. While 2.0 in most aspects can be seen as a natural extension of 1.0, there are differences too. JSON-RPC 1.0 was designed to be peer-to-peer with the main transport protocol being TCP. HTTP was also supported but with limitations. JSON-RPC 2.0 is explicitly client-server even though the change is only in the terminology used in the specification. The client-server nature of JSON 2.0 ensures, however, that HTTP as a transport is more naturally supported.

Let's start with the basics.

This is a JSON 1.0 request:

{"method":"add","params":[3,4],"id":0}

"method" attribute specifies the remote method, "params" attribute has a value of an array with the RPC arguments. The order matters and has to be the same as the arguments on the remote side. JSON-RPC requests and responses are connected with the "id" attribute because requests and responses does not have to be ordered. The response following a request - or over a full duplex bearer like TCP arriving at any time - does not have to be the response to the request just sent. It may be a response to an earlier request and the "id" parameter guarantees that requests can be coupled with their responses.

This is a JSON 1.0 response:

{"error": null, "result": 5, "id": 0}

"result" is the return value of the RPC call, "id" is the same as the corresponding request and "error" carries the error object if there was an error during the RPC processing. RPC 1.0 does not define the error value beside stating that it is an object.

JSON-RPC 2.0 extends JSON-RPC 1.0 in many important ways.
  • In order to provide backward compatibility, JSON-RPC 2.0 requests and responses must all have "jsonrpc": "2.0" attribute.
  • Error object is now specified. It has "code", "message" and "data" attributes (more about that in the specification).
  • It specifies oneway messages that JSON-RPC 2.0 calls notifications. Notifications are requests without "id" attribute. No response can be sent to notifications.
  • There are batched requests and responses. Batch is formed by having an array as top-level element and adding JSON-RPC request or response objects tothis array. E.g. the following is a batched request:
[{"method":"add","params":[2,3],"id":0,"jsonrpc":"2.0"},
{"method":"mul","params":[4,5],"id":1,"jsonrpc":"2.0"}]


and the following is a batched response:

[{"error": null, "jsonrpc": "2.0", "result": 5, "id": 0},
{"error": null, "jsonrpc": "2.0", "result": 20, "id": 1}]


The following is a JSON-RPC 2.0 error response:

{"jsonrpc": "2.0","error": {"message": "ServiceRequestNotTranslatable", "code": 1000}, "result": null, "id": 12}

That was a lot of explanations, let's see something that works!

Click here to download the example program.

This is the same client-server batch calculator that you have seen in the previous post. Use the deployment instructions of the previous post to try out the example program. If you want to try the example on a real phone, I recommend deploying the server part on the real Google App Engine infrastructure as described in the previous post.

The example program uses JSON-RPC 2.0 as communication protocol between the client and server instead of our homegrown protocol. The biggest change is on the server side, which is implemented as a Python application for Google App Engine. Instead of obscure parsing code, you see remote methods defined like this:

@ServiceMethod
def add(self, v1,v2):
return v1+v2

and the rest is done by the server-side Python JSON-RPC 2.0 framework which I hacked out of this JSON-RPC 1.0 implementation. The client side does not use any framework at all which is a major problem with this example program. So far I have not been able to find any decent JSON-RPC 2.0 client-side framework suitable for Android. If you have an idea, please comment!

Thursday, October 21, 2010

Client-server communication with JSON

A long time ago, XML was the emerging, fashionable technology. Those times have clearly passed and nowadays XML is just another boring piece of basic technology. If you want to be trendy, you have to go for something else. Luckily, there is a new in thing, JavaScript Object Notation, JSON. In this post, I show you a simple client-server application based on Android, JSON and Google App Engine.

JSON is based on JavaScript object literals and is a textual representation of complex data instances. It has 4 basic types: strings, numbers, booleans and null. Objects and arrays can be constructed out of these 4 types, object is really just the JSON slang for key-value pairs. The entire thing is specified in RFC 4627 so I stick to just some simple examples so that you have that JSON feeling if you don't want to read the RFC.

This is a JSON number: -4.7e12
This is another JSON number: 0
This is NOT a JSON number: 00012 (no leading zeros!)
This is a JSON string: "JSON"
This is another JSON string: "\u005C" (always 4 hexa digits after the \u, just one character, the \)
This is not a JSON string: 'notJSON'
These are JSON literals: true, false, null
This is a JSON array: [1,"hello",null]
This is a JSON array having two other JSON arrays as elements: [[1,"hello"],[2,"hallo"]]
This is a JSON object: {"a":1,"b":"hello"}
This is a JSON object having another object associated with "a" key and an array with "b" key:
{"a":{"aa":1,"bb":2},"b":[3,4]}

I think this is enough for starter because it is evident that JSON has quite powerful data structures. Arrays are equivalent of Lisp lists while objects can be represented as lists of dotted pairs.

How does JSON compare to the incumbent serialization standard, XML? JSON is fashionable in mobile communication because it is somewhat more bandwidth-efficient than XML. E.g. the last example can be written in (ugly) XML like this (XML tag mangling due to blog engine limitations):

[l]
[a]

[aa]1[/aa]

[bb]2[/bb]

[/a]

[b]

[k]3[/k]

[k]4[/k]
[/b]
[/l]

Not counting the white spaces, this is 58 characters while the JSON representation is just 32 characters, about 45% less. The situation is worse if we consider many "well-designed" XMLs out there (like e.g. SOAP documents) which are full of namespace declarations, long tag names, etc. JSON with its simple yet powerful syntax definitely has a space in mobile application design.

Click here to download the example program.

I will demonstrate the power of JSON with a simple client-server example application. This application is a batch-mode calculator. The user first enters operations on the device's UI. These operations are not calculated immediately. When the user selects the "Process list" menu item, the collected operations are sent to the server in one batch. The server calculates the operations that are sent back to the client which displays the results. As you might have guessed, the client and the server communicates with JSON data structures.



The client part needs to be deployed on Android emulator (let's start with the emulator, I will return to this) like usual. The server part, however, requires the Google App Engine SDK for Python. Download the SDK, edit server/ae_jsoncalcserver.sh so that it suits your directory layout then start the shell script (Windows users will have to turn this one-line shell script into a .bat file). If you access localhost:8080 with your browser, you get a short message that GET is not supported. If you get there, you installed the server part properly.

Now you can start the client on the emulator and you can start to play with it. The most interesting part is the process() method in JsonCalc.java. I structured the program so that the generation of JSON request and the parsing of the response is all there. The org.json package has been part of Android from the beginning and this package is really easy to use. For the server part, I used the simplejson JSON parser for Python.

The Android client logs the JSON requests and responses in the Android log (adb logcat). I represented the batch operation and its result as array of objects. Request and response objects have different keys.

Request example:
[{"op":2,"v1":2,"id":1,"v2":3.3},{"op":3,"v1":4,"id":2,"v2":6},{"op":4,"v1":6.6,"id":3,"v2":2.2}]

Response example:
[{"id": 1, "result": -1.2999999999999998}, {"id": 2, "result": 24}, {"id": 3, "result": 2.9999999999999996}]


In case you want to try it out on a real phone, you'd better deploy the server part on Google infrastructure. Then edit Config.java in the client source so that APP_BASE_URI points to your server's address and you are ready to go. When you recompile, make sure that you delete the client/bin/classes directory - chances are that javac will not notice your changes you made to Config.java because of its broken dependency resolution.

Monday, October 11, 2010

Push service from Google

Update: the first version of this post contained factual errors. Thanks to Tejas for pointing these out, you can find his experiences with C2DM here. The example program was also updated to fix these errors.

Android 2.2's most important improvement is without doubt the significantly faster virtual machine. The API, however, got some new features too. For me, the most intriguing new feature is the access to Google's experimental push services.

I have already posted about push feature in relation to iBus//Mobile asynchronous communication package so push is not new to Android. There are also open source alternatives like MQTT, Deacon or Urban Airship. It has also been present in certain Google applications from the beginning. Whenever Android Gmail application sends a notification when a new mail arrives on the server, you see push in operation. It is pretty fast, according to my experiences, even though time to time it takes longer time for the notification to arrive. 2.2 opened that already existing mechanism to general-purpose applications, even though the push service is still in early beta.

In order to do push, either one has proper push bearer (a network mechanism able to deliver unsolicited messages to the device) or such mechanism is simulated. Currently only two real push bearers are deployed widely on mobile networks, SMS and Blackberry's proprietary push solution. SMS is costly and the Blackberry solution is available only for Blackberry so the push bearer has to be simulated. The most common simulation method relies on the device to open a TCP connection to the push server. As the TCP stream is full-duplex, the server can send the push message to the device, provided that the stream is still alive. And that's the tricky bit whose complexity should not be underestimated. TCP streams time out if there is no communication on them and the other side does not necessarily notice it. A constant ping-pong traffic needs to be generated to prevent this. Frequency, however, is critical. Too frequent ping-pongs and the data cost associated to ping-ponging will be unacceptable and the battery is drawn down quickly. Too rare ping-pongs and the device or the server will not notice that the TCP stream was closed, only after a long timeout. For the user, it means that he did not get his urgent message immediately, only after, say, an hour.

There is no perfect solution to this problem, therefore it is a good feeling that Google made its best to create a simulated push bearer, along with the server that is able to keep that many TCP sockets open and shared that infrastructure with us. The Google push architecture has the following elements.

  • The device that authenticates and registers with the push server (Cloud to Device Messaging in Google lingo) and provides authentication ticket to the 3rd Party Application Server.
  • 3rd Party Application Server that uses the authentication ticket generated by the device to send push requests to the push server.
  • Push server provided by Google that authenticates push requests from 3rd Party Application Servers and delivers the messages to the devices.
Click here to download the example program.

Cloud to Device Messaging (C2DM) has many tricky aspects but I wanted to create a test program that is as simple as possible. To try it out, you need the following:

  • A PC connected to the Internet so that Google C2DM servers can be reached (and C2DM servers can reach us).
  • Android emulator with 2.2+Google API AVD created (API level 8).
  • Google App Engine SDK for Python.
  • A Google account that you register with Google for push. It is better not to use your real e-mail address because you have to insert the password for this account into the server script. Use aexp.push as package name for the application in the registration form.
Do the following.
  • Unpack the download package. Start the Android emulator. Create a Google account if no such account exists (Settings/Accounts & sync). This account does not have to be the same that you registered for push, it can be any Google account that you know the password for.
  • Enter the "client" directory in the download package, update client/src/aexp/push/Config.java with the Google account you registered for push (C2DM_SENDER)
  • Open server/pushserver/pushserver.py and update it with your push Google account (lines 165 and 167). Unfortunately here you have to insert the password for your push Google account into the server script.
  • Enter the "server" directory, customize "ae_pushserver.sh" according to your directory layout and start it. Google App Engine SDK starts.
  • Start the Push application on the emulator, select the account using the Menu and wait for the "Registered" message to appear. Now the server application is ready to deliver push messages.
  • Go to http://localhost:8080 with your browser and type a message. Select the account from the drop-down list that you configured on the device and click the submit button. The message should appear on the emulator screen.


If you want to do the same trick from a real phone, you need a server with fix IP address or a fix DNS name and run the App Engine SDK there. Or better, deploy it on the real thing, on Google infrastructure. In any case, make sure that you update the server address in client/src/aexp/push/Config.java (and delete client/bin/classes directory as there are occassionally troubles with javac's dependency resolution).

About the application. The application reuses c2dm utility library from the JumpNote sample application. When the account is selected, the application registers with the push server using the phone user's credentials and the application ID that you registered with Google. Then comes the interesting bit. After the application registers with the push server, it sends the registration ID to the application server. The application server will use the registration ID to talk to the push server when it wants to do push. The server uses the Google account that you selected for push for authentication and therefore it needs the password for this account.

The implementation in the example application server program is a not efficient as it always requests an authentication token before it sends the push message. In an efficient implementation the token can be requested only once and can be used to authenticate many messages. Eventually the token expires and only then should new authentication token requested.

The client application also sends the client account name to the server but it is not used for authentication, it is only needed so that you can select the push target by account name in the web application's drop-down menu.