Tuesday, January 29, 2008

Hacking into Android's Bluetooth support

I have been mentioning that wicked Bluetooth service for two posts but eventually something else was more interesting. Well, finding something more interesting was easy because it turned out pretty soon that the Bluetooth support in m3-rc37a version of the emulator is half-ready at best. I cannot be sure but it seems that the hcid daemon on which Android's Bluez port is based is completely missing or at least is not launched properly along with other system services.

Is there a point then to deal with Bluetooth given the current state of the emulator? IMHO, yes, because when the shiny new version of the emulator comes out, applications will be ready to run. At the present state of affairs this approach means that we have to live with a bit (lot!) of hacking.

For the moment, I was interested in the discovery process. Discovery in the present implementation does not work but that did not prevent me simulating it. By taking apart org.bluez.IBluetoothService and android.server.BluetoothService classes with the jad tool, it quickly became evident that the discovery process is based on 4 intent messages.
  • org.bluez.intent.action.DISCOVERY_STARTED - delivered when the discovery process starts.
  • org.bluez.intent.action.REMOTE_DEVICE_FOUND - delivered when one device is discovered. It has 3 parameters in the Intent's Bundle: "address" (String) is the Bluetooth address of the device, "class" (Integer) is the class of the Bluetooth device and "rssi" (Short) is the Received Signal Strength Indicator.
  • org.bluez.intent.action.REMOTE_NAME_UPDATED - delivered when the user-friendly name of one device is discovered. This message follows an earlier REMOTE_DEVICE_FOUND message and carries the user-friendly name ("name" (String)) along with the Bluetooth address ("address" (String)).
  • org.bluez.intent.action.DISCOVERY_COMPLETED - delivered when the discovery process finishes.
You can download the example program from here.

The discovery simulator itself is an interesting piece of software because of some common Android programming tricks it employs. The discovery simulator is located in the BtDetectSimulatorService class that the main activity launches with a startService call (it is therefore the other type of service, not the one seen at the service invocation entry). Normally, the discovery would just be started with a method call on org.bluez.IBluetoothService but the implementation is not that ready. The service abuses the half-implemented Bluetooth classes (like the android.server.BluetoothService class, don't ever do what I did here!), starts the Bluetooth service and sends out intent messages. These intent messages are captured by an intent receiver (BtIntentReceiver) and are used to update the UI of the main activity.

Two small details are interesting here. The intent receiver and the main activity call each other like ordinary Java objects. The code exploits the fact that the main activity is a singleton and just connects the activity and the intent receiver instances by Java references. This is possible because these two entities are part of the same package, hence they run in the same VM instance. Here is the proof (619 is the PID of the VM process):

D/BTINTENTRECEIVER( 619): org.bluez.intent.action.DISCOVERY_STARTED
D/BTDETECT( 619): Action = org.bluez.intent.action.DISCOVERY_STARTED


The other interesting bit is the way BtDetectSimulatorService delivers the events. There is a delay among the intent broadcasts (Bluetooth discovery may be a lengthy process). This delay could not be coded with the good old Thread.sleep call in onStart method because onStart would have run too long and the Android application manager would kill the process. In addition, no broadcasts would be delivered until onStart ends which means that all the broadcasts would be delivered at once. Instead, observe how an android.os.Handler instance is used to schedule Runnable tasks after a certain delay.

And at last, the fun part. The image below shows, how Android's own Phone application crashes terribly when our application stops the Bluetooth service (it was us who started it, so we stop it nicely). That characterizes pretty much the readiness of the Bluetooth support in the current emulator.

Saturday, January 26, 2008

About binders

I wanted to go back to my Bluetooth exercise once again but then I stumbled into this beautiful sample program from John Guilfoyle. I decided immediately that before playing again with that wicked Bluetooth service, I have to test-drive again the service framework and the low-level mechanisms behind it.

My Android experiences with services were so far restricted to repeating the sample program in the documentation. This time I wanted to try something more ambitious: connecting two applications through services. In order to do that, it is worth getting behind the facade of the Android service framework. This thread contains a lot of information that I try to recap here.

The workhorse of the Android IPC is the "IBinder class". As its name implies, android.os.IBinder is really an interface, implemented by the real worker objects, namely android.os.BinderNative and android.os.BinderProxy. The binder provides a simple functionality of synchronous method invocation. Calling the transact() method on the binder is able to transmit a function code, a java.os.Parcel instance and some flags. The result is returned in another Parcel instance. Look for this method signature:

public final boolean transact(int code, Parcel data, Parcel reply, int flags ) throws DeadObjectException;

The client calls the transact method and the other side of the binder, in the other side of the process separation will receive a callback.

protected boolean onTransact(int code, Parcel data, Parcel reply, int flags);
The IPC has a thread pool for each process and the onTransact method is invoked in the context of a thread in the IPC thread pool. Parcel is the abstraction of the data the IPC framework is able to serialize and deserialize, you can look at the documentation of the android.os.Parcel object to see, what data types are supported. The most interesting data type is the IBinder itself (readStrongBinder, writeStrongBinder) so it is possible to pass binder endpoints to another processes. The interface mechanism that the aidl tool provides is really simple after that. Aidl takes the interface description and turns it into Java code that serializes and deserializes invocation parameters and return values to and from Parcels. Take for example our trusty IAdderService which has a method with this signature:

int add( in int i1, in int i2 );

Aidl has to generate a code that serializes two integers into the request Parcel and deserializes the return integer from the response Parcel in transact() because this is the client side of the stub. In onTransact method, as it is the receiver side of the call, it has to deserialize the input parameters, invoke the service-side implementation of the method then serialize the result. And it does exactly that (excerpts from the IAdderService.java generated by the aidl tool):

public int add(int i1, int i2) throws android.os.DeadObjectException { android.os.Parcel _data = android.os.Parcel.obtain();
android.os.Parcel _reply = android.os.Parcel.obtain();

int _result;

try {

_data.writeInt(i1);

_data.writeInt(i2);

mRemote.transact(Stub.TRANSACTION_add, _data, _reply, 0);

_result = _reply.readInt();

}

...

public boolean onTransact(int code, android.os.Parcel data, android.os.Parcel reply, int flags) {
try {
switch (code)

{

case TRANSACTION_add:

{
int _arg0;
_arg0 = data.readInt();

int _arg1;

_arg1 = data.readInt();

int _result = this.add(_arg0, _arg1);

reply.writeInt(_result);
return true;
}

}

The service framework does not provide any means to find out, what is the serialization and deserialization stub on the client and the service side which I consider a drawback. This is a reason why it is impossible to discover the service interfaces (one can retrieve the client-side binder but not the interface stub, therefore it is impossible to know, how to talk to the service without knowing the interface stub from some other, independent source). The framework simply assumes that the client- and the server-side stubs match, if not, then it is the programmer's fault.

I was now ready to realize my dream to connect two applications by means of service invocations. The best way is, of course, Intent-based invocation but I wanted to work that around just for the sake of hacking. As Dianne Hackbod pointed out, however, Android application lifecycle makes the direct connection impossible. Depending on resource situation, Android apps just come and go and therefore the binders connecting them may stop functioning without any warning. One has to go through services because service dependencies are respected by the application manager. So I created the following simple application (actually, two applications and one service).

You can download the example program from here.

The task is to receive a callback from Application 2 (called SClientApp) by Application 1 (called HostappClient). The connection between the two applications goes through a service (HostappServiceImpl, co-located with HostappClient). Both applications bind this service in a usual way. In addition to that, HostappClient provides the HostappServiceImpl with a callback binder. Why binder, why not interface? It could be interface and it would be more elegant to do so. The Windows version of the aidl tool in the current version (m3-rc37a) has an annoying bug that makes the import statement almost unusable so there is no way to import our own interface classes. This is not a problem, however, if we know the relationship of binders and interfaces. We just pass the binder reference and apply the interface stubs onto it when the binder arrives. The call flow is the following:
  • HostappClient binds HostappServiceImpl. When the service binding callback arrives, we pass the binder of HostAppClient's ISetTextService to HostappServiceImpl.
  • Now HostappClient launches SClientApp as subActivity. The first thing SClientApp does is to bind HostAppServiceImpl. When the service is bound succesfully, SClientApp passes the hugely valuable text result to HostAppServiceImpl which in turn calls HostappClient's ISetTextService whose binder we have received previously.
  • One interesting bit remained: the use of android.os.Handler in HostappClient. When HostappClient receives the service callback, it cannot directly manipulate the UI, only the UI thread is allowed to do that. The service callback runs in an IPC thread, not the UI thread, therefore it uses a Handler instance to pass a Runnable to the UI thread which is then executed there (the Handler executes the Runnable in the context of the thread it was created in and our Handler instance was created in the context of the UI thread).

Huh, pretty complicated instead of returning that hugely important string in the intent invocation bundle? And that is not the end, because our example program is not perfect. It is not prepared for low-memory situations when each of our three players may be destroyed and resurrected by the application manager. The worst-case scenario is this:
  • HostappClient is launched and when it binds HostappServiceImpl, these two entities are in the memory.
  • When HostappClient makes an intent invocation to SClientApp, both HostappClient and HostappService may be destroyed by the application manager.
  • SClientApp now binds HostappServiceImpl and these two are in the memory now. The IBinder instance passed by HostappClient is now lost.
  • When SClientApp finishes, SClientApp and HostappServiceImpl may both be destroyed again. Our hugely valuable text data passed by SClientApp to HostappServiceImpl is lost then.
  • HostappClient is resurrected but it has nothing to receive from HostappServiceImpl.
A robust implementation of HostappServiceImpl would store setMyText() invocations in a persistent storage, e.g. SQLite database if the callback to HostappClient fails (DeadObjectException or appService==null in HostappServiceImpl.setMyText). When the binder is set by HostappClient's invocation to setCallback(), HostappServiceImpl would check its storage and would immediately send back setMyText() callbacks to HostappClient from the persistent storage. This exercise is left, however, to the interested reader. :-)

Sunday, January 20, 2008

Using the service directory

Originally I wanted to play with Bluetooth which seemed easy but then - as usual with Android - unexpected complications occured. By the way, does anyone know, how to get the default Adapter activated? :-)

Playing with Bluetooth brought me back to services and I was surprised once again how well Android documentation keeps the secret that Android is a service-based platform. Actually, I found, that the Android system's architecture is pretty similar to that of the Symbian. The system is built in client-server fashion. System services (all located in the android.server package) run in their own virtual VMs. Whenever a client wants to talk to those services, they go through the service framework that I played with in this blog entry. The client side sees the services through client-side stubs whose name normally start with I (like IServiceManager). All the other classes on the client side are purely convenience methods, building on the functions offered by the interface class.

Android is superior to Symbian in that it has a proper service directory. This functionality was added to Symbian later, this is the ECom framework. General Symbian servers are not registered, however, in a service directory which is a major drawback of the platform. Android servers are nicely listed in Android's local service directory but of course this feature is not documented at all. :-)

I went after the service directory by decompiling the Android emulator's framework library with Pavel Kouznetsov's wonderful Jad tool. The android.jar is located under the root directory of the Android SDK. I found that the service directory is accessible via the android.os.ServiceManagerNative class.

IServiceManager sm = ServiceManagerNative.getDefault();

The android.os.IServiceManager is the service interface class through which the default service manager can be accessed. I put together a simple program to list these, you can download the program from here.

The display of this program is like this:

This is nice and good but how can one access one of those services? The example program demonstrates it by accessing a less known system service, the Statistics service.

The key lines are the following:

IServiceManager sm = ServiceManagerNative.getDefault();
IBinder statBinder = sm.getService( "statistics" );
IStatisticsService statService = IStatisticsService.Stub.asInterface( statBinder );


The service name "statistics" is the same name that was displayed by the previous program. Unfortunately, there seems to be no programmatic way to get from the IBinder (which is a client-side stub) to the IStatisticsService (or any other service interface class) without knowing, what is the real interface the binder supports. It is the programmer's responsibility to know the interface the service supports and obtain it correctly using the I class belonging to the service (like IStatisticsService this time).

Convenience classes using the service interface classes are more suitable for the application programmer to talk to services. Services may have other means to interact with applications, like Intent broadcasts. In the present state of Android with a lot of areas under development, consequently very poor or not-existing documentation, service interface classes may in many cases be the only way to activate features of the platform.

Wednesday, January 16, 2008

Phonecalls

Update: example program and further information for Android 2.2. 

Again, there was a bit of a break in the blog. One reason is personal: I moved to London to take a new job. The other reason is that this time I went into the hairy issue which is call handling in Android.

I worked with the m3-rc37a version of the emulator and I can tell you that emulator crashes are really common if you play with the telephony stuff in this version. Don't be surprised if the example program distributed with this blog entry does not show the behaviour I promise but crashes instead. It just happens with this emulator version.

You can find the example program here.

The task I gave myself is catching incoming and outgoing calls which is easy. There needs to be just an IntentReceiver defined in the manifest and the onReceiveIntent method implemented in the intent receiver class. The manifest entry declares the filter for the IntentReceiver (again, the XML is mangled due to limitations of the blog engine).

[receiver class=".PhoneIntentReceiver"]
[intent-filter]
[action android:value="android.intent.action.PHONE_STATE" /]
[/intent-filter]
[/receiver]


PHONE_STATE is a general intent for signalling all sorts of phone events. The example program dumps the event bundle when the event comes in. In case of outgoing call, the event is delivered with state=OFFHOOK when the call starts then state=IDLE when the call finishes (state is a key in the intent bundle ("extras") and its string values are OFFHOOK and IDLE). In case of the incoming call (you can simulate it by telnet localhost 5554 then issuing the gsm call phone_number command on the console) the state transition is RINGING then IDLE again.

So far so good. Having a notification on incoming and outgoing calls is less useful, however, if the caller or the called number is not known. This turned out to be a really hairy issue for which I did not find the answer (just desperate help requests). I remembered the good old Series60 days and went for the call log.

The call log is surprisingly co-located in the database of the Contacts application. Try this:

adb shell
#cd /data/data/com.google.android.providers.contacts/databases

# sqlite3 contacts.db

SQLite version 3.5.0
Enter ".help" for instructions
sqlite> .dump

...

CREATE TABLE calls (_id INTEGER PRIMARY KEY,number TEXT,number_key TEXT,number_type TEXT,date INTEGER,duration INTEGER,type INTEGER,person INTEGER,new INTEGER);

INSERT INTO "calls" VALUES(1,'+44444','44444+','Home',1200263250247,0,3,NULL,0);
INSERT INTO "calls" VALUES(2,'+4444445','5444444+','Home',1200263859817,5,2,NULL,1);
INSERT INTO "calls" VALUES(3,'+1234','4321+','Home',1200263990161,2,2,NULL,1);

There is the log. Unfortunately (or fortunately? :-)) Android enforces strict database separation for applications so the Contacts database cannot be accessed from within another Android application. Luckily, the Contacts application decided to share the data as Content Provider. There is even a facilitator class: CallLog.Calls. Dumping the call log is relatively easy after that (look at the PhoneIntent class which is an Activity that makes this dump on pressing a button).

Now the only thing remained to see how the database is updated with regards to phone calls. It turned out that the number is not visible when the call goes out or comes in (RINGING or OFFHOOK state) but is accessible through the content provider when the IDLE transition event comes in. I was not able, therefore, to capture the number of the currently ongoing call but I was able to capture the number of the call that has just finished.

Testing the example application was a pain for me. When working with calls, the emulator tended to crash on my PC without installing any applications. In order to monitor the event transitions, you have to bring up the phone application because it does not come to the front by itself. You can do that by pressing some digits at the emulator's opening screen. If you don't bring the phone app to front manually, you won't be able to answer or reject the call.




I propose that you don't run adb logcat while the call goes on - it tends to increase the chance of emulator crash. You can examine whether the number was retrieved succesfully from the call log by running adb logcat after the call was finished.

Wednesday, January 9, 2008

Service invocation performance

The whole service invocation exercise would not be complete without having an estimate of the performance of a service invocation. Activity invocation turned to be rather slow, due to the complex activity lifecycle transitions the invocation involves. Now those transitions would not occur when invoking services so I expected improvements here.

The measurement client can be downloaded from here. Note that this is just the client therefore the service application from the Invoking services post material also needs to be installed. Don't forget to update the sdk-folder and android-tools properties in the Ant build files. The client and the service applications need to be installed separately with adb install as described at an earlier post.

There are no surprises in the performance measurement client. As there are no lifecycle transitions in the client, the measurement could be performed in a simple for cycle. The average invocation time turned to be around 4 msec which is about 5-10 times faster than the activity invocation performance but is still something like 1000 times slower than a simple in-process Java call.

Monday, January 7, 2008

Invoking services

Update: please check out the updated example program for Android SDK 1.5.

I was not blogging very actively in the past few days and the reason for that was ... erm ... partying. The parties seem to be over for some days so I can return to my favourite pastime of finding service-oriented architecture (SOA) traits in Android.

Activity invocation is just one way of Android's service-oriented design. Android has a complete IDL-based invocation framework built into it. I was particularly curious about it because this sort of service invocation does not have to go through the service lifecycle management that slows down activity invocation considerably.

First, about services. Similarly to intents, I was a bit surprised to find out that the same Service class is used to cover two different service abstractions.
  • If the service is started with Context.startService then it is a long-running background task whose lifecycle is not related to the lifecycle of the application that started it. Beside passing a start-up parameter bundle to this kind of service, Android platform does not provide any means of communicating with the service.
  • If the service is started with Context.bindService then the lifecycles of the service and that of the application using it are tied. The application using the service is notified about the state of the service by means of service lifecycle events. It is also possible to communicate with this type of service using the Android IDL (AIDL) framework.
As I wanted to play with AIDL, this second type of service was of interest for me. That model reminds me of OSGi and its ServiceTracker except that OSGi was much more cumbersome to program and did not not provide any client-service separation whatsoever.

  • The client requests connection to the service (by specifying an intent, as with activities). This launches the service (in separate address space)
  • Once the service is ready to accept requests, the client is notified by means of a registered listener object (conforming to the ServiceConnection interface).
  • The notification also carries the address of a stub object that can be used to invoke the service. If the service crashes or closes unexpectedly, the client is notified by the same service connection object.
  • Meanwhile, the service can be invoked through the stub objects that the aidl tool generates from the service description.
The example code can be downloaded from here.

The service description (with the extension of .aidl) is actually quite similar to a Java file. This is the AIDL of the example program.

package aexp.aidl;

// Adder service interface.

interface IAdderService {

int add( in int i1, in int i2 );
}


Currently one service can have only one interface although Dianne Hackborn from Google hopes for improvement here.

The convention of the aidl tool is that AIDL files are mixed with the Java files. This is how the tool finds them and converts them to Java stub files. In spite of the documentation's statement that only the Eclipse plugin converts AIDL to Java stub file, the Ant script generated by activityCreator invokes the aidl tool before compiling the Java file so if the AIDL is located correctly (in the Java source directory corresponding to the package declaration). In the picture below you can see IAdderService.aidl and the IAdderService.java Java stub file generated by the aidl tool.



Although AdderService is located in the same package as the activity, it has a life (and lifecycle) of its own. The service is declared in the AndroidManifest.xml file (again, XML fragment is garbled so that the blog engine does not corrupt it).

[service class=".AdderServiceImpl"/]

The service declaration could have intent filters as activities may have. In our example we target it directly, using its class name. In order to demonstrate the service lifecycle, the lifecycle events are logged.

The client is located in a separate application. The client also depends on the Java stub file generated by the service project so this file has to be copied by hand into the client project.


The interesting bit here is the AdderServiceConnection embedded class that acts as the service state listener. Just because the bindService call was issued, the service is not available! (actually, we can't obtain the stub object until the ServiceConnection object is notified by its onServiceConnected method). Note that the interfaceName argument of bindService is always null and this argument is supposed to disappear altogether.

Here is the snippet invoking the service.

private void invokeService( int i1, int i2 ) {
TextView t = (TextView)findViewById(R.id.invokeserv_result);
if( service == null )
t.setText( "Service not available" );
else {
try {
int result = service.add( i1,i2 );
t.setText( Integer.toString( result ) );
} catch( DeadObjectException ex ) {
t.setText( "Service invocation error" );
}
}
}


The code invoking the service needs to be prepared for the following exceptional cases:
  • Service is not yet available (onServiceConnected was not yet called) or has gone (onServiceDisconnected was called).
  • For some reason, the invocation was not succesful (DeadObjectException).
I expect that service invocation performance is superior to activity invocation performance but let's see that later. :-)