It’s also a good idea to include the EGL library in order to set up a GL context. Use -lEGL in the Android.mk
file and include EGL/egl.h and EGL/eglext.h in your code.
Native Audio with OpenSL ES
The native audio API in Android is based on the OpenSL ES 1.0.1 from Khronos Group. This API provides a way
to perform native audio operations that is generally faster than the Java alternatives. If your application requires
very low-latency audio, use this API. To include the API, use the following flag in your Android.mk file:
LOCAL_LDLIBS += -lOpenSLES
Use the following include statements in your code to utilize both the standard OpenSL ES API and the
Android-specific extensions:
278 Part III: Pushing the Limits
#include <SLES/OpenSLES.h>
#include <SLES/OpenSLES_Platform.h>
#include <SLES/OpenSLES_Android.h>
#include <SLES/OpenSLES_AndroidConfiguration.h>
Porting a Native Library to Android
Often you will have the source code for an existing, usually open-source, library that you want to integrate into
your application. However, doing so can be difficult because these libraries are often built around the traditional
build system (usually a GNU makefile generated by the tools Autoconf and Automake) for Linux or other
UNIX operating systems. You could copy all the required source files and write a new Android.mk file from
scratch that configures the build and includes the right headers. However, this approach can prove very time-
consuming and error-prone, but luckily there’s a way around this issue.
The Android NDK provides the capability to generate a stand-alone toolchain that works in the same way as
the standard toolchain on your computer. By referring to this toolchain when building the library, you can
use the build system the source code is configured for and build a binary library that you can include in your
application. This process makes it easy to port an existing native library written in C/C++ and reduces the
complexity of your builds as well.
To build the stand-alone toolchain, run the following command:
$NDK/build/tools/make-standalone-toolchain.sh --platform=android-5
--install-dir=<install path>
This command creates a new stand-alone toolchain for the ARM ABI and Android API level 5. If you don’t
provide the install-dir parameter, a compressed archive with the toolchain is placed in your system’s
temp directory.
Building a Native Library
The library for the example in this section is the audio codec library called Opus. This is a free, open-source audio
codec that provides an excellent result for interactive speech as well as streaming music over the Internet. This
library uses technology from both the SILK codec from Skype and the CELT codec from Xiph.org. I don’t go into
the details of how Opus works; instead, I focus on the fact that you can encode and decode between a large
audio frame recorded in PCM format and a much smaller Opus packet.
You can download the source code and read more about the Opus codec at http://opus-
Even though this example uses the Opus audio codec, the same method applies to most other native libraries
that support Linux. Simply download your library and perform the same operations shown in this section.
Chapter 14: Native Code and JNI 279
After you download and extract the library, you need to configure your environment before building.
$ export PATH=$TOOLCHAINS/arm-linux-androideabi-4.6/bin/:${PATH}
$ export CC=arm-linux-androideabi-gcc
$ export CXX=arm-linux-androideabi-g++
These commands prefix your path so that the binaries in your toolchain have precedence over the default ones
for your operating system. It will also set two environment variables used by the build systems to determine
which compiler architecture to use.
Install all your stand-alone toolchains in a common directory and assign an environment variable. This
way, you can easily refer to a specific toolchain.
This command generates the file Makefile used for compiling and installing the library at the specified
position (note that <project path> requires an absolute path).
opus-1.0.2 $ ./configure --prefix=<project path>/jni/libopus --host=
arm-eabi
The preceding command needs to be executed in the directory where you extracted the Opus source code and
in the same terminal where you configured your environment. It sets up the build parameters for your sources.
The host parameter tells the configure script which CPU is to be targeted. The prefix parameter decides where
the compiled binaries and header files are installed. Use the directory of your Android application as a prefix if
that is where you wish the compiled library to be placed. Note that if the jni and libopus directories are
missing, they will be created.
Before you can continue and build the library, you probably need to manually edit the generated makefile. In a
normal UNIX environment, you can have several versions of the same library installed, which are identified by an
internal version number as well as their filename (for instance, libXYZ.1.2.3.so). The Android NDK doesn’t
support this version information, so before you build the library, you need to tell the build system to avoid
applying any version information. You do so by editing the file Makefile that was generated by the configure
script. Open the file in a text editor and find the line that appears as follows:
libopus_la_LDFLAGS = -no-undefined -version-info 3:0:3
Now remove the –version-info argument and replace it with the argument –avoid-version, which
tells the build system not to add this information in the compiled result. The result will appear as follows:
libopus_la_LDFLAGS = -no-undefined –avoid-version
Now you’re ready to compile and install your library in your project’s jni directory:
opus-1.0.2 $ make ; make install
280 Part III: Pushing the Limits
The preceding line is actually two commands, make and make install. The first command compiles the
code according to the configuration set up by the configure script. make install copies all the resulting
binaries and other files to the installation directory. You have now successfully installed the compiled Opus as
an ARM binary in the jni directory of your project. You’re ready to configure the NDK makefiles and implement
the JNI bridge to call the library.
Static Library Module
The library you just built consists of the actual library binary named libopus.so as well as a number of C
header files. The header files are required in order to call the functions inside the library itself. When building
your native code with the NDK, be sure to define a separate module for the native library. This is why the output
path was <project path>jni/libopus and not simply <project path>/jni. The Android.mk file
you place in the libopus directory looks like this:
LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)
LOCAL_MODULE := libopus
LOCAL_SRC_FILES := libopus.so
LOCAL_EXPORT_C_INCLUDES := $(LOCAL_PATH)/include
include $(PREBUILT_SHARED_LIBRARY)
The makefile configuration described in the preceding examples defines the module as a shared library and
exports the include files found in the include directory. Because no sources need to be compiled for this
module (you already did that in the previous section), you specify only the libopus.so file as the source files
to be included.
JNI Module for Native Library
It’s time to create a new directory named opus_jni next to the libopus directory. This is where your JNI
wrapper module for the native library is implemented. Start with the Android.mk file that will import your
module.
LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)
LOCAL_MODULE := opus_jni
LOCAL_SRC_FILES := opus_jni.c
LOCAL_LDLIBS := -llog
LOCAL_SHARED_LIBRARIES := libopus
include $(BUILD_SHARED_LIBRARY)
$(call import-module,libopus)
This Android.mk file tells the NDK to include the static local library libopus in your build, just as was done
in the native-activity sample shown in “Android NDK Build Scripts,” earlier in this chapter. The next step is to
define the Java class that will provide the native methods.
The following class loads the library named opus_jni, and for each new instance, a new encoder and a new
decoder are initialized. You then have two public methods, encodePcm() and decodeOpus(), that do the
actual codec work. Finally, there is a destroy() method that frees up the memory used by the codec. You
now use the javah tool, as shown earlier, to generate the header file for these functions. After you have the
header file, you can start implementing the JNI code.
Chapter 14: Native Code and JNI 281
public class OpusCodec {
{
System.loadLibrary(“opus_jni”);
}
public OpusCodec(int sampleRate, int channels) {
initOpusEncoder(sampleRate, channels);
initOpusDecoder(sampleRate, channels);
}
private native void initOpusDecoder(int sampleRate, int channels);
public native int decodeOpus(byte[] encodedData,
int encodedLength,
short[] pcmOutput,
int pcmLength);
private native void initOpusEncoder(int sampleRate, int channels);
public native int encodePcm(short[] pcmData,
int pcmLength,
byte[] encodedData,
int encodedLength);
public native void destroy();
}
Here is the entire JNI code, piece by piece, starting with the header files and definitions:
#include <opus/opus.h>
#include <opus/opus_defines.h>
#include <opus/opus_types.h>
#include <opus/opus_multistream.h>
#include <android/log.h>
#include <string.h>
#include <stdlib.h>
#include <time.h>
#include “com_aptl_opus_OpusCodecTest_OpusCodec.h”
#define LOG_TAG “OpusTest”
#define LOGD(LOG_TAG, ...) __android_log_print(ANDROID_LOG_DEBUG, LOG_TAG,
__VA_ARGS__)
#define LOGV(LOG_TAG, ...) __android_log_print(ANDROID_LOG_VERBOSE, LOG_
TAG,
__VA_ARGS__)
#define LOGE(LOG_TAG, ...) __android_log_print(ANDROID_LOG_ERROR, LOG_TAG,
__VA_ARGS__)
OpusDecoder* gOpusDecoder;
OpusEncoder* gOpusEncoder;
JNIEXPORT jint JNI_OnLoad(JavaVM *vm, void *reserved) {
return JNI_VERSION_1_6;
}
282 Part III: Pushing the Limits
As shown here, the locally generated header files for the JNI functions and the headers for the Opus codec
that are located in the libopus module are included. The two global variables, OpusDecoder and
OpusEncoder, are the references to the decoder and encoder objects in the Opus library.
You also define the shorter log functions as well as the default JNI_OnLoad function to tell the system that it
supports the correct JNI version.
/*
* Class: com_aptl_opus_OpusCodecTest_OpusCodec
* Method: initOpusDecoder
* Signature: (II)I
*/
JNIEXPORT void JNICALL Java_com_aptl_opus_OpusCodecTest_OpusCodec_
initOpusDecoder
(JNIEnv *env, jobject obj, jint sampleRate, jint channels) {
int error;
gOpusDecoder = opus_decoder_create(sampleRate, channels, &error);
LOGD(LOG_TAG, “Decoder initialized: %d”, error);
}
/*
* Class: com_aptl_opus_OpusCodecTest_OpusCodec
* Method: initOpusEncoder
* Signature: (II)I
*/
JNIEXPORT void JNICALL Java_com_aptl_opus_OpusCodecTest_OpusCodec_
initOpusEncoder
(JNIEnv *env, jobject obj, jint sampleRate, jint channels) {
int error;
gOpusEncoder = opus_encoder_create(sampleRate, channels,
OPUS_APPLICATION_VOIP, &error);
}
The two init functions call the respective create function from the Opus library. In this example, you
provide the encoder with a parameter telling it to optimize its processing for VoIP applications.
/*
* Class: com_aptl_opus_OpusCodecTest_OpusCodec
* Method: decodeOpus
* Signature: ([BI[BIZ)I
*/
JNIEXPORT jint JNICALL Java_com_aptl_opus_OpusCodecTest_OpusCodec_
decodeOpus
(JNIEnv *env, jobject obj, jbyteArray in, jint inLength, jshortArray
out,
jint outLength) {
opus_int32 encodedBytes;
jbyte* jniIn;
jshort* jniOut;
if(in != NULL) {
Chapter 14: Native Code and JNI 283
jniIn = (*env)->GetByteArrayElements(env, in, NULL);
jniOut = (*env)->GetShortArrayElements(env, out, NULL);
encodedBytes = opus_decode(gOpusDecoder,
jniIn,
inLength,
jniOut,
outLength, 0);
(*env)->ReleaseByteArrayElements(env, in, jniIn, JNI_ABORT);
(*env)->ReleaseShortArrayElements(env, out, jniOut, 0);
} else {
jniOut = (*env)->GetShortArrayElements(env, out, NULL);
encodedBytes = opus_decode(gOpusDecoder,
NULL,
0,
jniOut,
(*env)->GetArrayLength(env, out), 1);
}
return encodedBytes;
}
The decode function allows the input to the opus_decode function to be null, which handles situations
where you’re missing an incoming packet of Opus data because of network latency and want the decoder to
handle the situation. The Opus library is designed to handle such cases and will generate a PCM package to fill
out missing audio.
/*
* Class: com_aptl_opus_OpusCodecTest_OpusCodec
* Method: encodePcm
* Signature: ([BI[BI)I
*/
JNIEXPORT jint JNICALL Java_com_aptl_opus_OpusCodecTest_OpusCodec_
encodePcm
(JNIEnv *env, jobject obj, jshortArray in, jint inLength, jbyteArray
out,
jint outLength) {
opus_int32 encodedBytes;
jshort* jniIn;
jbyte* jniOut;
jniIn = (*env)->GetShortArrayElements(env, in, NULL);
jniOut = (*env)->GetByteArrayElements(env, out, NULL);
encodedBytes = opus_encode(gOpusEncoder,
jniIn,
inLength,
jniOut,
outLength);
284 Part III: Pushing the Limits
if(encodedBytes == OPUS_BAD_ARG) {
LOGE(LOG_TAG, “OPUS_BAD_ARG”);
} else if(encodedBytes == OPUS_BUFFER_TOO_SMALL) {
LOGE(LOG_TAG, “OPUS_BUFFER_TOO_SMALL”);
} else if(encodedBytes == OPUS_INTERNAL_ERROR) {
LOGE(LOG_TAG, “OPUS_INTERNAL_ERROR”);
} else if(encodedBytes == OPUS_INVALID_PACKET) {
LOGE(LOG_TAG, “OPUS_INVALID_PACKET”);
} else if(encodedBytes == OPUS_UNIMPLEMENTED) {
LOGE(LOG_TAG, “OPUS_UNIMPLEMENTED”);
} else if(encodedBytes == OPUS_INVALID_STATE) {
LOGE(LOG_TAG, “OPUS_INVALID_STATE”);
} else if(encodedBytes == OPUS_ALLOC_FAIL) {
LOGE(LOG_TAG, “OPUS_ALLOC_FAIL”);
}
(*env)->ReleaseShortArrayElements(env, in, jniIn, JNI_ABORT);
(*env)->ReleaseByteArrayElements(env, out, jniOut, 0);
return encodedBytes;
}
The encode function takes PCM data represented as an array of short and encodes. The result ends up in the
byte array that the Java side can work with.
The destroy function is the last part in this file. It’s important to call this native method from your Java code as
follows once you’re done with the OpusCodec instance; otherwise, you’ll leak memory.
/*
* Class: com_aptl_opus_OpusCodecTest_OpusCodec
* Method: destroy
* Signature: ()V
*/
JNIEXPORT void JNICALL Java_com_aptl_opus_OpusCodecTest_OpusCodec_destroy
(JNIEnv *env, jobject obj) {
opus_decoder_destroy(gOpusDecoder);
opus_encoder_destroy(gOpusEncoder);
}
You now have a fully functioning JNI wrapper for the Opus codec optimized for VoIP applications. With just
some small modifications, you can craft this implementation to your own needs and build your own VoIP
application or adapt it for streaming music.
Summary
In this chapter, I introduced you to the Android NDK and how you can use it to write and build native code for
your Android applications. You can choose to write your entire application in native, which is many times the
case when it comes to 3D games where there’s an existing 3D engine available, or you can combine it with Java
Chapter 14: Native Code and JNI 285
code and write native C code where further optimization is needed. Also, thousands of open-source libraries
are on the Internet that you may find useful, and with the techniques shown in this chapter, you can easily port
them to Android and create a simple JNI wrapper to access them from Java.
This chapter certainly isn’t a comprehensive discussion of all things regarding the NDK and JNI. However, for
further details, please refer to the official documentation in the Android NDK as well as the JNI specifications
from Oracle, and check out “Further Resources,” below.
Further Resources
Android Developers Site
“JNI Tips” specific to Android development: http://developer.android.com/training/
Oracle Websites
Oracle. “Java SE Documentation.” Here you can find information on JNI Documentation: http://docs.
oracle.com/javase/7/docs/technotes/guides/jni/index.html
Oracle. “Java Native Interface Documentation.” Here you can find specification for JNI: http://docs.
oracle.com/javase/7/docs/technotes/guides/jni/spec/jniTOC.html
Chapter 15
The Hidden Android APIs
Android developers know that tons of features in the platform aren’t accessible through the official and public
APIs. For instance, although Android has classes for sending SMS (SmsManager and SmsMessage), there is no
official API for receiving them. Nevertheless, you can find several applications on the Google Play Store site that
provide full-featured SMS clients, and many apps are built around incoming SMS. Although a simple Google
search will give the necessary information for how to receive SMS in an Android application, you’ll come upon
many other cases where the hidden platform APIs in Android could be useful in applications.
In this chapter, I explain how and where to find hidden APIs. I also describe different methods for accessing
them and how to do so in a safe and secure way.
Most of the hidden Android APIs are also protected by permissions that have a protectionLevel of
signature or system (refer to Chapter 12). Although you can’t use these APIs in regular applications you
publish on the Google Play Store, you can still use them in applications you write for a custom firmware. Doing
so allows you to access these APIs without changing the Android platform. (See Chapter 16 for more on how to
build a custom firmware.)
Official and Hidden APIs
The official APIs are all the classes, interfaces, methods, and constants found in the SDK documentation.
Although these APIs are usually enough for most applications, at times you’ll want to access something more,
but don’t know how to find them in the official APIs.
When Google publishes the Android SDK, it contains a JAR file (android.jar) that you use as a reference
when compiling your code. You can find this file via <sdk root>/platforms/android-<API level>/.
The file contains only empty classes, meaning that it includes only the public and protected declarations and all
code within the methods has been scrubbed away. This JAR file is generated as part of the SDK when you build
the platform.
The scrubbed android.jar file is generated when the SDK is built by inspecting each source file and
excluding every field (such as constants), method, and class with the JavaDoc annotation @hide (see Figure 15-1).
This means that these symbols are still accessible to the implementation running on the device but they aren’t
visible at compile time.
288 Part III: Pushing the Limits
Figure 15-1 The source code for the hidden class CountryDetector . Note the @hide annotation for the JavaDoc .
Some APIs in Android are hidden automatically, without the @hide annotation. You can usually find these APIs
in the package com.android.internal, which is never part of the android.jar but does contain lots of
code for internal use in Android. You can find some of the other hidden APIs in the system applications. These APIs
usually have ContentProvider information for the system providers that isn’t included in the official SDK.
Discovering Hidden APIs
The easiest way to find the hidden APIs is to search for them in the Android source code. The source code for the
Android platform is vast, but fortunately several online sites have indexed the code and made it searchable. One
such site is AndroidXRef (http://androidxref.com), which allows you to search all the open source code
for all officially released Android versions (see Figure 15-2).
Chapter 15: The Hidden Android APIs 289
Figure 15-2 The Search dialog box at AndroidXRef
Another way to look for hidden APIs is through the View Source link on the Android API reference site (see
Figure 15-3). This site doesn’t provide a search method like AndroidXRef does, but it’s easier to access directly
from the official API documentation.
Although searching is good if you know where and what to look for, finding the code that does what you need
can be difficult. You can find most of the hidden APIs in the frameworks project. All of the APIs in the android
package can be found here as well as most of the APIs in com.android.internal.
Often the hidden API you’re looking for is part of a class that is public. For instance, the WifiManager has
several public, unhidden, methods but also a number of useful hidden methods and fields. In other cases, the
class is hidden from the public API, such as the CountryDetector class shown earlier in Figure 15-1.
290 Part III: Pushing the Limits
Figure 15-3 The View Source link for Settings .Global on the API reference site
Safely Calling Hidden APIs
Constant fields, such as broadcast actions or provider Uri, are major parts of the useful hidden APIs. You can
copy these fields into your own code and use them just as they are. The easiest way to do so is to copy the
entire class (for instance, directly from AndroidXRef) and place it in your project. If these hidden APIs have been
changed in the different Android API levels, you can keep a copy of each version in its own package. This way,
you can use a hidden API and still target multiple Android versions.
For most situations in which you need to use a hidden API that consists of constants (such as
broadcast actions), I recommend copying the hidden constants from the Android source code.
For APIs that require compile-time linking—that is, interfaces, classes, and methods—you have two choices. You
can compile your application with a modified SDK that contains a JAR file with all the classes and interfaces you
need. The other solution is to use the Reflection API in Java to dynamically look up the classes and methods you
want to call. Each method has its pros and cons, and your choice will depend on the situation.
Chapter 15: The Hidden Android APIs 291
Modifying the SDK will effectively bind your code to the device you use when generating the modified
android.jar (see the section “Extracting Hidden APIs from Device”) and could result in your code crashing
on other devices if you’re not careful. However, there’s no performance penalty with this solution. Using the
Reflections API allows you to target multiple Android versions but can penalize performance because it requires
a runtime lookup of classes and methods. I discuss both these approaches in the following sections.
Extracting Hidden APIs from a Device
To do compile-time linking to the hidden APIs, you first need to extract and process the libraries from a device.
You can extract these libraries either from an emulator instance or from a device because they’ll be used only
for compiling your code. Because this work requires a number of files from a device, I recommend that you
create an empty working directory. Since you may want to perform this task for multiple API versions, you can
create one working directory per API level.
$ adb pull /system/framework .
pull: building file list...
pull: <files pulled from device>
63 files pulled. 0 files skipped.
4084 KB/s (35028810 bytes in 8.374s)
Run the preceding command from your working directory, and you’ll see that it pulls all the files from the
directory /system/framework on the device. Once the extraction is completed, you can list the files, and the
output should look somewhat like the following (it may vary depending on the device’s manufacturer and the
version of Android you’re using):
$ ls
am.jar ext.jar
am.odex ext.odex
android.policy.jar framework-res.apk
android.policy.odex framework.jar
android.test.runner.jar framework.odex
android.test.runner.odex ime.jar
apache-xml.jar ime.odex
apache-xml.odex input.jar
bmgr.jar input.odex
bmgr.odex javax.obex.jar
bouncycastle.jar javax.obex.odex
bouncycastle.odex mms-common.jar
bu.jar mms-common.odex
bu.odex monkey.jar
com.android.future.usb.accessory.jar monkey.odex
com.android.future.usb.accessory.odex pm.jar
com.android.location.provider.jar pm.odex
com.android.location.provider.odex requestsync.jar
com.android.nfc_extras.jar requestsync.odex
com.android.nfc_extras.odex send_bug.jar
com.google.android.maps.jar send_bug.odex
com.google.android.maps.odex services.jar
com.google.android.media.effects.jar services.odex
com.google.android.media.effects.odex settings.jar
com.google.widevine.software.drm.jar settings.odex
292 Part III: Pushing the Limits
com.google.widevine.software.drm.odex svc.jar
content.jar svc.odex
content.odex telephony-common.jar
core-junit.jar telephony-common.odex
core-junit.odex uiautomator.jar
core.jar uiautomator.odex
core.odex
These files are all the Java-based system libraries on your Android device. They are the optimized DEX files that
are loaded by the Dalvik VM. The next step is to decide which file contains the hidden APIs you want to convert
back to Java class files that you can use when compiling your application. Most of the hidden APIs are placed in
framework.odex, whereas the crypto-libraries are in the bouncycastle.odex file.
Starting with Android 4.2, several hidden APIs that used to be in framework.odex are now placed
in other files. For instance, the hidden Telephony class is now optional (because not all Android
devices have telephony support) and can now be found in telephony-common.odex.
Once you know which file you need to convert, you download a tool named Smali that can convert the
optimized DEX files (.odex) to an intermediate format (.smali). You can then convert this intermediate
format back to Java class files using another tool named dex2Jar. You can download Smali at https://
code.google.com/p/smali, and you can find dex2Jar at https://code.google.com/p/dex2jar.
Download both and extract them to an appropriate location (for instance, next to your working directory). Start
by converting the ODEX file to the intermediate format as shown here:
$ mkdir android-apis-17
$ java -jar ~/Downloads/baksmali-2.0b5.jar -a 17 -x framework.odex -d . -o
android-apis-17
When you run this command from the same directory where you pulled the files from the device, the file
framework.odex is converted to a number of SMALI files placed in the correct package structure in the
directory android-apis-17. Next, you need to convert these files into a single DEX file.
$ java -jar ~/Downloads/smali-2.0b5.jar -a 17 -o android-apis-17.dex
android-apis-17
You can repeat the two preceding steps for each file you need to convert. For instance, on Android 4.2, the
hidden Telephony class is placed in telephony-common.odex. This way, you can create a single JAR file in
the end with all the hidden classes you need, even if they’re contained in different ODEX files from the start.
Finally, you need to use the dex2Jar tool to convert the DEX file into a JAR file containing all the Java class files.
$ ~/Downloads/dex2jar-0.0.9.15/d2j-dex2jar.sh android-apis-17.dex
dex2jar android-apis-17.dex -> android-apis-17-dex2jar.jar
The resulting JAR file contains all the classes, both hidden and public, from the original ODEX file (or files). To
use this file instead of the default android.jar in your SDK, simply rename it to android.jar and replace
Chapter 15: The Hidden Android APIs 293
it with the one in your SDK (for instance, <sdk root>/platforms/android-17/android.jar).
Remember to back up the original file in case you want to revert to the original SDK without the hidden APIs.
This approach provides you with a set of platform APIs that are guaranteed to work only on the device
you extracted the files from. Because this is the baseline for all other Android devices, I recommend
doing this only from a Nexus device with an official factory image. Perform this only on a non-Nexus
device if you need to use the hidden, proprietary APIs implemented by the device’s manufacturer (for
instance, a hidden camera extension API or something similar).
Error Handling for Modified SDK
When utilizing the approach for using the hidden APIs described previously, it’s difficult to know whether the
method signatures from your extracted classes match the ones that users have on their devices. Although
modifying the SDK may work on the device you use for development, a user who installs your application may
have a device where the hidden APIs are modified by the vendor. When that happens, your application will
throw a NoSuchMethodException or ClassNotFoundException.
You can deal with this situation a couple of ways. You can combine this approach with the use of reflections
(described in the next section) to detect the presence of your hidden APIs. In this way, you have the benefit of
both solutions, which I recommend. Another way is to simply catch the exception so that you can perform some
graceful degradation of the functionality.
Whatever you do, be sure to perform some error handling when calling hidden APIs. At the very least, you can
limit the availability of your application to the devices you’ve tested on. You always want to avoid having your
application crashing on a user’s device.
Calling Hidden APIs Using Reflections
The Reflections API in Java (found in the java.lang.reflect package) gives you a safer approach than
modifying the SDK does because you can use it to detect the presence of an API (or lack thereof) before calling
it. However, because all binding and invocation of hidden APIs occur in runtime, this method is also slower than
the alternative method described in the previous section.
Calling a hidden API using Reflections is a two-step process. First, you need to look up the class and methods
you want to call and store a reference to the Method object. After you have this reference, you can invoke
the method on an object. The two steps are shown in the following code, where you look up the method for
checking the state of Wi-Fi tethering:
public Method getWifiAPMethod(WifiManager wifiManager) {
try {
Class clazz = wifiManager.getClass();
return clazz.getMethod(“isWifiApEnabled”);
} catch (NoSuchMethodException e) {
throw new RuntimeException(e);
}
294 Part III: Pushing the Limits
}
public boolean invokeIsWifiAPEnabled(WifiManager wifiManager,
Method isWifiApEnabledMethod) {
try {
return (Boolean) isWifiApEnabledMethod.invoke(wifiManager);
} catch (IllegalAccessException e) {
throw new RuntimeException(e);
} catch (InvocationTargetException e) {
throw new RuntimeException(e);
}
}
The preceding example shows a fairly simple invocation using the Reflections API. If the hidden method
takes parameters, you need to provide the classes for those in the call to Class.getMethod(). Also, in this
example, the only error handling is to throw a RuntimeException. In your own application, you should
handle errors properly and do a graceful degradation of your application’s feature set.
Never assume that methods you retrieve using Reflections are available on all devices. If they’re hidden, the
manufacturer may have modified them and changed their signature (number of parameters, for instance).
In the early days of Android, this situation was quite common because many features were missing in the
platforms added by manufacturers. However, now you can usually expect the API to be there, just take care to
do proper error handing and feature fallback when using the hidden APIs.
Examples of Hidden APIs
In this section, I show a few examples of how hidden APIs are used. These are some of the typical scenarios I’ve
discovered that developers are asking for.
Receiving and Reading SMS
The most commonly requested hidden API in Android is related to receiving and reading SMS. Although the
public API contains two permissions, RECEIVE_SMS and READ_SMS, the actual API for performing these
actions is hidden.
An application that must be able to receive an SMS must declare the use of the RECEIVE_SMS permission and
implement a BroadcastReceiver that is triggered for incoming SMS.
public class MySmsReceiver extends BroadcastReceiver {
// Hidden constant from Telephony.java
public static final String SMS_RECEIVED_ACTION
= “android.provider.Telephony.SMS_RECEIVED”;
public static final String MESSAGE_SERVICE_NUMBER = “+461234567890”;
private static final String MESSAGE_SERVICE_PREFIX = “MYSERVICE”;
public void onReceive(Context context, Intent intent) {
String action = intent.getAction();
if (SMS_RECEIVED_ACTION.equals(action)) {
Chapter 15: The Hidden Android APIs 295
// “pdus” is the hidden key for the SMS data
Object[] messages =
(Object[]) intent.getSerializableExtra(“pdus”);
for (Object message : messages) {
byte[] messageData = (byte[]) message;
SmsMessage smsMessage =
SmsMessage.createFromPdu(messageData);
processSms(smsMessage);
}
}
}
private void processSms(SmsMessage smsMessage) {
String from = smsMessage.getOriginatingAddress();
if (MESSAGE_SERVICE_NUMBER.equals(from)) {
String messageBody = smsMessage.getMessageBody();
if (messageBody.startsWith(MESSAGE_SERVICE_PREFIX)) {
// TODO: Message verified - start processing...
}
}
}
}
The preceding code shows a BroadcastReceiver that listens for the Intent action android.provider.
Telephony.SMS_RECEIVED (remember to add this to the intent-filter in the manifest as well). The
only “hidden” parts in this example are this Intent action and the String to retrieve SMS data from the
Intent (“pdus”).
For reading already received SMS, you need to query a hidden ContentProvider and declare the use of the
permission READ_SMS. The hidden class Telephony, found in the android.provider package, provides
all of the information needed. The best way to use this class is to simply copy it to your own projects and refactor
it to suit your package structure. Because it also contains calls to other hidden classes and methods, you must
either remove or refactor these calls to make your code compile. Depending on how much of this hidden API
you’ll need to use, sometimes it’s enough to simply copy a number of constant declarations instead of the
entire class.
@Override
public void onActivityCreated(Bundle savedInstanceState) {
super.onActivityCreated(savedInstanceState);
mAdapter = new SimpleCursorAdapter(this,
R.layout.sms_list_item, null,
new String[] {Telephony.Sms.ADDRESS, Telephony.Sms.BODY,
Telephony.Sms.DATE},
new int[] {R.id.sms_from, R.id.sms_body, R.id.sms_received},
CursorAdapter.FLAG_REGISTER_CONTENT_OBSERVER);
setListAdapter(mAdapter);
getLoaderManager().initLoader(0, null, this);
}
public Loader<Cursor> onCreateLoader(int id, Bundle args) {
Uri smsUri = Telephony.Sms.CONTENT_URI;
296 Part III: Pushing the Limits
return new CursorLoader(getActivity(), smsUri, new String[] {
Telephony.Sms._ID,
Telephony.Sms.ADDRESS,
Telephony.Sms.BODY,
Telephony.Sms.DATE},
null, null, Telephony.Sms.DEFAULT_SORT_ORDER);
}
Here are two methods shown from a custom ListFragment, which loads a Cursor from the Content
Provider for SMS and loads that into a SimpleCursorAdapter. The Uri for the provider and the name
of the columns are all the content used from the Telephony class.
Wi-Fi Tethering
Android smartphones can enable Wi-Fi tethering, which makes it possible to create a mobile Wi-Fi hotspot that
allows other devices (usually your laptop) to connect to the Internet when you’re traveling. This feature is a very
popular one on Android, but it has caused some problems for application developers.
When the user enables Wi-Fi tethering, the state of the Wi-Fi is neither on nor off, but “unknown” if you query it
through the public API. In a previous example (see the “Calling Hidden APIs Using Reflections” section), I showed
how to detect whether Wi-Fi tethering is enabled using hidden method isWifiApEnabled(). A number of
other hidden methods in the WifiManager class provide you with more information about Wi-Fi tethering.
private WifiConfiguration getWifiApConfig() {
WifiConfiguration wifiConfiguration = null;
try {
WifiManager wifiManager =
(WifiManager) getSystemService(WIFI_SERVICE);
Class clazz = WifiManager.class;
Method getWifiApConfigurationMethod =
clazz.getMethod(“getWifiApConfiguration”);
return (WifiConfiguration)
getWifiApConfigurationMethod.invoke(wifiManager);
} catch (NoSuchMethodException e) {
Log.e(TAG, “Cannot find method”, e);
} catch (IllegalAccessException e) {
Log.e(TAG, “Cannot call method”, e);
} catch (InvocationTargetException e) {
Log.e(TAG, “Cannot call method”, e);
}
return wifiConfiguration;
}
The preceding code shows how you can retrieve the WifiConfiguration for the Wi-Fi tethering settings
on a device. Note that calling these methods requires that your application have the permission android.
permission.ACCESS_WIFI_STATE. All Wi-Fi networks that an Android device has configured (that is,
connected to) can be enumerated as a list of WifiConfiguration objects through the WifiManager.
getConfiguredNetworks(). In each of the WifiConfiguration objects retrieved through this
method, the preSharedKey is set to null for obvious security reasons. However, when retrieving the
WifiConfiguration object for the Wi-Fi tethering settings as shown in the preceding code, you’ll find the
Chapter 15: The Hidden Android APIs 297
clear-text password present in the preSharedKey variable. In this way, your application can retrieve both the
name for the access point that is created when you activate Wi-Fi tethering and the password.
Although this feature can be considered a security flaw, it’s good to know that the permission needed to
activate Wi-Fi tethering requires your application to be signed with the system certificate. Thus, even if an
application can read the password, there’s no way for it to activate Wi-Fi tethering without the user’s consent.
Hidden Settings
An Android device has hundreds of different settings that are available through the class Settings. Besides
providing access to the value for each setting, it publishes a number of Intent actions that you can use
to launch a specific part of the settings UI. For instance, to launch the settings for airplane mode, you use
Settings.ACTION_AIRPLANE_MODE_SETTINGS when creating the Intent. Figure 15-4 shows how the
file Settings.java looks like when viewed through AndroidXRef.
Figure 15-4 Some of the hidden constants in the source file Settings.java
298 Part III: Pushing the Limits
A number of hidden setting keys and Intent actions are in the Settings class, some of which can be very
useful when your application needs to figure out details about the device or when you want to present a
shortcut within your application to a certain system setting.
Summary
This chapter introduced how you can discover and use the hidden APIs in the Android platform. Although only
a few examples are shown, the number of available hidden APIs is quite large. Most of these APIs are not only
hidden but also protected with permissions with the signature or system protection level, which makes
them unusable for most Android developers. However, as you will see in the next chapter, they can be an
efficient method for building advanced applications with access to system APIs if you build them for a device
with a custom firmware.
Some of the APIs are simple constants used to access ContentProviders, Intent actions for launching
Activities or settings keys for reading hidden system settings, whereas others are methods that you need
to invoke.
Although most applications will never require these APIs, in some situations, you’ll benefit by using an API that
is officially unavailable. Using the hidden APIs in a smart way will allow you to further enhance your apps.
Further Resources
Websites
An index of all the Android source code sorted according to specific versions: http://android
The Java tutorials on the Reflection API: http://docs.oracle.com/javase/tutorial/reflect
The utility for converting ODEX files to DEX format: https://code.google.com/p/smali
The utility for converting DEX files to JAR files with Java classes: https://code.google.com/
Chapter 16
Hacking the Android
Platform
Although I expect most of you will be developing Android applications, sometimes using the NDK (refer to
Chapter 14) and sometimes using hidden APIs (refer to Chapter 15), some of you may have specific reasons
to modify the Android platform. Google provides the source code for the Android platform as an open-
source project called (you guessed it) Android Open Source Project (AOSP) just so third-party developers can
experiment and modify Android to suit their needs.
You can download all the code and compile it, but you also need a supported device if you want to test
your modifications on real hardware. Fortunately, all the devices announced under the Google Nexus brand
allow you to do so, and many manufacturers, such as Sony Mobile and HTC, allow you to unlock their devices
and flash them with a custom firmware as well. A company working on a low-level software feature that it
intends to license to handset manufacturers can use this approach to demonstrate its products on real devices,
for example.
Until Google released Android as an open-source project and provided open devices, this approach required
expensive development platforms from chipset vendors like Qualcomm and Texas Instruments. Today, hundreds
of independent developers spend a significant amount of time hacking on their own version of the Android
platform. Some of these developers have organized themselves into community groups focusing on building
custom firmware for Android devices that users can download and install.
In this chapter, I describe how you can build your own firmware or a custom ROM for an Android device,
including how to access and download the AOSP source code, set up the build environment, compile the
code, and flash the firmware to a device. In this chapter, I’m using the Galaxy Nexus, but you can use any of the
Google Nexus-branded devices with the same instructions. I also provide some general directions on how to
proceed with building custom firmware for other manufacturers’ devices.
In addition, I describe how to modify and extend the Android platform. Because the Android source code is
very large, I selected a few simple examples that will introduce you to the platform’s design. If your company is
developing custom solutions for the Android platform, you should find this information valuable.
I end this chapter talking about how you can contribute your changes to the Android platform to the AOSP.
If you think a feature is missing and not prioritized by Google, you can use this approach to make it happen.
( Note: This isn’t a guarantee Google will accept your patch, but you’ll never know unless you try.)
300 Part III: Pushing the Limits
A word of caution: The things described in this chapter can, if you’re not careful, break your device. If
you need to do this, use a device that is not your primary phone and know that your actions could
render the device unusable or break the warranty. When you flash your device, all its content will be
deleted, so be sure to back up content you want to save.
Several of the major Android device manufacturers allow you to flash a custom firmware on their devices, which
you do by unlocking the bootloader so that the device will accept firmware images with signatures other than
the device’s manufacturer. Doing so usually affects the warranty of the device, but if your goal is to demonstrate
a new low-level feature to that manufacturer, it might be worth doing. I briefly cover some of the major
manufacturers approach to unlocking the bootloader later in this chapter.
Unlocking Your Device
All Android devices come with a locked bootloader that you need to unlock before you can flash them with
new firmware (this also applies to flashing the factory firmwares from the link in the section “Flashing Factory
Images”). You can unlock all Google Nexus devices with a simple command. The following instructions apply to
Google Nexus devices.
Start by connecting your device to your computer and ensure that USB Debugging is enabled (see Chapter 1).
Next, reboot the device to the bootloader using adb as shown here:
$ adb reboot-bootloader
Your device will reboot into the bootloader, as shown in Figure 16-1, where you can issue fastboot
commands, which are used to unlock the device and flash new firmware images.
Figure 16-1 A Galaxy Nexus device in fastboot mode with the bootloader locked
Chapter 16: Hacking the Android Platform 301
Next, perform the following command, which unlocks the bootloader so you can flash your custom firmware
images:
$ fastboot oem unlock
It’s also possible to lock the bootloader again by issuing the command fastboot oem lock.
The screen of the device changes (see Figure 16-2) and shows information about the effects of unlocking
the bootloader. Press Volume Up/Down to select your choice and press the Power button to confirm. When
unlocked, the device returns to the fastboot mode and allows you to flash a new firmware image.
Figure 16-2 The warning box presented when you try to unlock the bootloader on a
Galaxy Nexus device
Flashing Factory Images
When you build your custom firmware and test your new features, the device is likely to crash. Although
generally you can return to fastboot mode and flash a new firmware image, sometimes it may be useful to
flash the factory image to the device. You may also want to do so when testing an older version of Android.
Download the factory image for your device from https://developers.google.com/android/
nexus/images and extract the archive into an empty directory.
In the directory, you’ll find a shell script named flash-all.sh, which will flash the entire factory firmware
to the device and restore it to its original state. After the process is complete (it could take a few minutes,
depending on the device and your computer), the device reboots into the freshly installed Android image.
302 Part III: Pushing the Limits
Unlocking Non-Google Nexus Devices
Most Android devices on the market aren’t Google Nexus brands, and not all of these device’s manufacturers
allow you to unlock the bootloader and flash a custom firmware. However, more and more manufacturers are
realizing the positive benefits of allowing advanced users and third-party companies to unlock and flash their
devices with custom firmware, so the list of officially unlockable devices is growing all the time.
Sony Mobile was the first manufacturer to announce that its devices could be unlocked, beginning with its
2011 device. Not every device after 2011 from Sony Mobile can be unlocked, and sometimes the devices sold
through operators can’t be unlocked. You can find more information about unlocking the Sony Mobile Android
devices at http://unlockbootloader.sonymobile.com.
HTC also provides an official method for unlocking its devices. Some devices require a firmware update before
they can be unlocked. More information is available at http://www.htcdev.com/bootloader.
Motorola has an official bootloader unlock program covering some of its devices. To find out if this program
covers your device, visit https://motorola-global-portal.custhelp.com/app/standalone/
bootloader/unlock-your-device-a.
Other manufacturers may also allow you to unlock your device. Visit their support sites for more information.
Also, even if there is no official way of unlocking your device, there may be an unofficial workaround. A good
place to start is on the XDA Developers forum (http://www.xda-developers.com). Remember that
unlocking your device this way will usually void the warranty. Also, even if you manage to unlock the device,
most manufacturers don’t provide a full set of hardware drivers necessary to build a working custom firmware.
Refer to the each manufacturer’s information on the details.
Community-Supported Firmwares
Ever since the first Android devices were released, a stable and growing community of developers has worked
on releasing custom firmware that users can download and flash onto their own devices. The most well-known
group is CyanogenMod, and it has custom firmware for a wide selection of devices. You can find more
information about its work at http://www.cyanogenmod.org.
Other groups have focused on devices from specific vendors. For instance, a group called the FreeXperia Project
is building high-quality community firmware for the Sony Mobile Xperia devices. Visit its site at http://free
xperiaproject.com for more information.
You can usually find the best source for community-developed custom firmware at the XDA Developer forums
(see http://forum.xda-developers.com). At this site, you can find information about most devices
that have been released.
The Android Source Code
The Android Open Source Project is the publically available source code for the Android platform, and Google
provides it at http://source.android.com. This site includes not only the source code for the Android
platform, but also for the SDK and other development tools. The project is structured around a number of Git
Chapter 16: Hacking the Android Platform 303
modules. Each module is part of the overall system, such as the Linux kernel, native libraries, the Dalvik VM, or a
system application.
Setting Up the Build Environment
You can work with the Android Open Source Project on Linux and Mac OS X. The build instructions tend to
be updated on a regular basis, with support for new versions of the host operating systems and updated
development tools, so I’m not covering the instructions on how to set up the environment on your system. You
can find the latest instructions for setting up the build environment for AOSP at http://source.android.
Because the default partition on OS X is case-insensitive, for Mac OS X, you need to create a new case-sensitive
disk image. Also, downloading the source code (using repo sync) the first time will take quite a while—a lot
of code must be downloaded.
Nexus Binaries
Although the Android platform is open-source, all the hardware drivers for the Nexus devices are not. For
instance, the graphics driver for Galaxy Nexus is a proprietary library from Imagination Technologies, and the
source code for this hasn’t been released yet. Therefore, you will also need to download all the binary drivers for
your Nexus device before you proceed with building a custom firmware. You can find the available binaries at
https://developers.google.com/android/nexus/drivers.
You download all the binary drivers available for your device to the root of the Android platform projects you’ve
just downloaded. Run the following command to extract the archive, and you’ll be prompted with a license
agreement:
$ tar xzvf imgtec-maguro-jdq39-bb3c4e4e.tgz
x extract-imgtec-maguro.sh
$ sh ./extract-imgtec-maguro.sh
The license for this software will now be displayed.
You must agree to this license before using this software.
-n Press Enter to view the license
After you agree to the terms of the license, you have new projects under the vendor directory. These are
usually only headers, makefiles, and binary .so files that will be installed on the custom firmware when you
finish building.
It’s important to include all the binary drivers available, or the custom firmware you’ve built probably won’t
boot correctly.
Building and Flashing
After all the source code is downloaded and the development environment is set up, you can start building the
custom firmware. Make sure ccache is installed and configured (see http://source.android.com/
source/initializing.html#setting-up-ccache) because it will speed up subsequent builds.
304 Part III: Pushing the Limits
Building the platform the first time will take a bit of time (about 11⁄2 hours on my MacBook Pro with 16GB RAM).
First, you need to load the build environment into your current shell and tell the build system which device and
configuration you want to build.
$ source build/envsetup.sh
$ lunch full_maguro-userdebug
The first command loads the build system and sets all the right environment variables. The lunch command lets
you select the build target (when it’s called without arguments, you can manually select the target). In this case,
you want to build the userdebug configuration for the Galaxy Nexus (“maguro”). The userdebug configuration
allows you to access adb with root access and provides more privileges that enable you to debug the device and
write to the system partition, which is useful if you want to install a new system application later on.
Because building can take a significant amount of time, I suggest building on a high-end computer with a fast
disk (preferably a SSD)—although it’s possible to compile the platform on a low-end computer. To speed things
up, instruct the build system to use additional threads while building by giving the argument –jN to the make
command, where N is twice the number of hardware threads your computer supports. Generally, computers
can support two hardware threads per CPU core (using hyper-threading), so a computer with one CPU with four
cores will support eight threads.
$ make –j8
This is a good time to go and make some more coffee.
When the build is complete and, hopefully, successful, you can reboot your connected device in fastboot mode
and flash the firmware.
$ adb reboot-bootloader
$ fastboot flashall –w
When the process is complete, your device should boot up with your new custom firmware. Try running the
following commands from a terminal to confirm that you have root access:
$ adb root
restarting adbd as root
$ adb remount
remount succeeded
$ adb shell
root@android:/ #
The first command restarts the adb daemon on the device to run as root, giving it additional privileges. The
second command remounts the system partition on the device in read/write mode, allowing you to install
new system applications under /system/app. The final command simply confirms that you have root access
by opening a shell on the device. If the command line starts with root@android, you know that you have
root access.
Congratulations, you now have your very own custom firmware on which you have full root control and the
possibility to install new system and platform-signed applications.
Chapter 16: Hacking the Android Platform 305
Writing System Applications
As I describe in Chapter 3 and 12, Android’s permission system has various protectionLevels that describe
who can use a specific permission. If the permission has its protectionLevel set to system, the application
that uses this permission must initially be installed on the system partition (/system/app, to be specific). In
this way, third-party developers can build applications that the manufacturer installs on the system partition
and which then have access to system-level permissions. Note that updates to this application will retain the
same permissions, even though the updates are placed on the data partition. This is how Google’s Android
applications (such as Gmail, Google+, and Google Map) can gain additional permissions even though the device
manufacturer doesn’t sign them.
As a third-party developer, as long as you can get your application in the system partition for a device, you can
provide upgrades to users through the Google Play Store.
Along with the challenge of getting your system application on a manufacturer’s device, you should
also define a new system permission that the manufacturer then adds to the system permissions. By
using this permission in your system application, you can effectively filter the application on Google
Play to make it invisible to users who don’t have these modifications.
To install an APK as a system application, you need to have write access to the system partition (see the end
of the “Building and Flashing” section). Then you can push the new APK to the system application folder as
shown here:
$ adb push <apk file> /system/app
After the application is placed on the system partition, the package manager sets it up properly. Your
application can now use permissions requiring protectionLevel system, even though it isn’t signed with
the manufacturer’s platform certificate.
The signature permissionLevel is usually set for even more-sensitive permissions. For instance,
the permission android.permission.FORCE_STOP_PACKAGES has its protectionLevel set
to signature, meaning that only applications signed with the vendor platform certificate can use APIs
that require this. Third-party developers who want to sell their solution to a manufacturer, which requires
signature permissions can do so by providing a project for the platform build with an unsigned APK, similar
to how hardware drivers are distributed for the Nexus devices.
Platform Certificates
You can find the default test keys used to sign your custom firmware under <aosp root>/build/target/
product/security. Because these keys are available publically, never consider them safe. Refer to the
README file in this directory for additional information and instructions on how to generate your own platform
keys. However, you can use them for developmental purposes when you want to test new system applications.
The format of these keys is different from the one used to sign Android applications using the Android SDK or
Android Studio, so before you can use them in your development environment, you need to convert them.
306 Part III: Pushing the Limits
$ openssl pkcs8 -inform DER -nocrypt -in platform.pk8 -out platform.pem
$ openssl pkcs12 -export -in platform.x509.pem -inkey platform.pem -out
platform.p12 -name android-platform -password pass: <password>
$ keytool -importkeystore -destkeystore platform.keystore -srckeystore
platform.p12 -srcstoretype PKCS12
You now have a keystore with the format recognized by the Android development tools. Simply use this
keystore when you want to sign your application with the platform certificate. Doing so allows your application
to use all permissions requiring signature permissionLevel. Also, you can now set the sharedUserId
attribute in the manifest to android.uid.system, which gives the application the same user ID as the
system server and grants access to additional system-level APIs.
To use this certificate in your application, copy the keystore to your application and add the following to your
Gradle build:
android {
signingConfigs {
debug {
storeFile file(“platform.keystore”)
storePassword “password”
keyAlias “android-platform”
keyPassword “password”
}
}
// ... omitted for brevity ..
}
Now subsequent debug builds from Android Studio must use the extracted platform key when signing your
application.
Remember: Never use this certificate for applications published on the Google Play Store. Only use it
for testing and demonstration.
Writing Signature-Signed Applications
In Chapter 15, I discuss how to use the hidden platform APIs in the WifiManager to detect whether tethering
is enabled. Using some of the additional hidden APIs together with the platform key, you can build your own
application for controlling Wi-Fi tethering, as shown in the following example:
<?xml version=”1.0” encoding=”utf-8”?>
<manifest xmlns:android=”http://schemas.android.com/apk/res/android”
package=”com.aptl.systemlevelapps”
android:sharedUserId=”android.uid.system”
android:versionCode=”1”
android:versionName=”1.0” >
Chapter 16: Hacking the Android Platform 307
<uses-permission android:name=”android.permission.CHANGE_WIFI_STATE”
/>
<uses-sdk
android:minSdkVersion=”17”
android:targetSdkVersion=”17” />
<application
android:allowBackup=”true”
android:icon=”@drawable/ic_launcher”
android:label=”@string/app_name”
android:theme=”@style/AppTheme” >
<activity
android:name=”.EnableTetheringActivity”
android:label=”@string/app_name” >
<intent-filter>
<action android:name=”android.intent.action.MAIN” />
<category android:name=”android.intent.category.LAUNCHER” />
</intent-filter>
</activity>
</application>
</manifest>
First, you add sharedUserId with the value android.uid.system to the manifest element to give your
application the correct user ID for controlling the Wi-Fi. Even though your application is now signed with the
platform certificate, you still need to declare the permissions your application requires, which is why you need
to add the permission CHANGE_WIFI_STATE to the manifest.
public class EnableTetheringActivity extends Activity {
private static final String TAG = “EnableTetheringActivity”;
private WifiManager mWifiManager;
private Method mSetWifiApEnabledMethod;
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
mWifiManager = (WifiManager) getSystemService(WIFI_SERVICE);
try {
Class clazz = WifiManager.class;
mSetWifiApEnabledMethod = clazz.getMethod(“setWifiApEnabled”,
WifiConfiguration.class, boolean.class);
} catch (NoSuchMethodException e) {
Log.e(TAG, “Error retrieving method setWifiApEnabled()”, e);
}
setContentView(R.layout.activity_main);
}
public void doEnableWifiTethering(View view) {
try {
if(mWifiManager.isWifiEnabled()) {
mWifiManager.setWifiEnabled(false);
}
308 Part III: Pushing the Limits
EditText ssidNameView = (EditText) findViewById(R.id.ssid_name);
String ssidName = ssidNameView.getText().toString();
EditText wifiPasswordView =
(EditText) findViewById(R.id.wifi_password);
String wifiPassword = wifiPasswordView.getText().toString();
WifiConfiguration wifiConfiguration = new WifiConfiguration();
wifiConfiguration.SSID = ssidName;
wifiConfiguration.preSharedKey = wifiPassword;
mSetWifiApEnabledMethod.invoke(mWifiManager,
wifiConfiguration, true);
} catch (IllegalAccessException e) {
Log.e(TAG, “Illegal access.”, e);
} catch (InvocationTargetException e) {
Log.e(TAG, “Invocation error.”, e);
}
}
}
The Activity shown here retrieves the hidden method setWifiApEnabled() from the WifiManager
class. When the click-callback doEnableWifiTethering() is called, the SSID and password are retrieved
and used to construct a default WifiConfiguration, which is then used as a parameter when invoking the
Method object.
You now have a way of controlling the state of Wi-Fi tethering through a standard Android application, which
just happens to be signed with the platform certificate.
Hacking the Android Platform
In this section, I describe how to work with the AOSP source code. The code examples are in Java, but the same
approach applies to native development as well.
Setting Up Your IDE
Although Android Studio works fine for developing regular Android applications, I recommend using
IntelliJ IDEA CE (Community Edition) for working with the AOSP source code. You can find this IDE at www.
jetbrains.com/idea. It’s also possible to work with AOSP in Eclipse, but I recommend IntelliJ IDEA CE
because of its superior performance and code navigation support.
Before opening the IDE and importing the source code, make a complete build as described in the section
“Building and Flashing” earlier in this chapter. Doing so generates the Java source files for platform resources,
which makes it easier to work with from the IDE.
After you have a complete build, you can generate the IntelliJ project files (and Eclipse .classpath file) by
executing the following command in the project root directory:
$ development/tools/idegen/idegen.sh
Read excludes: 3ms
Traversed tree: 50027ms
Chapter 16: Hacking the Android Platform 309
Now, you have two IntelliJ IDEA files in the root named android.ipr and android.iml.
Start IntelliJ IDEA CE and open the project file (android.ipr)—the first time you do this, IntelliJ will build an
index of all the source code, so it will take some time.
You can now start hacking the AOSP source code.
Android Projects
The Android platform consists of a number of projects, each in its own Git repository. Because there are more
than 300 of them, there’s not enough room in this book to cover and explain every project in the AOSP source
code. Instead, I’ll describe some of the more important ones, based on their path in the file system. Some
consist of several Git repositories (for instance, the packages project).
You can find a complete list of the projects in the file <aosp root>/.repo/project.list.
Frameworks/base
The central (and according to some, most important) project in the Android platform is frameworks/base.
This project contains most of the sources for the system server as well as many other components running
on an Android device. Here, you’ll also find the implementation of most of the Android APIs (except those
belonging to the java and javax packages). If you want to add a new system service or modify the Android
APIs, you make your changes here.
Packages
You’ll find all of the standard system applications in the packages project, divided into apps,
inputmethods, providers, screensavers, and wallpapers folders. The apps folder contains the
applications the user can see in the launcher, such as Email, Phone, and Calculator. The Providers folder
contains the system ContentProviders, such as the media or contacts provider. The Inputmethods
folder contains the default soft keyboards found in Android.
If you want to modify one of the existing system applications, this is where you start. Developers often start
modifying the default launcher application in the AOSP sources. You can find this app under packages/
apps/Launcher2.
Vendor
The vendor directory is not a single project but the place where vendor-specific platform projects
are contained. This is usually where the binary hardware drivers are placed. If you intend to provide a
manufacturer with your own project for the Android platform, such as a shared library with some proprietary
implementations, place it in this directory as well. Also, it’s best to place vendor-specific applications that
should be preinstalled in this directory. In fact, most vendor-specific code should end up here; this way, device
manufacturers can keep the Android Open Source Project code clean from source code that isn’t open-source.
310 Part III: Pushing the Limits
Here is the Android.mk for the Wi-Fi and Bluetooth drivers from Broadcom for the Galaxy Nexus:
LOCAL_PATH := $(call my-dir)
ifeq ($(TARGET_DEVICE),maguro)
include $(CLEAR_VARS)
LOCAL_MODULE := bcm4330
LOCAL_MODULE_OWNER := broadcom
LOCAL_SRC_FILES := bcm4330.hcd
LOCAL_MODULE_TAGS := optional
LOCAL_MODULE_SUFFIX := .hcd
LOCAL_MODULE_CLASS := ETC
LOCAL_MODULE_PATH := $(TARGET_OUT_VENDOR)/firmware
include $(BUILD_PREBUILT)
endif
You can find this makefile in <aosp root>/vendor/broadcom/maguro/prebuilt. The content of this
file is very similar to what a normal Android.mk looks like when you’re doing native development using the
Android NDK. The source is the binary file bcm4330.hcd, and its output is in the firmware directory on
the device.
When building your own vendor extension, refer to the existing build files for other projects and construct your
own. For Android applications (APK) that you want to include in the system partition, refer to the build files
found in the packages directory for examples of how your Android.mk should look.
Android Linux Kernel
When you’re building Android for a Nexus device, note that the Linux kernel is usually prebuilt and can be found
under <aosp root>/device (for instance, <aosp root>/device/samsung/tuna for the Galaxy
Nexus). If you need to modify the kernel for your device, you must download the Linux source code for that
device, modify the build configuration, build the image, and copy it to the right location.
The Linux kernel has a different license (GPLv2) than the Android platform (usually Apache License v2). The
interesting thing with the Linux kernel is that this license guarantees that you can get access to the source code
for every single Android device. While modifying the Linux kernel for your own custom firmware is beyond
the scope of this book, you can refer to the instructions from the chipset vendor for each device to learn more
about kernel development on its hardware platform.
Adding a System Service
When modifying the Android platform, a common task is to add a new system service. These are the services
that Android developers get when calling Context.getSystemService(). In the following code, I
show you how to add a new system service by modifying the code in frameworks/base. I call this service
HomeDetector, and its task is to send a broadcast when the device enters the location defined as home. This
class belongs to the location APIs; thus it is placed in the android.location package.
Chapter 16: Hacking the Android Platform 311
public class HomeDetector {
private static final String TAG = “HomeDetector”;
/**
* Sticky broadcast indicating that the device arrives
* or leave the defined home wifi.
*
*/
public static final String ACTION_HOME_LOCATION_CHANGED =
“android.location.HOME_LOCATION_CHANGED”;
/**
* Extra <code>boolean</code> indicating if the device arrived
* to the home location or not.
*/
public static final String EXTRA_AT_HOME = “atHome”;
private IHomeDetector mHomeDetectorService;
public HomeDetector(IHomeDetector homeDetector) {
mHomeDetectorService = homeDetector;
}
/**
* Set the home wifi.
*
* @param homeWifi
* @hide Should only be available to system applications
*/
public void setHomeWifi(WifiInfo wifiInfo) {
try {
mHomeDetectorService.setHomeWifi(wifiInfo);
} catch(RemoteException e) {
Log.e(TAG, “setHomeLocation.”, e);
}
}
}
The class HomeDetector will be the class that is returned when an application calls Context.
getSystemService() with the parameter Contex.HOME_DETECTOR (which you also need to add to the
Context class). This class takes an instance of HomeDetectorService as a constructor parameter, which
will contain the actual implementation for the service.
When adding a new system service in Android, the standard approach is to define an AIDL file that will be
used internally by the system server. You create this AIDL in the same package as the HomeDetector class
(android.location).
package android.location;
import android.location.Location;
312 Part III: Pushing the Limits
oneway interface IHomeDetector {
void setHomeWifi(in WifiInfo homeWifi);
}
The AIDL is compiled as an ordinary application-specific AIDL and allows you to implement the other parts of
your new system service.
The following code goes into SystemServer.run() in the package com.android.server. This is the
central class for all system services.
try {
Slog.i(TAG, “Home Detector”);
homeDetector = new HomeDetectorService(context);
ServiceManager.addService(Context.HOME_DETECTOR, homeDetector);
} catch (Throwable e) {
reportWtf(“starting Home Detector”, e);
}
The preceding code simply creates an instance of HomeDetectorService and starts the service.
Here is the actual implementation of the new service:
public class HomeDetectorService extends IHomeDetector.Stub {
private static final String TAG = “HomeDetectorService”;
public static final String HOME_WIFI_SSID = “homeDetector.ssid”;
public static final String HOME_WIFI_BSSID = “homeDetector.bssid”;
public static final String SET_HOME_WIFI_PERMISSION
= “android.permission.SET_HOME_WIFI”;
private Context mContext;
private boolean mCurrentState = false;
public HomeDetectorService(Context context, WifiManager wifiManager) {
mContext = context;
WifiInfo wifiInfo = wifiManager.getConnectionInfo();
if(wifiInfo != null) {
String ssid = SystemProperties.get(HOME_WIFI_SSID);
String bssid = SystemProperties.get(HOME_WIFI_BSSID);
mCurrentState = wifiInfo.getSSID().equals(ssid)
&& wifiInfo.getBSSID().
equals(bssid);
}
sendHomeDetectionBroadcast(mCurrentState);
IntentFilter intentFilter =
new IntentFilter(WifiManager.NETWORK_STATE_CHANGED_
ACTION);
mContext.registerReceiver(new WifiListener(), intentFilter);
}
Chapter 16: Hacking the Android Platform 313
public void setHomeWifi(WifiInfo homeWifi) throws RemoteException {
mContext.enforceCallingPermission(SET_HOME_WIFI_PERMISSION,
“Missing permission “ + SET_HOME_WIFI_PERMISSION);
SystemProperties.set(HOME_WIFI_SSID, homeWifi.getSSID());
SystemProperties.set(HOME_WIFI_BSSID, homeWifi.getBSSID());
}
public void sendHomeDetectionBroadcast(boolean state) {
Intent homeDetectorBroadcast =
new Intent(HomeDetector.ACTION_HOME_LOCATION_
CHANGED);
homeDetectorBroadcast.putExtra(HomeDetector.EXTRA_AT_HOME, state);
mContext.sendStickyBroadcast(homeDetectorBroadcast);
}
class WifiListener extends BroadcastReceiver {
@Override
public void onReceive(Context context, Intent intent) {
NetworkInfo networkInfo = intent.
getParcelableExtra(WifiManager.EXTRA_NETWORK_
INFO);
boolean newState = false;
if(networkInfo.getState().equals(NetworkInfo.State.CONNECTED))
{
String ssid = SystemProperties.get(HOME_WIFI_SSID);
String bssid = SystemProperties.get(HOME_WIFI_BSSID);
WifiInfo wifiInfo =
intent.getParcelableExtra(WifiManager.EXTRA_WIFI_
INFO);
newState = wifiInfo.getSSID().equals(ssid) &&
wifiInfo.getBSSID().
equals(bssid);
}
// Only send new broadcast if changed
if(newState != mCurrentState) {
mCurrentState = newState;
sendHomeDetectionBroadcast(mCurrentState);
}
}
}
}
The usual place to put a new system service is under com.android.server.<category>, where
<category> is the type of service you created (in this case, location). Note the call to Context.
enforceCallingPermission(), which verifies that the calling process has the right to call this method.
In this case, I created a passive service, but if your service needs to run its own thread, you need to implement
the Runnable interface and initialize a new Looper in the run() method, as shown here:
314 Part III: Pushing the Limits
@Override
public void run() {
// Set the correct thread priority
Process.setThreadPriority(Process.THREAD_PRIORITY_BACKGROUND);
Looper.prepare(); // Prepare the Looper
// Create a custom Handler for processing
mServiceHandler = new MySystemServiceHandler();
init(); // Perform initialization
Looper.loop(); // Start this thread as a Looper
}
With the run() method just shown, your background service can post Messages to the custom Handler
that has its own dedicated thread. In this way, you can perform blocking operations in your service that
otherwise could block the entire system.
The final step is to declare the new permission required for the setHomeWifi() method, which you do by
editing the AndroidManifest.xml of the system server, located in <aosp root>/frameworks/base/
core/res/. Locate the permissions belonging to the location group (android.permission-group.
LOCATION) and add the new permission there.
<!-- Allow an application to set the WiFi which should
be considered “home” -->
<permission android:name=”android.permission.SET_HOME_WIFI”
android:permissionGroup=”android.permission-group.LOCATION”
android:protectionLevel=”signature|system”
android:label=”@string/permlab_setHomeWifi”
android:description=”@string/permdesc_setHomeWifi” />
In this case, only system or signature applications are allowed to use this permission. If you want to create
a new public API that anyone can access, consider whether the protectionLevel should be normal or
dangerous.
Speeding Up the Platform Development Cycle
After you add your new system service, you’re ready to build the code and push the changes onto your device.
Because building the entire platform from scratch every time would take a significant amount of time, you can
use some tricks to reduce the turnaround time for this kind of development.
For example, instead of building the entire platform, build only the project you’ve changed. In the earlier
example in “Adding a System Service,” you modified the project frameworks/base, so give this project as a
parameter to the make command so that only that part is rebuilt.
$ make –j8 frameworks/base
This command compiles the code and builds the binaries for that specific project. Then you can push only the
changed binaries onto the Android device with the following commands:
Chapter 16: Hacking the Android Platform 315
$ adb shell stop
$ adb sync
$ adb shell start
The first command stops the Android system server, allowing you to safely push the new binaries onto your
devices without crashing them. The second command pushes the actual binaries. This command can also take
a second parameter stating which directory you want to synchronize. Finally, you call the third command to
restart the system server. Your newly modified system server will now start up together with the new service
you just added. This procedure lets you iterate through the development cycle quickly and try out small
changes to the system service and libraries, without the time-consuming task of having to shut down the
phone and flash a new firmware.
The preceding procedure doesn’t produce a new firmware image for each build. To do this, you have
to perform a new, full build. However, because you (hopefully) have enabled ccache and haven’t
deleted your build output directory, the second time around shouldn’t take too long.
Contributing to AOSP
Google is doing most of the new development on the Android platform, but a significant amount of code is
contributed by other companies and individuals through the Android Open Source Project. Manufacturers of
Android devices and chipset vendors are the largest contributors, but smaller companies also fix many bugs and
develop small features. Although Google has the final say in what ends up in the Android platform, it is always
interested in high-quality features that others are willing to contribute.
The first thing to do is visit the Android contribution discussion forum (https://groups.google.com/
forum/?fromgroups#!forum/android-contrib) and describe your new proposal there. If you want to fix a bug, visit the issue tracker for the Android platform (https://code.google.com/p/android/
issues/list) and report it there (don’t forget to check if it has already been reported!). When you feel comfortable that your proposed fix is okay, you can start your contribution process.
Contributions to one of the Android projects are described in detail at http://source.android.com/
source/contributing.html. Remember that your fix needs to maintain high-quality standard and conform to the Android Code Style Guidelines (see http://source.android.com/source/code-
style.html). Don’t expect that your contribution will get accepted right away or that the processing will go fast. Google deals with a huge number of contributions, and most never make it into the platform for various
reasons. Be patient and make sure that your code is well-tested and documented.
Figure 16-3 shows the workflow of a patch. As you can see, you must take a number of steps before your
contribution is accepted, but these steps help to guarantee that your code and feature is of high quality.
316 Part III: Pushing the Limits
Image reproduced from work created and shared by the Android Open Source Project (http://source.android.com/source/life-of-a-patch.html)
and used according to terms described in the Creative Commons 2.5 Attribution License.
Figure 16-3 The Android Open Source Project Contribution Workflow
Chapter 16: Hacking the Android Platform 317
Summary
In this chapter, I explained the basics of developing the Android platform. I showed how you can unlock your
Google Nexus device and flash it with a custom firmware and how you can download the entire Android
platform source code and relevant binary drivers to build your own version. I explained some of the reasons for
doing so from a business perspective. By working closely with a device manufacturer, your business can get its
proprietary third-party solutions that require system-level access onto a device and still be able to upgrade the
application through the Google Play Store.
I showed how you can build a system application and push it onto the system partition to test it on your custom
firmware. Next, I explained the steps for signing an application with the platform certificate in order to gain
access to permissions requiring signature permissionLevel.
I completed this chapter with an example for modifying the Android platform by adding a new system service.
Although the example is very simple and could just as well be created as a normal application, it demonstrates
how and where you should place your new code in the Android platform.
I believe that all Android developers would benefit from a full understanding of how the Android platform is
built and works, as well as how the platform certificates protect the sensitive platform APIs.
If you come up with a new feature for the Android platform, consider contributing it back to the Android Open
Source Project. There’s no guarantee it will be accepted by Google and merged into the Android source code,
but it’s a good way to practice participating in a large open-source project. Doing so also is great experience in
developing high-quality code—because the requirements from Google are very high.
Further Resources
Websites
Developing for the Android Open Source Project: http://source.android.com
The Gerrit site for reviewing patches to AOSP: https://android-review.googlesource.com
The main discussion group for Android platform and technologies: https://groups.google.com/
forum/?fromgroups#!forum/android-platform
The main discussion group for help with building the Android source code: https://groups.google.
com/forum/?fromgroups#!forum/android-building
The discussion group for those who want to port Android to a new device: https://groups.google.
com/forum/?fromgroups#!forum/android-porting
The main discussion group for those who want to contribute to the Android platform through AOSP:
https://groups.google.com/forum/?fromgroups#!forum/android-contrib
The public issue tracker for bugs in the Android platform: https://code.google.com/p/android/
XDA Developers forums: http://forum.xda-developers.com
Chapter 17
Networking, Web Service,
and Remote APIs
A majority of all Android applications perform some kind of network communication over the Internet. Even
single-player games today provide social integration with an online web service. So, the data an application
generates should be stored online so that users will have access to the same data from other devices.
As a developer of Android applications, you need to consider all aspects of networking. If your application
wastes networking resources, it will consume more power and possibly increase the cost to users because of
mobile data charges.
In this chapter, I cover how to perform network calls as efficiently as possible. I introduce my favorite APIs for
HTTP-based communication. Then I move on to describe three different types of web services and how you can
integrate them into your application. If you’ll be using a web service in your application, you can likely integrate
it by using a technique very similar to one of the three I describe.
I complete this chapter with guidelines for reducing the power your application consumes in regard to network
operations. All network calls come with an additional drain on the battery, and you want to reduce this drain as
much as possible.
Android Networking
Although Android supports both TCP and UDP communication, most of the network calls for applications are
done over HTTP, which is built on top of TCP. In this section, I write mostly about HTTP communication and
related topics. I briefly cover connectionless protocols, like mDNS, in Chapter 18.
The first and most important rule to remember when it comes to networking on Android is: Never perform
network calls on the main thread. Starting with Android version 3.0 (Honeycomb), the main thread is protected
by the system, and your application will crash if it tries to do a network call from the main thread.
Second, and almost as important: Always do networking on Android from a Service and avoid performing
network calls from within an Activity . There are several reasons for doing so, but the most important one is
that you need to consider the fast changes in the Activity state if you perform network operations there.
The user might press Home in the middle of a network call, just to return to your application a second later. By
moving all network operations to a Service, you can avoid this issue. It also makes for a better overall design
because with this approach, you reduce the amount of code in your Activities, making it less complicated.
320 Part III: Pushing the Limits
However, in many cases you will be forced to perform network operations directly from an Activity,
as I show later in this chapter in the section “Foursquare API Using OAuth2” Please consider these cases as
exceptions to the rule. As much as possible, try to perform the network calls from a Service.
I recommend using either a callback interface or the LocalBroadcastManager to communicate network
results between the Activity and the Service. Another way is to store the results from a network call
directly into a ContentProvider and let the provider notify registered clients about the changes to any
data, as shown in Figure 17-1.
4. ContentProvider
1. User triggers a
Activity
notifies all connected
network refresh
clients
Content
3. Service insert data to
Service
Provider
ContentProvider
2. Service fetches data
from the Internet
Internet
Figure 17-1 Network calls from a Service inserting data into a ContentProvider
that notifies an Activity
HttpUrlConnection
The Android API has two APIs for HTTP communication, the Apache HttpClient and HttpUrlConnection.
Both have the same level of support in regard to features, but only HttpUrlConnection is recommended
because it’s the one being actively maintained by Google. Google has also implemented a number of useful
features that you otherwise would have to implement yourself, such as transparent response compression and
response cache.
private void enableHttpResponseCache() {
try {
long httpCacheSize = 10 * 1024 * 1024; // 10 MiB
File httpCacheDir = new File(getCacheDir(), “http”);
Class.forName(“android.net.http.HttpResponseCache”)
.getMethod(“install”, File.class, long.class)
.invoke(null, httpCacheDir, httpCacheSize);
} catch (Exception httpResponseCacheNotAvailable) {
}
}
The response cache is available from Android 4.0 (ICS) on, so if your application will also run on an earlier
version of Android, use the preceding approach to initiate the cache using Reflections. If your application will
target Android 4.0 at the minimum, you can use the following code to install the response cache:
Chapter 17: Networking, Web Service, and Remote APIs 321
try {
HttpResponseCache httpResponseCache = HttpResponseCache.
install(new File(getCacheDir(), “http”), CACHE_SIZE);
} catch (IOException e) {
Log.e(TAG, “Error installing response cache!”, e);
}
Choose a suitable cache size for your application. If you’re retrieving only small amounts of data, pick a cache
of only a few megabytes. The cache is application-private, so you don’t have to worry about leaking cached
responses to other applications on the device.
A Simple HTTP GET
Most network calls from applications will be in the form of simple HTTP GET requests. Here is an example of
how to perform such a request:
public JSONObject getJsonFromServer(URL url,
long lastModifiedTimestamp) {
try {
HttpURLConnection urlConnection = url.openConnection();
urlConnection.setRequestMethod(“GET”);
urlConnection.setInstanceFollowRedirects(true);
urlConnection.setIfModifiedBecause(lastModifiedTimestamp);
urlConnection.setUseCaches(true);
urlConnection.connect();
if (urlConnection.getResponseCode()
== HttpURLConnection.HTTP_OK) {
if (urlConnection.getContentType().
contains(“application/json”)) {
int length =
(HttpURLConnection) urlConnection.
getContentLength();
InputStream inputStream = urlConnection.
getInputStream();
String jsonString = readStreamToString(inputStream, length);
return new JSONObject(jsonString);
}
} else {
// TODO: Error handling...
}
} catch (IOException e) {
Log.e(TAG, “Error perform HTTP call!”, e);
} catch (JSONException e) {
Log.e(TAG, “Error parsing JSON!”, e);
}
return null;
}
private String readStreamToString(InputStream inputStream, int length)
throws IOException {
try {
BufferedReader bufferedReader =
322 Part III: Pushing the Limits
new BufferedReader(new InputStreamReader(inputStream));
StringBuilder stringBuilder = new StringBuilder(length);
char[] buffer = new char[length];
int charsRead;
while ((charsRead = bufferedReader.read(buffer)) != -1) {
stringBuilder.append(buffer, 0, charsRead);
}
return stringBuilder.toString();
} finally {
inputStream.close();
}
}
This example is a typical one for using HttpUrlConnection to make an HTTP GET and parse the response to
a JSONObject. The important thing to note here is the readStreamToString() method. Although you
may find code examples for how to read an InputStream to a String online, the preceding shows the
correct way of reading from a stream. Without any exception, you should always exhaust an InputStream
when reading. If you don’t, you leave data waiting in the lower layers of the platform that could waste resources
and keep your device from going to power saver mode, which is one of the most common mistakes when doing
network calls, so keep this in mind whenever you’re reading from an InputStream.
Uploading Files to a Server
Many applications will send data, like images or other files, to an online server. This has turned out to be one of
the more complex issues for developers because the standard Java APIs (including Android) doesn’t provide an
obvious, straightforward method for this issue. Sending data using HTTP involves doing an HTTP POST request
with the data in the body. However, the body requires some special formatting, and a number of header fields
must be set correctly.
The following example shows the necessary steps for posting a file to a server using HTTP POST.
public int postFileToURL(File file, String mimeType, URL url)
throws IOException {
DataOutputStream requestData = null;
try {
long size = file.length();
String fileName = file.getName();
// Create a random boundary string
Random random = new Random();
byte[] randomBytes = new byte[16];
random.nextBytes(randomBytes);
String boundary = Base64.
encodeToString(randomBytes, Base64.NO_WRAP);
HttpURLConnection urlConnection
= (HttpURLConnection) url.openConnection();
urlConnection.setUseCaches(false);
urlConnection.setDoOutput(true);
urlConnection.setRequestMethod(“POST”);
Chapter 17: Networking, Web Service, and Remote APIs 323
// Set the HTTP headers
urlConnection.setRequestProperty(“Connection”, “Keep-Alive”);
urlConnection.setRequestProperty(“Cache-Control”, “no-cache”);
urlConnection.setRequestProperty(“Content-Type”,
“multipart/form-data;boundary=” + boundary);
// If larger than MAX_FIXED_SIZE - use chunked streaming
if (size > MAX_FIXED_SIZE) {
urlConnection.setChunkedStreamingMode(0);
} else {
urlConnection.setFixedLengthStreamingMode((int) size);
}
// Open file for reading...
FileInputStream fileInput = new FileInputStream(file);
// Open connection to server...
OutputStream outputStream = urlConnection.getOutputStream();
requestData = new DataOutputStream(outputStream);
// Write first boundary for this file
requestData.writeBytes(“--” + boundary + CRLF);
// Let the server know the filename
requestData.writeBytes(“Content-Disposition: form-data; name=\””
+ fileName + “\”;filename=\”” + fileName + CRLF);
// ...and the MIME type of the file
requestData.writeBytes(“Content-Type: “ + mimeType + CRLF);
// Read the local file and write to the server in one loop
int bytesRead;
byte[] buffer = new byte[8192];
while ((bytesRead = fileInput.read(buffer)) != -1) {
requestData.write(buffer, 0, bytesRead);
}
// Write boundary indicating end of this file
requestData.writeBytes(CRLF);
requestData.writeBytes(“--” + boundary + “--” + CRLF);
requestData.flush();
return urlConnection.getResponseCode();
} finally {
if (requestData != null) {
requestData.close();
}
}
}
The important part here is to understand the boundary that is used to tell the server where the file data
starts and ends in the request body. Also, notice the check on whether the file size is larger than MAX_
FIXED_SIZE (in bytes), where chunked streaming or fixed-length streaming mode is used. For chunked
streaming, the parameter 0 means “system default” for the chunk size, which is what most clients should use
324 Part III: Pushing the Limits
in this mode. Chunking basically means that the data is sent to the server in parts, with each part prepended
with the size. Chunked streaming can be more efficient for memory use and reduces the risk of getting an
OutOfMemoryException. However, using fixed-length streaming mode is usually faster, although it requires
more memory while performing the streaming.
Volley
The standard HttpUrlConnection works well for most occasions, but it easily becomes quite complex
when you’re doing a lot of different HTTP requests. You still need to perform the requests on a background
thread and to ensure that everything is closed and shut down properly when your application is done. To make
things a bit easier, Google has started working on a new HTTP library that intends to make networking for
Android applications easier and faster. You can retrieve the latest version of the Volley library by cloning the
Git repository.
$ git clone https://android.googlesource.com/platform/frameworks/volley
The Volley library is currently under development but is already stable enough for use in applications. It
provides a very easy-to-use API for doing the typical network calls, and it handles all background threading and
other low-level things for you.
The following example shows how you can use Volley in a Service that makes HTTP GET calls and treats the
response as JSON data:
public class VolleyExample extends Service {
private LocalBinder mLocalBinder = new LocalBinder();
private RequestQueue mResponseQueue;
@Override
public void onCreate() {
super.onCreate();
mResponseQueue = Volley.newRequestQueue(this);
mResponseQueue.start();
}
@Override
public void onDestroy() {
super.onDestroy();
mResponseQueue.stop();
mResponseQueue = null;
}
public void doJsonRequest(String url,
Response.Listener<JSONObject> response,
Response.ErrorListener error) {
JsonObjectRequest jsonObjectRequest
= new JsonObjectRequest(Request.Method.GET,
url, null, response, error);
mResponseQueue.add(jsonObjectRequest);
}
Chapter 17: Networking, Web Service, and Remote APIs 325
@Override
public IBinder onBind(Intent intent) {
return mLocalBinder;
}
public class LocalBinder extends Binder {
public VolleyExample getService() {
return VolleyExample.this;
}
}
}
You pass in two callbacks, one for a successful response and one for error. You don’t need to worry about
starting a new background thread for the network call or initiating the connection yourself; this is all taken
care of by the Volley library. All you need to do is add the new request to the RequestQueue so that it gets
processed.
public void doJsonUpdateRequest() {
Response.Listener<JSONObject> responseListener
= new Response.Listener<JSONObject>() {
@Override
public void onResponse(JSONObject response) {
handleJsonResponse(response);
}
};
Response.ErrorListener errorListener
= new Response.ErrorListener() {
@Override
public void onErrorResponse(VolleyError error) {
handleError(error);
}
};
mService.doJsonRequest(API_URL, responseListener, errorListener);
}
In the preceding example, you define a callback for a successful response and an error callback. Both of these
will run on a thread other than the main thread, so if you intend to modify your UI, be sure to post that update
using a Handler or an AsyncTask.
Currently, the Volley library doesn’t support uploading arbitrary binary files. If you need to do
uploading, use the method described in the earlier section, “Uploading Files to a Server.” However,
continue to use Volley for other requests if possible.
If you’re doing lots of different HTTP requests, I highly recommend using the Volley library. Because it’s also
hosted as a part of the Android Open Source Project, it’s likely to become a central part of the Android platform
in the future.
326 Part III: Pushing the Limits
OkHttp and SPDY
A big problem with HTTP is that it allows only one request and response at a time per connection, which forces
browsers and other clients to spawn multiple sockets in order to perform requests in parallel. This issue is less
problematic for clients because there will be relatively few connections from one application at any given time;
however, it makes a huge difference for servers. In 2009, Google started working on an updated HTTP protocol
to address these issues. The result was a new wire protocol named SPDY (pronounced speedy) that allows
multiple HTTP requests to be multiplexed over a single socket connection. This protocol has become a de facto
open standard for the next generation of HTTP. This protocol doesn’t replace HTTP; instead, it modifies how
requests and responses are sent over the wire. The IETF working group for HTTP has announced that it will start
the work on HTTP 2.0 and use the SPDY protocol as a starting point.
If you have full control of both the clients and the server, it may be worth investigating the use of SPDY as an
alternative to regular HTTP/1.1—doing so will greatly reduce the network’s load and increase the performance.
SPDY is already seeing good support in the major web browsers, and there are several implementations for
different platforms available, including Android.
If you choose to use SPDY as your communication protocol, I recommend that you choose the third-party
library called OkHttp, which is developed by Square, Inc., and available as open source on GitHub (http://
square.github.io/okhttp). The library is available as a Maven dependency by including the following
line in your Gradle script:
compile ‘com.squareup.okhttp:okhttp:1.1.1’
This library is simply a new and improved HTTP client with support for the SPDY protocol. It uses the
HttpUrlConnection interface, so switching to this in your existing code should require little work.
The following code shows how you can use the OkHttp library in Android:
public class OkHttpExample {
private final OkHttpClient mOkHttpClient;
public OkHttpExample() {
mOkHttpClient = new OkHttpClient();
}
public String okHttpDemo(URL url) throws IOException {
HttpURLConnection urlConnection = mOkHttpClient.open(url);
InputStream inputStream = null;
urlConnection.setRequestMethod(“GET”);
urlConnection.connect();
if (urlConnection.getResponseCode()
== HttpURLConnection.HTTP_OK) {
inputStream = urlConnection.getInputStream();
return readStreamToString(inputStream,
urlConnection.getContentLength());
}
return null;
}
Chapter 17: Networking, Web Service, and Remote APIs 327
private String readStreamToString(InputStream inputStream,
int length)
throws IOException {
try {
BufferedReader bufferedReader =
new BufferedReader(new InputStreamReader(inputStre
am));
StringBuilder stringBuilder = new StringBuilder(length);
char[] buffer = new char[length];
int charsRead;
while ((charsRead = bufferedReader.read(buffer)) != -1) {
stringBuilder.append(buffer, 0, charsRead);
}
return stringBuilder.toString();
} finally {
inputStream.close();
}
}
}
When you create a new instance of the OkHttpClient, it will set up everything you need, such as connection
polling and response cache. This implementation is extremely fast even on regular HTTP requests, so using
it may it be a good idea in general. When using OkHttp for SPDY communication, you’ll notice a huge
improvement in performance for your network calls.
Web Sockets
Web Sockets is the latest darling of the web and is an extension running on top of standard HTTP. It allows for
asynchronous message-based communication between a client and a server. It starts with a regular HTTP GET
request that contains special HTTP headers indicating that the client wants to upgrade the connection to a Web
Socket connection.
Here is an example of the clients GET request for initiating a Web Socket connection:
GET /websocket HTTP/1.1
Host: myserver.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: MjExMjM0MTI0MTI0MTI0MTIzCg==
Sec-WebSocket-Protocol: chat
Sec-WebSocket-Version: 13
Origin: http://myserver.com
If the client request is accepted, the server responds with the following:
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: HSmrc0sMlYUkAGmm5OPpG2HaGWk=
Sec-WebSocket-Protocol: chat
328 Part III: Pushing the Limits
The values in the preceding headers aren’t valid for every case but should be calculated according to
the Web Socket protocol specification. Normally, you don’t need to bother about these details when
using a readymade library for Web Socket communication.
When a Web Socket is established, both sides (client and server) can send messages to each other asynchronously.
More importantly, you now have a way to quickly notify the client from your server. Messages can be either text
or binary and are usually quite small. If you need to transmit a large file between the client and server, stick to
standard HTTP instead. Web Sockets is intended for sending notifications with a relatively small payload.
I recommend that you choose a suitable data format when communicating over Web Sockets. In most cases,
JSON is sufficient and allows for a simple implementation. JSON messages should be sent as text-messages over
Web Sockets. For more advanced scenarios with mixed-type data, I recommend using Google Protocol Buffers,
as I describe in Chapter 9.
Although you can implement your own Web Sockets client using the standard Socket class in Android, I highly
recommend that you use one of the many third-party libraries available. You can choose from several; the one I
show here is the one I consider most stable at the moment, the Web Socket implementation for Java by Nathan
Rajlich (also known as TooTallNate), which you can find at http://java-websocket.org. In addition,
it contains a server implementation for Web Sockets, which is useful in Chapter 18. To use this library, simply
include the following dependency in your Gradle file:
compile ‘org.java-websocket:Java-WebSocket:1.3.0’
When using Web Sockets through this library, you don’t use the HttpUrlConnection: instead, you create a
WebSocketClient and connect to a URI.
The following code is a complete example of using this library to connect to a Web Socket online:
public class ChatService extends Service {
private static final String TAG = “ChatService”;
private ChatWebSocketClient mChatWebSocketClient;
private ChatClient mChatClient;
private LocalBinder mLocalBinder = new LocalBinder();
public IBinder onBind(Intent intent) {
return mLocalBinder;
}
public IBinder onBind(Intent intent) {
return null;
}
public void connectToChatServer(URI serverUri) {
new ChatWebSocketClient(serverUri).connect();
}
Chapter 17: Networking, Web Service, and Remote APIs 329
public void disconnect() {
if (mChatWebSocketClient != null) {
mChatWebSocketClient.close();
}
}
public void setChatClient(ChatClient chatClient) {
mChatClient = chatClient;
}
public void sendMessage(String message) {
if(mChatWebSocketClient != null) {
mChatWebSocketClient.send(message);
}
}
public boolean isConnected() {
return mChatWebSocketClient != null;
public interface ChatClient {
void onConnected();
void onMessageReceived(String from, String body, Date timestamp);
void onDisconnected();
}
private class ChatWebSocketClient extends WebSocketClient {
public ChatWebSocketClient(URI serverURI) {
super(serverURI);
}
@Override
public void onOpen(ServerHandshake serverHandshake) {
// Called when the Web Socket is connected
mChatWebSocketClient = this;
if(mChatClient != null) {
mChatClient.onConnected();
}
Notification notification = buildNotification();
startForeground(1001, notification);
}
@Override
public void onMessage(String message) {
// Called when a text message is received
if(mChatClient != null) {
try {
JSONObject chatMessage = new JSONObject(message);
String from = chatMessage.getString(“from”);
String body = chatMessage.getString(“body”);
330 Part III: Pushing the Limits
Date timestamp =
new Date(chatMessage.getLong(“timestamp”));
mChatClient.onMessageReceived(from, body, timestamp);
} catch (JSONException e) {
Log.e(TAG, “Malformed message!”, e);
}
}
}
@Override
public void onMessage(ByteBuffer bytes) {
// Called when a binary message is received
}
@Override
public void onClose(int code, String reason, boolean remote) {
// Called when the connection is closed
mChatWebSocketClient = null;
if(mChatClient != null) {
mChatClient.onDisconnected();
}
stopForeground(true);
}
@Override
public void onError(Exception e) {
// Called on in case of communication error
}
}
private class LocalBinder extends Binder {
public ChatService getService() {
return ChatService.this;
}
}
}
This is a guide for implementing Web Socket support in your code. In a real-life application, you should add
additional security and error checks. The important thing here is that when you call WebSocketClient.
connect(), it spawns a new thread on which this Web Socket will live, which means you don’t have to do the
background thread yourself when using this library.
The ChatClient extends the WebSocketClient class and implements the different event callbacks. All
these callbacks occur on the thread that the Web Socket is running on, so never block these calls because doing
so will stall any other traffic on the same Web Socket.
It’s good practice to use the onOpen() and onClose() callbacks for determining when the you can start
communicating over the Web Socket. In the preceding example, the mChatClient member is set and reset
(to null) in the respective method, allowing for a simple null check when sending a message.
Chapter 17: Networking, Web Service, and Remote APIs 331
I return to the topic of Web Sockets in Chapter 18, where I describe how you can communicate directly between
two devices.
Integrating Web Services
Most web services that you will use in your Android app usually fit into one of three categories: those that
don’t require authentication for a user, those that do require authentication for a user but lack a native SDK for
Android, and those that require user authentication and provide an SDK for Android. In this section, I provide
three examples that illustrate each of these categories.
Google Static Maps v2
In Chapter 13, I cover the use of the new and powerful Location API for Android. Although it’s fairly easy to use,
it may sometimes be more than you need. If your application needs to show only a static map that the user
won’t interact with, you can use Google Static Maps v2 API instead, which is a web service from Google that
allows you to fetch a piece of Google Maps as a bitmap image (PNG, GIF, or JPEG).
Here is a simple static method for retrieving a static map:
public class StaticMapsFetcher {
public static final String BASE_URL
= “http://maps.googleapis.com/maps/api/staticmap”;
// TOOD Create this before release!
public static final String API_KEY = null;
public static final String UTF8 = “UTF-8”;
private static final String TAG = “StaticMapsFetcher”;
public static Bitmap fetchMapWithMarkers(String address,
int width,
int height,
String maptype,
List<String> markers) {
HttpURLConnection urlConnection = null;
try {
StringBuilder queryString = new StringBuilder(“?”);
if (address != null) {
queryString.append(“center=”).
append(URLEncoder.encode(address, UTF8)).
append(“&”);
}
if (width > 0 && height > 0) {
queryString.append(“size=”).
append(String.format(“%dx%d”, width, height)).
append(“&”);
}
if (maptype != null) {
queryString.append(“maptype=”).
append(maptype).append(“&”);
}
332 Part III: Pushing the Limits
if (markers != null) {
for (String marker : markers) {
queryString.append(“markers=”).
append(URLEncoder.encode(marker, UTF8));
}
}
if (API_KEY != null) {
queryString.append(“key=”).append(API_KEY).append(“&”;
}
queryString.append(“sensor=false”);
URL url = new URL(BASE_URL + queryString.toString());
urlConnection = url.openConnection();
urlConnection.connect();
if (urlConnection.getResponseCode()
== HttpURLConnection.HTTP_OK) {
BufferedInputStream bufferedInputStream
= new BufferedInputStream(urlConnection.
getInputStream());
return BitmapFactory.decodeStream(bufferedInputStream);
} else {
return null;
}
} catch (IOException e) {
Log.e(TAG, “Error fetching map!”, e);
} finally {
if (urlConnection != null) {
urlConnection.disconnect();
}
}
}
}
Note that the API_KEY is left as null; this is allowed, but not recommended, for the Google Static Maps API.
Before you release an application that uses this API, you should create a new API key using the Google API
Developer Console.
Also note how the use of URLEncoder.encode() for the values of each parameter. This guarantees that
whatever the value is, it will be encoded properly in the URL. When a web service is called, an error stating a bad
request is common because the server cannot parse the request parameters properly.
This example shows how the simplest web services can be integrated into an Android application. They
are one-way, don’t require authentication, and have only a few parameters. You can find more information
about the parameters for Google Static Maps API API at https://developers.google.com/maps/
Foursquare API Using OAuth2
When you use a web service that requires a user account, you need to authenticate the user in some way.
However, because your application will be a third-party client for this service, you need some way to provide
Chapter 17: Networking, Web Service, and Remote APIs 333
authentication that doesn’t compromise the user’s credentials, meaning the username and password shouldn’t
pass through your code. To solve this issue, a standard named OAuth was developed. This standard is now on
its second version and is called OAuth2. It was designed so that web applications can allow one website to
integrate with a second one on behalf of a user. To use it on Android, you route the authentication through a
WebView that displays the service authentication web page for the user.
Although it’s possible to implement all of the steps for OAuth2 yourself, I recommend using a third-party library
that hides away some of the complexity. The one I recommend is called Scribe and can be found at https://
github.com/fernandezpablo85/scribe-java. Scribe provides a simple wrapper on top of the
HttpUrlConnection class for OAuth2-enabled services, so it’s an excellent use for Android.
In this OAuth2 example, the user’s friends are displayed on Foursquare. Although many of the API calls for
Foursquare don’t require authentication (like searching for venues), getting a list of Foursquare friends will
obviously require authentication by the user.
The following code shows an Activity that you can use to get authorization for a user’s Foursquare account
using OAuth2:
public class OAuthActivity extends Activity {
public static final String CLIENT_ID
= “<Client ID from foursquare.com/developer>”;
public static final String CLIENT_SECRET
= “<Client SECRET from foursquare.com/developer>”;
public static final Token EMPTY_TOKEN = null;
public static final String ACCESS_TOKEN = “foursquare.access_token”;
private static final String TAG = “FoursquareOAuth2”;
private OAuthService mOAuthService;
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
}
@Override
protected void onResume() {
super.onResume();
mOAuthService = new ServiceBuilder()
.provider(Foursquare2Api.class)
.apiKey(CLIENT_ID)
.apiSecret(CLIENT_SECRET)
.callback(“oauth://foursquare”)
.build();
String authorizationUrl =
mOAuthService.getAuthorizationUrl(EMPTY_TOKEN);
WebView webView = (WebView) findViewById(R.id.oauth_view);
WebViewClient webViewClient = new WebViewClient() {
@Override
public boolean shouldOverrideUrlLoading(WebView view,
334 Part III: Pushing the Limits
String url) {
if (url.startsWith(“oauth”)) {
Uri uri = Uri.parse(url);
String oauthCode = uri.getQueryParameter(“code”);
Verifier verifier = new Verifier(oauthCode);
new GetTokenAccess().execute(verifier);
return true;