Why do some Android phones cause our app to throw an java.lang.UnsatisfiedLinkError?

Question

We're experiencing a java.lang.UnsatisfiedLinkError on some of the Android phones that are using our app in the market.

Problem description:

static
{
    System.loadLibrary("stlport_shared"); // C++ STL        
    System.loadLibrary("lib2"); 
    System.loadLibrary("lib3"); 
}

Crashes the app in on of the System.loadLibrary() lines with a java.lang.UnsatisfiedLinkError. java.lang.UnsatisfiedLinkError: Couldn't load stlport_shared from loader dalvik.system.PathClassLoader[dexPath=/data/app/app_id-2.apk,libraryPath=/data/app-lib/app_id-2]: findLibrary returned null

Solution approach

We started running some custom diagnostics on all our installs to check if every lib is unpacked in the /data/data/app_id/lib folder.

PackageManager m = context.getPackageManager();
String s = context.getPackageName();
PackageInfo p;
p = m.getPackageInfo(s, 0);
s = p.applicationInfo.dataDir;

File appDir = new File(s);
long freeSpace = appDir.getFreeSpace();

File[] appDirList = appDir.listFiles();
int numberOfLibFiles = 0;
boolean subFilesLarger0 = true;
for (int i = 0; i < appDirList.length; i++) {

    if(appDirList[i].getName().startsWith("lib")) {
        File[] subFile = appDirList[i].listFiles(FileFilters.FilterDirs);   
        numberOfLibFiles = subFile.length;
        for (int j = 0; j < subFile.length; j++) {
            if(subFile[j].length() <= 0) {
                subFilesLarger0 = false;
                break;
            }
        }
    }
}

On every test phone that we have numberOfLibFiles == 3 and subFilesLarger0 == true. We wanted to test if all libs are unpacked properly and are larger then 0 byte. In addition we're looking at freeSpace to see how much disk space is available. freeSpace matches the amount of memory that you can find in Settings --> Applications at the bottom of the screen. The thinking behind this approach was that when there is not enough space on the disk available that the installer might have problems unpacking the APK.

Real world scenario

Looking at the diagnostics, some of the devices out there do NOT have all 3 libs in the /data/data/app_id/lib folder but have plenty of free space. I'm wondering why the error message is looking for /data/app-lib/app_id-2. All our phones store their libs in /data/data/app_id/lib. Also the System.loadLibrary() should use a consistent path across installation and loading the libs? How can I know where the OS is looking for the libs?

Question

Anyone experiencing problems with installing native libs? What work arounds have been successful? Any experience with just downloading native libs over the internet when they are not existent and storing them manually? What could cause the problem in the first place?

EDIT

I now also have a user who runs into this problem after an application update. The previous version worked fine on his phone, an after an update the native libs seem missing. Copying the libs manually seem to cause trouble as well. He is on android 4.x with a non rooted phone without custom ROM.

EDIT 2 - Solution

After 2 years of spending time on this problem. We came up with a solution that works well for us now. We open sourced it: https://github.com/KeepSafe/ReLinker

Answer 1

EDIT: Since I got another crash report yesterday for one of my apps I dug a bit deeper into the matter and found a third very likely explanation for that problem:

Google Play Partial APK Update Goes Wrong

To be honest, I did not know about that feature. The APK file name suffix "-2.apk" made me suspicious. It is mentioned in the crash message of this question here and I could also find that suffix in the crash report of that customer of mine.

I believe the "-2.apk" hints at a partial update that probably delivers a smaller delta to Android devices. That delta apparently does not contain native libraries when they did not change since the previous version.

For whatever reason the System.loadLibrary function tries to look up the native library from the partial update (where it doesn't exist). It's both a flaw in Android and in Google Play.

Here is a very relevant bug report with an interesting discussion about similar observations: https://code.google.com/p/android/issues/detail?id=35962

It looks like Jelly Bean may be flawed in terms of native library installation and loading (both crash reports that came in were Jelly Bean devices).

If this is really the case I suspect that some forced change in the NDK library code may fix the problem, like changing some unused dummy variable for each release. However, that should be done in a way that the compiler preserves that variable without optimizing it away.

EDIT (12/19/13): The idea of changing the native library code for each build unfortunately does not work. I tried it with one of my apps and got an "unsatisfied link error" crash report from a customer who updated anyway.

Incomplete APK installation

That's unfortunately just out of my memory and I cannot find a link anymore. Last year I read a blog article about that unsatisfied link issue. The author said that this is a bug in the APK installation routine.

When copying native libraries to their target directory fails for whatever reason (device ran out of storage space, maybe also messed up directory write permissions...) the installer still returns "success", as if native libraries were just "optional extensions" to an app.

In this case the only workaround would be reinstalling the APK while making sure there's enough storage space for the app.

However, I cannot find any Android bug ticket nor the original blog article anymore and I searched for it for quite a bit. So this explanation may be in the realms of myths and legends.

"armeabi-v7a" directory takes precedence over "armeabi" directory

This bug ticket discussion hints at a "concurrency issue" with installed native libraries, see Android bug ticket #9089.

If there is an "armeabi-v7a" directory present with just a single native library, the whole directory for that architecture takes precedence over the "armeabi" directory.

If you try to load a library that is just present in "armeabi" you'll get an UnsatisfiedLinkException. That bug has been flagged as "works as intended" by the way.

Possible workaround

In either case: I found an interesting answer to a similar question here on SO. It all boils down to packaging all native libraries as raw resources to your APK and copy on first app start the correct ones for the current processor architecture to the (app private) file system. Use System.load with the full file paths to load these libraries.

However this workaround has a flaw: since the native libraries will reside as resources in the APK Google Play won't be able to find them and create device filters for the mandatory processor architectures anymore. It could be worked around by putting "dummy" native libraries into the lib folder for all target architectures.

Overall I do believe this issue should be properly communicated to Google. It seems as if both Jelly Bean and Google Play are flawed.

It usually helps to tell the customer with that problem to reinstall the app. This is unfortunately not a good solution if app data loss is a concern.