Mobile Development Platform Performance

APRIL 7, 2015 // By Kevin Ford

Last month I published my Mobile Technology Decision Making White Paper and I feel that goes a long way to helping choose what technology to use in the mobile space. One question I get asked frequently is how do the different mobile development platforms compare to each other from a performance perspective. While I've heard a lot of anecdotal information I have not seen a lot of formal comparisons. At Magenic we normally work with native (iOS, Android, WP), Cordova and Xamarin so I wanted to compare these. I plan on this being the first post on this topic.

For some background on testing methodology using Android and iOS.

The Development Platforms

  • Native (Objective-C 64 bit and Java)
  • Cordova (Multi-Device Hybrid Apps) using Intel's App Framework for the UI
  • Classic Xamarin (64 bit unified beta for iOS)
  • Xamarin.Forms (64 bit unified beta for iOS with beta version of Xamarin.Forms, note latest version of the unified API in the beta/alpha channels could not be used as it is not supported by Xamarin.Forms)

The Devices

  • iPad Mini (non Retina) running iOS 8.1.1 (12B435)
  • ASUS K00F running Android 4.2.2

The Test Apps

Applications were made for each of the development platforms that are functionally similar. There was little (if no) effort to make them look exactly the same or even look "good".  But they looked about the same.  There were some differences such as Java, Classic Xamarin and Xamarin.Forms rendered the tabs on the top in Android as expected while the JavaScript library showed them on the bottom.

The Timing Methodology

Due to difficulties in knowing when things are "done", particularly with JavaScript, timings were handled via stopwatch.  Each timing was taken ten times and the results were averaged.  It should noted that hand timings have an accuracy of about 2/10 of a second so that does give us an approximate margin of error.

Test 1: Test App Size

The size of the application can impact how much bandwidth it takes to deploy and also have some impact on load times.  For Android the size of the APKs was examined.  For iOS I looked at Settings to find out how much space the apps took up on disk/

Development Platform Size
Android
Java 166kb
Cordova 433kb
Classic Xamarin 3.5mb
Xamarin.Forms 4.7mb
iOS
Objective-C (64 bit) 644kb
Cordova 2.7mb
Classic Xamarin 12.1mb
Xamarin.Forms 16.9mb

When it comes to application size Xamarin shows the extra size involved in the overhead of the .Net framework.  There was an attempt to reduce the size of the deployed Xamarin application by using the "Link SDK assemblies only" setting.  I am surprised in a very small application how large the difference is.  However, from experience in "real" applications the difference is much less consequential as graphics and frameworks get added to the projects.

Test 2: Load Times
I wanted to see how long it took the application to load into memory.  While the initial load time is important, many mobile applications tend to stay in memory so it tends to have a limited impact.  For this test I made sure to close all applications before each timing.

Development Platform Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 Test 8 Test 9 Test 10 Test Avg.
Android
Java 1.47 .81 .9 1.24 1.22 1.06 1.05 .98 1.07 1.05 1.085
Cordova 4.13 3.89 4.03 3.94 4.01 3.88 4.06 3.98 4 3.86 3.978
Classic Xamarin 1.73 1.78 1.7 1.59 1.68 1.78 1.59 1.77 1.68 1.74 1.704
Xamarin.Forms 2.92 2.76 2.72 2.75 2.89 2.82 2.57 2.76 2.71 2.74 2.764
iOS
Objective-C 1.19 1.23 1.16 1.28 1.4 1.35 1.13 1.18 1.16 1.13 1.221
Cordova 1.95 1.77 1.43 1.69 1.73 1.87 1.76 1.59 1.75 1.61 1.715
Classic Xamarin 1.31 1.29 1.26 1.34 1.32 1.2 1.29 1.24 1.33 1.22 1.28
Xamarin.Forms 1.92 1.76 1.77 1.9 1.76 1.8 1.86 1.74 1.85 1.77 1.813

In all cases the vendor native technologies loaded the fastest.  Classic Xamarin loaded nearly as fast as the native languages.  Xamarin.Forms and Cordova had the slowest load times.  The Cordova load time on Android was particularly bad while on iOS the load times were close enough to not be a huge factor.

Test 3: Loading a List from Azure Mobile Services
In this test I wanted to look at getting data from an external service so I loaded 1000 records from Azure Mobile Services.  For Xamarin iOS 64 bit I had to modify the Azure Mobile Services to be compatible with the unified API.  The timings were taken from pushing the button to load the list until the results visibly came back and were displayed on a list on the screen.

Java:

mobileServicesClient.getTable(Registration.class).top(1000).execute(new TableQueryCallback<Registration>() {
       @Override
       public void onCompleted(List<Registration> result, int count, Exception exception,
                               ServiceFilterResponse response) {
     final List<Registration> registrations = result;
     runOnUiThread(new Runnable() {
               @Override
               public void run() {
                   resetList(registrations);
               }
           });
       }
    }
);

Objective-C:

-(IBAction) btnFetchClicked:(id)sender {

    MSTable *table = [client tableWithName:@"Registration"];

    MSQuery *query = [table query];    

    query.fetchLimit = 1000;
    [query readWithCompletion:^(NSArray *items, NSInteger totalCount, NSError *error) {
        if(error) {
            NSLog(@"ERROR %@", error);
        } else {
            tableData = items;
            [self.grdRegistrations reloadData];
        }
    }];
}

JavaScript:

client.getTable("Registration")
    .take(1000)
    .read()
    .done(function (results) {
        var lstRegs = $("#lstRegistrations");
        lstRegs.empty();
        for (var i = 0; i < results.length; i++) {
            var listItem = "<li>" + results[i]["screenname"] + "</li>";
            lstRegs.append(listItem);
        }
        lstRegs.refresh();
        $.refresh();
    }
)

Xamarin (All Versions):

public static async Task<IList<Registration>> GetRegistrationsAsync()
{
    return await mobileService.GetTable<Registration>().Take(1000).ToListAsync();
}

 

Development Platform Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 Test 8 Test 9 Test 10 Test Avg.
Android
Java 2.25 2.31 2.41 2.39 2.41 2.32 2.68 2.42 2.16 2.34 2.369
Cordova 2.23 2.14 2.02 2.02 2.38 2.14 2.12 2.02 1.94 2.48 2.149
Classic Xamarin 1.83 1.68 1.76 1.64 1.71 1.83 1.61 1.63 1.84 1.85 1.738
Xamarin.Forms 1.99 1.76 2.32 1.91 1.9 1.58 1.93 2.02 2.03 1.64 1.908
iOS
Objective-C 2.38 2.44 2.24 2.3 2.34 2.32 2.32 2.35 2.2 2.27 2.316
Cordova 3.57 2.18 2.07 1.95 1.97 2.05 2.04 1.93 2.2 1.96 2.192
Classic Xamarin 2 1.87 1.88 2.06 1.74 1.9 1.81 1.94 1.75 1.96 1.891
Xamarin.Forms 2.11 2.01 2.23 1.96 1.95 2.07 2.12 2.16 2.08 2.1 2.079

*results in seconds>

In many ways this test is showing how well the Azure Mobile Services libraries perform on the different platforms.  Unsurprisingly Xamarin, with it's underpinnings of a .Net implementation, performs the best in this test.  I was surprised to see the libraries for the native technologies perform the worst, both on Android and iOS.

I have heard that the Xamarin.Forms lists can perform poorly with large data sets.  These results did not show that, at least with lists of up to 1000 records.

Test 4: Prime Number Calculation

In the final of my first series of tests I wanted to try out a CPU intensive operation.  I created a Sieve of Eratosthenes on each of the platforms.  My first plan was to calculate all prime numbers up to 50,000,000.  This required some special handling of the method's array for both Objective-C and JavaScript.  In the case of Objective-C I had to malloc memory to support arrays that large.  Also for Objective-C and JavaScript I had to initialize the array items to 0.  To keep the timings the same I did the array item initialization to 0 on all platforms even though it could have been left out for .Net (and Java I believe).  It that was done, the .Net timings would have been even better.

I did end up having to settle for only calculating primes up to 5,000,000.  The reason for this is that the JavaScript performed so poorly that I was unwilling to wait for it to complete 10 times.

Java:

private int getPrimesFromSieve(int maxValue)
{
    byte[] primes = new byte[maxValue + 1];
    for (int i = 0; i <=maxValue; i++)
    {
        primes[i] = 0;
    }
    int largestPrimeFound = 1;

    for (int i = 2; i <=maxValue; i++)
    {
        if (primes[i - 1] == 0)
        {
            primes[i - 1] = 1;
            largestPrimeFound = i;
        }

        int c = 2;
        int mul = i*c;
        for (; mul <= maxValue;)
        {
            primes[mul - 1] = 1;
            c++;
            mul = i*c;
        }
    }
    return largestPrimeFound;
}

Objective-C:

- (int) getPrimesFromSieve: (int) maxValue {
    Byte *primes;
    primes = (Byte *) malloc(maxValue * sizeof(Byte));
    for (int i=1; i<=maxValue; i++)
    {
        primes[i-1] = 0;
    }    

    int largestPrimeFound;
    largestPrimeFound = 1;

    for (int i=2; i<=maxValue; i++)
    {
        if(primes[i-1] == 0)
        {
            primes[i-1] = 1;
            largestPrimeFound = i;
        }

        int c=2;
        int mul = i*c;
        for(; mul <= maxValue;)
        {
            primes[mul-1] = 1;
            c++;
            mul = i*c;
        }
    }
    return largestPrimeFound;
}

JavaScript:

function getPrimesFromSieve(maxValue) {
    var primes = new Uint8Array(new ArrayBuffer(Number(maxValue)));
    for (var i = 0; i <=maxValue; i++) {
        primes[i] = 0;
    }
    var largestPrimeFound = 1;

    for (i = 2; i <= maxValue; i++) {
        if (primes[i - 1] == 0) {
            primes[i - 1] = 1;
            largestPrimeFound = i;
        }

        var c = 2;
        var mul = i * c;
        for (; mul <= maxValue;) {
            primes[mul - 1] = 1;
            c++;
            mul = i * c;
        }
    }
    return largestPrimeFound;
}

Xamarin (All Versions):

public static int GetPrimesFromSieve(int maxValue)
{
    var primes = new byte[maxValue + 1];
    for (var i = 0; i <=maxValue; i++)
    {
        primes[i] = 0;
    }
    var largestPrimeFound = 1;

    for (var i = 2; i <=maxValue; i++)
    {
        if (primes[i - 1] == 0)
        {
            primes[i - 1] = 1;
            largestPrimeFound = i;
        }

        var c = 2;
        var mul = i*c;
        for (; mul <= maxValue;)
        {
            primes[mul - 1] = 1;
            c++;
            mul = i*c;
        }
    }
    return largestPrimeFound;
}

 

Development Platform Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 Test 8 Test 9 Test 10 Test Avg.
Android
Java 4.31 4.31 4.2 4.33 4.39 4.37 4.32 4.45 4.34 4.4 4.342
Cordova 91.69 95 94.31 94.4 94.73 94.1 94.1 91.8 93.63 97.75 94.151
Classic Xamarin 4.27 4.25 4.15 4.32 4.51 4.41 4.22 4.12 4.14 4.19 4.258
Xamarin.Forms 4.21 4.17 4.31 4.3 4.2 4.34 4.29 4.36 4.22 4.19 4.259
iOS
Objective-C 5.04 5.49 5.38 4.86 4.8 5.02 5.03 4.83 4.84 4.85 5.014
Cordova 66.96 67.36 67.22 67.3 67.17 67.44 67.13 67.11 67.58 67.64 67.291
Classic Xamarin 4.41 4.42 4.35 4.34 4.49 4.37 4.17 4.27 4.39 4.28 4.349
Xamarin.Forms 4.51 4.33 4.31 4.31 4.33 4.4 4.41 4.4 4.33 4.46 4.379

*results in seconds

While I was expecting that JavaScript would be slower, I was unprepared for how much worse it was for this type of operation.  This would make Cordova problematic for highly CPU bound work were performance is important (I wonder what that means for server side Node.js...).  CPU bound work is not as important in many of today's mobile applications but given the history of increased performance for mobile CPUs and what happened in the PC market in the 1990's, it is likely that in the future CPU bound work will be more prevalent in mobile applications.

Having said that, using native HTML commands can be very fast.  In the past I've testing loading JSON using HTML commands vs. the JSON.Net library on Xamarin and found them nearly comparable.

I was also surprised that Xamarin performed better than Objective-C by a noticeable amount.  It also performed better than Java on Android but by a very marginal amount, well within the .2 second margin of error that manual timings give us.

That's it for my first installment of performance tests.  Much of this code was made more difficult for Xamarin due to the flux around the 64 bit Unified iOS API.  For next month I'll take a look at loading large JSON strings and perhaps something else.

I hope this is useful or you.  If you have any ideas for performance tests I can perform, I'd love to hear them.

Continue to read part 2: Mobile Development Platform Performance Part 2 (Native, Cordova, Classic Xamarin, Xamarin.Forms)

Kevin Ford is a Practice Lead at Magenic. This is republished from his blog and can be found here. If you’d like to speak to us directly, contact us or give us a call at 877-277-1044.

Categories // Mobile
Tags // Java, Objective-C, Performance, Windows Azure, Xamarin, Xamarin.Forms

Get Started

Contact Us