Mobile Development Platform Performance Part 2 (Native, Cordova, Classic Xamarin, Xamarin.Forms)

April 22, 2015 // By Kevin Ford

To see Part 1 of this blog series, click here.

This is the second in the series comparing sample test applications using vendor native development technologies (Objective-C, Java), Cordova (using Microsoft's Multi-Device Hybrid Apps), classic Xamarin and Xamarin Forms. This time I decided to focus on IO using the different frameworks. For this I used SqLite to save and retrieve records and also just writing and reading from plain text files.

For some background on testing methodology using Android and iOS.

The Development Platforms

• Native (Objective-C 64 bit and Java)
• Cordova (Multi-Device Hybrid Apps) using Intel's App Framework for the UI
• Classic Xamarin (64 bit unified beta for iOS)
• Xamarin.Forms (64 bit unified beta for iOS with Xamarin Forms version 1.3.1)

The Devices

• iPad Mini (non Retina) running iOS 8.1.1 (12B435)
• ASUS K00F running Android 4.2.2

The Test Apps

Applications were made for each of the development platforms that are functionally similar. There was little (if no) effort to make them look exactly the same or even look "good". But they looked about the same.

The Timing Methodology

Due to difficulties in knowing when things are "done", particularly with JavaScript, timings were handled via stopwatch. Each timing was taken ten times and the results were averaged. It should noted that hand timings have an accuracy of about 2/10 of a second so that does give us an approximate margin of error. In the previous post I showed the 10 individual timings and the average. This time to save me some typing I'm just showing the average of the ten timings.

Test 1: Test App Size

As mentioned in the last set of test, the size of the application can impact how much bandwidth it takes to deploy and also have some impact on load times. For Android the size of the APKs was examined. For iOS I looked ipa files for ad-hoc deployment.

A few interesting things to look at here. First is the Cordova apk is actually smaller than the vendor native tools on Android. I suspect this has to do with differences in size of the SqLite libraries for both of the platforms. Also while the Xamarin APKs are larger, they are not as large as they were on the first set of tests. For this I used the linking option of Link All Assemblies. This was probably important for Xamarin Forms which is, of course, just a library on top on Classic Xamarin. For whatever reason I was not able to get the Link All Assemblies option to stay with Xamarin iOS and you see the results with Link SDK assemblies only instead.

Test 2: Load Times

Like the last test I verified how long the applications took to load into memory. The results were similar to the set of tests I did previously.

As last time vendor native technologies load the fastest. Xamarin Classic follows closely with Xamarin Forms somewhat after that. In all cases, Cordova is the slowest loading. It is interesting that on iOS Xamarin Forms loaded almost as slowly as the Cordova app. Of course for iOS, nothing really took that long to load.

Test 3: Adding 1,000 records to SqLite

I originally wanted to add 10,000 records but found that too slow on my Android device and also it didn't scroll well under Cordova where my implementation had no auto paging. Offline storage is a common need with mobile applications and SqLite is one of the few solutions that span all platforms

Java:

public void addRecord(String firstName, String lastName, int index, String misc) throws Exception {
    if (dbConn == null) {
        openConnection();
    }

    ContentValues values = new ContentValues();
    values.put("firstName", firstName);
    values.put("lastName", lastName + index);
    values.put("misc", misc);
    dbConn.insertOrThrow(TABLE_NAME, null, values);
}

Objective-C:

- (void)addRecord:(NSString*)firstName withLastName:(NSString*)lastName withIndex:(int)index withMisc:(NSString*)misc withError:(NSError**)error {
    NSString *sqlStatement = NULL;
    char *errInfo;
    *error = nil;

    if (dbConn == nil) {
        [self openConnection:error];
        return;
    }
    
    sqlStatement = [NSString stringWithFormat:@"%@%@%@%@%d%@%@%@", @"INSERT INTO testTable (firstName, lastName, misc) VALUES ('", firstName, @"', '", lastName, index, @"', '", misc, @"')"];
    
    int result = sqlite3_exec(dbConn, [sqlStatement UTF8String], nil, nil, &errInfo);
    
    if (result != SQLITE_OK) {
        NSMutableDictionary *errorDetail = [NSMutableDictionary dictionary];
        [errorDetail setValue:[NSString stringWithFormat:@"%@%s", @"Error writing record to database: ", errInfo] forKey:NSLocalizedDescriptionKey];
        *error = [NSError errorWithDomain:@"testDomain" code:101 userInfo:errorDetail];
    }
}

JavaScript:

function addRecords() {
    $.ui.blockUI();
    var maxValue = 999;
    var successCount = 0;
    var db = window.sqlitePlugin.openDatabase({ name: "testDB.db" });

    for (var i = 0; i <= maxValue; i++) {
        var lastName = "person" + i.toString();
        db.executeSql("INSERT INTO testTable (firstName, lastName, misc) VALUES (?,?,?)", ["test", lastName, "12345678901234567890123456789012345678901234567890"], function (res) {
            successCount++;
            if (successCount === maxValue) {
                $.ui.popup({
                    title: "Success",
                    message: "All records written to database",
                    doneText: "OK",
                    cancelOnly: false
                });
                $.ui.unblockUI();
            }
        }, function (e) {
            $.ui.popup({
                title: "Error",
                message: "An error has occurred adding records: " + e.toString(),
                doneText: "OK",
                cancelOnly: false
            });
            $.ui.unblockUI();
            return;
        });
    }
}

Xamarin (All Versions):

public void AddRecord(string fName, string lName, int i, string m)
{
    if (dbConn == null)
    {
        OpenConnection();
    }

    var testRecord = new TestTable {firstName = fName, id = 0, lastName = lName + i, misc = m};

    dbConn.Insert(testRecord);
}

Xamarin Android Alternate:

public void AddRecord(string fName, string lName, int i, string m)
{
    if (dbConn == null)
    {
        OpenConnection();
    }

    var testRecord = new TestTable {firstName = fName, id = 0, lastName = lName + i, misc = m};

    dbConn.Insert(testRecord);
}

*results in seconds

Right off the bat you might notice that Xamarin on Android did poorly. Very poorly. Slower than native Android or Cordova by a large margin. One thing I noticed is that in native Android I used the SQLiteOpenHelper but this is only available in the Android API. I used the method I did because it was cross platform and you can see the results on Android. SQLiteOpenHelper is available in the SqLite Xamarin Library for Android and on a hunch I tried it instead and you can see those timings under Xamarin Classic Alternate for Android and the timing were virtually the same as they were for Java. I suspect if I used this for Xamarin.Forms I would have gotten about the same advantages. The lesson from this is just because there is a cross platform version of the API, it doesn't mean it will perform well. Buyer beware.

Test 4: Querying SqLite Records

I wanted to test reading the 1,000 records I just wrote. I only did the 1,000 I originally wrote because I didn't want to deal with paging solutions for Cordova. A 10,000 record test performed very poorly when just creating a 10,000 for HTML table.

Java:

public ArrayList getAllRecords() throws Exception {
    if (dbConn == null) {
        openConnection();
    }

    ArrayList returnValue = new ArrayList();
    String sqlStatement = "SELECT * FROM " + TABLE_NAME;
    Cursor cursor = dbConn.rawQuery(sqlStatement, null);

    if (cursor.moveToFirst()) {
        do {
            returnValue.add(cursor.getString(1) + " " + cursor.getString(2));
        } while (cursor.moveToNext());
    }

    return returnValue;
}

Objective-C:

- (NSMutableArray*)getAllRecords:(NSError**) error {
    NSString *sqlStatement = NULL;
    NSMutableArray *results;
    sqlite3_stmt *sqlResult;
    char *errInfo;
    *error = nil;
    
    if (dbConn == nil) {
        [self openConnection:error];
        return nil;
    }
    
    sqlStatement = @"SELECT * FROM testTable";
    results = [[NSMutableArray alloc] init];

    int result = sqlite3_exec(dbConn, [sqlStatement UTF8String], nil, nil, &errInfo);
    
    result = sqlite3_prepare_v2(dbConn, [sqlStatement UTF8String], -1, &sqlResult, nil);
    
    if (result == SQLITE_OK) {
        while (sqlite3_step(sqlResult)==SQLITE_ROW) {
            NSString* firstName = [NSString stringWithFormat:@"%s", (char*)sqlite3_column_text(sqlResult, 1)];
            NSString* lastName = [NSString stringWithFormat:@"%s", (char*)sqlite3_column_text(sqlResult, 2)];
            NSString* fullName = [NSString stringWithFormat:@"%@ %@", firstName, lastName];
        
            [results addObject:fullName];
        }
        sqlite3_finalize(sqlResult);
        return results;
    } else {
        NSMutableDictionary *errorDetail = [NSMutableDictionary dictionary];
        [errorDetail setValue:@"Error loading records" forKey:NSLocalizedDescriptionKey];
        *error = [NSError errorWithDomain:@"testDomain" code:101 userInfo:errorDetail];
        return nil;
    }
}

JavaScript:

function showSqlRecords(sql) {
    var db = window.sqlitePlugin.openDatabase({ name: "testDB.db" });
    db.executeSql(sql, [], function (res) {
        try {
            var lstRecords = $("#lstRecords");
            lstRecords.empty();
            var lastValue = res.rows.length;
            for (var i = 0; i < lastValue; i++) {
                var listItem = "

• " + res.rows.item(i).firstName + " " + res.rows.item(i).lastName + "

";
                lstRecords.append(listItem);
            }
        } catch (err) {
            alert(err.toString());
        }
    });
}

Xamarin (All Versions):

public IList GetAllRecords()
{
    if (dbConn == null)
    {
        OpenConnection();
    }

    var results = (from t in dbConn.Table() select t ).ToList(); 

    var returnList = new List();

    foreach (var result in results)
    {
        returnList.Add(string.Format("{0} {1}", result.firstName, result.lastName));    
    }
    return returnList;
}

Xamarin Android Alternate:

public IList GetAllRecords() 
{
    if (dbConn == null) {
        OpenConnection();
    }

    var returnValue = new List();
    var sqlStatement = "SELECT * FROM " + TABLE_NAME;
    var results = dbConn.RawQuery(sqlStatement, null);

    if (results.MoveToFirst()) {
        do 
        {
            returnValue.Add(results.GetString(1) + " " + results.GetString(2));
        } while (results.MoveToNext());
    }
    return returnValue;
}

*results in seconds

Reading records in Android was similarly bad for Xamarin on Android until I switched to the SQLiteOpenHelper implementation and then performance came online with the native example. As usual the interpreted JavaScript in Cordova slower but probably won't be a factor unless there is a large operation reading thousands of lines.

Test 5: Writing Lines to a File

I also wanted to test writing lines to a text file and how that performs. For each platform I write 1,000 lines to a file.

Java:

public void writeLineToFile(String line) throws Exception {
    if (!textFile.exists()) {
        this.createFile();
    }
    if (fileHandle == null) {
        this.openFile();
    }

    fileHandle.write(line);
}

Objective-C:

- (void)writeLineToFile:(NSError**)error withTextToWrite:(NSData*)textToWrite {
    *error = nil;

    if (fileHandle == nil) {
        [self openFile:error];
    }
    
    if (*error == nil) {
        [fileHandle seekToEndOfFile];
        [fileHandle writeData:textToWrite];
    }
}

JavaScript:

file.createWriter(function (fileWriter) {
    fileWriter.seek(fileWriter.length);
    var count = 0;
    var line;
    var message = "Writing line to file at index: ";
    var maxLines = 999;
    fileWriter.onwriteend = function(evt) {
        count += 1;
        if (count <= maxLines) {
            line = message + count + "\\n";
            fileWriter.write(line);
        } else {
            $.ui.unblockUI();
            $.ui.popup({
                title: "Success",
                message: "All lines written to file.",
                doneText: "OK",
                cancelOnly: false
        });
    }
};
line = message + count + "\\n";
fileWriter.write(line);

Xamarin (All Versions):

public void WriteLineToFile(String line) 
{
    if (!File.Exists(filePath)) 
    {
        this.CreateFile();
    }
    if (streamWriter == null) 
    {
        this.OpenFile();
    }

    streamWriter.WriteLine(line);
}

*results in seconds

When it comes to writing lines to a text file the vendor native technologies are the undisputed leaders in high performance. Xamain on iOS is close and a little slower on Android. Cordova comes in at 3-4 times slower than the other technologies. Some of this may come from the interpreted looping logic and not the plugin but the reality is that the overhead of the interpreted code that calls into the plug happens if you write one line or a thousand.

Test 6: Reading Lines from a File

Compliment to test 5, reading those 1,000 lines from a file and displaying them on a list.

Java:

public ArrayList readFileContents() throws Exception {

    BufferedReader reader = new BufferedReader(new FileReader(textFile));
    String line;
    ArrayList returnValue = new ArrayList();

    while((line = reader.readLine()) != null){
        returnValue.add(line);
    }
    reader.close();
    return returnValue;
}

Objective-C:

- (NSArray*)readFileContents:(NSError**)error {
    *error = nil;

    if (fileHandle == nil) {
        [self openFile:error];
    }
    
    if (*error == nil) {
        [fileHandle seekToFileOffset: 0];
        NSData* fileContents = [fileHandle readDataToEndOfFile];
        NSString *fileString = [[NSString alloc] initWithData:fileContents encoding:NSUTF8StringEncoding];
        return [fileString componentsSeparatedByString:@"\r\n"];
        
    } else {
        return nil;
    }
}

JavaScript:

file.file(function(innerFile) {
    var reader = new FileReader();

    reader.onerror = function(e) {
        alert("Error");
    }
    reader.onloadend = function (e) {
        var lines = e.target.result.split("\\n");
        var lstFileLines = $("#lstFileLines");
        lstFileLines.empty();
        if (lines.length > 1) {
            for (var i = 0; i < lines.length - 2; i++) {
                var listItem = "

• " + lines[i] + "

";
                lstFileLines.append(listItem);
            }
        }
    };
    reader.readAsText(innerFile);
}, function(ex) {
    $.ui.popup({
        title: "Error",
        message: "Error occurred opening file: " + ex.description,
        doneText: "OK",
        cancelOnly: false
});

Xamarin (All Versions):

public IList ReadFileContents()
{
    if (!File.Exists(filePath))
    {
        this.CreateFile();
    }

    var returnValue = new List();

    using (var streamReader = new StreamReader(filePath))
    {
        string line;
        while ((line = streamReader.ReadLine()) != null)
        {
            returnValue.Add(line);
        }
    }
    return returnValue;
}

*results in seconds

You want fast, go for native. Xamarin is as good or nearly as good except for Form on Android. This may have more to do with the performance of the list display than the actual reading of the list. That should be comparable to classic Xamarin. Same comments on Cordova as before, it's slower. If this is a factor or not depends a lot about the type of application.

I hope some of you find these comparisons helpful. The lessons I learned from this set of tests were:

• It is self-evident that not all implementations of code even on the same development platform perform the same. Having said that, be careful of wrappers around libraries that have unified APIs to allow for cross platform code. It may not perform as well as the specialized libraries for Android or iOS.
• While it is hard to differentiate slowness in code that comes from it being interpreted and what comes from the libraries they are using, to some extent it doesn't matter. In production apps the Cordova JavaScript code will be interpreted as it calls into plugins like SqLite so both will be a factor.
• The linking settings on Xamarin can have a huge difference in the size of the app. If you are using external libraries where you are not likely to use all the functionality use the Link All Assemblies option if possible.

Thanks everyone.

Source code for the tests can be found here: Performance Tests Source

Kevin Ford is a Practice Lead at Magenic. If you’d like to speak to us directly, contact us or give us a call at 877-493-9369.

Categories // Mobile
Tags // Cordova, Java, Objective-C, Performance, Windows Azure, Xamarin, Xamarin.Forms