Quick JSON serializer performance test (Json.NET vs ServiceStack)

SKallSKall Sami M. KallioUSMember ✭✭✭

No pun intended but this is just a quick test between two popular JSON (de)serializers available for Xamarin iOS and Android. I will run iOS comparison later on but from previous experiments Android shows more difference between the two libraries.

Source code is available at GitHub in case you want to extend the tests or run them yourself: https://github.com/sami1971/SimplyMobile

Tests were compiled and ran in Release mode.

[NUnitLite] NUnit automated tests loaded.
[Runner executing:  Run Everything]
[M4A Version:   ???]
[Board:     tuna]
[Bootloader:    PRIMELA03]
[Brand:     samsung]
[CpuAbi:    armeabi-v7a armeabi]
[Device:    maguro]
[Display:   JOP40D.I9250XWMA2]
[Fingerprint:   samsung/yakjuxw/maguro:4.2.1/JOP40D/I9250XWMA2:user/release-keys]
[Hardware:  tuna]
[Host:      DELL123]
[Id:        JOP40D]
[Manufacturer:  samsung]
[Model:     Galaxy Nexus]
[Product:   yakjuxw]
[Radio:     unknown]
[Tags:      release-keys]
[Time:      1359045633000]
[Type:      user]
[User:      dpi.sec]
[VERSION.Codename:  REL]
[VERSION.Incremental:   I9250XWMA2]
[VERSION.Release:   4.2.1]
[VERSION.Sdk:       17]
[VERSION.SdkInt:    17]
[Device Date/Time:  11/6/2013 11:07:32 PM]
TextSerializationTests
JsonNetTests
    [FAIL] TestBase.CanSerializeInterface : Newtonsoft.Json.JsonSerializationException : Could not create an instance of type TextSerializationTests.IAnimal. Type is an interface or abstract class and cannot be instantated. Path 'Pets[0].Name', line 1, position 62.
          at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateNewObject (Newtonsoft.Json.JsonReader reader, Newtonsoft.Json.Serialization.JsonObjectContract objectContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, Newtonsoft.Json.Serialization.JsonProperty containerProperty, System.String id, System.Boolean& createdFromNonDefaultConstructor) [0x00000] in <filename unknown>:0 
          at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject (Newtonsoft.Json.JsonReader reader, System.Type objectType, Newtonsoft.Json.Serialization.JsonContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, System.Object existingValue) [0x00000] in <filename unknown>:0 
          at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal (Newtonsoft.Json.JsonReader reader, System.Type objectType, Newtonsoft.Json.Serialization.JsonContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, System.Object existingValue) [0x00000] in <filename unknown>:0 
          at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList (IWrappedCollection wrappedList, Newtonsoft.Json.JsonReader reader, Newtonsoft.Json.Serialization.JsonArrayContract contract, Newtonsoft.Json.Serialization.JsonProperty containerProperty, System.String id) [0x00000] in <filename unknown>:0 
SimplyMobile.Text.JsonNet.JsonSerializer took 1583ms deserializing 10000 iterations.
    Passed TestBase.DeserializationSpeed
SimplyMobile.Text.JsonNet.JsonSerializer took 425ms serializing 10000 iterations.
    Passed TestBase.SerializationSpeed
JsonNetTests : 2583.71 ms
ServiceStackTests
    Passed TestBase.CanSerializeInterface
SimplyMobile.Text.ServiceStack.JsonSerializer took 364ms deserializing 10000 iterations.
    Passed TestBase.DeserializationSpeed
SimplyMobile.Text.ServiceStack.JsonSerializer took 187ms serializing 10000 iterations.
    Passed TestBase.SerializationSpeed
ServiceStackTests : 1332.855 ms
TextSerializationTests : 3936.737 ms
Tests run: 6, Passed: 5, Failed: 1, Skipped: 0, Inconclusive: 0
«1

Posts

  • SKallSKall Sami M. Kallio USMember ✭✭✭

    Another run without the headers and the one failed interface deserialization.

    Json.NET:

    JsonNetTests
    SimplyMobile.Text.JsonNet.JsonSerializer took 1717ms deserializing 10000 iterations.
        Passed TestBase.DeserializationSpeed
    SimplyMobile.Text.JsonNet.JsonSerializer took 423ms serializing 10000 iterations.
        Passed TestBase.SerializationSpeed
    JsonNetTests : 2547.028 ms
    

    ServiceStack:

    ServiceStackTests
    SimplyMobile.Text.ServiceStack.JsonSerializer took 468ms deserializing 10000 iterations.
        Passed TestBase.DeserializationSpeed
    SimplyMobile.Text.ServiceStack.JsonSerializer took 189ms serializing 10000 iterations.
        Passed TestBase.SerializationSpeed
    ServiceStackTests : 1119.476 ms
    
  • SKallSKall Sami M. Kallio USMember ✭✭✭
    edited November 2013

    Same code on iPad2 with iOS 7.

    LLVM & Thumb-2 instructions checked:

    Json.NET - deserialization 2657ms, serialization 1332ms
    ServiceS - deserialization 1185ms, serialization  907ms
    

    Unchecked:

    Json.NET - deserialization 2899ms, serialization 1447ms
    ServiceS - deserialization 1235ms, serialization  985ms
    
  • SKallSKall Sami M. Kallio USMember ✭✭✭

    Some interesting finding with WP8. While ServiceStack is slightly faster on iOS and completely obliterates Json.NET on Android, the tables are turned in WP8. ServiceStack still handles serialization faster but takes twice the time in de-serializing (tested with a low end Lumia 521, next up will be a Lumia 1020 but I doubt that will have much effect other than total times).

  • CheesebaronCheesebaron Tomasz Cielecki DKMember, University ✭✭✭✭

    I don't know why you are running these benchmarks, but the speed of ServiceStack has already been benchmarked and proved faster than many alternatives: http://www.servicestack.net/benchmarks/NorthwindDatabaseRowsSerialization.1000000-times.2010-02-06.html

  • SKallSKall Sami M. Kallio USMember ✭✭✭

    Those are for desktop (and I believe Windows .NET runtime, not Mono) benchmarks. The results can be quite different on different platforms and certainly on something so different as Xamarin.iOS is.

    These serializers also implement a common interface so you can use them both in PCL's through DI. With some configuration it is also possible to mix-and-match them if necessary.

    The portable version of protobuf-net seems to need some work as it currently has trouble passing the unit tests.

  • YSharpYSharp Cyril Jandia USMember

    Thank you for sharing Sami. Always interesting/good to know. For now this little one still is pretty tightly coupled to the full .NET:

    https://github.com/ysharplanguage/FastJsonParser#Performances

    (significant use of IL emit), but if anyone from this community does have firm interest in that sort of effort, I might consider trying the code downgrade for it to be able to run on our dear mobiles. Just let me know (via the GitHub).

    Cheers,

  • SKallSKall Sami M. Kallio USMember ✭✭✭
    edited December 2013

    Your code compiles without changes for Android. I assume it is only a deserializer without a serializer? All I can say from the below times is that if you are running on Android and using large data sets you should avoid Json.NET if possible.

    [Model:     SAMSUNG-SGH-I747]
    [VERSION.Release:   4.0.4]
    
    FastJson took 1798ms deserializing 100000 iterations.
    JsonNet took 16899ms deserializing 100000 iterations.
    ServiceStack took 1905ms deserializing 100000 iterations.
    
    FastJson took 1341ms deserializing 1 iterations.
    JsonNet took 11005ms deserializing 1 iterations.
    ServiceStack took 3188ms deserializing 1 iterations.
    

    It seems like your deserializer fails on DateTimeOffset.

  • SKallSKall Sami M. Kallio USMember ✭✭✭

    I created a hybrid FastJson/ServiceStack plugin serializer for my test framework:

    https://github.com/sami1971/SimplyMobile/blob/master/Core/Plugins/SimplyMobile.Text.FastJson/JsonSerializer.cs

    For WP8 support the code needs to change from SortedList to Dictionary (seems easy change) and some changes to enum handling (GetEnumUnderlyingType is not supported).

    iOS, now that's another story...

  • CheesebaronCheesebaron Tomasz Cielecki DKMember, University ✭✭✭✭
  • SKallSKall Sami M. Kallio USMember ✭✭✭

    Thanks for the hint. I was able to compile Sigil and Jil for Android, wrap the JSON (de)serializer inside a common interface but speed tests have to wait until later tonight.

  • YSharpYSharp Cyril Jandia USMember
    edited December 2013

    @SKall... yes, it's only a deserializer.

    Date/time is precisely the one data type with the poorest (i.e., most naive) support as of now. They're merely groked thru this one:

        private DateTime ParseDateTime(int outer)
    
  • YSharpYSharp Cyril Jandia USMember
    edited December 2013

    I just took care of having GetEnumUnderlyingType for all platforms hopefully @ latest commit eb288af83a

    I'll have to work on the replacement for SortedList on WP8 later tonight, as the code in a few places relies internally on that computed lexicographical order.

    Thanks,

  • YSharpYSharp Cyril Jandia USMember

    Replacement for SortedList (n/a on WP8) taken care of @ commit 7aee70ef74 (and also updated the NuGet, with v1.7).

    Thanks for the hints,

  • SKallSKall Sami M. Kallio USMember ✭✭✭

    Device:

    [Model:     SAMSUNG-SGH-I747]
    [VERSION.Release:   4.0.4]
    

    Deserializing:

    FastJson took 1843ms deserializing 100000 iterations.
    Jil took 379ms deserializing 100000 iterations.
    JsonNet took 16926ms deserializing 100000 iterations.
    ServiceStack took 2004ms deserializing 100000 iterations.
    

    Serializing:

    Jil took 627ms serializing 100000 iterations.
    JsonNet took 4816ms serializing 100000 iterations.
    ServiceStack took 2418ms serializing 100000 iterations.
    

    It looks though that Jil failed all of the sanity checks.

  • YSharpYSharp Cyril Jandia USMember

    I didn't know Jil would come also with a parser/deserializer... it seems a tough one to beat, then!

  • SKallSKall Sami M. Kallio USMember ✭✭✭

    Turned out Jil required a different interface, Serialize<(T obj) instead of Serialize(object obj) to work.

    Serialization speed in order:

    Jil             1014ms  100%
    ServiceStack    2075ms  205%
    JsonNet         4648ms  458%
    

    Deserialization speed in order:

    Jil           1656ms    100%
    FastJson      1871ms    113%
    ServiceStack  2197ms    133%
    JsonNet      17147ms    1035%
    
  • YSharpYSharp Cyril Jandia USMember
    edited December 2013

    @SKall... if you get a chance, I'd be especially interested to know about the parsers perfs on the device re: these two specific tests I prepared at my project page:

    1) the "highly nested arrays" test, with an input JSON I borrowed from Peter Ohler's "Oj" page

    2) the "dicos" test, with an input JSON using an enum type and a list of dictionaries

    These tests are interesting I think because of the discrepancies between implementations we can observe easily on the desktop, and also for their non-trivial payloads. Just 10K and 100K iterations should be enough to give us a good idea, I suppose (to account for init warm up, etc)

    Thank you again for this feedback above.

    Cheers,

  • SKallSKall Sami M. Kallio USMember ✭✭✭

    FastJson with Dicos, 10k:

    SimplyMobile.Text.FastJson.JsonSerializer
    ... Done, in 1,627 ms. Throughput: 5,679,164.11 characters / second.
        Memory used : 23,116,744
    

    Jil:

    [FAIL] FileLoadTests.DicosTest : System.TypeInitializationException : An exception was thrown by the type initializer for Jil.Deserialize.ISO8601StyleTypeCache`1
      ----> System.Exception : Only dictionaries with strings for keys can be deserialized
    

    Json.NET:

        [FAIL] FileLoadTests.DicosTest : Newtonsoft.Json.JsonSerializationException : Cannot deserialize the current JSON array (e.g. [1,2,3]) into type 'System.Collections.Generic.IDictionary`2[TextSerializationTests.SomeKey,System.String]' because the type requires a JSON object (e.g. {"name":"value"}) to deserialize correctly.\r\nTo fix this error either change the JSON to a JSON object (e.g. {"name":"value"}) or change the deserialized type to an array or a type that implements a collection interface (e.g. ICollection, IList) like List<T> that can be deserialized from a JSON array. JsonArrayAttribute can also be added to the type to force it to deserialize from a JSON array.\r\nPath 'Dictionaries[0]', line 4, position 7.
    

    ServiceStack:

    SimplyMobile.Text.ServiceStack.JsonSerializer
    ... Done, in 2,484 ms. Throughput: 3,719,806.76 characters / second.
        Memory used : 5,032,880
    
  • SKallSKall Sami M. Kallio USMember ✭✭✭

    Highly nested does not come with a POCO so it's a no-go.

  • SKallSKall Sami M. Kallio USMember ✭✭✭

    Tiny test with FastJson, 100k:

    SimplyMobile.Text.FastJson.JsonSerializer
    ... Done, in 4,380 ms. Throughput: 2,945,205.48 characters / second.
        Memory used : 28,245,096
    

    Jil: NA, see above

    Json.NET:

    SimplyMobile.Text.JsonNet.JsonSerializer
    ... Done, in 51,339 ms. Throughput: 251,270.96 characters / second.
        Memory used : 28,977,208
    

    ServiceStack:

    SimplyMobile.Text.ServiceStack.JsonSerializer
    ... Done, in 8,210 ms. Throughput: 1,571,254.57 characters / second.
        Memory used : 29,599,992
    
  • YSharpYSharp Cyril Jandia USMember
    edited December 2013

    @SKall... Thank you, Sami, great feedback :)

    On this one test run, three remarks:

    1) interesting, because providing support for deserialization into instances of

    (I)Dictionary<TKey, TValue>
    

    (where "TKey" is something else than just "string") is a use case that I wanted to support, and I just didn't know yet if Jil would pass this one.

    2) and on the same, btw, I found out that (on the desktop and for JSON.NET v5.0 r8, anyway) JSON.NET is in fact able to deserialize OK that very shape of input:

    "Dictionaries" : [
       [ { "Key": "Key0", "Value": "value at key 0" },
         ... ]
         ...
    ]
    

    but then you need to use the POCO "DictionaryDataAdaptJsonNetServiceStack" that I prepared for it and for ServiceStack (as opposed to just the "DictionaryData" which is more straightforward C# to express the intent, and that my parser groks fine, yes ;-)

    3) finally, the difference between the respective memory consumptions most likely precisely comes from the fact that we instantiate a List-of-List-of-KeyValuePairs in the former POCO, more lightweight than the List-of-Dictionaries in the latter (with its simpler expression in C# coming at the cost of .NET's Dictionaries being "heavier" to allocate than Lists)

    So, maybe make sure you use a JSON.NET >= v5.0 r8 if you want it to pass that, and get the corresponding timing for the curious.

    Thanks again,

    Cheers,

  • SKallSKall Sami M. Kallio USMember ✭✭✭

    Tiny tests with modified person DTO, no pun intended on the naming:

        public class TinyPersonNoDic
        {
            public int Id { get; set; }
            public string Name { get; set; }
            public Status Status { get; set; }
            public string Address { get; set; }
            public List<int> Scores { get; set; }
        }
    

    FastJson:

    SimplyMobile.Text.FastJson.JsonSerializer
    ... Done, in 4,001 ms. Throughput: 3,224,193.95 characters / second.
        Memory used : 28,215,792
    

    Jil:

    SimplyMobile.Text.Jil.JsonSerializer
    ... Done, in 6,520 ms. Throughput: 1,978,527.61 characters / second.
        Memory used : 28,267,208
    

    Json.NET:

    SimplyMobile.Text.JsonNet.JsonSerializer
    ... Done, in 48,890 ms. Throughput: 263,857.64 characters / second.
        Memory used : 28,373,496
    

    ServiceStack:

    SimplyMobile.Text.ServiceStack.JsonSerializer
    ... Done, in 7,996 ms. Throughput: 1,613,306.65 characters / second.
        Memory used : 25,442,584
    
  • SKallSKall Sami M. Kallio USMember ✭✭✭

    Highly nested deserialized to object, FastJson:

    SimplyMobile.Text.FastJson.JsonSerializer
    Count: 10k
    ... Done, in 2,600 ms. Throughput: 988,461.54 characters / second.
        Memory used : 60,146,176
    

    Jil:

    SimplyMobile.Text.Jil.JsonSerializer
        [FAIL] FileLoadTests.HighlyNestedTest : System.TypeInitializationException : An exception was thrown by the type initializer for Jil.Deserialize.NewtonsoftStyleTypeCache`1
      ----> System.ArgumentException : labels must have at least one element
    

    Json.NET:

    SimplyMobile.Text.JsonNet.JsonSerializer
    Count: 10k
    ... Done, in 9,573 ms. Throughput: 268,463.39 characters / second.
        Memory used : 117,266,592
    

    ServiceStack:

    SimplyMobile.Text.ServiceStack.JsonSerializer
    Count: 10k
    ... Done, in 179 ms. Throughput: 14,357,541.90 characters / second.
        Memory used : 9,889,672
    
  • YSharpYSharp Cyril Jandia USMember
    edited December 2013

    I suspect something went wrong with this one:

    SimplyMobile.Text.ServiceStack.JsonSerializer
    Count: 10k
    ... Done, in 179 ms. Throughput: 14,357,541.90 characters / second.
        Memory used : 9,889,672
    

    My bad probably, though: I might have to make more robust the check on that speed test's outcome, to make sure that whichever parser at hand is actually deserializing... anything and completely!

  • YSharpYSharp Cyril Jandia USMember
  • SKallSKall Sami M. Kallio USMember ✭✭✭
    edited December 2013

    Finally, HighlyNested deserialized into a POCO class (courtesy of json2csharp):

    FastJson:

    SimplyMobile.Text.FastJson.JsonSerializer
    Count: 10k
    ... Done, in 1,473 ms. Throughput: 1,744,738.63 characters / second.
        Memory used : 17,236,544
    

    Jil failed after a long run with an exception:

    System.ArgumentException : labels must have at least one element

    Json.NET:

    SimplyMobile.Text.JsonNet.JsonSerializer
    Count: 10k
    ... Done, in 16,584 ms. Throughput: 154,968.64 characters / second.
        Memory used : 35,209,688
    

    ServiceStack:

    SimplyMobile.Text.ServiceStack.JsonSerializer
    Count: 10k
    ... Done, in 3,948 ms. Throughput: 650,962.51 characters / second.
        Memory used : 27,885,448
    
  • YSharpYSharp Cyril Jandia USMember

    Cool! Thank you, Sami, that's indeed nice input, and quite encouraging for me, especially since I just had no clue until two days ago of where that might take us / "it" :)

    Laterz, Cheers,

  • SKallSKall Sami M. Kallio USMember ✭✭✭

    It does seem like a hybrid FastJson/ServiceStack serializer class might be a good and fast option for Android. If I am working with a backend built for the mobile client then it is worth noting ServiceStack has one nice function built in, being able to deserialize interfaces (by injecting the class name in the JSON output on serialization).

    All changes and tests are now on GitHub. You can use interface ITextSerializer (or IJsonSerializer if you want to be specific) from a PCL library SimplyMobile.Text and then inject the desired serializer at runtime. This way you don't need any code changes on iOS where FastJson isn't available.

    I will try WP8 sometime later next week.

  • YSharpYSharp Cyril Jandia USMember
    edited December 2013

    @SKall... that makes sense and sounds good. I'll give a try at implementing that ServiceStack's interface deserialization capability in my System.Text.Json.JsonParser (that you call "FastJson"), as well.

    Seems like it's a useful feature nice to have around, yes.

    The only (and significant) reserve I have for now re: this lean parser, is it does need, still, much more of conformance/unit tests, before people consider using it any seriously in their apps for Android (or WP8 later on).

    I can only encourage them to contribute their input test cases at the project home, and to raise over there all the bug-related issues they may find. I'd like to tackle all such those in priority, before whatever feature requests.

    Cheers,

  • SKallSKall Sami M. Kallio USMember ✭✭✭

    Quick test results with Json.NET and ServiceStack on iPod:

    Dicos, 10k:

    Json.Net: fails the same as in Android
    ServiceS: 5,109 ms, Memory used : 807,936
    

    Highly Nested, 10k:

    Json.NET: 23,327 ms, Memory used : 17,137,664
    ServiceS: 09,505 ms, Memory used : 12,575,744
    

    TinyTest, 100k:

    Json.NET: 59,608 ms, Memory used : 24,399,872
    ServiceS: 27,966 ms, Memory used : 23,929,856
    
«1
Sign In or Register to comment.