Best Practice For Cross Platform Data Synchronization

thedigitalseanthedigitalsean USUniversity ✭✭
edited September 2014 in Cross Platform with Xamarin

I have been doing a lot of research and am trying to wrap my head around an architecture for cross platform data sync. The current architecture of my app is shared code in a PCL, using SQlite.Net (and SQLiteNetExtensions) for local storage. The server-side team has chosen Cassandra clusters running in Azure VMs on the backend which we will interface to with a REST API via ServiceStack. We have a few tables that can be edited at multiple locations so these tables may be vulnerable to conflicts. I am trying to figure out a good synchronization strategy for these tables. I haven't used it before but I think I want something like Android's Sync Adapter but cross platform. Unfortunately I haven't found such a solution.

So far I have found the following:
1) Couchbase
Looks great and has turn-key sync solution for mobile. However, we are currently headed down the path with Cassandra. I suppose I could store just the conflict-vulnerable tables in Couchbase but this seems les than ideal.

2) Azure Mobile Services
I just found this and am not even sure if I can use it in with Cassandra. Does anyone have experience with this?
https://github.com/conceptdev/xamarin-forms-samples/tree/master/TodoAzureSync

3) Microsoft Sync Framework
I found a few ports of this to Xamarin. But it appears to be a port of a codebase that Microsoft abandoned. I'm not sure if it's something I can really trust. Again, I also wonder about how to integrate with my Cassandra backend.
https://github.com/Mimetis/SyncWinRT
https://github.com/wmeints/SyncFrameworkToolkit

4) Roll my own. Not having done this before does anyone have suggestions for best practices here?

I apologize for the long post but I'm having trouble finding any discussions on what the suggested method is on this for Xamarin. Thanks!

Posts

  • thedigitalseanthedigitalsean USUniversity ✭✭
    edited September 2014

    Anwering my own questions:

    1) Couchbase Mobile

    Seems like the preferred "turn key" solution for Xamarin at the moment. Although it looks like a great solution, it's not made for the back-end my team wants to use so I unfortunately cannot use it.

    2) Azure Mobile Services

    The claim is that it supports "any" data store but in all my research I found it only supports SQL, MongoDB, and Azure Tables. Also I didn't see an option for on premise hosting, which is one of my other requirements. But for others, this one is a strong contender with Couchbase Mobile. Especially if you want to use something other than Couchbase as your data store.

    3) Microsoft Sync Framework (now called Sync Framework Toolkit)

    On the surface this actually seems feasible if you use an ADO.Net capable driver and want to create a port for your data store (one does not exist for Cassandra that I found. But others might already be developed. I saw someone made one for PostgreSQL for example). Regardless, there didn't seem to be a lot recent stuff out there on this and the open source projects for Xamarin clients have a single contributor. This seemed risky for a serious project.

    4) Open source options:

    SymmetricDS - looks promising. Support a few different data stores and is adding more. But no Cassandra for me sadly.
    StrongLoop - Another promising one, but is written for Node.js. I read that you can potentially get Node.Js to run in .NET but this seemed like it would be an undertaking and be adding a lot of overhead I didn't want.

    5) Use native solutions: Android Sync Adapter, Restkit, others

    Since I am using a PCL for cross platform SQLite local storage, the architecture of then using platform specific synchronization libraries looked too messy.

    6) Roll Your Own

    Unfortunately this is what I think I'm left with. Our current data model design doesn't seem too complex at the moment so we hope to contain it with something light weight. None of the above solutions panned out for a Cassandra back end.

    For those with different back ends I would recommend 1, 2, or 4.

  • ahmedkhanahmedkhan USUniversity ✭✭

    Interesting rolling your own is not trival, conflict free is a real headache.... Sync is really important to enterprise apps -you'd think it would get more attention and its unfortunate that Microsoft Sync Framework is a bust. I don't understand why it was abandoned by MS.

    I'm really interested in Azure's sync capability. - azure mobile services have been great, but I'm not sure if they can work from a PCL?

    Last year, we implemented a Version of SymmetricDS in our enterprise iOS app to sync data with a sql server source with the intention that it will eventually become cross platform for android. However the resulting sync client and server was very complex and maintaining the sync to tables was not easy. Also the java based Symmetric server was difficult to setup with our microsoft ecosystem and sql server and maintain with our enterprise admins -we had to create a new instance of the data that was mirrored in a clean sync-able table from the original sql server instance which had a lot of stored proc based dependencies.. however the end result does work (data is synced between server and mobile app) but we still have to expand the client to work with android..

    also to add to your list : Zumero? https://components.xamarin.com/view/zumero/ (sql server only)

  • KenWenyonKenWenyon USMember, University

    Did you end up rolling your own or did you find something since them...

    I have a similar issue I am trying to resolve. Feel free to call me cell to discuss 561-756-0814.

    Ken

  • thedigitalseanthedigitalsean USUniversity ✭✭

    We are still going down the "roll your own" path. It's quite dissapointing that there is nothing out there for this.

  • GeorgiosTaskosGeorgiosTaskos USUniversity ✭✭

    I would like to hear what was the outcome and lessons from this. I'm on the same route to "rolling your own".

  • thedigitalseanthedigitalsean USUniversity ✭✭

    For the project using Cassandra, it didn't pan out since management changed direction before we were able to complete it. For the follow on project, largely due to the challenges with sync, I was able to convince the team to switch to Couchbase on the backend. We are woking through that implementation now. Sorry I don't have a good answer.

  • cristophcristoph ROMember, Beta

    Hey guys, have a look at recently released SiaqodbCloud (open source under Apache license) which allows you to sync Siaqodb from mobile with MongoDB or CouchDB. You can host SiaqodbCloud on premise, more info here.

  • ThomasBurkhartThomasBurkhart DEMember ✭✭✭✭

    @thedigitalsean Hi, how was your experience with Couchbase so far?

  • thedigitalseanthedigitalsean USUniversity ✭✭

    @ThomasBurkhart The experience with Couchbase has been pretty good. I was a little disappointed that while there is a framework in iOS where you can store/retrieve using objects, there wasn't such a framework for Android and you are left using Dictionaries which is a bit cumbersome. To combat this I wrote a base Couchbase data model I called DocBaseModel that all my data models inherit from. It has all the CBL meta data (_id, _rev, channels, etc) and uses reflection so that it convert itself to/from a Dictionary using the the name of its properties as the key. Then I wrote some wrappers for saving, getting, querying, etc. Unfortunately Couchbase doesn't support PCL either so I had to do this in a shared project.

    My only other gripe with the framework is that they currently don't let you get at the actual JSON neither in string format nor JObjects. You have to use their Dictionary<string,object> methods which tries to do the serializing/deserializing for you. I have run into at least one situation where their automatic serializing caused me some issues and there isn't a way to customize it currently. I was able to work around it but it wasn't ideal. The devs say they are working to get potentially give access to the raw json in a future release.

    Beyond that it has been pretty good. I feel like these issues are minor vs trying to work through bugs in hand-rolled synchronization and conflict detection. We haven't deployed yet but so far so good.

  • ThomasBurkhartThomasBurkhart DEMember ✭✭✭✭

    Thanks, sounds promising. I will give it a try.

  • ThomasBurkhartThomasBurkhart DEMember ✭✭✭✭

    @thedigitalsean Ah, I just got what you meant with missing Object-Document-Mapper in couchbase. I really don't know why this is so as they already use jason.net.

  • ThomasBurkhartThomasBurkhart DEMember ✭✭✭✭

    @thedigitalsean I'm not sure, but I think I found an extremely easy way to store an object. Please correct me if I'm wrong.
    The idea is to put my Class into the Dictionary:

                var c = new TestClass() {TestString = "SimpleStringfield", TestDate = DateTime.Today,TestStringProperty = "TestStringPropertyContent",
                                          TestStringList = new List<string>(new string[] { "item1","item2","item3"})};
    
    
                var manager = Manager.SharedInstance;
                var db = manager.GetDatabase("test_database");
                if (db == null)
                {
                    System.Diagnostics.Debug.WriteLine("--------------------------Could not open Database");
                }
    
                var doc = db.CreateDocument();
                var properties = new Dictionary<string,Object>() { {"type","day"}, {"day",c} };
    
                var rev = doc.PutProperties(properties);
    
                var docread = db.GetDocument(doc.Id);
                JObject testobject = (JObject)docread.GetProperty("day");
    
                var o = testobject.ToObject<TestClass>();
    

    At least "o" has the same values as "c".

  • thedigitalseanthedigitalsean USUniversity ✭✭

    I think we are getting off topic. But looks like you are off to a good start.

  • jonojono USMember ✭✭
    edited March 2016

    Everything u have mentioned is pretty much technology specific. The only real established cross platform technology choice would be a pattern, namely Event Sourcing.

    Events drive your system, your sync implementation with be platform specific - a projection of your events into a "view". The events themselves are versioned and ordered. Communicating events with different components is a done deal with heaps of guides availible online for whatever technology u choose. This is an eventual consistancy pattern.

    The downside of this approach is that you loose performance over say a specialised technology specific platform like sql sync framework.

    If your new to the pattern, Martin Fowler has a good writeup. The system i work on was originally written for sql2sql syncing using syncFramework. Its a brute force approach but it works and is fast taking advantage of bulk loading. the ES approach is a more chatty, but is platform agnostic. For instance, we sync excel spreadsheets with a sql database. The great part of this approach is you only write one adapter, once for each platform - and extending your sync process is just a matter of writing event processesors.

    I use DDD to generate events from from an AggRoot, write this to the eventStore (sql table), then broadcast the event to clients (using whatever transport/queue/protocal you need). Then its just a matter of ensuring ur clients process the events without errors, and retrying if they failed because ordering is the key. Im at the point where i can build my DomainModel using a POCO design, cross-compile from C# to TypeScript and implement optomistic updates on the client to acheive offline mode.

    The great part about this pattern for us, we use DDD and event sourcing for everything, including analytics, in the UI aka a 1-way-dataflow (aka "Flux" -- Flux is just event sourcing rebadged afterall) Many platforms, shared code, one pattern, many uses.

    Ive been working towards open sourcing our framework. I feel there is a missing link here, atleast from within the C# community.

  • MahaLJMahaLJ USMember ✭✭

    what about Sync Framework Toolkit for Mobile Devices? any one try it? for Xamarin Forms!

  • FrtySixAndTwoFrtySixAndTwo USMember ✭✭

    Any additional updates on experiences with a sync service for Xamarin mobile app? We are in the process of gathering information to decide on one to use. We're using Azure App Services sync for a smaller app now although I have some concerns with whether this can really handle a large app with a lot more data moving around right now. It seems like Couchbase Mobile may be the best available option to try out.

  • GizmoGizmo CLMember ✭✭

    I was asking kinda the same question now in 2016...but I havent received replies. Has anyone being able to create a Forms app with offline sync functionality?. I'm looking to sync a remote database (sql server) with my Xforms app (sqllite).
    Azure seems to be the only solution, but you are somehow forced to use their databases and not your own/ or your clients.

  • BojanPanjevicBojanPanjevic USMember
    edited November 2016

    @Gizmo Take a look at Zumero http://zumero.com
    We're evaluating it right now for enterprise app we're working on and it seem to be working fine. You can sync with a MS SQL (on-prem or in SQL Azure) and XamarinForms.

  • WTDEVWTDEV USMember ✭✭
    edited November 2016

    @Gizmo +1 for checking out Zumero. We too are currently evaluating it and looks good. @BojanPanjevic are you using version 3?..

  • @RolfDalhaug Updated to Zumero 3.0 yesterday. So far so good.

  • PlamenYovchevPlamenYovchev USMember ✭✭
    edited March 2017

    Hi guys, we are also looking at the Zumero but we are experiencing issues with updates and inserts. Could you please take a look at this description. I've also attached a screencast.

    We’ve downloaded the Zumero Manager and installed it on our server. We tested it with the Zumero Test Client and everything seems to be working fine. We’ve made select, update and insert queries and successfully synced them with our database.
    But when we generated the Xamarin.Forms demo application by Zumero Generator we have problems with inserts and deletes of the records from our table.
    We’ve included only one table just for test. There are records in the tables and they appear in the grid, so this makes us to believe that this is not a connection problem.

    We are trying to update exising record or to insert new record but we are receiving an Exception Message which is: “Constraint”.

    Almost all of our table primary keys and the other IDs are Guids.

    Here is a screencast: https://www.screencast.com/t/aW6K9dGeav8

  • WTDEVWTDEV USMember ✭✭

    @BojanPanjevic how have you got on?

    @PlamenYovchev did you manage to resolve this?

  • PlamenYovchevPlamenYovchev USMember ✭✭

    No, we've decided to implement our own synchronization.

  • biaparbiapar ITMember ✭✭✭
    edited July 2017
  • batmacibatmaci DEMember ✭✭✭✭✭

    Has anybody found a good solution using sql server? my experience with azure sql sync is not so good for small apps and my current app is quite large i dont believe that it will be a good solution.

  • UnreachableCodeUnreachableCode USMember ✭✭✭
    edited December 2018

    I'm looking at this problem space right now. I have an existing backend system that uses MySQL. I have a RESTful API that I call for some pretty small operations, but I am now looking at the best way to download a large amount of data from our backend to multiple Android devices using SQLite, most likely a few times a day. Our app is written in Xamarin Forms. I've thought about rolling my own system with what we have just now. But I'm worried about many client devices constantly polling a remote database (maybe only when connected via WiFi). Is this the smartest way to do this?

    I also should note that I don't need all the data from the MySQL server on the SQLite databases. Does offline syncing allow configuration for subsets of data to be synced?

  • batmacibatmaci DEMember ✭✭✭✭✭

    Concept is very complicated. you understand this once you go into it deeper. i have tried azure sync which is now deprecated. I tried also Aws coginito sync, easy to implement, fast, reliable, expensive, I think also json size is restricted.
    I tried also Aws DynamoDB, complicated to set up with rules and complication but once it is set, it is fast enough, reliable and doesnt require Api in between. It will be expensive also after 12 free months usage.
    Finally I decided to use Azure SQL+Web Core Api+JWT token authorization using Firebase... It is quite complicated process if you use login and social media logins. and on azure database, I save data as json. So basically I did kind of key value syncronization, In another words, i use Sql database as NoSQL. of course here your best pick will be to use a real NoSQL database. because using relational database synchronization is slower. Why i didn't use it, because of NoSql database prices are higher for budget developers. Or if you have to use RD, you shouldnt use FK relations.

    here is a perfect documentation to read and learn https://adrianhall.gitlab.io/develop-mobile-apps-with-csharp-and-azure/
    I read that probably 2 times before i jumped into all of these :)

Sign In or Register to comment.