Porting Google WebRTC app to C# - Android (and iOS in the future, MVVMCross) - native lib problem

KennethThorman.1544KennethThorman.1544 DKMember
edited December 2013 in Xamarin.Android

Hi

Edit: Attempting to fix markup

Main machine setup
*Unbuntu for building the WebRTC code
*Window 7 and Visual Studio 2013 with Xamarin.iOS and Xamarin.Android for building apps

Background:
I wish to build a cross platform app that supports WebRTC (real time communication via video/audio).
Currently this is in the starting stages, and if I manage to get this developed further then this will be included in a main app.
For now I am just working on porting and implementing the Java WebRTC demo app in C#.

Current stage:
I have managed to port the Java demo app to c# so it builds, there are very likely some conversion errors and problems, but the app builds with no errors.
I am now stuck at getting marshaling interop exceptions when trying to invoke the JNI enabled methods in the .so file I built following

http://devryazantsev.blogspot.ru/p/building-webrtc-trunk-for-andoid-and.html

After following this blogposting, and copying the resulting files to my windows machine you end up with

libwebrtc-video-demo-jni.so (located at: webrtc\video_engine\test\android\libs\armeabi-v7a)

main app - java files
1. webrtc\video_engine\test\android\src\org\webrtc\videoengineapp

supporting classes - java files
1. webrtc\modules\audio_device\android\java\src\org\webrtc\voiceengine
2. webrtc\modules\video_capture\android\java\src\org\webrtc\videoengine
3. webrtc\modules\video_render\android\java\src\org\webrtc\videoengine

I have converted the Java files to C# equivalents (or at least I think I have -> it compiles)

Initially I had some problems using DllImport since it could not find the methods in the .so file but I found that method names
were scrambled, which I found by using this Android tools command on my Ubuntu build machine

/WebRTCDemo/trunk/webrtc/video_engine/test/android/libs/armeabi-v7a$ arm-linux-androideabi-nm -D libwebrtc-video-demo-jni.so

which yielded something like

00015358 T JNI_OnLoad
00015be4 T Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_AddRemoteRenderer
00016e3c T Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_CreateChannel
00015f14 T Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_EnableNACK
00015f58 T Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_EnablePLI
00015e04 T Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_GetCameraOrientation
00015acc T Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_GetCodecs
000153d4 T Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_GetVideoEngine
000154cc T Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_Init
000153d0 T Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_NativeInit
00015c30 T Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_RemoveRemoteRenderer
00015fb0 T Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_SetCallback
00015ea8 T Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_SetExternalMediaCodecDecoderRenderer
...

so after changing all my DllImport to using the wrangled names it is now loading the lib and registering all the functions.

I am running with the following mono diagnostics flags

adb shell setprop debug.checkjni 1
adb shell setprop debug.mono.env MONO_LOG_LEVEL=info 
adb shell setprop debug.mono.log gref,gc 

I am now stuck at lines 440-446 in the attached log

2-21 09:58:56.126 I/MonoDroid(18816): UNHANDLED EXCEPTION: System.Runtime.InteropServices.MarshalDirectiveException: Type Java.Lang.Object which is passed to unmanaged code must have a StructLayout attribute.
12-21 09:58:56.126 I/MonoDroid(18816): at (wrapper managed-to-native) WebRtc.ViEAndroidJavaAPI.Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_NativeInit (Android.Content.Context) 
12-21 09:58:56.126 I/MonoDroid(18816): at WebRtc.ViEAndroidJavaAPI..ctor (Android.Content.Context) [0x00033] in d:\Downloads\WEBRTC\WebRtc\ViEAndroidJavaAPI.cs:30
12-21 09:58:56.126 I/MonoDroid(18816): at WebRtc.WebRTCDemo.startMain () [0x0004b] in d:\Downloads\WEBRTC\WebRtc\WebRTCDemo.cs:536
12-21 09:58:56.126 I/MonoDroid(18816): at WebRtc.WebRTCDemo.OnCreate (Android.OS.Bundle) [0x0028e] in d:\Downloads\WEBRTC\WebRtc\WebRTCDemo.cs:316
12-21 09:58:56.126 I/MonoDroid(18816): at Android.App.Activity.n_OnCreate_Landroid_os_Bundle_ (intptr,intptr,intptr) [0x00011] in /Users/builder/data/lanes/monodroid-mlion-monodroid-4.10.1-branch/d23a19bf/source/monodroid/src/Mono.Android/platforms/android-17/src/generated/Android.App.Activity.cs:2119
12-21 09:58:56.126 I/MonoDroid(18816): at (wrapper dynamic-method) object.468078ad-6f7a-4c7d-8532-4ad45f0259ae (intptr,intptr,intptr) 


After some Googling I reached one of Jonathan Pryor's postings http://forums.xamarin.com/discussion/325/native-library-integration

So my question is:

I was hoping to no have to use any Java Bindings Libraries but instead port the code to C# fully.

If I understand Jonathan's posting correctly, I cannot do it this way that I am doing, but instead I think I have 2 options

Making a Java Bindings Library that contains the needed jars (some of the files that are needed are not in jar files in the Java demo code, so I will have to modify the upstream WebRTC build process to make this work (which was something I hoped to avoid).

Do I have any other options, like manually registering the native methods using JNI?

Regards and thank you in advance
Kenneth Thorman


*** File excerpts supporting posting ***



webrtc\video_engine\test\android\src\org\webrtc\videoengineapp\ViEAndroidJavaAPI.java

public class ViEAndroidJavaAPI {

    public ViEAndroidJavaAPI(Context context) {
        Log.d("*WEBRTCJ*", "Loading ViEAndroidJavaAPI...");
        System.loadLibrary("webrtc-video-demo-jni");

        Log.d("*WEBRTCJ*", "Calling native init...");
        if (!NativeInit(context)) {
            Log.e("*WEBRTCJ*", "Native init failed");
            throw new RuntimeException("Native init failed");
        }
        else {
            Log.d("*WEBRTCJ*", "Native init successful");
        }
        String a = "";
        a.getBytes();
    }

    // API Native
    private native boolean NativeInit(Context context);

    // Video Engine API
    // Initialization and Termination functions
    public native int GetVideoEngine();
    public native int Init(boolean enableTrace);
    public native int Terminate();
...

C# port

    public class ViEAndroidJavaAPI
    {

        public ViEAndroidJavaAPI(Context context)
        {
            Log.Debug("*WEBRTCJ*", "Loading ViEAndroidJavaAPI...");
            Log.Debug("*WEBRTCJ*", "Calling native init...");
            JavaSystem.LoadLibrary("webrtc-video-demo-jni");

            if (!Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_NativeInit(context))
            {
                Log.Error("*WEBRTCJ*", "Native init failed");
                throw new Exception("Native init failed");
            }
            else
            {
                Log.Debug("*WEBRTCJ*", "Native init successful");
            }
            string a = "";
            Encoding.Default.GetBytes(a);
        }

        // API Native

        [DllImport("libwebrtc-video-demo-jni.so")]
        private static extern bool Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_NativeInit(Context context);

        // Video Engine API
        // Initialization and Termination functions
        [DllImport("libwebrtc-video-demo-jni.so")]
        public static extern int Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_GetVideoEngine();

        [DllImport("libwebrtc-video-demo-jni.so")]
        public static extern int Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_Init(bool enableTrace);

        [DllImport("libwebrtc-video-demo-jni.so")]
        public static extern int Java_org_webrtc_videoengineapp_ViEAndroidJavaAPI_Terminate();
...

Original Java sources at

Main app code
I cannot find this in the code.google.com repo so attaching the versions that I have of the Java originating files videoengineapp.zip
 

Supporting libs
https://code.google.com/p/webrtc/source/browse/#svn/trunk/webrtc/modules/audio_device/android/java/src/org/webrtc/voiceengine
https://code.google.com/p/webrtc/source/browse/#svn/trunk/webrtc/modules/video_capture/android/java/src/org/webrtc/videoengine
https://code.google.com/p/webrtc/source/browse/#svn/trunk/webrtc/modules/video_render/android/java/src/org/webrtc/videoengine


Regards and thank you in advance (2nd time - I know :) )
Kenneth Thorman

Posts

  • I have created a github repo with the current code

    https://github.com/kenneththorman/webrtc-app-mono-mvvmcross

  • My reluctance to manually interfere with the upstream build process ruled out using a Java Bindings Library since 2 needed classes was not supplied in a jar file.
    
    Based on the 2 files that where missing from the jars in the Google WebRTC build script (they are compiled into the WebRTC application itself) not as a jar).
    
    I manually created a new jar file called ViEAndroidJavaAPI.jar. This was created on the Ubuntu machine using the following command
    
    XXX@ubuntu:~/WebRTCDemo/trunk/webrtc/video_engine/test/android/bin/classes$
    jar cvf ViEAndroidJavaAPI.jar
    org/webrtc/videoengineapp/IViEAndroidCallback.class
    org/webrtc/videoengineapp/ViEAndroidJavaAPI.class
    added manifest
    adding: org/webrtc/videoengineapp/IViEAndroidCallback.class(in = 218)
    (out= 173)(deflated 20%)
    adding: org/webrtc/videoengineapp/ViEAndroidJavaAPI.class(in = 2845)
    (out= 1303)(deflated 54%)
    
    Then a new Java Bindings Library project was added to the solution. The jar was added to the new project and the references updated to point to the new project instead of the old manual c# file
    
    The branch and the commit can be seen at
    
    https://github.com/kenneththorman/webrtc-app-mono-mvvmcross/commit/e914d27bd9f228aeeb39591e3b3bb27989294df8
    
    I however get a strange Java error in some generated code 
    
    Error   32   package org.webrtc.videoengineapp does not exist
            org.webrtc.videoengineapp.IViEAndroidCallback,
        webrtc-app-mono-mvvmcross\WebRtc.Mono.Droid\obj\Debug\android\src\webrtc\mono\droid\WebRTCDemo.java 8   28  WebRtc.Mono.Droid
    
    I am still fairly new to both Android development and the way Mono/Xamarin interacts with Java and JNI so I am very far from fully understanding how all the pieces are fitting together.
    
    Any insights would be appreciated.
    
    Thank you in advance
    Regards
    Kenneth Thorman
    
  • KennethThorman.1544KennethThorman.1544 DKMember
    edited December 2013

    I managed to make some more headway in the second approach which was using a Java Binding Library.

    The changes can be seen in the following commit https://github.com/kenneththorman/webrtc-app-mono-mvvmcross/commit/da7f2d402c5985b5d23f600d39e3e170b024a3bd

    I have decided to document my approach here step by step, so it might help others in the future.

    I am now able to call the native JNI wrapped C functions of the webrtc project. Currently I am stuck at the following native error

    12-23 10:50:35.532 D/dalvikvm( 2270): Trying to load lib /data/data/WebRtc.Mono.Droid/lib/libwebrtc-video-demo-jni.so 0x4295c588
    assertion "javaAmClassLocal" failed: file "../../webrtc/modules/audio_device/android/audio_manager_jni.cc", line 62, function "static void webrtc::AudioManagerJni::SetAndroidAudioDeviceObjects(void*, void*, void*)"
    12-23 10:50:35.532 D/dalvikvm( 2270): Shared lib '/data/data/WebRtc.Mono.Droid/lib/libwebrtc-video-demo-jni.so' already loaded in same CL 0x4295c588
    12-23 10:50:35.532 D/*WEBRTCJ*( 2270): Calling native init...
    12-23 10:50:35.532 D/*WEBRTCJ*( 2270): Native init successful
    12-23 10:50:35.537 D/*WEBRTCN*( 2270): Create VoiceEngine
    12-23 10:50:35.537 E/mono-rt ( 2270): Stacktrace:
    12-23 10:50:35.537 E/mono-rt ( 2270): 
    12-23 10:50:35.542 E/mono-rt ( 2270):   at  <0xffffffff>
    12-23 10:50:35.542 E/mono-rt ( 2270):   at Org.Webrtc.Videoengineapp.ViEAndroidJavaAPI.VoE_Create (Android.Content.Context) [0x00048] in c:\Users\kenneth.thorman\Documents\GitHub\webrtc-app-mono-mvvmcross\Org.W12-23 10:50:35.542 E/mono-rt ( 2270):   at (wrapper managed-to-native) object.wrapper_native_0x40857d2d (intptr,intptr,intptr,Android.Runtime.JValue[]) 
    12-23 10:50:35.542 E/mono-rt ( 2270):   at (wrapper delegate-invoke) .invoke_bool__this___intptr_intptr_intptr_JValue[] (intptr,intptr,intptr,Android.Runtime.JValue[]) 
    12-23 10:50:35.542 E/mono-rt ( 2270):   at Android.Runtime.JNIEnv.CallBooleanMethod (intptr,intptr,Android.Runtime.JValue[]) [0x00040] in /Users/builder/data/lanes/monodroid-mlion-monodroid-4.10.1-branch/d23a19bf/source/monodroid/src/Mono.Android/src/Runtime/JNIEnv.g.cs:224
    12-23 10:50:35.542 E/mono-rt ( 2270):   at Org.Webrtc.Videoengineapp.ViEAndroidJavaAPI.VoE_Create (Android.Content.Context) [0x00048] in c:\Users\kenneth.thorman\Documents\GitHub\webrtc-app-mono-mvvmcross\Org.Webrtc.Videoengineapp\obj\Debug\generated\src\Org.Webrtc.Videoengineapp.ViEAndroidJavaAPI.cs:945
    12-23 10:50:35.542 E/mono-rt ( 2270):   at WebRtc.Mono.Droid.WebRTCDemo.setupVoE () [0x00001] in c:\Users\kenneth.thorman\Documents\GitHub\webrtc-app-mono-mvvmcross\WebRtc.Mono.Droid\WebRTCDemo.cs:841
    12-23 10:50:35.542 E/mono-rt ( 2270):   at WebRtc.Mono.Droid.WebRTCDemo.startMain () [0x00058] in c:\Users\kenneth.thorman\Documents\GitHub\webrtc-app-mono-mvvmcross\WebRtc.Mono.Droid\WebRTCDemo.cs:531
    12-23 10:50:35.542 E/mono-rt ( 2270):   at WebRtc.Mono.Droid.WebRTCDemo.OnCreate (Android.OS.Bundle) [0x0028e] in c:\Users\kenneth.thorman\Documents\GitHub\webrtc-app-mono-mvvmcross\WebRtc.Mono.Droid\WebRTCDemo.cs:309
    12-23 10:50:35.542 E/mono-rt ( 2270):   at Android.App.Activity.n_OnCreate_Landroid_os_Bundle_ (intptr,intptr,intptr) [0x00011] in /Users/builder/data/lanes/monodroid-mlion-monodroid-4.10.1-branch/d23a19bf/source/monodroid/src/Mono.Android/platforms/android-17/src/generated/Android.App.Activity.cs:2119
    12-23 10:50:35.542 E/mono-rt ( 2270):   at (wrapper dynamic-method) object.4f62e96e-01d0-4b0c-81ae-46a84f3b0135 (intptr,intptr,intptr) 
    12-23 10:50:35.542 E/mono-rt ( 2270):   at (wrapper native-to-managed) object.4f62e96e-01d0-4b0c-81ae-46a84f3b0135 (intptr,intptr,intptr) 
    12-23 10:50:35.542 E/mono-rt ( 2270): 
    12-23 10:50:35.542 E/mono-rt ( 2270): =================================================================
    12-23 10:50:35.542 E/mono-rt ( 2270): Got a SIGSEGV while executing native code. This usually indicates
    12-23 10:50:35.542 E/mono-rt ( 2270): a fatal error in the mono runtime or one of the native libraries 
    12-23 10:50:35.542 E/mono-rt ( 2270): used by your application.
    12-23 10:50:35.542 E/mono-rt ( 2270): =================================================================
    12-23 10:50:35.542 E/mono-rt ( 2270): 
    The program 'Mono' has exited with code 0 (0x0).
    
  • KennethThorman.1544KennethThorman.1544 DKMember
    edited December 2013

    Attaching the full debugOutput.log for the previous mentioned error in the posting above here. This is the debug log I get when trying to run the app on a real hardware device Samsung Galaxy 2 (Model: GTI9100) - running Android 4.1.2

  • DennisWeluDennisWelu USUniversity, Developer Group Leader ✭✭

    Kenneth, nice job with getting as far as you have! I'm trying to find a Xamarin friendly approach to native android webrtc clients so definitely appreciate sharing the repo. Any interesting updates since December I should know before looking through code?

  • DennisWeluDennisWelu USUniversity, Developer Group Leader ✭✭

    Doh...just found your subsequent blog post (and related posts) here: http://kenneththorman.blogspot.com/2014/01/webrtc-app-c-xamarin-part-2-attempt-2.html. Cool!

  • Hi Dennis

    You might find this repo (another port) more interesting https://github.com/kenneththorman/appspotdemo-mono this is actually a reference implementation of the android app that is a client to the google apprtc.appspot.com WebRTC site.

    I have got it so far as to showing the local video on phone android mobile phones and tablets, there seems to be a problem with the WebRTC signaling in my port. I am currently comparing the adb logcat logs between the working Java version and my Mono port.

    I have not so far been able to make headway, but then again I did not have time to look at this project until just today (been busy with other things since my last commit).

    Filtered Java log:

    ?: D/AppRTCDemoActivity(28331): Waiting for ICE candidates...
    ?: I/libjingle(28331): Capture delay changed to 190 ms
    ?: I/libjingle(28331): Captured frame size 640x480. Expected format NV21 640x480x30
    ?: W/libjingle(28331): Warning(webrtcvideoengine.cc:1414): webrtc: Too long processing time of Incoming frame: 69ms
    ?: D/AppRTCDemoActivity(28331): Creating offer...
    ?: D/AppRTCDemoActivity(28331): Sending OFFER
    ?: I/libjingle(28331): Ignored line: c=IN IP4 0.0.0.0
    ?: I/libjingle(28331): Ignored line: c=IN IP4 0.0.0.0
    ?: I/libjingle(28331): Created channel for audio
    ?: W/libjingle(28331): Warning(webrtcvideoengine.cc:1414): webrtc: SetStorePacketsStatus already set, number: 600
    ?: I/libjingle(28331): NACK enabled for channel 0
    ?: I/libjingle(28331): NACK enabled for channel 0
    ?: W/libjingle(28331): Warning(webrtcvideoengine.cc:1414): webrtc: SetStorePacketsStatus already set, number: 600
    ?: I/libjingle(28331): Created channel for video
    ?: I/libjingle(28331): Session:3161569035716025754 Old state:STATE_INIT New state:STATE_SENTINITIATE Type:urn:xmpp:jingle:apps:rtp:1 Transport:http://www.google.com/transport/p2p
    ?: I/libjingle(28331): Setting local voice description
    ?: I/libjingle(28331): Add send ssrc: 2413872825
    ?: I/libjingle(28331): Setting receive voice codecs:
    ?: I/libjingle(28331): ISAC/16000/1 (103)
    ?: I/libjingle(28331): opus/48000/2 (111)
    ?: I/libjingle(28331): G722/16000/1 (9)
    ?: I/libjingle(28331): ILBC/8000/1 (102)
    ?: I/libjingle(28331): PCMU/8000/1 (0)
    ?: I/libjingle(28331): PCMA/8000/1 (8)
    ?: I/libjingle(28331): CN/32000/1 (106)
    ?: I/libjingle(28331): CN/16000/1 (105)
    ?: I/libjingle(28331): CN/8000/1 (13)
    ?: I/libjingle(28331): red/8000/1 (127)
    ?: I/libjingle(28331): telephone-event/8000/1 (126)
    ?: I/libjingle(28331): Changing voice state, recv=0 send=0
    ?: I/libjingle(28331): Setting local video description
    ?: I/libjingle(28331): AddSendStream {id:ARDAMSv0;ssrcs:[1496681501];ssrc_groups:;cname:+ZSFYflgMquUtnG7;sync_label:ARDAMS}
    ?: I/libjingle(28331): Add send ssrc: 1496681501
    ?: I/libjingle(28331): Changing video state, recv=0 send=0
    ?: I/libjingle(28331): Setting voice channel options: AudioOptions {}
    ?: I/libjingle(28331): Set voice channel options.  Current options: AudioOptions {}
    ?: I/libjingle(28331): Local and Remote descriptions must be applied to get SSL Role of the session.
    ?: I/libjingle(28331): Transport: audio, allocating candidates
    ?: I/libjingle(28331): Transport: audio, allocating candidates
    ?: I/libjingle(28331): Transport: video, allocating candidates
    ?: I/libjingle(28331): Transport: video, allocating candidates
    ?: I/libjingle(28331): Jingle:Net[wlan0:192.168.1.0/24]: Allocation Phase=Udp
    ?: I/libjingle(28331): Jingle:Port[:1:0::Net[wlan0:192.168.1.0/24]]: Port created
    ?: I/libjingle(28331): Adding allocated port for audio
    ?: I/libjingle(28331): Jingle:Port[audio:1:0::Net[wlan0:192.168.1.0/24]]: Added port to allocator
    ?: I/libjingle(28331): AllocationSequence: UDPPort will be handling the STUN candidate generation.
    ?: I/libjingle(28331): Jingle:Net[wlan0:192.168.1.0/24]: Allocation Phase=Relay
    ?: I/libjingle(28331): Jingle:Port[:1:0:relay:Net[wlan0:192.168.1.0/24]]: Port created
    ?: I/libjingle(28331): Adding allocated port for audio
    ?: I/libjingle(28331): Jingle:Port[audio:1:0:relay:Net[wlan0:192.168.1.0/24]]: Added port to allocator
    ?: I/libjingle(28331): Jingle:Port[audio:1:0:relay:Net[wlan0:192.168.1.0/24]]: Trying to connect to TURN server via udp @ 8.34.221.6:3478
    ?: I/libjingle(28331): Jingle:Net[wlan0:192.168.1.0/24]: Allocation Phase=Tcp
    ?: I/libjingle(28331): Jingle:Port[:1:0:local:Net[wlan0:192.168.1.0/24]]: Port created
    ?: I/libjingle(28331): Adding allocated port for audio
    ?: I/libjingle(28331): Jingle:Port[audio:1:0:local:Net[wlan0:192.168.1.0/24]]: Added port to allocator
    ?: I/libjingle(28331): Jingle:Net[wlan0:192.168.1.0/24]: Allocation Phase=SslTcp
    ?: I/libjingle(28331): All candidates gathered for audio:1:0
    

    Equivalent Mono log

    ?: D/AppRTCDemoActivity(16707): Waiting for ICE candidates...
    ?: I/libjingle(16707): Capture delay changed to 190 ms
    ?: I/libjingle(16707): Captured frame size 800x600. Expected format NV21 800x600x35
    ?: W/libjingle(16707): Warning(webrtcvideoengine.cc:1414): webrtc: Too long processing time of Incoming frame: 73ms
    ?: W/libjingle(16707): Warning(webrtcvideoengine.cc:1414): webrtc: Too long processing time of Incoming frame: 12ms
    ?: W/libjingle(16707): Warning(webrtcvideoengine.cc:1414): webrtc: Too long processing time of Incoming frame: 12ms
    ?: W/libjingle(16707): Warning(webrtcvideoengine.cc:1414): webrtc: Too long processing time of Incoming frame: 12ms
    ?: W/libjingle(16707): Warning(webrtcvideoengine.cc:1414): webrtc: Too long processing time of Incoming frame: 17ms
    ?: W/libjingle(16707): Warning(webrtcvideoengine.cc:1414): webrtc: Too long processing time of Incoming frame: 21ms
    ?: W/libjingle(16707): Warning(webrtcvideoengine.cc:1414): webrtc: Too long processing time of Incoming frame: 12ms
    ?: W/libjingle(16707): Warning(webrtcvideoengine.cc:1414): webrtc: Too long processing time of Incoming frame: 13ms
    ?: D/AppRTCDemoActivity(16707): Stats: id: googLibjingleSession_4993710414640259294, type: googLibjingleSession, timestamp: 1.389979164759263E12, values: [googInitiator: false],
    ?: D/AppRTCDemoActivity(16707): Stats: id: googTrack_ARDAMSa0, type: googTrack, timestamp: 0.0, values: [googTrackId: ARDAMSa0],
    ?: D/AppRTCDemoActivity(16707): Stats: id: googTrack_ARDAMSv0, type: googTrack, timestamp: 0.0, values: [googTrackId: ARDAMSv0],
    

    So all in all it seems like my webrtc signaling is off, somehow.

    I am trying to track it down, but any help is of course appreciated.

    Regards
    Kenneth

  • The " Warning(webrtcvideoengine.cc:1414): webrtc: Too long processing time of Incoming frame:" are also to be found in the JAva version of the log so this is not a problem, however the I believe the missing "Creating offer", "Sending offer" is the problem currently preventing my port from functioning.

  • DennisWeluDennisWelu USUniversity, Developer Group Leader ✭✭
    edited January 2014

    Have you considered wrapping in a WebView for making the WebRTC client connection and have it coordinate with the main container C# app (stuff like this http://docs.xamarin.com/recipes/android/controls/webview/call_c#_from_javascript/). It seems like that would allow the hard stuff to be done by the chrome on android webview. Doesn't help on the iOS side since its not supported in ios webviews, so you'd be back to native libraries there.

    EDIT: ah wait, not the same thing. Chrome on Android, which supports it, is not the same as the Android browser.

  • DennisWeluDennisWelu USUniversity, Developer Group Leader ✭✭

    @KennethThorman.1544, I looked at the other repo you have there. It appears that you've moved away from the first approach of porting the WebRTCDemo over to instead porting the libjingle sample that works with the apprtc site. So...I'm still absorbing all this but my understanding is that libjingle takes you down an XMPP approach for signaling, as it includes an implementation of a STUN and relay server. Whereas the first approach would allow you to have a native WebRTC library but you would separately pull in something like rfc5766-turn-server as a STUN/TURN server to use. That allows you to decide separately how you talk to your client apps. Does that sound about right?

  • Have you considered wrapping in a WebView for making the WebRTC client connection and have it coordinate with the main container C# app (stuff like this http://docs.xamarin.com/recipes/android/controls/webview/call_c#_from_javascript/). It seems like that would allow the hard stuff to be done by the chrome on android webview. Doesn't help on the iOS side since its not supported in ios webviews, so you'd be back to native libraries there.

    Yes I did consider this (if I understand you correctly), and decided against it. I have in the past done webview native integration which at the basic level is not hard to do. However the webview currently supported on Android does not support WebRTC.

    According to https://developers.google.com/chrome/mobile/docs/webview/overview they shipped a new version of the webview in Adnroid 4.4 which is built on top of Chrome - yes was my immediate thoughts when I read this, but then later rereading it it states

    For the most part, features that work in Chrome for Android should work in the new WebView.

    Chrome for Android supports a few features which aren't enabled in the WebView, including:WebGL 3D canvas, WebRTC, WebAudio, Fullscreen API, Form validation

    So all in all it seems like we are on our own as far WebRTC implementation goes, no nicely fully implemented version in the webview :(

  • @KennethThorman.1544, I looked at the other repo you have there. It appears that you've moved away from the first approach of porting the WebRTCDemo over to instead porting the libjingle sample that works with the apprtc site. So...I'm still absorbing all this but my understanding is that libjingle takes you down an XMPP approach for signaling, as it includes an implementation of a STUN and relay server. Whereas the first approach would allow you to have a native WebRTC library but you would separately pull in something like rfc5766-turn-server as a STUN/TURN server to use. That allows you to decide separately how you talk to your client apps. Does that sound about right?

    The AppRtcDemo is using the Google App Engine to handle the signaling, but the code leaves the signaling up to the developer and it is there in the code to change. My basic reason was that when I got the WebRTCDemo UI working/compiling, the UI looked like nothing I had ever seen before and what I needed was basically a native component that pretty much acted and behaved like the apprtc.appspot.com website.

    When I built the native .so (http://kenneththorman.blogspot.dk/2014/01/webrtc-app-c-xamarin-part-1-building.html) it also build a full android apk that you can install and test. That was the app I already tried on my devices which worked the same way as apprtc.appspot.com. That that was the code I was actually looking for, and as embarrassing it is to admit it - the repo https://github.com/kenneththorman/webrtc-app-mono was me porting the "wrong" Java app code. When I found out and figured out how to work with JNI (mainly through using the Java Binding Library) I went looking in the official WebRTC code again to find the app that I tested in the Java version. This is the repo at https://github.com/kenneththorman/appspotdemo-mono.

    So basically the repos are proof of me not being familiar with the official WebRTC code base and not really knowing what code is which :)

    Regards

    Kenneth Thorman

  • As an alternative would be to keep the official https://github.com/kenneththorman/appspotdemo-mono app in Java install it side by side with your mono version and make sure it accepted external Intents. Then you could invoke the webrtc app in Java from .NET. You would have to maintain your changes in the WebRTC app in Java and the changes in your own app in C#. Not totally unreasonable, but annoying and I really wanted to in time build an open source .NET library for WebRTC that could be plugged into MVVMCross so you just could add WebRTC support to your app either on IOS or Android easily.

    The signaling would be located behind a nice interface, which you could implement as you liked, and there would be a reference signaling implementation in a PCL (portable class library) using websockets, for a peer-peer meeting.

    My current schedule has brought me back to reality though, and currently I have not much time at all to dedicate to this. I will still try to reach the end goal but it will most likely take some time.

    In addition - I would love to skip the C# - JBL (Java Binding Library)/JNI - JNI c++ wrapper - C++ and replace the JNI C++ wrapper with a clean wrapper that could be P/Invoked directly from C#, I think that currently is beyond my skills, I have not touched C++ in many years, and I do not know the WebRTC code base, so I think we need to expand the team a little to get this optimization in.

    Any takers :)

    Regards

    Kenneth Thorman

  • DennisWeluDennisWelu USUniversity, Developer Group Leader ✭✭

    So basically the repos are proof of me not being familiar with the official WebRTC code base and not really knowing what code is which :)

    I hear you. It's a confusing repo with "examples" in different major portions of the tree. I've done some double-takes and can't say I understand it all yet.

    As an alternative would be to keep the official https://github.com/kenneththorman/appspotdemo-mono app in Java install it side by side with your mono version and make sure it accepted external Intents.

    This is an interesting idea but probably not something that works for me. I'm going to need a single app that can be controlled through a C# interface. And I only have a short couple days to prove it's doable without integrating a full service WebRTC API provider like vLine, OpenTok, Weemo, etc who seem to have figured out a proprietary mobile approach. But I'll take a look and see if I can make some progress today.

    I appreciate all the work you've done to this point!

    Dennis

  • Hi Dennis

    I have been spending some hours I could pull out of my schedule to add Log.Debug statements locally to try to track down where there is a problem with the appspotdemo.

    As mentioned before currently I am not really debugging the code, but instead relying on log comparison between the adb logcat log from the working Java version and then comparing this with the Mono version of the log. And then adding some additional Log statements.

    I have not been able to track down what is causing this at this moment, an extra pair of eyes has never hurt.

    Regards
    Kenneth

  • DennisWeluDennisWelu USUniversity, Developer Group Leader ✭✭

    Hey Kenneth just an update. Pretty quick I realized there was no way I was going to push through that in time constraints. I'm now evaluating the IceLink product from Frozen Mountain (http://www.frozenmountain.com/products/icelink) because 1) it's a drop-in library giving a common api across platforms, 2) it allows me to use my own servers instead of signing up for a service, and 3) it allows an option to BYOTS (bring your own TURN server) yet it bundles one in as well (UDP only right now). They support Xamarin...except for the WebRTC part, but that is coming "end of February" so not too long from now. Might work...still evaluating. You can look at the native Android (java) and iOS (obj-c) examples to see how it will work.

  • Hi Dennis

    I was just able to get a single simple commit in that now allows debugging in VS.NET. That is now on GitHub.

    The currently only know problem with the code is that the webrtc signaling is not working correctly. So something is not wired correctly in the converted code. I will attempt to see if I can get some more time to look at this.

    Regards

    Kenneth Thorman

  • Hi Dennis

    I have not managed to solve the entire problem yet but have moved closer I believe.
    I am pretty sure it is the C#/Javascript integration that is failing.

    I have made a few commits today and I hope that I am aable to push through.

    Regards
    Kenneth

  • DennisWeluDennisWelu USUniversity, Developer Group Leader ✭✭

    Nice job! Sounds like you are close. It looks more certain that we are going to end up going the IceLink route. But one thing I noticed in the repo you have is that there was a couple places where virtual methods were being hidden, and a lot of inner class runnables where simple actions could be used. I sent a pull request your way with some of the changes I started making but mostly just so you can review the changes and see if there's anything there that might help. At this point I'll probably focus on the IceLink approach.

    Thanks and good luck! I'll check back on your progress at some point.

    Dennis

  • LarryGugerLarryGuger USMember

    Hi All,

    I am from the Microsoft C++ team and we are very interested in talking to developers that are incorporating C++ libraries into their Xamarin apps. If you are doing that I would be very excited if you would reach out to me directly. My email alias is larrygug. The domain is microsoft.com. :-)

    Larry

  • KiranRaoKiranRao USMember

    Kenneth / Dennis - were either of you successful at getting WebRTC running on Xamarin + iOS?

  • DennisWeluDennisWelu USUniversity, Developer Group Leader ✭✭

    I did...this turned out to be the best solution :smile: https://www.frozenmountain.com/products-services/icelink

  • KiranRaoKiranRao USMember

    Thanks for the response. Was trying to avoid a paid solution, but it seems that it will cost more to try and develop our own solution.

    Question for you - you mentioned that IceLink comes bundled with a server, but from their documentation - it looks like you have to use a separate STUN service (like their WebSync)?

    Related note - Twillio is offering a free STUN service now.

  • DennisWeluDennisWelu USUniversity, Developer Group Leader ✭✭

    IceLink has a community edition that's free but of course has constraints (like WAN only works for some number of seconds). Not sure if that might work. Otherwise yeah, it's some $ to purchase.

    In terms of the server, you need both a STUN server and a server to facilitate the signaling. They can be on the same box of course if that works for your situation, but they typically aren't because the STUN server needs to sit on a public static IP. You can use one of the public STUN servers (here's a link I saved from a couple years ago with some public STUN's: https://gist.github.com/yetithefoot/7592580) but of course it's public (privacy concerns?) and no guarantee on performance/reliability. If peer-to-peer cannot work due to network topology of the participants then you would want a TURN server to do relay...that is often on the same box the STUN server is on (but doesn't have to be).

    IceLink provides an implementation of STUN/TURN that makes it easy to set up the server part. It's worked great. You just need to provision the server somewhere and manage it...

    So then there's the signaling server. They have WebSync set up so you can pretty much just plug it in to IceLink and away you go. Again you just need to host it somewhere. In my case I elected to avoid the secondary cost for WebSync. I was going to have my own API server anyway so I set up the signaling through that. There are IceLink examples that illustrate how to manage the signaling process using your own communication mechanism - I used SignalR. The gotcha I found with SignalR is messages actually are not guaranteed to be received in the same order they are sent. But I worked through that.

    Hope that helps!

  • KiranRaoKiranRao USMember

    This is great info, thanks! Will dig into it.

Sign In or Register to comment.