GSoC’22 at Matrix.org
Hey there, GSoC ’22 has finally come to an end and I am very excited to share that I have successfully completed the program. I was selected for GSoC’ 22 at Matrix for the making the WebRTC stuff more stable for daily use in FluffyChat. In this blog, I would like to share my experience with this amazing program and also my journey so far.
The Google Summer of Code, often abbreviated to GSoC, is an international annual program in which Google awards stipends to students who successfully complete a free and open-source software coding project during the summer. The program is open to university students aged 18 or over. Wikipedia
fyi- This is my first blog so it will suck (you can probably tell I copied the intro from somewhere) :D
TLDR: Most of my work for GSoC can be found in the FluffyChat repository and some patches in the Dart Matrix sdk.
FluffyChat merge requests — https://gitlab.com/famedly/fluffychat/-/merge_requests?scope=all&state=all&author_username=techno_disaster
Dart Matrix sdk — https://gitlab.com/famedly/company/frontend/famedlysdk/-/merge_requests?scope=all&state=all&author_username=techno_disaster
The problem:
The main task FluffyChat listed as an issue for GSoC at matrix.org was making calls more stable. This included several other sub tasks like notifying the user for an incoming call (in background and terminated states), giving the user haptic and audio feedback when call state changes (answered, declined/ended), show errors in case something breaks, etc.
The original issue can be found here — https://gitlab.com/famedly/fluffychat/-/issues/874
The solution:
The timeline which I submitted in my proposal before students are selected by a organization was fairly straightforward. In the first half of the coding period I would be working on implementing a way to notify the user about incoming calls, this was supposed to work when the app was in background and terminated as well, implement giving user haptic and audio feedback once a call is answered or ended. The second half I would just pick random issues which needed to be fixed in FluffyChat along with adding integration tests.
Of course no plan can ever run without any hiccups. For the first task where I had to implement a way for FluffyChat to notify any incoming calls, I chose to write native Android code and popup a calling UI like most other applications. This would have been easy if only the whole thing was written native :D, turns out you can only execute a top level function on a push notification, so bringing up the calling UI and other various matrix sync stuff would require several code fragments (native and flutter because the matrix sdk was in the flutter engine) to be handled manually. Sounds really error prone and complicated, I gave up on this anyway. My next solution was to just bring up the app (appear on top) if the app is in background and launch the app if terminated (not sure why Android allows this? but yayy). This would mean listening for m.call.invite push notifications and running flutter code in background. Thankfully FluffyChat already had unified push implemented to decode push notifications and so there was already a flutter engine to run dart code on a push notification. This worked out fairly well along with enabling the show on lockscreen bool to emulate a incoming call which basically opens the app and shows the calling UI even when the screen is locked. But as you probably expected, not all OEMs allow this and this was a really flaky solution. So coming to our last pivot, Callkeep. I found out that Android allows you to use the telecom library and delegate calls to the native dialer. Fun. Turns out this already had a library from the same author as flutter_webrtc (thanks Duan :D). So all I had to do was implement this and boom, FluffyChat now registers as a calling account (consider this a esim) and all calls are passed through to the Android native dialer app. This worked very well, could answer directly from notifications, wakes up device when terminated, etc. It also supported other nice features some OEMs like swapping between 2 calls, holding the other one. The only downside to this was that these calls were now shown in the call log of the dialer (some might say this impedes the user experience). So we decided to have Callkeep and the appear to top feature both and allow the user to select which one they would like for incoming calls.
The next tasks were fairly easy, adding haptic and audio feedback for call state updates just included calling the vibration library while listening to the call state updates stream and error handling was just put in a SnackBar with tons of try catches.
The second half of the timeline was fairly smooth sailing as well (except for the part where I had exams and had to implement most of the stuff in the last 5 days :D /me is writing this blog 5 hours before the deadline). For the second half I decided to implement group calls (msc3401) instead of integration tests and my mentor (Krille) was fine with this as well. The MSC itself had already been implemented in the dart matrix sdk and so I just had to do the client implementation and some patches in the dart matrix sdk.
Most of my work for GSoC can be found in the FluffyChat repository or some fixes in the Dart Matrix sdk.
FluffyChat merge requests— https://gitlab.com/famedly/fluffychat/-/merge_requests?scope=all&state=all&author_username=techno_disaster
Dart Matrix sdk — https://gitlab.com/famedly/company/frontend/famedlysdk/-/merge_requests?scope=all&state=all&author_username=techno_disaster
Lastly, there is still a ton of work to be done to make calls stable for daily usage but it’s getting there. Next tasks on my list include checking for a active group call in a room, a better UI for desktop mode during group calls, etc. Special thanks to my mentor Krille for helping me around dart matrix sdk and for allowing me to have a really flexible timeline wouldn’t have finished without this, Nico for helping me with WebRTC and matrix stuff in general (and being a mentor when no one asked him to be :P) and Duan for all the WebRTC stuff. and the whole matrix community for their immense support and mentorship :D
Update (10 Nov): Had a small trip to Germany and after having a talk with Krille, Nico and a few other people, we came to the conclusion that asking users for the calling account permission or even the appear on top permission would be a bit too much. Although this works pretty nicely I would not accept this in a production grade app. A few other disadvantages of this were no easy way to handle stuff on IOS and also missing video and group call functionalities in the native dialer via Callkeep. So we just decided to copy Whatsapp :D
Because IOS and Android handle calls differently and IOS needs VOIP stuff through Callkit and notifications that are decrypted via Pushkit and another Rust package, there will be some differences in the code structure but basically we show the user a native UI for incoming calls and then open up the app once they accept the call. This way we can use the already implemented UI for group calls and video calls.