Results: Free Software voice & video testing
Last weekend on Software Freedom Day the Manchester FSFE Fellowship group, assisted by additional participants in Britain and Germany, spent the afternoon testing Free Software alternatives to Skype.
Results
The 25 sets of results were recorded, and can be browsed, sorted, and searched below.
Six audio tests successfully passed (24%), as did five video tests (20%). Mumble was the most successful client, passing 100% of tests (audio only, video is not yet supported). XMPP passed four out of 14 audio tests, whereas SIP passed only one out of ten (both video and audio). Of nine apps tested, only Mumble, Pidgin, Jitsi, and Google Talk's web client achieved passes.
Conclusions
The clients tested performed more poorly than expected, probably due to network problems. One of the difficulties in testing was that generally there was little or no information about why the test had failed.
SIP clients couldn't connect successfully except when both testers used the same client, and had accounts on the same SIP server. This was surprising, especially considering that the accounts used for testing were is many cases paid for and commercially supported.
The only client and protocol which consistently did what it promised was Mumble, which had 100% test pass rate. Unlike all other clients, Mumble uses its own protocol, and also offers audio conferencing and text-to-speech by default. Mumble users are constrained to using the same server however, unlike SIP and XMPP users who should theoretically each be able to use a separate server of their choice.
It would have been useful to have a local SIP and XMPP server on the same network as the testers in order to better identify network related problems. This could have helped determine whether failures stemmed from the client, network, or server..
Examining STUN and ICE configurations was beyond the scope of our tests, but as these technologies seem critical to whether calls succeed or fail, they merit careful examination when choosing or configuring SIP and XMPP servers.
FSFE Fellows will be pleased to note that XMPP calls using FSFE's own XMPP server were often successful: several volunteers could see and hear each other when both were using their @jabber.fsfe.org accounts.
A Skype alternative?
Skype uses a variety of notorious methods to punch through network obstacles like firewalls, and to ensure it can locate the intended call recipient wherever they may be. It also benefits from being a centralised system where, if necessary, all roads lead to Rome (or a Skype managed server, at least).
Contrastingly, Free Software systems play by conventional rules of network traffic, and try and connect callers using established procedures. They also have to communicate with a variety of different servers, which may operate in ways they don't know or expect. In addition to this, some of the chat clients that we tested have only recently added support for more than text-based chatting (Gajim in Debian testing is too old for video, for example), and probably add a few problems of their own into the mix.
Based on our test results, those looking for a Free Software Skype alternative should use either Mumble, or SIP with all callers on the same server and using the same client.
Moving Forward
We had hoped to publish a conclusive compatibility table of chat clients, providing a reference for people needing to know "what works with what", and "what is easiest". Instead, we have table of subjective results, and a list of questions:
- How can we determine whether an XMPP server supports audio and video ('Jingle')?
- How can we gather more detailed information about why a call failed?
- How can we determine whether a SIP or XMPP server supports STUN (and should therefore be less prone to network issues)?
If you have answers to these questions, please share them in the comments of the blog edition of this article.
Table
More data is available in the table source code.
| From Client | From OS | From DE | To Client | To OS | To DE | Report | Local | type | Audio | Video | Notes | Account 1 | Account 2 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Pidgin 2.7.3 | Crunchbang | openbox | Pidgin 2.1 | Fedora 17 | Gnome | Anna | true | XMPP | pass | pass | |||
| Pidgin 2.7.3 | Crunchbang | openbox | gajim 0.15 | Fedora 17 | Gnome | Anna | true | XMPP | fail | pass | |||
| Jitsi 1.0-build.3967 | Ubuntu 12.04 | Gnome/Unity | Gajim 0.15 | Fedora 17 | Gnome | Jamie | true | XMPP | fail | fail | Options were greyed out on Gajim | ||
| Pidgin 2.10.6 | Ms Win XP | na | Pidgin 2.7.3 | crunchbang | openbox | David | false | XMPP | fail | fail | |||
| Empathy 3.4.2.3 | Fedora 17 | Gnome | Empathy 3.4.2.3 | Ubuntu 12.04 | Gnome/Unity | Sam | true | XMPP | fail | fail | only text chat worked - no indication of a call request was received for either video or voice calls | ||
| snom220 hardphone snom220-SIP 3.56z | Snom Linux | NA | Wookey | false | SIP | fail | fail | call connection failed: 'No such domain' - local DNS failure | [email protected] | [email protected] | |||
| snom220 hardphone snom220-SIP 3.56z | Snom Linux | NA | wookey | false | SIP | fail | fail | call connection failed: 'No such domain' - local DNS failure | [email protected] | [email protected] | |||
| snom220 hardphone snom220-SIP 3.56z | Snom Linux | NA | n900 app | Maemo 5 | hildon | wookey | false | SIP | fail | fail | call connection failed: 'No such domain' - local DNS failure | [email protected] | [email protected] |
| n900 21.2011.38-1 | Maemo 5 | Hildon | linphone | wookey | false | SIP | fail | fail | call connection failed: 'unable to call, contact offline | [email protected] | [email protected] | ||
| n900 21.2011.38-1 | Maemo 5 | Hildon | n900 app | Maemo 5 | hildon | wookey | false | SIP | fail | fail | call connection failed: 'unable to call, contact offline | [email protected] | |
| n900 21.2011.38-1 | Maemo 5 | Hildon | snom220 hardphone snom220-SIP 3.56z | Snom Linux | NA | Sam | false | SIP | fail | fail | call connection failed: no response" 1st time, call unauthorised 2nd time" | [email protected] | [email protected] | |
| n900 21.2011.38-1 | Maemo 5 | Hildon | n900 21.2011.38-1 | Maemo 5 | Hildon | Sam | false | SIP | fail | fail | call connection failed | [email protected] | [email protected] | [email protected] |
| Jitsi 1.0-build.3967 | Ubuntu 12.04 | Gnome/Unity | Jitsi 1 | Fedora 17 | Gnome | Jamie | true | XMPP | fail | fail | ICE failed | [email protected] | |
| Pidgin unknown | Ubuntu 12.04 | Gnome/Unity | Pidgin 2.1 | Fedora 17 | Gnome | Jamie | true | XMPP | fail | fail | ICE failed | ||
| mumble 1.2.3 | Fedora 17 | Gnome | Murmur murmur-static_x86-1.2.3-412 | Debian Wheezy | Sam | false | mumble | pass | na | using bens server, it just works | |||
| Pidgin 2.7.3 | Crunchbang | openbox | kopete 1.2.5 | Fedora 17 | Gnome | Anna | true | XMPP | fail | fail | kopete couldn't initiate call with pidigin, only pidgin could initiate with kopete | ||
| Kopete 1.0.0 | Crunchbang | openbox | Pidgin 2.1 | Fedora 17 | Gnome | Anna | true | XMPP | fail | fail | |||
| Jitsi 1.0-build.3967 | OSX 10.6.8 | NA | Pidgin? ? | Ubuntu | KDE | Robin | false | XMPP | pass | pass | notes | ||
| Jitsi 1.0-build.3967 | OSX 10.6.8 | NA | Jitsi 1.0-build.3967 | Fedora | Gnome | Robin | true | SIP | pass | pass | Audio one one was pretty garbled, both using SIP2SIP | ||
| Jitsi 1.0-build.3967 | OSX 10.6.8 | NA | Jitsi 1.0-build.3967 | Fedora | Gnome | Robin | true | SIP | fail | fail | failed using different servers: SIP2SIP and localphone | ||
| linphone 3.5.2 | Fedora 17 | Gnome | Jitsi 1.0-build.3967 | OSX 10.6.8 | NA | Sam | true | SIP | fail | fail | failed using different servers: SIP2SIP and linphone | ||
| Pidgin 2.10.3 | Kubuntu 12.04 | KDE | Google Talk, webclient ??? | OSX, Android | NA | Steffi | true | XMPP | pass | pass | |||
| Pidgin 2.10.3 | Kubuntu 12.04 | KDE | Google Talk, webclient ??? | ??? | NA | Steffi | false | XMPP | pass | fail | Video was fine, audio worked but the quality was very low | ||
| Psi 0.14 | Fedora 17 | Gnome | psi+ 0.15.5337 | OSX | NA | Sam | true | XMPP | fail | fail | |||
| Psi/Psi+ v0.15.5339-webkit | Fedora 17 | Gnome | psi+ Psi+ v0.15.3910 | KKubuntu 12.04 | KDE | Sam | true | XMPP | fail | fail | auth requests never appeared, voice chat enabled via pssimedia client |