Last weekend on Software Freedom Day the Manchester FSFE Fellowship group, assisted by additional participants in Britain and Germany, spent the afternoon testing Free Software alternatives to Skype.
Results
The 25 sets of results were recorded, and can be browsed, sorted, and searched on the news article edition of this post.
Six audio tests successfully passed (24%), as did five video tests (20%). Mumble was the most successful client, passing 100% of tests (audio only, video is not yet supported). XMPP passed four out of 14 audio tests, whereas SIP passed only one out of ten (both video and audio). Of nine apps tested, only Mumble, Pidgin, Jitsi, and Google Talk’s web client achieved passes.
- Chat testing results table
Conclusions
The clients tested performed more poorly than expected, probably due to network problems. One of the difficulties in testing was that generally there was little or no information about why the test had failed.
SIP clients couldn’t connect successfully except when both testers used the same client, and had accounts on the same SIP server. This was surprising, especially considering that the accounts used for testing were is many cases paid for and commercially supported.
The only client and protocol which consistently did what it promised was Mumble, which had 100% test pass rate. Unlike all other clients, Mumble uses its own protocol, and also offers audio conferencing and text-to-speech by default. Mumble users are constrained to using the same server however, unlike SIP and XMPP users who should theoretically each be able to use a separate server of their choice.
It would have been useful to have a local SIP and XMPP server on the same network as the testers in order to better identify network related problems. This could have helped determine whether failures stemmed from the client, network, or server..
Examining STUN and ICE configurations was beyond the scope of our tests, but as these technologies seem critical to whether calls succeed or fail, they merit careful examination when choosing or configuring SIP and XMPP servers.
FSFE Fellows will be pleased to note that XMPP calls using FSFE’s own XMPP server were often successful: several volunteers could see and hear each other when both were using their @jabber.fsfe.org accounts.
A Skype alternative?
Skype uses a variety of notorious methods to punch through network obstacles like firewalls, and to ensure it can locate the intended call recipient wherever they may be. It also benefits from being a centralised system where, if necessary, all roads lead to Rome (or a Skype managed server, at least).
Contrastingly, Free Software systems play by conventional rules of network traffic, and try and connect callers using established procedures. They also have to communicate with a variety of different servers, which may operate in ways they don’t know or expect. In addition to this, some of the chat clients that we tested have only recently added support for more than text-based chatting (Gajim in Debian testing is too old for video, for example), and probably add a few problems of their own into the mix.
Based on our test results, those looking for a Free Software Skype alternative should use either Mumble, or SIP with all callers on the same server and using the same client.
Moving Forward
We had hoped to publish a conclusive compatibility table of chat clients, providing a reference for people needing to know “what works with what”, and “what is easiest”. Instead, we have table of subjective results, and a list of questions:
- How can we determine whether an XMPP server supports audio and video (‘Jingle‘)?
- How can we gather more detailed information about why a call failed?
- How can we determine whether a SIP or XMPP server supports STUN (and should therefore be less prone to network issues)?
If you have answers to these questions, please share them in the comments!
I spent time to look for this kind of study, thanck’s for sharing
Well done on the analysis, and thank you, it is needed. Though I think the most important test for a Skype alternative is: Can I run the client, set up a username and call anyone else without thinking about what server I am on.
The features a true replacement should have are:
1. Peer to peer address book that works out which server to use to connect to a contact without user interaction.
2. Sip routing agent that can, using peers, get around firewall and nat issues so that calling just works.
3. provide conference call features which choose the best connected/resourced client as the host for mixing etc.
Anything less and users will just use Skype.
@Chris Lee: good points, thanks. Some Free Software chat clients integrate account registration for those who need them (Linphone client automatically registers new accounts on their own SIP server, for example). Developers have a difficult balance to strike between ease of configuration and choice of account provider (which is the point of having federated technologies in the first place).
opentelecoms.org has good information about how to fulfill the first two requirements that you stipulate for replacing skype – I recommend reading around it (thanks to Daniel Pocock for introducing me to the site). I’m not so sure about your third point though (conf call mixing).
It’s worth remembering that the needs of Skype users within organisations are different to those of the general public. An internal Mumble or XMPP server could probably work very well in a controlled environment where staff share a single server and use standardised clients. But I agree with you that we’ve a way to go before Free Software alternatives can serve the needs of everyone.
Just think about ENUM! With this technology everyone is reachable. I can`t beleve that no one fights against the blocking of SIP-protocol by the telephone/internet provider. Just look at pbxes.com how it can be realised.
What about Ekiga? Isn’t it free software?
As unfortunate as it is, my additional requirement for a Skype replacement is the availability of Mac and Windows clients.
@Marco: yes Ekiga is Free Software, and it was on the list of clients to test. Nobody got around to testing it however. I’ve used it before and I know it’s quite popular.
@Thomas Koch: That shouldn’t be a big problem – several of the clients that we tested were cross platform, and people actually tested calls on different operating systems. Pidgin, Gajim, Psi, and Jitsi, for example, are all available for GNU/Linux, Mac, and Windows.
@samtuke: I always wanted to replace Skype with Ekiga (it’s probalby the most complete and cross-platform tool). The only problem is to instruct non-aware people… I have to explain how to install and use it (and, more important, why!) to my wife’s sister and her husband… but I have to explain it in Japanese… and I’m Italian! I don’t think it will be simple! 😉
I know what you mean! Hopefully these pages will be useful to you:
Ten reasons to boycott Skype
More arguments against Skype
Skype and censorship
FSFE now hosts a dedicated mailing list for Free real-time communication discussion and coordination:
free-rtc@lists.fsfe.org
https://lists.fsfe.org/mailman/listinfo/free-rtc
Feel free to send questions, ideas, and proposals there.