Keynote Systems released a benchmark study ranking the audio quality of major VoIP providers in the U.S. as perceived by end-users. For the study, Keynote ranked six leading VoIP providers on critical performance factors that influence the end-user experience using Keynote's VoIP Perspective measurement service.
Included in the study were AT&T CallVantage, Packet 8, Primus Lingo, SkypeOut, Verizon Voicewing and Vonage in the New York and San Francisco metro areas.
Key conclusions included:
- Vonage ranked first overall as the most reliable VoIP service provider with a normalized rank of over 80 points on a scale of 0-100. Keynote found a significant gap in reliability between Vonage and the lowest ranking providers. In spite of its high average, Vonage still has room for improvement on the Dropped Calls performance factor.
- AT&T CallVantage led all other providers in audio clarity with a noticeable gap between AT&T CallVantage and its competitors.
- "Internet telephone service is not yet up to the standards to which users are accustomed when using standard 'plain old telephone service' (POTS) and VoIP providers have some work to do to capitalize effectively on the growing consumer adoption of VoIP service"
"VoIP reliability and audio clarity are important factors limiting the widespread adoption of VoIP in consumer markets. Consumers are unsure whether VoIP can live up to the dial-tone reliability and crystal-clear communication quality they have come to expect with traditional phone service over the years," said Dharmesh Thakker, senior product manager for service level solutions at Keynote.
Methodology: During a five-week period from May 21 to June 25, 2005, Keynote placed 163,000 VoIP calls. The VoIP calls were carried on three business-class networks: AT&T, Sprint and UUNET. In addition, the study captured the impact of the last-mile on call quality by measuring each of the six providers on residential DSL lines from SBC and Verizon and residential cable lines from Comcast and Time Warner Cable as well.
Keynote placed coast-to-coast calls from VoIP to standard telephones (PSTN) between San Francisco and New York once every 30 minutes. Additionally, local loop VoIP to PSTN calls were placed every 30 minutes for each of the provider and network combinations to emulate a contained environment and accurately track the network impact on call quality. Calls were also placed from and to traditional phones every 30 minutes between the two metro areas to understand what residential customers can expect when switching from traditional phone lines to VoIP.
See
Network Diagram
To quantify the end-user experience with call quality over each VoIP provider and network combination, Keynote identified ten key performance factors based on multiple network and audio fidelity metrics that are collected during the call placement:
- Service Availability -- indicating the reliability of the service
- Outage Hours -- how many minutes the service was unavailable during the study collection period and on a daily basis
- Average number of call attempts -- how many times, on average, a user must call to establish a connection
- Dropped Calls -- the number of times a call is dropped in the midst of a conversation
- Audio Delay -- once the call is established, the average lag between utterances. High audio delay leads to overlapping and unpleasant conversations in real-life.
- Listening quality (or Mean Opinion Score -- MOS) measured using the Perceptual Evaluation of Speech Quality scale (PESQ) -- the internationally recognized ITU-T P.862 standard
- Audio delay consistency over time
- Audio delay geographic uniformity between New York and San Francisco
- MOS consistency over time
- MOS geographic uniformity between New York and San Francisco
The full report is available for sale by Keynote.
http://www.keynote.com
