What if you knew what your customers’ experience really was, rather than just what your switch engineers told you it was? Can you test the actual user experience of data, as well as voice, services? California-based wireless test solutions company, Comarco, together with Swiss-based SwissQual are making a splash in Europe with a range of products based on algorithms that can measure voice and data quality — from a user’s perspective.
Keith Dyer talks to Comarco Wireless Test Solutions senior vp Gregory Maton to find out how operators can test and benchmark the most advanced of services.
Mobile Europe: Can you tell us a bit about your experience in this market and the reasons for your move into the European market?
Greg Maton: Comarco has been in this drive test business for almost 15 years now. We had the first benchmarking product in the market place and have had great success in the US market place as well as other areas in North and South America. In Europe, though, we have had mixed results so what we did was seek out a partner and we found an excellent one in SwissQual.
ME: How has the SwissQual partnership enabled you to address the European market more effectively?
GM: Prior to our partnership coming together, SwissQual had established themselves as a leader in the area of drive test products using pioneering objective voice and video Quality of Service algorithms. The SwissQual algorithms are able to produce valid mean opinion scores (MOS) in the presence of low bit rate vocoders, 3G techologies, VoIP, etc. In addition to producing objective scores, the algorithms can pinpoint the cause of any audio or video degradation, which is unique to our industry. They have contributed significantly to ITU standards in this area. In addition to SwissQual technology, they have a deep understanding of the European market which we were lacking.
Once Comarco and SwissQual found each other it made perfect sense to jointly develop a drive test tool that brought both company’s expertise together. We started with a clean sheet of paper and the result is our Seven.Five product family that is designed to test 2G, 2.5G, and 3G networks by emulating the subscriber experience. SwissQual has been selling Seven.Five in Europe for about two years now and with the proliferation of 3G operators we are seeing a tremendous interest in the product.
ME: And how is that different from the traditional testing operators carry out?
GM: Traditionally, operators have been looking at switch based data, engineering parameters collected from drive test tools, and huge volumes of protocol messages to get an understanding of how their networks perform. We are saying to the carriers that they must emulate the customer and subsequently focus on areas where the customer experience is poor. The Seven.Five has a wide range of available tests that model customer usage of their wireless devices, including voice quality analysis in the presence of background noise, picture messaging, web browsing, and soon video telephony. Key performance indicators are available for all of these tests so engineers can understand the network elements that contribute to customer perceived problems.
ME: So how does the Comarco approach enable operators to target their resources better?
GM: If you look at our Seven.Five product it is a rack of cell phones that place calls any way the carrier likes, and they typically like to mimic how a customer would use their phone. Then we run our algorithms against these tests and come back with objective scores on network quality. And behind all that is the engineering data necessary to optimise the network. So if a guy is doing a voice test and sees an area where his voice scores are twos and threes instead of threes and fours, he can go in, understand what caused the voice quality impairments and look at the engineering data behind it so that he can the take corrective action.
In this domain we’ve got a complete suite of products depending on the carrier and his specific needs. And the products range right up from a phone and a laptop — which is a basic technician type tool. So we can give an RF profile along with the engineering data from the phone, up to benchmarking where we can add up to 16 different calling modules. For example in Brazil we have a customer who has 12 calling modules because they’ve got so many different technologies and so many different frequency bands with so many different carriers.
ME: So, how does an operator understand his network quality, without spending a fortune, hiring engineers to drive expensive boxes all over the place?
GM: As you can imagine, it can be extremely expensive to hire engineers to be out in the field with costly capital equipment. This has been frustrating for operators since they do not get good efficiency out of engineers driving networks. I would argue that people in the drive test environment are probably spending 85-90% of the time testing network elements where there are really no issues. That’s a waste of good engineering resources. They ought to be working on problem areas. So what we want to do is help operators to find a problem area inexpensively, and then let the engineers spend 90% of their time fixing the problems and invest the engineering resources they need there.
SwissQual has developed QualiPoc, an application that runs today on Symbian based GSM/ GPRS phones. This is an application that runs strictly on the phone, and measures basic network quality, which is accessibility — if you go to make a call and it is blocked; retainability — measuring dropped calls; and then voice quality with the SwissQual algorithms. Many data tests are also available, including SMS, MMS, web browsing, and FTP.
The objective there is now we can put thousands of these probes in a network very cost effectively and then pick up data points on a random sample, based on what you and I typically do with our cellphones, not based on a drive test tool that’s going up and down the street, maybe several hundred metres away from where the person is really using their phone.
QualiPoc gives carriers a really good look at their network performance from their customer’s perspective because the phones are located where customers use their phones. With QualiPoc running, the phone can wake up, make a call, do some voice exchanges with the receiving phone, score it, end the call and ftp the data to a central server. That data gets built up as you get more and more of these Qualipoc devices out in the field, and carriers can really identify their weak spots. And where they identify their weak spots they can now send in an engineer with one of our high end engineering tools to really get in and understand what’s going wrong and get it fixed.
So that’s our product strategy. Our Seven.Five platform supports the following four product areas; optimisation, benchmarking, RF scanning, and post-processing. The SwissQual Qualipoc product compliments Seven.Five nicely by providing a lower cost method to obtain bulk network quality data that provides an excellent snapshot of the network performance. Once you find your network’s weak areas, you can bring in Seven.Five configured for Optimisation to evaluate the network at the deepest levels and then fix them. And then on top of that you lay Seven.Five configured for benchmarking which gives carriers a good snapshot of their network versus other carriers in the market.
ME: What is it about these quality test tools that makes them different from other tools in the market?
GM: There’s a tremendous amount of value in the SwissQual algorithms, combined with our hardware and software architecture, that enables us to perform tests that truly emulate customer behavior. The primary algorithm we use is SquadLQ (listening quality). Audio is fed into the algorithm and scores come out, along with other information that allows you to understand the problems associated with degraded audio. That’s what we would consider an intrusive algorithm. We inject clean speech into the network and, at the other end, measure the coded speech against the reference speech. The differences are where we come up with scores. But what’s happening now is that carriers are adding a lot of network elements that improve voice quality in the presence of noise or echo. We have an algorithm that measures audio quality in noisy backgrounds, so we can score how effective their noise cancellation is, and that is unique in the industry.
We have another set of algorithms in relation to echo cancellation. If, for instance, you are talking on the phone and we detect echo we will score that accordingly. The other thing we can do is actively introduce echo. We can vary the distance between the initial speech and the “echo speech” to find out how effective the network elements are at stripping out echo. And probably the most powerful algorithm where we are really ahead of the networks, though much more so in the US than in Europe, is our video quality algorithm. We can actually score streaming video clips.
There is also a non-intrusive speech quality algorithm that allows it to listen to a conversation without it being a reference clip. So it enables us to observe speech and score it. It is powerful in that you can score audio from any source. Also, the algorithm can run in the back end of the networks so we could be listening to audio quality all the time, and scoring it, and that way you could measure audio quality at the switch based on your customer’s actual usage.
ME: Video is a good example, but what else about the way operators’ own businesses have changed have impacted on your market and technology?
GM: In our original product we had proprietary circuitry that would connect to phones – but we found that as we went to 2.5 and 3G we needed to connect to data systems. We had to look at how customers are using their phones. So for data people are doing WAP, SMS (if you want to consider that data) MMS, hooking them to their computers and using them as modems, surfing the web, sending an email. So we had to come up with a platform that allowed us to connect a phone to our system and make it behave like a customer behaves. We’ve integrated a special embedded Windows XP based computer for each phone. The platform’s unique in that we can run Internet Explorer and measure how the network performs, just like a customer is using it to browse websites. We can send and receive emails from the product, and measure critical network parameters and key performance indicators as well, that allow us to rank the carriers’ performance. One customer spent millions on software in his network to enhance the performance of Internet Explorer from a customer’s perspective, and we were the only drive test tool that could use IE as a browsing test, so he could measure the before and after experience after his investment.
ME: So mobile data has been the driver?
GM: The primary driver for buying the Seven.Five product was data testing on GPRS networks. Currently, EDGE and UMTS are the big drivers. As these new technologies come on line, the carrier deploying them as well as their competitors want to know how they perform from a customer perspective.
As a result, we are seeing UMTS and EDGE in almost every RFP we get. Looking forward, we are seeing a tremendous interest in video telephony and streaming video. As these services start showing up in the market place, we expect that this will drive further demand for network testing. SwissQual and Comarco are uniquely positioned in the market place to respond to this demand with Qualipoc and Seven.Five.
Although the mobile operators are also sizing up the smart meter opportunity, BT says long range radio is a better technical solution than mobile because “Unlike mobile, it can provide truly nationwide coverage and dependable reception indoors.”
“The fact it operates on dedicated licensed spectrum is also important as it is ideal in ensuring the security of supply and protection of consumer data while meeting the needs of the Energy Industry,” a BT statement said. BT said it has taken 18 months sizing up the technical options.
The UK Government’s proposed smart metering initiative is due to cover 28 million homes and small business properties by 2020 or sooner. The Government will publish a prospectus that will provide details about the project and the possible commercial opportunities. The partners will review this document and will then formally launch their proposal in September, BT said.
Olivia Garfield, BT Group Strategy Director, said: “Smart meters will use telecommunications to deliver important environmental benefits and so BT is determined to be at the heart of the project. It is vital that any solution is designed for ubiquitous coverage of homes and is thoroughly secure and resilient. We believe that long range radio is the only technology to offer nationwide coverage and we will release more detail in a series of events in September.”