OpEx Reduction Strategies for Today’s DSL World
Providing telephony service in the old POTs world, was a relatively straight forward and well understood business. Telephones were simple and reliable devices. Customer service faults were mainly attributable to cable pair faults and the occasional POTs switch fault. Over decades, Telco’s evolved very effective customer fault management processes, supported by night routining of copper lines to proactively address copper line problems.
In today’s IP world, cable pair faults are still an issue but the overall problem set is orders of magnitude more complex. Consumers today expect high-speed data connections to web servers, reliable Voice over IP communications and quality video entertainment services and all for very little money.
When it all stops working, consumers turn to their DSL service provider to fix the problem. Unfortunately analysing the problems can be very complex and the cause can often lie in the customer’s own equipment or even in other service provider’s networks, but the ISP is left to sort out the mess. There is no way that the ISP can escape this responsibility. Many have tried, but the customers have no one else to turn to. The harder the service provider tries to shirk this responsibility, the greater the cost of failure. One need only look at the appalling mess some of today’s ‘low cost’ ISPs have managed to get themselves into: - unhappy customers voting with their feet, inadequate numbers of employees trying to knife and fork through festering problems and the cost of fighting customers over disputed bills. Such an out of control situation with dissatisfied customers and uncontrolled costs is clearly disastrous.
There is however a very simple way to control and minimise the OpEx associated with service provision and that is to implement an effective fault management process.
When a customer rings the ISP’s call centre to report a problem, it is absolutely imperative that the service provider can very quickly and definitively assign the fault to either: -
- The customer’s own equipment
- The ISP’s network
- Other service providers networks
This problem analysis needs to be done while the customer is still on the line otherwise the costs will increase significantly. If the ISP can definitively show that the fault lies in the customer’s own equipment or outside of his network then he can bounce the fault without redress.
The technology to do this is readily available, proven and cost effective.
To accurately assign the fault, the ISPs fault management system must be able to verify if the customer’s copper line is in working order, whether the assigned port on the DSLAM line card is functioning properly and whether the backhaul to the ISP’s server is operational. This can be definitively achieved in real time by using a Test Access Switch to switch a Testhead into the customer line circuit.
The Testhead consists of two parts: -
- A ‘Golden Modem’ and an IPTV and VOIP service test module
- A copper line test module
The Test Access Switch splits the customer line at the Splitter / DSLAM interface. It physically connects the ‘Golden Modem’ to the customers DSLAM port which in turn pings the ISP’s server. If this works, then there is nothing wrong with the DSLAM Port or the ISP’s network. The service test module can also be used to automatically verify that the IP services sold to the customer meet the specified performance criteria. It can carry out DSL speed tests and IPTV and VOIP service testing.
The Test Access Switch also physically connects the customer’s copper line to the copper line test module. This checks the condition of the customer’s line and identifies any line faults that may be present as well as checking the customer’s line termination.
If the measurements show that the copper line is electrically sound and properly terminated and that the DSLAM port is working then the problem by a process of elimination must lie in the customer’s equipment.
The test results are automatically interpreted and presented as clear definitive messages on the call centre operator’s screen. The customer is happy. His ISP clearly knows what he is doing; the ISP has definitively identified the cause of the fault and targeted remedial action is initiated as appropriate.
The most important thing however is that the service provider has only expended 2 Minutes of call centre operative time definitively identifying the source of the problem, without having to resort to expensive skilled resource. The word ‘definitively’ cannot be stressed too greatly. If the fault identification process is not 100% definitive then the ISP opens the door to the uncontrolled costs of failure.
Many ISPs have had the misfortune of having been persuaded by their DSLAM vendors to invest in integrated ‘SELT, DELT and MELT’ as a universal solution to customer service measurement.
- SELT = Single Ended Line Test
- DELT = Double Ended Line Test
- MELT = Metallic Line Test
The DSLAM can generate masses of statistics on DSL performance, it can also pull off statisticss from the customer’s DSL modem: -at least as long as the customer has not decided to use an incompatible modem! Loads of ‘useful’ information can be derived from this data. This is all very impressive but unfortunately it is only of very limited use in practice in solving real customer service problems, because SELT and DELT only works while there is a DSL connection established.
The customer however only really gets hot and bothered in the converse situation, when his or her DSL connection does not work at all. This will definitely guarantee a call to the ISP’s helpline. Under these circumstances SELT and DELT will tell you that it is not possible to establish a DSL connection. It however cannot tell you why! The fault may be due to a faulty line card port on the DSLAM or a faulty DSL modem at the customer’s end. But then again it could be neither of these, it could be a real line fault!
SELT and DELT enable the ISP’s call centre operative to confirm that there is a fault but it cannot identify the source of the fault.
Some DSLAM vendors claim that they can test the copper line using their built in ‘MELT’ functionality. Unfortunately this is another half truth. Yes the DSLAM can carry out AC tests on the line but these tests are of little value as they cannot identify common line faults such as Leg to Ground leakage. All of the really useful line tests require DC coupling of the test circuitry to the line. This is however not possible on the line card because the DSLAM port is AC coupled to the line via a line transformer. - One of our major customers carried out an exercise to validate DSLAM ‘MELT’. They took ten years worth of real line fault data and checked how many of these ‘real’ faults could have been detected using the DSLAM MELT test functionality. The answer was a single digit percentage!
Faced with a ‘don’t know’ answer from the integrated SELT DELT and MELT, the unfortunate operative at the ISP’s call centre is left with few options. He can only ask the customer to cold start his PC and Modem and check the status of the modem indicator LEDs. These LEDs again only verify that there is no service but cannot identify the cause of the fault. – ‘Yes we can verify that you have no service Sir ’ does not do very much for the customer who was complaining about of this fact.
Unable to sort out the problem, the call centre operative escalates the fault to the ISP’s ‘technical helpdesk’ who lacking a proper testhead and test access switch will not be able to do much either. The first thing they will do is to further annoy the customer by asking him again to cold start his PC and Modem. –‘technical professionals’ generally do not trust the actions of the ‘unskilled’ call centre operatives. If this fails they will try to eliminate the line. In the case of an ‘unbundled’ copper pair this will mean asking the incumbent operator to do a proper line test. If the line tests OK, then the ISP will usually be charged for the test. If it tests OK it still leaves the big question open – has the DSLAM line card port failed or is there a problem with the customer’s DSL modem. The only way that this can be checked is to send a technician to the DSLAM to manually break the line and to carry out tests with a hand held tester or plug a known good working modem onto the line at the customer’s premises. In either case it means expensive truck rolls to resolve the problem.
It is fairly easy to see that the ISP’s ‘technical helpdesk’ will need to grow at least linearly with the network to keep on top of customer problem reports. Experience has shown that ISPs never really consider this in assessing their ongoing operational costs. Those that understand this, install a TAM and Testhead in the first place! In most cases, the technical help desk is seen as an unforeseen ‘cost of failure’ and is therefore under resourced. As a consequence, this cost of failure grows even further because additional call centre operatives are required to manage fed-up customers who are not getting their problems fixed and are now also refusing to pay their bills.
Another unforeseen problem is that technical helpdesks do not scale very well. Experience has shown that customer support costs grow non-linearly with network growth. The support costs per customer actually increase as the network expands! The reason for this is simple. Technical helpdesks are originally set up using small numbers of technical experts who can relatively effectively knife and fork their way through manageable numbers of problems.
As the network grows and the number of problem reports grow, there is pressure to increase the size of the support team and even greater pressure is applied to control these unforeseen costs. The result is that lower skilled technicians are employed who rely on pre-defined faulting processes and red and green light testers rather than relying on the high quality technician’s inherent analytical skills and ability to drive the test equipment. This cuts down the manpower cost of each truck roll, but the number of truck rolls required increase as these guys are unable to tackle more complex problems. –In a case study we carried out for one of our customers we found that a technician had been dispatched to the exchange on eight occasions to test the DSLAM card for a fault over a six month period. On each occasion the result was ‘no fault found’ but the customer kept on experiencing difficulties establishing a DSL connection. A proper analysis of the problem showed that the problem was due to a defective DSL modem.
It is however never too late to retrofit Test Access Switches and Testheads and avoid this massive cost of failure. UTEL are global experts in supplying and fitting retrofit TAM solutions. UTEL have retrofitted over 10 million DSL lines in Europe with Test Access Switches over the past few years and have developed dedicated TAM retrofit solutions for a wide range of DSLAMs. The retrofitting of UTEL Test Access Switches has helped global players to turn their businesses around by enabling them to achieve a year on year decrease in OpEx in an expanding market. Specialist DSL technical support teams, that were growing year on year, have become virtually redundant and customer service transaction times have also been dramatically reduced. Most importantly the ability to automatically identify the source of customer problems has greatly improved the suppliers reputation in the market place.
If you would like further information please contact sales@utel.co.uk