Page 5
Working with local dial tone providers. Generally speaking, if you have done your homework, there is no reason for the Telco to dispatch a Tech to your site. In fact, this is likely to slow getting the problem resolved. However, understand that you now probably know more about the telephone network than the person who enters the repair tickets. Explaining in detail the troubleshooting you have done (and even the nature of the problem) is likely to be futile. Your best bet is to give a very brief description of the problem, and politely insist that they have this ticket referred to the "trunking group" because it is a "network problem" and to have someone call you back. When they do call you back, be sure to inquire if they are with the trunking group. Then explain to them in detail what the problem is, the testing you have done and the results thereof, and the fact you have been unable to find any explanation other than a network problem.
In some cases, local procedure will insist that a Tech be dispatched. You should make a "friendly" attempt to explain that this wasting their money and your time, but don't make too big an issue of this. However, when the Tech arrives, stay with him/her and be friendly and helpful as they work. Offer coffee (or pizza if it's lunch time). Explain that you are sorry that this is probably a waste of their time, as you believe it is a network problem. Offer to show them the air studio(s) - chances are they'll be interested to see your station(s). Be sure to mention all of the stations at your location, as they may be a listener. And be sure to give him/her T-shirts, coffee mugs, or whatever you have before s/he leaves. Make this person your ally and be sure to find out how to get back in touch later. The goal is for this person to think you are ok, and understand that the problem you are working together to solve is having an effect on the listeners. It is also good to build up a "blue collar" rapport with the Tech. Let him/her know that the boss is breathing down your neck (particularly if things have already dragged on for some time).
In the meanwhile, pay attention to what the Tech is doing. If the problem really is a network problem, they may be unable to find any "problems with the line" without your assistance. Don't let him or her depart at this point (that is one reason you have to stay with the Tech). Now explain to the Tech how to create the problem and do so. This may mean having the far end call into the Tech's test equipment, or dialing long distance, etc. Or, you may have to demonstrate the problem using your equipment.
The goal is to convince this person that there is a problem and to have him/her call to the trunking group to get it fixed (not bad as s/he gets to sit around drinking coffee chatting with you and seeing the radio station while waiting for this to happen).
Once you have someone from the trunking group involved, explain to them the tests you have done, and the results. If s/he tries to blame your equipment explain why this can't be. Then ask if s/he has an alternative explanation. If there is a Telco Tech at your site, s/he should be able to verify that they saw the results you describe.
At this point, the procedure is identical to what we described above for working with long distance carriers. The person from the trunking group will set up a trace and you will demonstrate the problem (see above) so they can determine where the problem lies.
CASE STUDIES
Case 1 - Long Distance Access Problems
The following case history is from a codec user in Phoenix. Customer reported problems placing Circuit Switched Data calls to a particular site. This site was a long distance call. We made some test calls and rapidly collected the following information on the symptoms:
· Problem occurred with outbound long distance calls only. Local and inbound calls worked 100% of the time. We were able to duplicate the same pattern calling to a third site in a third city. · The problem occurred nearly 100% of the time. Out of 20 test calls made (at roughly 10 am to 1 PM, plus a few calls the morning before) 19 showed the problem. · Each of the codecs obtained "lock" (frame) and received its own audio back. Careful examination showed no mixing of audio from one site to the other (the network has no ability to mix two data streams together, so this proved the problem was not due to improper mix minuses). · The same results were obtained with three different long distance carriers. · Voice calls to the third location were unaffected (voice calls to the site of the original destination were not attempted).
We concluded that the problem was that some network element had been left in a diagnostic mode (bi-lateral loopback). Based on the fact that only long distance calls were involved, combined with the fact it occurred with multiple long distance carriers, indicated the problem was with the trunks to the Access Tandem (refer to Figure 5). We recommended that the customer contact the local phone company and that he should request that this matter be referred to their trunking group. That this was "a tandem trunk problem and involved data calls only".
After over 24 hours of wrangling, the customer was able to talk to someone in the trunking group. This person was skeptical, so the customer conferenced on one of our Support Engineers (by this time it was 5:30 PM). A few calls were attempted, but none failed. Telco Tech was very skeptical and wanted to close out the ticket. Our tech called me in. I verified that the Telco had a trace setup and immediately had the customer repeatedly dial a number at our office in hopes of having the problem occur again.
I explained to the Telco that there was no known way that each of the codecs would get their own audio back were it not for something in the network that was in loopback mode. He gave the usual reply that "if there were something wrong in the network we'd get thousands of complaints" and added "this would show up on our trouble logs".
I asked if they had separate trunk groups for voice and data. Reply was no…all trunks are 64 kbps capable. However, I reminded him that there might be a subset of trunks used for data. He denied this, until I reminded him that without looking at the routing tables it was impossible to tell. I persisted, and ask him to explain how these symptoms could occur. He could not. At this point, the customer had made over 20 calls and the problem had not occurred. I was beginning to think we'd need to resume the next day as apparently traffic had diminished to the point the problem was not going to occur.
Fortunately, the problem began occurring again. The Telco traced several calls and was seeing good and bad calls going over the same trunk groups (we both knew that it was unlikely a single trunk would be in loopback, more likely it would be at least 24 channels - e.g. a T1).
The Telco was becoming rather defensive by now. I suggested the customer contact another station in Phoenix to see if the problem occurred dialing from there. We did, and that location was not experiencing the problem, however we were seeing it on about 60% of calls from the original site at that point.
The Telco determined that the second customer's line was served out of a different CO from the customer having the problems. However, he insisted that they shared the same tandem trunks and tandem switch. After asking a number of questions, we learned that the customer's CO did not have any tandem trunks, rather it had interoffice trunks to another CO, that then routes calls to the tandem trunks. The configuration is shown in figure 7.
Figure 7 - Tandem access via inter-office trunk to a second CO
I told him to trace the test calls on these interoffice trunks, and the problem was narrowed down in about 4 calls to a single T1 that carried 24 members of a much larger trunk group. "Busying out" that T1 caused the problem to cease, proving that this was where the problem lay.
We learned that because the trunk group was substantially larger than the 24 channels of this T1, the statistics on "trunk group occupancy" had not revealed anything due to the small percentage of trunks involved, and hence did not show up on the Telco's trouble logs. Score 1 for persistence/common sense and 0 for the Telco surveillance systems.
• 
