BGP, OSPF, Spans, and Bounces – Working for a Service Provider

Disclaimers

First, I would like to say that in general, I like my job. Second, it should be mentioned that I am in a group that is often looked down upon. Third, I am good at my job.

It should also be mentioned that anything contained in this post is my own personal opinion and does not in any way reflect the views of the company I work for, my department, my bosses, my peers, or any other organizational unit or entity within my company.

Now that the disclaimer is out of the way…

Background

Let’s get something straight. I consider my job to be vital to the continued operations and profitability of the company I work for. I do not resolve issues on my own, but instead escalate them to the support group responsible for a particular piece of equipment. This is usually DNOC IP or DNOC ATM, although sometimes it can include our server group or applications support.

I watch Netcool. For those of you not familiar, it basically collects SNMP traps from all of the network-facing equipment in our company. By network-facing, I mean network backbone and service equipment. Customer equipment and building equipment – such as a Cisco 2960 switch – is not included. Netcool tells me when something bad happens. This could be as simple as a single IMA T1 going down or as bad as an entire market losing connectivity to the rest of the network.

I’ll say it again: I don’t resolve issues. However, that being said, I can tell you why an iBGP peer dropped and then re-established. And I can do it pretty quickly. I can generally tell you why anything that I receive an alarm for happens. I’ve been doing this for less than a year now, but I feel very confident in my abilities to narrow something down. This is where the problem comes in.

The Problem

If a routing protocol bounces for a few seconds, it is generally held for 24 hours. If it stays clear, the ticket is closed.

This is simple. Why must I send the ticket to another group for something so simple? I think I’m capable of periodically checking on an interface to see if it has remained stable. If we monitored these ourselves and escalated if necessary, we could save time for the support groups. And saving the time of people that get paid more than we do means saving the company money.

When we receive traps, I can easily find a root or common cause. Not always, but often.

I hold a NOC position and title. Why, then, does it feel like I am not a member of the NOC?

The Solution

We need to raise the standard for our group. We need to be more consistent as a group and provide better information as a group. We need to increase our knowledge as a group. Everyone needs to be on the same page.

If we can accomplish those things, perception of the group should become a little more positive. Over time, it should get to the point of trust. At that time, we should be allowed to be integrated more fully into the NOC and better fulfill our entry-level position expectations.

Summary

Working for a service provider is not all that it’s cracked up to be. If I did not have such an excellent boss, I would have jumped ship longer ago in search of greener pastures. No place is perfect, but not troubleshooting is a nightmare. Do not apply for a service provider position expecting sunshine and rainbows. You may find that it is either more than you bargained for or nowhere near as challenging as what you expected. I’m not sure if there is a happy balance anywhere in there. Unless, maybe, you want to be customer-facing.

Advertisements
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: