Roborooter.com

As you may know I work in a Network Operation Center or as my business card says "THE NOC". What is life like in the NOC?

My team is responsible for monitoring a large number of hardware devices (Are they working? Did something break? Is it fixed?, servers (Are they responding? How's their storage? Is something trying to kill it?) applications (Is it running? Why did it crash? Can it start up again?), and business processes (Did these jobs fail? Will they succeed before someone needs them? Can they be rerun?) It's a broad set of responsibilities and it involves working with many different teams.

One of the more interesting parts of my job is monitoring a new device or application. It usually goes like this. "We've just purchased something new and we want it to send it's alerts to you. Can you make that happen?" Why of course! Someone from that team will ask a few questions, turn on "SNMP" on the device, point the "traps" at our servers and walk away. My team will see these "alarms" and panic. It usually look like a cross between mayhem and spam. Neither are good and frankly I don't want either of them on my watch. There is sometimes some yelling. "Hey knock that off!" "No! This is important!" And then we sit down and work things out. Let me give you some background first.

SNMP for the purposes of this story is a simple way for things to send us messages. These messages when sent to us are called "traps". Things send us messages about anything they want to. Maybe they're hungry. Maybe they just turned on. More often then not they speak up when something goes wrong and it's usually along the lines of "Help I've lost power!" or "I HAD AN ERROR!" or "My eighth port on card 23 had too many collisions in the last 30 seconds!".

My systems listen to these messages and decides what's important and creates Alerts for my team to respond to. These are along the lines of "Pardon me sir, A switch named core-12 in New York is seeing too many errors on the following ports. Here are the procedures for this type of thing and here are the tickets from when they happened last week. Do have a nice day." I've trained my systems to be very polite.

When a device is set to send us all their messages, my system does it's best to guess what's important and ignores the rest. It's harsh but necessary. There are some standard messages that most devices know. (some useful some not) but there is such a plethora of useless chatter that it would be very troublesome to pay attention to all of it. That stuff gets recorded but not alerted. "Why yes that's very interesting that you just did that thing. I can tell it's very important to you, I'll write that down and put it over here for later."

Often the devices don't know how to tell you what's wrong, or tell when anything happens but not what happened. "I need help." or "Hey, I saw something!" Neither are helpful. Helpful devices will be very specific and tell you when problems go away. "I lost something." followed by "Hey I found it again!" is a very nice occurrence and my system knows it can resolve that Alert and we can all go on with life.

Today I found a device that didn't want to bother me. If it had an issue it would tell you, but only the first time. "I lost power! but.. I lost power a few weeks ago, I shouldn't bother you with that again. I'll keep it to myself." I don't understand why someone would make a device like that.

I wasn't honest before. Applications can talk SNMP too. It's not just for devices. Most of the applications that talk to my system do it poorly and I know who made them do it. They sit on the other side of my office and while I'm not a mean person, I do judge their work quite harshly. They're notorious for spam. You see, if you can tell a computer to send a message once, you can easily tell it to send a message a million times. Or even better, you can tell it to send you a message every time it thinks something. A ton of messages from an application that deals with city dog registration isn't helpful if all it tells you is "I thought about dogs." every time a dog comes up in conversation. In fact that's a bit rude. I'm often asked to ignore things "for now". Could you ignore that for very long?

To be fair application programmers don't want to be rude. Their application was made to do a task not alert about it. They usually do the task very well and that is most important. But when things go wrong (and they will, and soon) how they tell my team suddenly matters and is the difference between the right reaction and an very slow wrong reaction. Imagine if the dog registration program was handed a cat and decided it couldn't work with any more dogs until someone took this cat away. Does it even know what a cat is? Can it say "I have a cat and I'm stumped what to do with it. Its not very dog like at all." or will it just say "I can't think about this dog!". One message will make sense to me and I'll be able to help, the other will not and I wouldn't be able to guess what it meant.

I hold that it is important to talk to your NOC before sending us any new messages. But it's a step that's often forgotten.

I've barely scratched the surface of what life in the NOC can be like, But I should stop myself before I go on for too long. I hope this gives you a greater understanding some of what we do. I also hope it helps garner a greater respect for your local NOC workers in your neighborhood or city.

Good umbrellas are rare. Today I was coveting the BLUNT MINI. An umbrella that will probably never break and satisfies my inner engineer. It's designed to have a low wind profile, mitigate the risk of poking people in the eye, and distribute stress across it's structure. It doesn't look half bad either.

The problem with nice umbrellas are they cost a lot. The BLUNT Mini is $80 and I'll probably leave it somewhere before it breaks. I'd have to go though more then 20 of the $4 umbrellas you can buy on every corner in Manhattan when it rains. I'd rather not have tons of crappy umbrellas traveling around the world only to end up buried in a long island landfill but I wonder if the manufacturing costs (in money and environmental impact) of the Blunt outweigh a couple of them making that trip.

This is neither here nor there, I have a better use for the $80. I also don't mind getting wet.

I was listening to some dam fine journalism the other day.

Ira Glass of This American Life spent a few weeks in Glynn County, Georgia investigating a "drug court" that seems to have the power to legally imprison people without oversight, for indefinite amounts of time and relatively small charges. In the show's two acts he touched on two points I found very pertinent.

1. The judge behind the injustices, Amanda Williams, has enough power as not to be challenged by anyone in her county.
2. What she is doing is legal and the victims have no recourse. They can't appeal once they enter the Drug Court program.

There will always be people who overstep their power, make mistakes, make bad judgments, etc. It's why we have checks and balances. No one can rule absolutely. What makes me angry is that there is no checks on Judge Amanda Williams. She has overstepped her bounds (on the record) on multiple occasions, and it takes an out of state reporter to come in and show what's happening.

Since this is the kind of journalism and programming I want to encourage I've donated $10 to the "This American Life" show. If this is the kind of thing you want to encourage too, please donate at least a dollar, either via paypal or by buying the episode on the iTunes Store.

Listen to the episode or read the transcript (pdf).

Next up;

This Developers Life did an interview with Remon Zakaria, an Egyptian programmer who runs a successful business working on projects with American companies. Remon was an active participant in the recent revolution, against his family's wishes and with a company to run.

His turning point was when they shut off the internet and phones. He discovered that most phones still had data connections; they were limited to the networks they were on. To keep his team busy he had them develop a Twitter-like program that could run on every phone they could find (Nokia, Android, and weird java implementations on feature phones), but instead of requiring the internet they would scan the phone's subnet for other phones. They would connect with them, transferring messages and information about other phones. This led to a localized Twitter that could connect people in an area.

He talks about why older people trusted the TV more then any social media. That led to a great disagreement in what was actually going on in the streets and if his family should be involved. He also covers how life has changed since the revolution ended. It's worth a listen.

Herd these little guys on my way home the other day. No idea what kind of birds they are but they sure are loud. Good thing they're cute. They hung around a while and flew off.

I saw them visit the next day too.

(tron is brought to you by daft punk)