The Nerd Guru has an excellent Introduction/Refresher on Troubleshooting and it happens to be an entertaining read as well (with a site named “Nerd Guru” would you expect anything less than at least one summarization of the plot of Start Trek II to make his point?).

One of the most important things to do BEFORE you actually start any real troubleshooting is to define the problem. The most common mistake I see when people are trying to resolve some issue is that they don’t take the time to articulate exactly what the issue is in the first place. Here is an example:

User: “My computer doesn’t work.”
Admin: “What did you do to it?”
User: “Nothing.”
Admin: “Liar, you must’ve done something.”
User: “Really, I didn’t. I came in to the office, sat down, and tried to use it, and it isn’t working.”
Admin: “Ok, I’ll come to your cube and take a look.”

Notice anything wrong there? The first question the Admin asked should have been “what exactly do you mean by that?”. If he had, the User would have told him that he couldn’t log into email because it didn’t recognize his password. By narrowing down the problem the Admin could have resolved it remotely, without having to actually visit the User’s computer. By not asking any questions about the exact problem, the Admin not only wasted a bunch of time, but will also be pretty angry when he gets to the User’s computer and realizes that the computer is fine and the User’s password just needs to be reset. The Admin will leave thinking the User is an idiot and the User will be thinking that the Admin is a condescending jerk.

The point is that before doing anything else, you need to take the time to identify what the problem REALLY is. A good reference that makes this point pretty well and gives you some example questions to ask is Cisco’s System Troubleshooting Methodology. This is a very useful guide even if you don’t have any Cisco gear.

If you aren’t convinced about why you should spend some time taking a quick refresher on troubleshooting, I highly recommend you take a look at The Universal Troubleshooting Process. This site makes an excellent case for why you should define a troubleshooting process and practice it regularly, and it gives you lots of tips on troubleshooting and troubleshooting methodology. Brushing up on troubleshooting will save you time, money, and effort, and will make you look smarter and harder working in the eyes of your colleagues.

Links:

Troubleshooting Techniques [The Nerd Guru]
System Troubleshooting Methodology [Cisco]
The Universal Troubleshooting Process [Troubleshooters.com]

This article is copyright OPNET Technologies, Inc., and is reprinted from the original at www.itsnotthenetwork.com with permission.

On my system, I only had a few bridged interfaces, but there were still a lot of files there:

root@my-server:/proc/vmnet# ls
bridge0 hub0.1 hub0.5 hub1.1 hub1.5 userif22 userif50 userif66
bridge1 hub0.2 hub0.6 hub1.2 hub2.0 userif23 userif51 userif73
bridge2 hub0.3 hub0.7 hub1.3 hub3.0 userif24 userif64 userif74
hub0.0 hub0.4 hub1.0 hub1.4 netif0 userif4 userif65 userif75


A little exploring and I was able to surmise that hubX mapped to what we commonly know as vmnetX, with X usually being 0,1, or 8 for a default VMware configuration (in Windows land these are usually called “VMware Network Adapter VMnetX”). I think the hubX.Y files actually show which interfaces of which running machines are connected to that hub, kind of like link lights on a real hub, but I didn’t spend enough time to really be sure. In any case, I was able to determine that since I only have bridges configured on this system, the files bridge0, bridge1, and bridge2 were the ones I was after:


root@my-server:/proc/vmnet# cat bridge0
connected hub0.0 mac 00:50:56:00:00:00 ladrf 00:00:00:00:00:00:00:00 flags IFF_RUNNING,IFF_UP,IFF_PROMISC dev eth0
root@my-server:/proc/vmnet# cat bridge1
connected hub2.0 mac 00:50:56:00:00:00 ladrf 00:00:00:00:00:00:00:00 flags IFF_RUNNING,IFF_UP,IFF_PROMISC dev eth1
root@my-server:/proc/vmnet# cat bridge2
connected hub3.0 mac 00:50:56:00:00:00 ladrf 00:00:00:00:00:00:00:00 flags IFF_RUNNING,IFF_UP,IFF_PROMISC dev vmnet1


It is pretty plain to see that bridge0 connects hub0 (aka vmnet0) to eth0, bridge1 connects hub2 (aka vmnet2) to eth1, and bridge2 connects hub3 (aka vmnet3) to vmnet1 (I’ll explain this funky mapping another time).

Hopefully someday VMware will implement a handy tool like the Windows “Manage Virtual Networks” tool on Linux hosts, but until then, this will be good enough for me to at least tell what is going on on an unknown system.

This article is copyright OPNET Technologies, Inc., and is reprinted from the original at www.itsnotthenetwork.com with permission.

So here is the deal: the stuff in the SEO article is what I would currently consider to be the “best” way to optimize your site for search engines. By “best”, I mean the most likely to get you lasting long term results, provide value to the internet and to the rest of the community, and avoid potential issues if/when Google decides to change the rules. I’ve come to this conclusion based on a moderate-to-fair understanding of how search engines work, along with reading a huge pile of other SEO articles that are all probably made up. Having good content is key, and I am confident that if you have good content, your site will win over the long term. Now with that said, I have to note that the actual technical details of getting a high ranking in Google and other search engines do not precisely align with the way I have articulated things previously. If you want to talk about the topic further, I’d be happy to entertain a discussion about it, feel free to contact me.

Oh yea, and don’t ask about statistics, especially ones in a shiny Pie Chart with rounded corners. As a reasonably intelligent human, you should consider all statistics you see anywhere to be completely made up and purely for demonstration purposes, especially if they are on the Internet. Always ask for the source if someone is quoting statistics to you, and try to keep in mind that usually, statistics and other things like black and white statements and hyperbole are solely for the purpose of making a point.

1. Put good, reasonably well written, useful content on your site.
2. Have popular sites link to you.
3. Link to popular sites.

1. Put good, reasonably well written, useful content on your site. This is by FAR the most important thing you can do to get a good search engine ranking (seriously, this probably 90% of it). If you do a google search for “printing services”, you will notice that all the top sites (that aren’t ads) have something in common on their home pages: lots of words that say meaningful stuff. And not pictures of words, actual words. I notice that your site has a lot of pictures of words, but if I look at the source there isn’t much there that describes what you do. You’ve gotta say, in actual text on the page: what you do, why you are special or different than everyone else, and why someone should continue to look at your site and/or purchasesomething from you. And all that has to be in grammatically correct sentences that make sense and aren’t hidden using special tricks with CSS or JavaScript.If you really want to one-up everyone else, you could also provide some meaningful content on your site that helps people even if they don’t buy your services. For example, if you have a site that is a print shop, create some pages that have free useful articles and tips. Some examples I can think of are:
   • things to look out for when shopping around for a print house
   • questions to ask your printer before purchasing
   • tips to save money on printing costs
   • tips to really make an impact with what you are printing
   • the truth about paper quality
   • you get the idea
2. Have popular sites link to you. This counts for about 8% of the remaining 10. While the best way would be for popular sites to decide they like you so much that they spontaneously decide to link to you, but there are some places where you can get links “for free” to help things get started. Submit the your content to digg, del.icio.us, reddit, blinklist, stumbleupon, reddit, furl, and every other “social” news/bookmark service you can find. Have all the employees at your company go home and vote up your content from there (make sure they do it from home and not the office, cause it will be obviously bogus if 50 diggs come from 1 IP address).
3. Link to popular sites. Link to other sites that have good and popular content. This accounts for about 1.5% of your rank (with the final .5% being miscellaneous other stuff that isn’t worth bothering with). Don’t get too tricky with the linking though, cause if you are obviously trying to artificially inflate your pagerank google will squash you off the face of the Internet. And don’t for a second think you are smarter than they are and won’t get caught, or that they are too big and you are too small. There are a bazillion worker bees at google and they are all smarter than you and I put together. In addition to that, they have super smart robots that have already started taking over the world.

That is about all I have to say on the subject at the moment, but here are a few links to get you started on researching it more for yourself (I’m only putting a few links here now, will add more later when I’m back at my home computer with all my bookmarks).

• 101 Tips to Improve Your Web Presence
• Steve Pavlina: How to Make Money From Your Blog
• SEOmoz: Essentials
• 21 Great SEO Tips From Google Engineer Matt Cutts
• Lifehacker: Search Engine Optimization (SEO) made easy

Completed Walk-In Faraday Cage at Interop Las Vegas 2007

I learned a lot about RF and Wi-Fi, which I’ll be reporting on in the third and final part of this article, but here I want to detail the construction of the cage.

We started with a large aluminum frame that we had the onsite construction people build. If I could have had my way, I probably would have just used two-by-fours, but because it was on the show floor and such, we had to let the pros do it, and we weren’t about to start getting picky about the materials for the frame. We also had them cut a plywood floor for us, so once we got the screen down we could set the plywood on it and not have to worry about people tearing it up.

While they were building the frame and the rest of the show floor was getting prepped, I went to the local Lowe’s Home Improvement store and went looking for the aluminum screen we had decided to use for the build. Many people have asked me why aluminum instead of something like copper or brass, and the answer is basically because it is cheaper, more available, and the reports I was able to glean off the web suggested that an additional layer of cheap stuff was going to be far more effective than a single layer of expensive stuff (although two layers of expensive stuff would be nice, but I’ll give you tips on what to build your own out of in part three of this article). I was prepared to spend two hundred dollars to buy two 48 inch by 100 foot rolls (I had calculated the surface area of the frame to be int he neighborhood of 375 sq. feet, so this should give me enough to do a double layer all around). Amazingly, Lowe’s happened to have them on clearance that week, and I got the two rolls for $25 each! What a score. In retrospect, I should have purchased four rolls.

We also purchased a rivet tool to secure the screen to itself with, because the guys on the team thought my idea to just use zip ties was bogus (and it was, so I’m pretty glad they called me on it). Along with the rivet tool and rivets, we had to purchase washers to go on each side of the rivet. Annoyingly, even though they sell the rivets in boxes of 100, the washers were all sold in boxes of 20, so we had to buy 30 boxes to go with the three boxes of rivets we purchased! [If someone from the Arrow Fastener Company ever reads this, take note: sell washers in packs that are twice as large as the rivets.]

Then we started wrapping and riveting. It was slow going, but thanks to the help of the guys on the team (and some borrowed help form the NAC team), we were able to make reasonably quick progress.

Once we got the whole cage wrapped, it was time for a smoke test. I fired up AP Grapher on my laptop and checked outside the cage to see what I could see. 389 SSIDs showed up within 20 seconds of scanning! Talk about a noisy RF environment. I zeroed in on one that I knew was in our rack about 6 feet from the cage, and stepped in to measure the signal. -10dB. Buh. That sucks.

But we didn’t let the less than stellar results get us down. We kept on riveting and sealing up the places where the screen met and would likely allow waves in, and then added a second layer on top of the first and did some more.

After many more hours, we were ready to test again. This time we were able to see about a -35 to -40 dB. Now this was at least decent. Amazingly, we could still get quite reasonable Wi-Fi signal from nearby APs, but at least we were able to filter out the vast majority of the junk on the floor.

Oh yea, remember that whole thing about grounding in the last article? Well, we had the electricians at the show bring us a nice massive ground connection from the nearest electrical box, and we hooked it to the cage with some very nicer copper wire that Wej brought. Unfortunately, when we tested it, it didn’t seem to make a bit of difference. After multiple tests with the multimeter to make sure we were getting a decent electrical connection, Jerry put forth the theory that we were simply too far from the earth for the ground to be effective, and that the impedance of our ground connection was simply too much to get the 2.4Ghz waves from where we were to the actual earth. I’ll write a little more on this topic later in Part 3. We left all the ground wires connected though, cause they looked cool.

There are a lot of different products out there that I’ve played with for monitoring purposes, but NetSensory is really amazing with the variety of things you can do with it. You can use it to do all the basic stuff you are probably used to, like see who is using your bandwidth and what protocols are on the network, etc.; but you can also use it for so much more.

• You can find and debug application performance problems. Let’s say I’ve got some database servers that are feeding my website. If they start acting up and responding slowly, I can tell with NetSensory. I can also use it to tell if the webservers are appropriately spreading the load across the database servers. Or if the webservers are responding slowly or traffic is unbalanced to them
• You can manage changes in the network. Let’s say your boss wants to spend 50 grand on some load balancers: you can get the data necessary to determine if you should really do it, or if you should just cram a few a few extra sticks of RAM into your servers and pocket the remaining 49K (or I suppose you could let the company keep the money too).
• You can monitor your WAN traffic for usage and SLAs.
• You can spot worms and port scanners easily and track them down quickly.
• You can find and identify packet loss and other problems in the network.
• You can find rogue users or applications.
• And much more.

But, there is a problem: you’ve gotta know what you are doing. You can’t just launch the thing and expect it to tell you everything about everything, you need ot know your network and you have to actually LOOK at the data.

And, up until very recently, there was another problem: you couldn’t try it without convincing a sales critter that you are serious and then getting a demo appliance to put in your network (of which there are a limited quantity of, so not everyone could get one even if they were serious). But now, that isn’t so much of a problem, because we’ve released a version of the product that runs in VMware. The VMware version fills two rolls: it functions as a trial that ANYONE can download and use with minimal hassle, and it can be permanently licensed for use in a small office environment. The trial version only collects data for 5 days, but you can look at the data indefinitely, and because it is VMware, it is easy to reset and start another 5-day trial. It is also limited to only 5000 packets per second, so it will only work in small office environments (but larger environments can still use it as a test on a limited network segment, and can upgrade to a “full” appliance if desired).

So head on over to the download page and check it out. I think that most Network admins will be able to get some real use out of even just the 5-day trial, even if you never buy it. (Which sounds like something the company would be scared of, but they realize that the more people use the trial and do useful things with it, the more they will realize that NetSensory really rocks, and will eventually buy the product.)

So if you want to give it a try, head on over to the download page and give it a whirl. I think you’ll be happy if you do.

Some extra details you might want to know:

• The purchase of a license comes with Instructor Led Training (which you’ve got a decent chance of being taught by me).
• It also comes with a year of support. Yep, support is included.
• And perhaps one of the coolest things: NetSensory has these things called Insights that are used to display customized data in specific ways. YOU can write your own Insights that are custom for your environment, or download others that community members have written and shared from www.itsnotthenetwork.com/community-insights. These things are amazingly powerful, you can do a ton of stuff with them, including integrate scripts (perl, shell, whatever you want) into them.

NetSensory Virtual Appliance Trial Download from Network Physics

777 444 222 44 2 777 3

(the spaces are there for your readability, she doesn’t actually type them). This spells out R-I-C-H-A-R-D on the screen of the phone.

But when typing a text message, it doesn’t work like this. If she tried to type the same word, “Richard”, she would enter:

777 444 222 44 (beep) 2 (beep) 77 (beep) (beep) 7 (beep) 3 (beep)

and the screen showed Sprighabag?. What? Sprighabag?? What the heck is Sprighabag?? Stupid technology. (Actually, after the first beep, she probably got a little worried and may not have even continued typing. If she did, she certainly didn’t get past the two consecutive beeps, and likely put the phone down convinced that it was out to get her.)

The problem is the phone is trying to be helpful, and it ends up outsmarting itself (or her, I’m still not sure which). When texting, it goes into a special mode that uses “predictive text technology”, where it tries to make things simpler by lessening the number of key presses necessary to spell a word (sounds fancy, huh?). Instead of behaving the way that it used to, it now expects only one key-press per letter, and it guesses the word intended based on the fact that there are only a certain number of possibilities for each sequence.

Lets look a very simple example, the word “hi”:

The “old” way:

44 (wait till cursor starts blinking again) 444

The “new” way, with predictive text technology:

44

That’s it. Since the number 4 only has three possible letters: G, H, and I; there aren’t a lot of possible combinations that are actually words. Here are all the possibilities: GG, GH, GI, HG, HG, HI, IG, IH, and II. The phone has been programmed to know that none of those other choices make sense, so it assumes you want the word “HI”. Easy, huh?

Let’s play around with it a little more, this time with a more complicated message. Get your phone and type this into a new text message (on my mother’s phone, a Nokia 6061, you get to the new text message screen by pressing up on the four way navigation button–your phone might be different). Seriously, go do this.

(Don’t type the spaces, they are just there for readability here):

8447 0 47 0 2 0 8378 1

What did that spell? Neat, huh? Go ahead, take a minute to type some other stuff. Notice that pressing 1 gives you a period? I’ll talk more about that a little later. Fun now that you understand how it works, isn’t it?

Now let’s look at a slightly more complicated example. What if you want to type the word chef? Type the numbers 2433, which would spell chef, but also spells other valid words, like aged and aide. The phone first gives you Aged, but you can press * to have it display another possibility. In this case you need to press * twice before it realized that you want the word chef.

2433 * *
gives you:
Aged -> Aide -> Chef

Remember how earlier we learned that the 1 key will enter a period? Well, you can use it in conjunction with the * key to get other punctuation as well. Pressing it will cycle you through the -, ?, and more.

Now what if you want a word or abbreviation (or name) that it simply doesn’t recognize? For example, you boss sends you a text with the question “When can I expect to receive those TPS reports? Don’t forget to use the new cover sheet!”

You want to respond with “I’ll have them for you eod” (Eod is an abbreviation for “End of Day”).

So you type:

455 0 4283 0 8436 0 367 0 968 0 363

But 363 gives you end, which you don’t want, cause you just want the abbreviation “eod”, not to type the whole thing. So you press * to cycle through the possibilities.

363 * * * * * * * *
which cycles through the possibilities:
end -> foe -> doe -> ene -> eme -> enf -> dod -> fod -> end

Uh-oh, no eod. Now what? Well, on most phones at this point there is an option on the screen for “Spell” (in my mother’s case it is activated by pressing the middle key in the navigation pad, but your phone might be different). So you press “Spell”, and it brings you to a single text box that behaves the “old” way: you press each number several times to specify a particular letter. So now you can type

33 666 3

and then hit “Save”. Viola! eod

What if you just want to go back to the old way for all of your text messages? Or what if you want to type a word in all CAPS? Can you do that? Yes. By pressing the pound key (#, aka hash mark, aka number sign, aka cross hatch, aka octothorpe, aka comment symbol, etc) you can change the typing mode.

There are several possibilities, which you can scroll through by pressing the pound key repeatedly, and identify by looking for the little [image] symbol on your phone (it may be at the top or bottom of the screen, depends on the phone):

Predictive Text, Capitalize the first letter of each new sentence (the phone knows that each word after a period or question mark is the beginning of a new sentence)

Predictive Text, no capitalization

Predictive Text, All CAPS

No predictive text, Capitalize the first letter of each new sentence

No predictive text, no capitalization

No predictive text, all CAPS

Number mode (allows only numbers to be entered)

There are a few more tips to cover, but you should have the hang of it by now, and the “advanced” stuff changes a bit more depending on the type of phone you have. I’ll write more about them at another time.

Happy Texting!

Learning Linux Commands [Codejacked]

This article is copyright OPNET Technologies, Inc., and is reprinted from the original at www.itsnotthenetwork.com with permission.

[Update July, 2007: I know, I know, I've been terribly delinquent about posting the details from the show. I will soon, I promise. There will be two more parts: one with pictures and notes on construction, and one with lessons learned and other info. Thanks for being patient, check back soon for the next installments of the faraday cage series.]

This year at the Interop Labs we are building a walk-in Faraday cage as part of the VoIP over Wi-Fi demonstration. Since there are usually anywhere from 250-400 wireless access points “viewable” from the show floor, we need some way to remove all the junk in the RF spectrum in order to provide a controlled environment to test in and to effectively demonstrate the differences between VoIP Wi-Fi connections with and without QoS controls and burdened and unburdened Wi-Fi networks.

Anyway, since this walk-in size Faraday cage project was my idea, somehow I got saddled with making it happen. So the first thing to do was to build a proof-of-concept before ordering the necessary materials. On the last day of hotstage (about a month before the show we get together for just over a week to do the planning and build out of the Interop Labs booth, called hotstage) I went to the hardware store, got some aluminum mesh (the kind you put on a screen door or window), some 1/4 inch hardware cloth (something you might build a rabbit cage out of) and folded them up into a couple of box shaped containers to put something in that could measure radio waves. Then I fired up Kismet on my laptop and put it in each one. It was a spectacular success. Moments after placing my laptop inside the aluminum screen “box”, the signal went away. Unfortunately, after running several successive tests, it appeared that the signal went away about 90 seconds after launching Kismet, no matter what the circumstances (I could be standing next to the access point and it didn’t seem to make a difference.

So I went on the hunt to find some other Wi-Fi measurement tools to run on my Macbook Pro. Of course the obvious choice is Macstumbler, but it hasn’t been updated in a long time and crashes at launch. Then I found iStumbler and APGrapher, and both of those seemed to work well. So I fired them up, put my laptop into the proof-of-concept device, and…the signal didn’t change. Buh.

Ok, back to the drawing board. A little more research and a refresher in high-school physics reminded me that a faraday cage is basically a big “ground” wire extended into a 3D space that you could put stuff in. Doh! I didn’t ground any of my proof-of-concept tests (well, actually, I did hold it up against a metal wall socket at the warehouse that should have been grounded, but that didn’t seem to make a bit of difference). So I set out to build a new proof-of-concept, and this time I would be more meticulous about its construction and I would make sure it was well grounded.

I decided that this time, I’d build a frame, attach the mesh to it, and secure a copper wire to the mesh that I would then connect to the ground plug (third hole) of a standard wall socket. I choose to build a box roughly 18 inches X 12 inches X 12 inches, so I could put my laptop inside of it without much trouble.

Step one: buy the necessary components. Here is the list (pics down below in the assembly instructions):

1. Two 8′ lengths of 2″ X 2″ wood (I choose spruce “furring” strips, cause they were $1.99 at the local hardware store). This will be the main frame.
2. One 8′ length of 2″ X 1″ wood (again, spruce furring strips, even cheaper at $1.79). This will be frame for the hinged lid.
3. Hinge to attach lid to frame. I choose a 12″ piano hinge.
4. Aluminum screen to cover. I made some rough calculations and bought 10′ of 36″ wide mesh.
5. Copper wire for the group connection. I choose green 14 AWG stranded, cause it was reasonably cheap.
6. A bolt, some wing nuts, and washers to secure the copper wire to the mesh.
7. A plug the I could attach the copper wire to and insert into a socket.
8. Screws and staples to secure the frame together and the mesh to the wood (I simply used what I had laying around in my garage).

Step two: Cut the wood. I cut the 2″x2″ furring strips into four 18″ lengths and eight 12″ lengths. And the 1″x2″ strip into two 18″ lengths and two 12″ lengths.

Step three: Screw the frame together. This isn’t rocket science, so I’m not gonna give you detailed instructions on this part. Just remember to drill pilot holes for your screws so you don’t split the wood. Here is the assembled frame:

Step four: Build and attach the lid. I simply screwed the 1″x2″ pieces I had into a square and put them on the frame box with a piano hinge:

Step five: Attach the screen mesh. I will never regret the day I bought my air compressor and pneumatic stapler. I can only imagine the terrible hand cramps had I attempted this with a standard spring loaded stapler. If you don’t have a pneumatic stapler, I highly recommend you at least look into purchasing an electric one. To make sure that everything was tight, I tried to keep staples no more than 2 inches apart (usually about 1 inch, but towards the end I got a little tired and lazy, so the space between increased a littler). Here is the before and after:

Phoenix the cat seems curious about the frame and screen.

Step six: Attach the ground wire. Below is a shot of the parts mentioned above, along with Phoenix, one of our cats who was very interested in this project. Attaching the parts was pretty simple.

You will notice that I cut off the two prongs that get electricity. I wanted it to be perfectly clear that this plug is non-functional, and won’t create an electrified cage (cause that is a completely different project!). [NOTE: This is dangerous! Don't do it. If there are some cables crossed or your ground is wired incorrectly, you could create a lethal situation and die. Get an electrician to get you a dedicated ground wire to use for a project like this. If you kill or seriously injure yourself, don't blame me, I told you not to do this. Seriously, I'm not joking. Don't plug stuff into an outlet that connects to bare metal that you will be touching. That is just stupid.]

Step seven: Time to test! I plugged in the ground, got my laptop and started measuring Wi-Fi signals. The pictures speak for themselves:

DOH! It still has signal. WTF?!? Well, as it turns out, the ground connection in my house is quite bad. So I brought it into the office and tried it there:

Woohoo! Just to make sure though, I opened the door, let it reacquire signal, then closed the door again:

Success!!!

Check back after the show (May 20-25) for Part 2, where I will detail building the walk in cage.