A quick blurb on NSX ECMP configurations:

If you’re running ECMP, be sure to disable the Reverse Path Forwarding check on your interfaces, or at least set it to “loose.” NSX defaults to strict reverse path forwarding mode when RPF is enabled.

I learned this the hard way during a recent NSX deployment. Thankfully it was in a lab environment, as we wanted to ensure we had all of our ducks in a row before pushing this thing to production.


CCIE Study: 4 Months Deep

I’ve been informed that three months is when the wear-marks begin to show when doing a long period of study; and if that’s true I’d like to claim that four months is the start of the second-wind. I’ve been busier than ever this month yet I’ve made serious headway.

As none some of you may know, my wife and I are working through the process of adopting a child. This month the adoption agency issued us homework, consisting of a workbook, an online course, several videos, and two books. Having to read two books while studying for your CCIE is absurd, but it got done.

When I started studying for the CCIE, it had been a few months since my last vacation. I was able to get through the first few months, but all the added hours were starting to take their toll. It was unsustainable. A three-day-weekend pass to Seattle was a quick remedy. One must learn to rest instead of quitting.

So where am I now?

After three months of restarts, broken labs, and disappointments, I was getting pretty burnt out on re-learning the same topics over-and-over; but the fourth month has brought new life into the game. I’ve been able to move on to a topic that’s fresh to me: OSPF! I’ve never had the privilege of running an OSPF network (in production; I have labbed OSPF as much as I could). I’ve always wanted to — so much so that I even moved jobs so that I could participate in a migration to OSPF from EIGRP (which I’m in the middle of). Starting a new(-to-me) subject has really made all the difference, as I now have far more desire to study. I’m hoping to put together some OSPF posts in the future.

Speaking of OSPF (and the job with an OSPF network): We’re in the middle of an OSPF migration for VMware NSX implementation. There were some initial foibles of working with NSX (and doing so through another team), but we’ve gotten it up and running, and stable in a lab environment. I’m planning to put together a post on this as well.

What’s Next?

I’ve barely touched the surface of OSPF, and I’ve got a ton of reading and labbing to do on this one. I need to make up for a wealth of inexperience. Hoping to plow through “OSPF: Anatomy of a Routing Protocol,” knock out some labs, and view a lot of packet captures. Also anticipating a quick re-read of some of the EIGRP DUAL FSM states, path recalculation, and EIGRP OTP. Finally, I want to do a “speed run” on the INE ATC labs up to where I am now.


Mnemonic for EIGRP Flags

Just a short post; but this is how I remember the flags in the EIGRP (RTP) header:

Thanks, Wireshark 🙂

I use the mnemonic “I Can Remember EIGRP-flags.” (yes, it’s not perfect. Sue me).

Then you just need to remember that the byte used for the flags are just like binary bits (because they are binary bits). Each one is just the next largest binary number. (i.e. I=1, C=2, R=4, E=8).

CCIE Study: 3 Months Deep

Month three of my CCIE training has not been an easy one. I started off with a cold that was making things pretty foggy, while I was trying to learn several topics much further in-depth. On top of this, I was being beaten to death by a heavy load of flash cards, as I had increased my “new cards” load from 20 to 30. As I stated in my 2 Month update, I had a lot of catching up to do.

After the restarts, lab-rebuilds, and relearning I had been feeling pretty defeated. There’s something about running into failures right away that leaves you feeling pretty demotivated. These demotivating feelings can lead to distraction, and negative attitudes that only make studying even more difficult. These feelings stack up and can eventually make one ready to give up.

Thankfully, I had the guys over at RouterGods to talk me through this, as well as a few of my mentors and friends, and of course my awesome wife. There was also a really good video put out by one of the RouterGods members, as well as a blog post. My bible-study group was also very helpful.

On the subject of subjects being learned, I spent most of the month covering PPP, and PPPoE, CEF, basic IP routing, PBR, and RIP. I also managed to get caught up on flashcards, and even made some good breakthroughs on concepts that I didn’t quite understand.

The Big Takeaway

Don’t give up! In the Marines, I learned that you can’t just give up when life is rough. There’s a reason that a CCIE is considered an “expert.” Being an expert isn’t easy; If it was, there’d be a lot less morons in the world. Becoming an expert takes time, effort, and discipline. That’s why they say “The master has failed more times than the beginner has even tried.”

Moving Forward

So what’s on the agenda for next month? For my fourth month of self-abuse studying, I’ll be working on EIGRP and hopefully OSPF. As much as I’d like to do more, I think that those two topics are pretty deep and I’ll be lucky to get both of them in the month. I would even contend that while I have pretty deep experience with EIGRP, my OSPF knowledge is thin enough to ensure that learning won’t come quickly.

Additional Resources

PPP: How I Loathe Thee

PPP (Point to Point Protocol) is one of those protocols that I’m absolutely irritated about having to learn. I’ve never had to use or configure PPP in my life, and I had to ask around a couple of study groups to find someone who had. Prior to asking anyone, I thought PPP (and it’s bastard child, PPPoE) were outdated protocols, used only by aboriginal tribes who still had AOL discs. That being said I was wrong; I was informed that it is very much in use today for places that use T1/E1 for some of their more rural sites, and ISPs use it regularly (well, PPPoE) for connecting users to DSL.

The more unfortunate part of learning this technology is that the documentation isn’t very well organized. At best there are some books that cover it (namely The TCP/IP Illustrated v1, 2nd Ed.) and some Cisco documentation… but you don’t really get a good idea of what you need to configure with it. While I’m one CCIE certification away from being able to tell you what you need to configure with it, I can at least help you figure out the basics.

So What is PPP, anyway?

PPP is a media-independent encapsulation protocol. But why make yet another encapsulation protocol? There was already HDLC and Cisco’s proprietary HDLC, as well as SLIP, To answer the question of what is PPP, we need to answer a more fundamental question:

What problem does PPP solve that other protocols don’t?

A few simple answers:

  • HDLC is an ISO standard, but every vendor put their own spin on it.
    • These “spins” may as well be considered proprietary, as they don’t interoperate.
  • SLIP was lightweight and interoperable, but it doesn’t have many features
  • PPP has many features:
    • Open Standard (yay, interoperability!)
    • Authentication
    • Error Correction (Link Quality Management)
    • Load Balancing
    • Efficiency features (LFI, compression)

These aren’t the only features that PPP has over it’s competitors, but they are some of the more important ones. PPP is mostly used for it’s vendor interoperability, load-balancing, and authentication features.

What do you mean, poorly documented?

For starters, to find any usable Cisco configuration guides on PPP, you have to scavenge through the IOS 12.2(SR) code. Only there will you find the stuff that gives you a nice, from-the-ground-up explanation of PPP. Aside from that, there’s the “Network Technologies” page for PPP, with its various sub-pages and their sub-pages; these are plagued with broken links and documentation that is dense for those of us who are just getting started on it.

Before I found the page in the IOS 12.2(SR) code, I found a couple good pages on Wikipedia, and the always wonderful TCP/IP Illustrated v1, 2nd Ed.

What do I need to know about PPP?

There are some core things that are important about this. The first of which is that PPP uses a different frame format than Cisco’s HDLC.

At first glance, the cHDLC packet looks very similar to the PPP packet — and that’s actually because the PPP packet is largely based on HDLC. The difference really boils down the standardized “Protocol” field belonging to the PPP header (as opposed to cHDLC’s proprietary “Type” header), and the fact that PPP pads its payload with extra bits.

Another important thing here is that there are different phases that PPP uses when building it’s connection. You should probably make sure to remember them:

  1. Link Establishment
  2. Authentication (optional)
  3. Network Layer Protocol

Each one of these phases has a different protocol to handle it. The Link Establishment phase uses the Link Control Protocol to do a few things; namely, verifying that both devices speak PPP, and that they agree (more or less) on the parameters of the connection. The devices negotiate Authentication Type, whether or not they’ll use multiple physical connections (multilink PPP), if they’ll monitor the quality of the connection, and the MTU size for the link (called MRU). These are negotiated with a series of messages that you’ll of course need to memorize:

  • Configuration Request (CONFREQ): A list of proposed parameters for link establishment
  • Configuration Acknowledge (CONFACK): A copy of the CONFREQ message, with duplicate information, agreeing to the parameters.
  • Configuration Negative Acknowledge (CONFNAK): Contains the unacceptable parameters listed in the CONFREQ, but with acceptable options listed with it.
  • Configuration Reject (CONFREJ): Contains unacceptable or unrecognizable options. This doesn’t necessarily mean the connection will fail, but the rejected parameter/config will not be enabled.

Once the link is negotiated, authentication comes into play. The two most popular types of authentication are PAP (Password Authentication Protocol) and CHAP (Challenge-Handshake Authentication Protocol). Those are important too, and you’ll use them even more when you get to configuring PPPoE (so remember them too).

After the Authentication phase, the Network Layer Protocol phase begins. In this aptly-named phase, the devices must negotiate for each upperlayer protocol they’re going to use. I say upper-layer instead of layer 3 because CDP must also be negotiated, and it isn’t a layer 3 protocol.

Are you going to show me how to configure any of this stuff?

No. Mostly because I’m in the middle of learning it myself, but also because there are MANY use cases for this that one blog article can’t cover. This one has gone on long enough.

Additional Resources/References:

CCIE Study: 2 Months Deep

In my first month of studying I blew through several subjects worth of Videos-on-Demand, labs, and note-taking; I then promptly forgot most of it. This was due to not keeping up with that information, and not reading enough. I would learn a new subject only to forget the details of the old subject. The CCIE exam is not forgiving, and forgotten knowledge has no place in that world.

I realized the problem once I downloaded Jedediah Casey’s excellent Anki flashcard deck. I took the cards for subjects I thought I knew and added them to my daily study routine. The first attempt on the cards was enough for me to realize that I hadn’t read deeply enough, nor had I retained what I did know well enough. Prior to that I thought I had conquered topics at a CCIE-level, only to find that I barely knew what I was doing. The cards I had created prior to using this deck were a joke.

So for the last month, I re-studied Spanning Tree, Ethernet, and Layer 2. I read almost every article in the Cisco documentation on the subjects. I read blogs on them, memorized flash cards, and made more extensive use of the debug command. I made my own labs. I even spent time improving my lab so that debug messages weren’t such a pain to use.

The Big Takeaway

Don’t get discouraged; the learning process is more complicated than most of us think it is.

Actually plan what you’re going to study. It’s important to identify what you need to study, and make it a priority. Nobody becomes a doctor without a structured study plan, and I’m sure nobody becomes a CCIE that way either.

DO NOT just randomly blaze through topics. Brian McGahan, the instructor at INE lists a method for studying that goes as follows:

  1. Gain a high-level knowledge
  2. Basic hands-on experience
  3. Gain expert-level knowledge
  4. Expert-level hands-on experience

I’m sure I’m paraphrasing that a little bit, but the fundamental idea is there — At some point or another, you need to increase the depth of your studies. I made the mistake of not reading all of the documentation on a subject, or not reading the configuration guide or the command references. Once you’ve done the basic hands-on experience you should leave no stone unturned in your pursuit of routing knowledge.

The effectiveness of flashcards can make or break your studying. My flashcards were trash. Make flashcards that will challenge you. These are what’s going to make you retain the information; anything that doesn’t challenge you here is going to let your knowledge atrophy.

Moving Forward

I’m coming to the end of the catching-up phase of using the flashcards, but I suspect I’ll be totally done with re-learning information by the end of the month. This month will be consumed with re-learning the rest of the layer 2 technologies (PVLAN, CDP/LLDP, with a special focus on PPP/HDLC) followed by re-learning RIP and EIGRP if I can get that far.

Back to flashcards, in order to get “up-to-snuff” on retaining information, I had to increase the daily limit on “new” flashcards to 30, which has lead to some truly brutal days of studying them.

Additional Resources:

Study Pro-Tip:Console to Physical hosts with your ESXi Server

For the first few lab sessions I used a layer-3 port on my 3750G switches for console access. The plan worked well for the most part, though it had minor issues: debug messages typically don’t print to the vty lines, and if you were to make a mistake in your lab you could lose access to it until you made a physical change.

To be honest, I don’t have a lot of spare cash to buy spare equipment. That being said, the equipment I have access to has enough horsepower to get away with running an extra device on it. So I decided that I should take a minute away from studying to improve the quality of my lab.

What I Did

The basic idea is simple:

  • Plug your USB-to-Serial Adapters into your ESXi Host.
  • Create a Linux VM that will become your reverse-telnet server.
  • Configure “DirectPath I/O” (allow your VM to use the physical hardware).
  • Attach the devices to your VM
  • Set up your reverse telnet server (and allow via firewall)

I’ll walk you through each of the steps here below.

Create a Linux VM

Download your favorite Linux distribution and install it as a VM on your host. For any sort of server use I always recommend CentOS/RedHat or Debian (if you want fancy new packages Ubuntu also works in place of Debian). Once you have it installed, go ahead and plug your USB-to-Serial cables into your host.

Configure VMware Direct I/O

This part took some figuring out, but it really shouldn’t have. All you need to do is go to the “Configuration” tab under you ESXi host, and click on “Advanced settings.” There isn’t much on this screen, which makes it easy to look past it, but you’ll see an area that looks like so:

You’re going to want to click on the link that says “Configure Passthrough.” This will lead to another box that will let you select which devices you want to connect to your VMs. Typically you’ll choose whichever device looks like a USB controller.

Once this is complete you’ll need to reboot your host machine. Once that’s complete we can proceed to our next step.

Attach the devices to your VM

When the host comes back online you need to modify the hardware settings of your virtual machine. Make sure your virtual machine is powered off, then select your virtual machine and go to “Edit virtual machine settings.” Click “Add.” Choose the “PCI Device” option. You’ll be presented with the option to choose your PCI device. Choose the USB controller. You’ll need to do it for each USB controller you want the device to have access to.

Once that’s finished, you can boot up your virtual machine. Now all we have to do is…

Set up the reverse-telnet server

Log into your server and open up a terminal. The first thing we need to do is install ser2net. You can do this on CentOS like such:

sudo yum install ser2net

After installation, we need to see where our serial connections are mounted:

dmesg | grep tty

The output should be something along the lines of “/dev/ttyUSB0.” remember what they are because we’re about to modify a file that will use these values. Using your favorite text-editor, modify the /etc/ser2net.conf file. I used VIM, but you can use whatever floats your boat:

sudo vim /etc/ser2net.conf

We’re going to add the following lines to it. You’re should modify this file to meet the security needs of your environment.

ipv4,5001:telnet:0:/dev/ttyUSB0:9600 8DATABITS NONE 1STOPBIT 
ipv4,5002:telnet:0:/dev/ttyUSB1:9600 8DATABITS NONE 1STOPBIT 

Finally we’re going to start the service, enable it to run at boot-time, and configure our firewall rules. Keep in mind that this assumes you’re using the firewalld as well as systemd. If you’re not, you’ll have to do some research into what firewall you’re using.

sudo systemctl enable ser2net --now
sudo firewall-cmd --zone public --add-port 5001/tcp
sudo firewall-cmd --zone public --add-port 5002/tcp
sudo firewall-cmd --runtime-to-permanent

You should now be able to telnet to your device on the specified ports (tcp/5001-5002) and get console access to your devices.

Additional Resources: