Cisco MSE password recovery

If you are because you've lost the root password or/and GRUB password of your MSE “Cisco 3355 Mobility Services Engine”, please refer to the following Cisco doc : Cisco 3355 Mobility Services Engine Getting Started Guide (Chapter 2 section 40 “Recovering a Lost Root Password”).

Basically if you lost the root password, you’re fine and can follow the instruction recommended by Cisco.(described in the section two of this article)
The root password is your “Admin” password for the operating system running on your MSE Appliance.

As you will find out while trying to follow the steps of the Cisco documentation, you will need the “GRUB” password to access the options of the boot menu.
If you are not familiar with Linux, GRUB is permitting you to define what OS you want to boot and how you want it to be booted (there are a lot to say about how it works) I’ll keep it to the point of what we try to accomplish, but basically GRUB was initially implemented in your MSE with a password protection to prevent people to simply boot and choose different options including the one that will allow to change the root password of your MSE.
If you want to recover you root password and you know your GRUB password then go straight to section 2.

What usually happens… You’ve lost your MSE root password AND you don’t remember the GRUB password.
Cisco’s answer to this is straightforward : If you forget the GRUB password, you cannot login and you will need to contact TAC to arrange for an RMA.

In my case I knew there was a way around it, but you have to be careful because modifying your bootloader with linux can totally go wrong !
Please note that I DO NOT recommend anyone who has smartnet or any type of contract that would guarantee the support for their device, but if you’re out of contract and no more support and you’re one step away to send this device for recycle, you might want to consider this trick as an option.
In my personal experience with Linux I was confident enough to proceed with this hack rather than fighting with the logistic aspect of RMA.

First, you’ll need to burn an image of a live-DVD version of the following GNU/linux flavour (KNOPPIX).

Let me apologize for the length of this article. Many of you will find it way too detailed with “useless information”, but I think it’s important for people with less GNU/Linux skills to be able to have a brief idea about what are the linux commands doing to their system.

The Tools you will need:
- Knoppix Live-DVD- Keyboard
- Monitor with a VGA connector
- Power bar
- Laptop + power cord
- USB-Serial adapter DB-9
- Most common DB-9 console cable terminated with female connectors on both ends (not the Cisco blue cable)


You’ll need to have physical access to your MSE, bring your keyboard + Monitor to display via VGA port. Both keyboard and VGA ports would be situated at the back of the MSE appliance.

My laptop was on the console port of the MSE appliance. If you connect your laptop via serial only you won’t get everything that would go through the external monitor/keyboard and depending of the settings and boot stage you may only see and access the GRUB boot menu via your laptop and not the external keyboard/monitor.

Section 1 (Lost your GRUB password)

Insert your DVD in the MSE appliance and power-up/reboot the system.
Looking at the monitor display, wait for the Cisco splash page

You will only have few seconds to use the keyboard : hit <F12> Select Boot Device
Choose CD/DVD and be patient as the system will proceed with your request.

Two minutes later you should see the boot menu of your KNOPPIX LIVE-DVD

Type failsafe linux single












You will only have few seconds to type text before it time-out and proceed with default booting which is certainly not what you want (if you miss the opportunity you will have to reboot the server using the power button of the front panel of your MSE appliance).

Once booted it should look like this and you are now in a GNU/Linux shell.

Mounting the partition:

You need to mount the partition of your MSE appliance hard drive  on which the grub configuration file is stored. In our case the Linux Grub configuration file location is on the boot partition: /boot/grub/grub.conf

If you need to find your partition name use the fdisk -l command to get the partition listed / names:
# fdisk -l

Please double check the lines and make sure that you have a similar output :

You will have to mount your boot partition /dev/sda1 in order to be able to work on it, remember your current system was loaded via the LIVE-DVD so if you want to access your server’s hard drive and partition you have to mount it:

# mount /dev/sda1 /mnt

Removing/Changing The Grub Password:

You need to modify /boot/grub/menu.lst file:
# cd /mnt/grub

You will have to verify what’s in the folder by using this command:
# ls

You now have to generate a new grub md5 password it’s super easy, type :
#grub-md5-crypt

Choose your password and this will return you the hashed version of it.
You can write it down as we will need it to edit the menu.lst file with vi

BEFORE making any changes to the original menu.lst file, let’s make a backup of this file with the following command:
# cp menu.lst menu.lst.backup

Now we are safe to do the modifications using vi text editor. Type:
# vi menu.lst

PLEASE pay attention because vi isn’t easy to handle if it’s your first time.
Here is all you need to know.
If you messed up while editing the file (hit the following keys “Esc” then “:” and type “q!” this is to quit without saving.
“Esc” “:” “wq” will write the file and quit the text editor.
“Esc” “:” “q” will quit the text editor.

To do some editing and insert or modify text, you have to hit the following keys “Esc” “i” then you will be in “insert” mode which allow you to type text and make changes. In case you need to exit the insert mode you will hit “Esc”.

I will type the entire sequence as we modify the grub file.

Using the arrow keys of your keyboard place the selector on the line starting with “password --md5” then hit the the following keys “Esc” “i”
Password --md5 <replace this section with the previously generated md5 hash>

Place the selector to the timeout line and change the value to 5 (this will give you 5 seconds to hit a key and go around the GRUB menu option when you machine is booting). Once the modification is done, hit “Esc” “:” “wq”
There is obviously a way to script it all, but I don’t want to start explaining how to script this because if anything is different from your system compared to mine then you might end up in more trouble than just a lost password.

Now it’s time to wrap up, commit the changes and write nicely on the hard-drive and “eject”(unmount) the hard-drive from the Live-DVD environment.

# sync; sync

# cd /

# umount /mnt

# reboot

Section 2 (Change the system Root Password)

You will need to reset the root user password of your system before it’s all loaded and start your MSE service. Following the Cisco documentation, if you can’t access the grub option via the physical keyboard during the boot. Go to your laptop and just like you would console into a Cisco switch/router 96000 bauds.
When the appliance is booting/rebooting you will see the grub menu for few seconds so press “Esc” as soon as you see the following menu.

Then press “p” and you’ll be prompted for the GRUB password

Then press “p” and you’ll be prompted for the GRUB password

Now you will select the line beginning with “kernel” and press “e”.

Then you will be typing the following:
rhgb quiet consolettys0,9600 1
Press “Enter”

Now your GRUB menu will look like this, so it’s is time to hit “b”









Your laptop screen should look like this:

Now go to the keyboard/monitor and wait for the boot sequence to be done:


You’ll have the opportunity to change the system root password using this command:
#passwd

Proceed with the password creation and you should obtain a successful message at the end of the process

At this point everything is done and you can simply reboot the machine with :
#reboot

VOILA !

300-365 WIDEPLOY preparation.

Forewords.
I truly hope you will not feel too overwhelmed by the volume of documentation, materials, and video that I am going to list. I have desperately been looking at exam feedback and tips for months. I’ve searched for #300-365 or #wideploy on twitter and I have found NOTHING valuable. I couldn’t even find anyone mentioning that they’ve successfully passed the exam. In all honesty, I understand that the amount of documentation and the lack of proper study guides available for the CCNP Wireless is a little disheartening, but you can feel lucky that I’ve put the time into writing this article, providing you feedback and tips to prepare for the exam. I went through all the material and put a significant amount of time organizing my drafts and notes to share with you. I hope I can get you off to a better start preparing for the CCNP Wireless than I had.

Acknowledgement.
I am thanking my girlfriend for putting up with me spending all my free time on my preparation.
Thanks to my manager for supporting me and providing the vouchers for the exams.
I will never be grateful enough that Jerome Henry produced so much well presented content.
Special mention to Phil Morgan from NC-expert and Rasika for the motivational speech.
And finally, Thanks to all the Wireless enthusiast who delivered positiveness while I was posting about my failure on exam attempts.

My preparation journey towards the Cisco exam 300-365 WIDEPLOY

The numbers:
I like the numbers and fun facts that are usually at the end of most reports, so I’ll start with the fun stuff.
It took me three attempts to pass the CISCO exam 300-360 (WIDEPLOY). Here are my results:
Jan 30th FAIL 661/1000

Feb 14th FAIL 805/1000

Feb 20th PASS 850/1000

When I took the exams the passing score was 818/1000.

In total I travelled 2,100 km, spent 240$ on gas, not to mention the bad Canadian winter driving conditions with roadblocks for avalanche control, rocks falling from cliffs and a cracked windshield.

I’ve estimated my preparation time to be about ~ 200 hours (I am a slow reader).

The learning materials option:

As of this writing (Feb 2017), there are no official study guides available for the CCNP Wireless, but there are three main option available:
1.) The official Cisco page
https://learningnetwork.cisco.com/community/certifications/ccnp_wireless/wideploy/study-material
-I am planning to send my recommendation to Cisco and hopefully they will update their list

2.) On demand e-learning
https://learningnetworkstore.cisco.com/ccnp-wireless
-I’ve used the demo version and I wasn’t impressed with the content so I didn’t wanted to spend the money for it.

3.) Instructor lead course
https://learninglocator.cloudapps.cisco.com/GlobalLearningLocator/courseDetails.do?actionType=executeCourseDetail&courseID=7747

If you want to opt for this, try to familiarize yourself with the exam objectives and read at least once all the document provided by Cicso. I’ve attended 2 courses for the WITSHOOT and WISECURE with Phil Morgan@CCIE5224 from NC-Expert. If you really want to maximize your time/money, it would be wise to be prepared so when you’re sitting the class you will really have the benefits of having the instructor’s knowledge. I am really picky so I would only go with NC-Expert or NetworkDojo for instructor lead courses related to Cisco.
Please note that those classes are only a few days of intensive hours where you’ll be going through a lot of material. The instructor is not a magician, and you will still have to put the hours and effort in your preparation for the exam. They will be there to put the “polish coating” on what you already know.
If you haven’t prepared anything, which was my case for the WISECURE when I took the class last year, you will be swamped like I was and the instructor has to compensate the gap and spend time to get you at a the level you should be at (in other words you’ve just missed a great opportunity to make a better use of someone's knowledge and expertise)
I’ve received study and lab guides during the course so you will not walk away empty handed. I was pleased with the content and labs really helped reinforce the theory (and some people, like myself, learn better by doing hands-on testing).
A few more words about labing, if there is one thing that can be time consuming, it is setting up the lab, during the class the scenarios are all “ready” to get you to the exact subject you are studying. It would take you hours if you ever had to manage VMs and snapshots to make this entire thing work the way it does during the class.
When I took mine, one of the lab went sideways due to errors with the infrastructure, watching the instructor troubleshooting it live, during a lunch break, brought me to a point as “what does a CCIE looks like when troubleshooting…” What steps is he taking and how did he approach the problem, I was impressed to see how methodical my instructor was. You will also be in touch with other students and that could open up some doors for you. I know the financial aspect of instructor lead classes is big, but there is a lot more to take away than just some learning material and some random notes.

How to and where to start?
With Methodology and Discipline.

It is “Simple”, start a spreadsheet and list all the documentations recommended by Cisco. It is as simple as a copy and pasting into a spreadsheet.

Create some column to add the pdf link of each recommended learning materials and another column to have a check mark of documents you’ve printed. (Keep checking the Cisco site for potential updates or simply login with your cisco credential and hit the follow button,this will trigger emails as soon as changes are made to the page).

Whenever you’re following one of the learning material’s link pages it will open a new tab in your web-browser, leading you to the Cisco documentation site, and then you can go to “Download”.

Select the “PDF- This chapter” link

Copy and paste the link in your spreadsheet (you can also download the document on your computer or tablet).

Once you are done listing every possible links, it is up to you to decide which documents are worthwhile to print.

Now it’s time to load your printer trays and you know what to do next. I didn’t print all the documents, but focused on the material that was most pertinent to the exam objectives.
Talking about which,

THE First and most important document to read and print is :
THE EXAM OBJECTIVES !!! https://learningnetwork.cisco.com/community/certifications/ccnp_wireless/wideploy/exam-topics
The pdf version is here I printed the objectives during my own studying and covered it in notes.
You can easily expand all the sections and copy the entire content into a spreadsheet or another format.

If you want to do your own, create a spreadsheet then create additional sheets for each section and detail the sections by listing all the sub-categories. At that point you’re free to add all the details you want, such as references, links, notes and comments. I found it useful to list the recommended learning materials in each section so I could jump on it quickly and search for keywords related to the exam objectives.

I know it’s time consuming but trust me I’ve started without it and it’s simply impossible to keep track of your progress without it. You don’t have to go with too much granularity (you are not writing a study guide) I tend to think I overdid mine, but I can’t share my own spreadsheet with all the notes because I don’t want to get in trouble with Cisco and lose my credentials or even be banned from Cisco certification program. The Cisco exam policy is extremely strict, so I will only share this spreadsheet version which has enough content and notes to get you on the right track to compile your own and prepare for the exam. PLEASE don’t come back to me complaining that you failed and that I am to blame. The purpose of this document is not to tell you everything you need to know to pass, but rather to help you find the material and organize it in a logical manner, so you can be efficient in your studying and not waste as much time as I did. You will still have to put in the effort on your own to fully prepare for the exam!

Make PLANS
After you have printed the necessary documents I recommend picking up a binder, investing in a set of highlighters and hole punch and get going!
I am not saying that you have to print everything but I made some recommendation for printing on my list, you don’t have to follow it to a “T”. Either way it’s whatever works best for you.
Start thinking about a game plan. To stay on track I recommend setting up a study schedule. Use a calendar and try to schedule your reading and labing, set some realistic and achievable goals.
Don’t forget to take some time off for you well-being and your family, remain active (just a few hours of sports or activity with your wife/kids/friends won’t get you off track for your preparation).
Remember that “a man without goals is like a ship without a rudder”.

Take the time you need to prepare and don’t rush it.
“Recommended by the guy who took 3 attempts to pass…”

The reason I like to print some of the documents is to read them all and highlight what “feels” important to know regarding the exam objectives. No I will not do that part for you, the way I highlight my documents is simply the key concepts and subjects and details that are new to me or seems important to keep in mind for the exam.

Once you’ve spent quite some time reading all the recommended materials, the fun will start as you will go and get the good old CCNP Wireless (642-742 IUWVN) Quick Reference by uncle Jerome Henry.  (ISBN-10: 1-58714-311-9 ISBN-13: 978-1-58714-311-3)  you can find a kindle edition on amazon for about 20$ https://www.amazon.com/Wireless-642-742-IUWVN-Quick-Reference-ebook/dp/B0088476HU

If you don’t like amazon http://www.ciscopress.com/store/ccnp-wireless-642-742-iuwvn-quick-reference-9781587143113

My comments about this book,
Cisco doesn’t list it in the learning materials and I totally understand because much of the information is based on older AirOS code version. So there will be a lot of concepts that could mislead you since QoS isn’t the same between Airos 7 and 8. There are additional options in flexconnect that were added as the technology evolved. I still learned a LOT by reading it and found information in it that I couldn’t find anywhere else! I was super confused about PIM router addresses role/function and I’ve found a clear and concise explanation in the book. Multicast was well covered too. The VoWLAN part will get you to understand a lot of important concepts that will not be explained in design guides and best practice documents. It is also easier to approach than reading raw Cisco’s documentation.
It’s important to read the reference guide book AFTER you went through all the current and up to date documentation otherwise you might start losing your tracks on some of the details and what was is up to date. The opposite approach can work too but that depends on your base level of understanding.
Now that the gears are turning, go to the Cisco Live website and start watching the following videos (you need to register yourself to access all the content).

BRKEWN-2000 - Design and Deployment of Wireless LANs for Mobile Applications (2015 Cancun) - 2 Hours

BRKEWN-2016 - Branch Office Wireless LAN Design (2015 San Diego) - 90 Mins

BRKEWN-3014 - Best practices to deploy high-availability in Wireless LAN Architectures (2015 San Diego) - 2 HoursI could add a lot more videos but you can use the search function for the Cisco Live

BRKEWN-2670 - Best Practices for Configuring Cisco Wireless LAN Controllers (2015 San Diego) - 90 Mins

BRKEWN-2010 - Design and Deployment of Enterprise WLANs (2015 San Diego) - 2 Hourswebsite.
Optional (I’ve watched them even if I am not preparing for WITSHOOT yet):
BRKEWN-3011 - Advanced Troubleshooting of Wireless LANs (2015 San Diego) - 2 Hours

This is really for the WITSHOOT exam so bookmark this link BRKEWN-3000 - Analyzing and fixing WiFi issues - Cisco WLC tools and packet capture analysis techniques (2015 San Diego) - 90 Mins. If you have two hours same session :  BRKEWN-3000 - (2016 Berlin) - 2 Hours

Try to go over the slides at the end of the video or even pausing the video and take notes (elapsed time of the video and slide number so you can go back to it once you’ve watched it all)
I’ve learned a LOT of small details from the Cisco Live session that definitely helped me to have a better understanding of some concepts which are reinforcing the “foundations”. QoS was very informative and useful to help me demystifying the way the marking is done. Once you’ve watched it I highly recommend to watch uncle Jerome on youtube especially this one (https://www.youtube.com/watch?v=PhmhIojaEE8 ) just because you can’t get enough of the french accent, you should pay extreme attention to this video because this video aided my understanding of some tricky material that ultimately helped me answer a question correctly on the exam… even if it can be found in the documents of the proposed reading materials from Cisco’s page… My brain only grasped the information by watching that video in particular, but what works for me might not work for you.

To LAB or not to LAB ?

Here is a list of concept you might want to lab.

Mesh, try it all and check youtube if needed. Do some test and see what needs to be done to make it work. Associate a client, check what is the AP MESH configuration looks like.
Connect a device to the ethernet port of a Mesh AP where that mesh AP is connecting to a RAP and see what is needed to make it work.

High availability, simply know how things work by joining controllers in a mobility group. How the AP are failing over, try to change the priority of the AP and see how would they behave if one controller goes down. See how to configure backup controllers for the APs and if HA works between controllers that are not within the same mobility group.

Implement flexconnect local switching vs centrally switched and see what is needed to make 802.11r work.

Get a VM and install MSE a few times just so you know how it’s installed and if you have a physical appliance it’s even better, join your MSE to prime and get some mapping and location tracking done. Read section…. Creating and Applying Data Point and Calibration Models https://www.cisco.com/c/en/us/td/docs/wireless/mse/8-0/MSE_CMX/8_0_MSE_CAS/7_4_MSE_CAS_chapter_01010.html and try it !

If you feel like doing it, implement multicast and play around with the Bonjour gateway but I would rather recommend doing scenarios with fast roaming/QOS/AVC.

The labs will get you to crush the subjects and encounter issues, this is how you will learn the hard way but of course it’s time consuming, if your long term plan is to aim for the CCIE Wireless I would say that you can never get enough of home lab practise of crazy implementation.

While I am at it, here is some tips for the exam:
Know your table for RXSOP threshold on 2.4/5Ghz.
Know what it takes for flexconnect and roaming to work.
Get super familiar with high-availability (master the subjects listed in the exam objectives).
Know the functionalities of mDNS.
Know your QOS translation/marking I know the table is huge but try to master it by finding logic in it.
Know your ports. I am not talking about the obvious 80,23,21,53 and so on I am talking about ports related to CAPWAP SNPM ports used by MSE or related to mobility and controllers.
Keep track of time during the exam! Seriously!!!
If you get simulation/simlets during the exam, do not forget to click apply or save config!!!!! Be aware that switching between the little topology tabs, questions tabs and simulator tabs will make you lose whatever configuration you’ve entered unless you apply/save what you’ve done. In my case I was entering some config details but I forgot some details that were mentioned in the scenario tabs so I opened the scenario to find what I needed and when I switched back to the simulator interface, my settings were gone because I didn’t save my work.. This is why before moving onto the next question you may want to go back to each single step and make sure everything was applied/saved.

Now what? Well since you’re going for a CCNP level exam, you are already familiar with the exam booking procedure, so I will not cover that part.

The day of the exam:
BEFORE you enter the exam room, make sure they’ve provided you with the erasable pen/board it is a requirement from the exam center to provide it to you.

When you are entering the exam room take a seat, DO NOT START, do not even touch the mouse. Take the erasable board provided by the exam administrator and start writing whatever numbers/ports/table/protocols or whatever you can possibly remember.
THEN you test the mouse mobility/keyboard distance, adjust your chair and you’ll be ready to fire it up (the exam, not the chair).

I know I am focussing on “silly” details, trust me once the exam is started and the clock is ticking… Losing time to get your erasable pen replaced/ adjust your chair/ find that the cable of the mouse is tangled. You’ll be thinking of what I just wrote.
I find that the “time” you’re taking doing this is “relaxing” it’s about checking very simple things, free up your mind and slowly getting comfortable. Writing on erasable sheet without having a clock ticking in the background will allow you to write down as much as you can and that will save you precious time during the exam.

Why have I had to go through 3 attempts ?

My preparation for this exam took about 2 months 3-4 hours of reading/labing everyday. Prior to this exam let’s consider that I had no experience with MESH, very basic use of MSE.
I never used multicast and because of all the legacy devices I have to deal with I don’t have much experience with 802.11r.
I was familiar with flexconnect and flexconnect group / OEAP / VoWLAN / HA / Mobility. Once I was done reading few of the recommended materials from Cisco I went to take my first attempt I can certainly say that I knew I wasn't ready for the exam, but I also found out that a lot of the topics I was confident and familiar with were actually giving me issues, I’ve skipped reading on flexconnect and mobility when I started my journey and that was a big mistake. Don’t assume that you know the exam subjects just because you’re working with it everyday. Read it ALL you might even find useful information and discover mistake you did in the past.

I’ve failed my second attempt and missed the passing score by 13 marks which is probably one question of the entire exam. I can only recommend to read your score results and find the areas you have to improve. This is how I managed to make a better use of my preparation time for the next attempt and focus on some details, re-read chapters trying to understand where have I failed. You will see that your thinking will be sharper and you will get the “cisco” way of answering question that will get you the “pass” score.

Final recommendation:
When reading documentation, the sections you want to remember details about are:

• Operation mode
• Concept/technology XYZ “Summary” sections (usually bullet pointed)
• Limitation
• Compatibility
• Supported feature/clients
• General Deployment Considerations

Good Luck and don’t give up!

Do not hesitate to reach me if you have suggestions or questions about my article and spreadsheet.

Thanks,
Ben