NAPALM | the broadcast

2018-02-122018-02-27

Cisco IOS Service-Containers: Run an x86-VM inside a Router (Part 5: Install the Service-VM)

The CSR1000V-Router has already been prepared (Create a CSR1000V-Instance with nested Virtualiation support) and in (Part 4: Package the Service-VM into an OVA) we copied the OVA-Image of our „ubuntu-server“-VM to this Router.

Add the internal Network-Interface between Router and Service-VM

SERVICECONTAINER#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
SERVICECONTAINER(config)#int virtualportgroup 0
*Feb  3 19:05:17.206: %LINEPROTO-5-UPDOWN: Line protocol on Interface VirtualPortGroup0, changed state to up
SERVICECONTAINER(config-if)#ip addr 192.168.0.1 255.255.255.0
SERVICECONTAINER(config-if)#exit

SERVICECONTAINER(config)#virtual-service
SERVICECONTAINER(config-virt-serv-global)#signing level unsigned
% Support for unsigned packages has been user-enabled. Unsigned packages are not endorsed by Cisco Systems, Inc. User assumes all responsibility
SERVICECONTAINER(config-virt-serv-global)#ex

SERVICECONTAINER(config)#ip dhcp pool SERVICE
SERVICECONTAINER(dhcp-config)# network 192.168.0.0 255.255.255.0
SERVICECONTAINER(dhcp-config)# exit

SERVICECONTAINER(config)#virtual-service UBUNTU
SERVICECONTAINER(config-virt-serv)#vnic gateway virtualPortGroup 0
SERVICECONTAINER(config-virt-serv-vnic)#guest ip address 192.168.0.2
SERVICECONTAINER(config-virt-serv-vnic)#end
SERVICECONTAINER#
*Feb  3 19:06:27.474: %SYS-5-CONFIG_I: Configured from console by console

Install the Service-VM

SERVICECONTAINER#debug virtual-service all
virtual service all debugging is on
SERVICECONTAINER#term mon
% Console already monitors
SERVICECONTAINER#term width 0

SERVICECONTAINER#virtual-service install name UBUNTU package bootflash:ubuntu1604.ova
Installing package 'bootflash:/ubuntu1604.ova' for virtual-service 'UBUNTU'. Once the install has finished, the VM may be activated. Use 'show virtual-service list' for progress.

*Feb  3 19:07:43.518: VIRTUAL-SERVICE [UBUNTU]: Sending install req for [UBUNTU], path=bootflash:/ubuntu1604.ova, uri= uid=0
*Feb  3 19:07:43.529: VIRTUAL-INSTANCE: Message sent for INSTALL TDL request: Virtual-instance name: UBUNTU, UID: 0
*Feb  3 19:07:43.529: VIRTUAL-SERVICE: Started response timer for tid DD000001 - 30 minutes
*Feb  3 19:05:14.205: %IOSXE-4-PLATFORM: R0/0: kernel: dev->name [intsvc0]: dev_entry not populated

SERVICECONTAINER#show virtual-service list
System busy installing virtual-service 'UBUNTU'. The request may take several minutes...
Virtual Service List:


Name                    Status             Package Name
------------------------------------------------------------------------------
UBUNTU                  Installing         ubuntu1604.ova


*Feb  3 19:08:00.645: %VMAN-5-PACKAGE_SIGNING_LEVEL_ON_INSTALL: R0/0: vman: Package 'ubuntu1604.ova' for service container 'UBUNTU' is 'unsigned', signing level cached on original install is 'unsigned'
*Feb  3 19:08:09.216: VIRTUAL-SERVICE: Install response handler: VM[UBUNTU]: Owner IOSd trans_id 3707764737
*Feb  3 19:08:09.216: VIRTUAL-SERVICE [UBUNTU]: vm[UBUNTU] set owner [IOSd]
*Feb  3 19:08:09.225: VIRTUAL-SERVICE [UBUNTU]: application_name: 'ubuntu' application_vendor: '' application_version: '1.1'
*Feb  3 19:08:09.226: VIRTUAL-SERVICE [UBUNTU]: Default profile info: license_name: '', license_ver: ''
*Feb  3 19:08:09.226: VIRTUAL-SERVICE: Install pkg response for tid DD000001: rc=0, descr=Install Success
*Feb  3 19:08:09.226: VIRTUAL-SERVICE [UBUNTU]: License type: none, no license needed
*Feb  3 19:08:09.226: %VIRT_SERVICE-5-INSTALL_STATE: Successfully installed virtual service UBUNTU
*Feb  3 19:08:09.243: VIRTUAL-SERVICE: Received local transport activation request
*Feb  3 19:08:09.244: VIRTUAL-SERVICE: Enabling vman local transport

SERVICECONTAINER#show virtual-service list
Virtual Service List:

Name                    Status             Package Name
------------------------------------------------------------------------------
UBUNTU                  Installed          ubuntu1604.ova

*Feb  3 19:08:32.758: %ONEP_BASE-6-SS_ENABLED: ONEP: Service set Base was enabled by Default
*Feb  3 19:08:33.259: VIRTUAL-SERVICE: Local transport 'activation' request processed

Activate the installed Service-VM

SERVICECONTAINER#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
SERVICECONTAINER(config)#virtual-service UBUNTU
SERVICECONTAINER(config-virt-serv)#activate
SERVICECONTAINER(config-virt-serv)#end

% Activating virtual-service 'UBUNTU', this might take a few minutes. Use 'show virtual-service list' for progress.

*Feb  3 19:09:57.772: VIRTUAL-SERVICE [UBUNTU]: Activate CLI: appl->owner [IOSd]
*Feb  3 19:09:57.775: VIRTUAL-INSTANCE: Message sent for ACTIVATE TDL request: Virtual service name: UBUNTU, UID: 0
*Feb  3 19:09:57.775: VIRTUAL-SERVICE [UBUNTU]: Started virtual service (0) activate response timer - 30 minutes
*Feb  3 19:10:06.052: %SYS-5-CONFIG_I: Configured from console by console
*Feb  3 19:10:06.383: VIRTUAL-SERVICE [UBUNTU]: Activate response handler: got owner [IOSd]
*Feb  3 19:10:06.383: VIRTUAL-SERVICE: clnt_type 0: Interface counter is '1'
*Feb  3 19:10:06.383: VIRTUAL-SERVICE: Information for virtual port grp '0' is received
*Feb  3 19:10:06.384: VIRTUAL-SERVICE [UBUNTU]: Deliver intf response, vm =UBUNTU, counter=1
*Feb  3 19:10:06.384: VIRTUAL-SERVICE [UBUNTU]: Received interface id=0, type=1, state=1
*Feb  3 19:10:06.384: VIRTUAL-SERVICE [UBUNTU]: Received virtual port group interface 0 with service MAC 001e.e5b1.cfba, state: up
*Feb  3 19:10:06.385: VIRTUAL-INSTANCE: Message sent for IF MTU TDL message: appliance 'UBUNTU'
*Feb  3 19:10:06.385: VIRTUAL-SERVICE [UBUNTU]: Activate response handler: rsp_rc 0
*Feb  3 19:10:06.385: VIRTUAL-SERVICE [UBUNTU]: Deliver response: appliance_state 3 rsp_rc 0 if_notify name UBUNTU clnt_type 0 act_state 0
*Feb  3 19:10:06.385: %VIRT_SERVICE-5-ACTIVATION_STATE: Successfully activated virtual service UBUNTUconf t
*Feb  3 19:10:06.385: VIRTUAL-SERVICE [UBUNTU]: Stopped virtual service (1) response timer
*Feb  3 19:10:06.385: VIRTUAL-SERVICE: Delivered Virt-manager response message to virtual service 'UBUNTU' - Response: 'OK'
*Feb  3 19:10:06.385: VIRTUAL-SERVICE [UBUNTU]: set owner to 'IOSd' in appliance

SERVICECONTAINER#show virtual-service list
Virtual Service List:

Name                    Status             Package Name
------------------------------------------------------------------------------
UBUNTU                  Activated          ubuntu1604.ova

SERVICECONTAINER#show ip dhcp bind
Bindings from all pools not associated with VRF:
IP address      Client-ID/              Lease expiration        Type       State      Interface
                Hardware address/
                User name
192.168.0.2     001e.e5b1.cfba          Feb 04 2018 07:20 PM    Automatic  Active     VirtualPortGroup0

Access the VM using the (virtual) Serial-Console

SERVICECONTAINER#virtual-service connect name UBUNTU console
Connected to appliance. Exit using ^c^c^c

Ubuntu 16.04.3 LTS ubuntu-server ttyS0

ubuntu-server login: user
Password:
Last login: Sat Feb  3 20:23:27 CET 2018 on ttyS0
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-87-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

9 packages can be updated.
7 updates are security updates.


user@ubuntu-server:~$ who
user     ttyS0        2018-02-03 23:12

Logout: 3x [CTRL]+

user@ubuntu-server:~$ ^C
user@ubuntu-server:~$ ^C
user@ubuntu-server:~$ ^C

Access the VM using SSH via the internal Network

SERVICECONTAINER#ssh -l user 192.168.0.2
Password:
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-87-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

9 packages can be updated.
7 updates are security updates.

Last login: Sat Feb  3 23:12:21 2018
user@ubuntu-server:~$ who
user     pts/0        2018-02-03 23:13 (192.168.0.1)

Check the local python/NAPALM-Setup to get facts about the containing router

user@ubuntu-server:~$ python
Python 2.7.12 (default, Dec  4 2017, 14:50:18)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.

>>> import napalm

>>> iosdriver = napalm.get_network_driver('ios')
dpass', optional_args={'port': 22, 'dest_file_system': 'bootflash:'})sword='rmon

>>> router.open()

>>> print router.get_facts()
{u'os_version': u'Virtual XE Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.7.1, RELEASE SOFTWARE (fc6)', u'uptime': 12840, u'interface_list': [u'GigabitEthernet1', u'GigabitEthernet2', u'GigabitEthernet3', u'VirtualPortGroup0'], u'vendor': u'Cisco', u'serial_number': u'9SNHUBYAML', u'model': u'CSR1000V', u'hostname': u'SERVICECONTAINER', u'fqdn': u'SERVICECONTAINER.lab.local'}

2018-02-082018-02-06

Cisco IOS Service-Containers: Run an x86-VM inside a Router (Part 3: Create the Service-VM)

Create the „Ubuntu Server“-VM using KVM on top of the „Ubuntu Desktop“-VM
Run the „Virtual Machine Manager“

Create a new virtual Machine:

Virtual Machine Manager – create a new VM

(1) Create a new virtual Machine

(2) Select the Installation ISO
* and deselect „auto OS Detection“

(3) Choose Memory and CPU Settings

(4) Create a virtual Hard-Disk for the VM

(5) Give a name to the VM

vmm – 5 – Ready to begin the installation

Boot the KVM-VM („Ubuntu Server“)

Basic Setup:

Language: Englisch
my territory: other/Europe/Germany
locale: en_US.UTF_8
Keyboard: German
hostname: ubuntu-server
user / password
no home-directory encryption
timezone Europe/Berlin
use entire virtual disk (no need for „LVM…“ i think)
no HTTP-Proxy
no automatic updates
[x] SSH-Server
install GRUB

After the first login, as always:
sudo apt-get update && sudo apt-get upgrade –y sudo reboot

Enable the Virtual-„Console“-Port of the Ubuntu-Server-VM

enable a virtual „CON“ serial-port at the linux vm

sudo joe /lib/systemd/system/ttyS0.service

Example-Script for the 16.04 System-CTL Service:

[Unit]
Description=Serial Console Service

[Service]
ExecStart=/sbin/getty -L 9600 ttyS0 vt102
Restart=always

[Install]
WantedBy=multi-user.target

Reload systemctl, enable the TTY-service and then start it:

sudo systemctl daemon-reload sudo systemctl enable ttyS0 sudo systemctl start ttyS0

Verify the service state:
user@ubuntu-server:~$ sudo systemctl status ttyS0 ● ttyS0.service - Serial Console Service Loaded: loaded (/lib/systemd/system/ttyS0.service; enabled; vendor preset: en Active: active (running) since Sat 2018-02-03 20:21:20 CET; 1h 19min ago Main PID: 831 (login) Tasks: 0 Memory: 672.0K CPU: 196ms CGroup: /system.slice/ttyS0.service ‣ 831 /bin/login --

Add your „Network Application“ to the VM:
I want to use my Service-Container for network-management (Centralized access to device-configuration and other state-information using NAPALM and Linux: SNMP with Python ):
sudo apt-get install -y joe sudo apt-get install –y libsnmp-dev snmp-mibs-downloader sudo apt-get install –y gcc python-dev python-pip libssl-dev libffi-dev python-cffi pip install easysnmp pip install napalm

That’s all
Shutdown the VM

2018-02-072018-02-06

NAPALM Validation (Static rule)

Yesterday I attended a short presentation held by David Barroso introducing the NAPALM-„Validation“-Module.

Configuring the Network fully automated is just half the way to go.
Checking the Network-State the other half.

Python-Script
Had to try it out the easy-way using just python (without ansible) first:

import napalm

#just to get the DICT returned by the "compliance_report"-Method in a readable manner on the screen
import pprint
pp = pprint.PrettyPrinter()

iosdriver = napalm.get_network_driver('ios')

DEVICE="192.168.2.139"
USER="rmond"
PASS="rmondpassword"
router = iosdriver(hostname=DEVICE, username=USER,  password=PASS, optional_args={'port': 22, 'dest_file_system': 'bootflash:'})

router.open()
report = router.compliance_report("validate.yaml")
router.close()

pp.pprint(report)

Static YAML-File
Validating against the following static „validate.yaml“-File:

- get_facts:
    hostname: CSR-1-DC
    os_version: CSR1000V Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 15.4(2)S3, RELEASE SOFTWARE (fc2)
    interface_list:
        list:
            - GigabitEthernet1
            - GigabitEthernet2
            - GigabitEthernet3
            - GigabitEthernet4
            - VirtualPortGroup0
- get_interfaces_ip:
    GigabitEthernet1:
      ipv4:
       192.168.2.139:
         prefix_length: 25

leads to the following result

{u'complies': False,
 'get_facts': {u'complies': True,
               u'extra': [],
               u'missing': [],
               u'present': {'hostname': {u'complies': True,
                                         u'nested': False},
                            'interface_list': {u'complies': True,
                                               u'nested': True},
                            'os_version': {u'complies': True,
                                           u'nested': False}}},
 'get_interfaces_ip': {u'complies': False,
                       u'extra': [],
                       u'missing': [],
                       u'present': {'GigabitEthernet1': {u'complies': False,
                                                         u'diff': {u'complies': False,
                                                                   u'extra': [],
                                                                   u'missing': [],
                                                                   u'present': {'ipv4': {u'complies': False,
                                                                                         u'diff': {u'complies': False,
                                                                                                   u'extra': [],
                                                                                                   u'missing': [],
                                                                                                   u'present': {'192.168.2.139': {u'complies': False,
                                                                                                                                 u'diff': {u'complies': False,
                                                                                                                                           u'extra': [],
                                                                                                                                           u'missing': [],
                                                                                                                                           u'present': {'prefix_length': {u'actual_value': 24,
                                                                                                                                                                          u'complies': False,
                                                                                                                                                                          u'expected_value': 25,
                                                                                                                                                                          u'nested': False}}},
                                                                                                                                 u'nested': True}}},
                                                                                         u'nested': True}}},
                                                         u'nested': True}}},
 u'skipped': []}

Whats wrong with the Router?
„nested = True“ means – the issue is downward in the datastructure.

In the example, the first Object with „nested = False“ is „prefix_length“
The Prefix-Length („Subnet-Mask“) is wrong:

wanted: /25-Bit
configured: /24-Bit.

Validate the output of commands which use additional parameters
Example: Ping to the Default-Gateway

First: Get familiar with the datastructure returned by the command:

>>> pp.pprint(router.ping("192.168.2.1"))
{u'success': {u'packet_loss': 0,
              u'probes_sent': 5,
              u'results': [{u'ip_address': u'192.168.2.1', u'rtt': 0.0},
                           {u'ip_address': u'192.168.2.1', u'rtt': 0.0},
                           {u'ip_address': u'192.168.2.1', u'rtt': 0.0},
                           {u'ip_address': u'192.168.2.1', u'rtt': 0.0},
                           {u'ip_address': u'192.168.2.1', u'rtt': 0.0}],
              u'rtt_avg': 1.0,
              u'rtt_max': 2.0,
              u'rtt_min': 1.0,
              u'rtt_stddev': 0.0}}

add the following portion to the YAML-File:

- ping:
    _name: Default-Gateway
    _kwargs:
      destination: "192.168.2.1"
    success:
      packet_loss: 0

Result (Success)

>>> pp.pprint(router.compliance_report("validate.yaml"))
{'Default-Gateway': {u'complies': True,
                     u'extra': [],
                     u'missing': [],
                     u'present': {'success': {u'complies': True,
                                              u'nested': True}}},

Result (failure)

>>> pp.pprint(router.compliance_report("validate.yaml"))
{'Default-Gateway': {u'complies': False,
                     u'extra': [],
                     u'missing': [],
                     u'present': {'success': {u'complies': False,
                                              u'diff': {u'complies': False,
                                                        u'extra': [],
                                                        u'missing': [],
                                                        u'present': {'packet_loss': {u'actual_value': -1,
                                                                                     u'complies': False,
                                                                                     u'expected_value': 0,
                                                                                     u'nested': False}}},
                                              u'nested': True}}},

2018-02-052018-02-06

Cisco IOS Service-Containers: Run an x86-VM inside a Router (Part 1: KVM Development Environment)

Network Hosted Kernel Virtual Machine (KVM)

Many Cisco Routers allow you to host your own virtual machine directly inside a router.
DevNet: Cisco IOS KVM Service-Containers

CSR 1000v
ISR 4000
ASR 1000

I went through the documentation, especially Service-Containers-Tutorial.pdf and Ubuntu – Building a Service Container.pdf where very helpful.

But i didn’t want to go the „Ubuntu-Desktop-14.04-LTS“-way which was described.

„My“ Service-Container-VM should be:

Ubuntu 16.04 LTS

provides longer support (April 2021) Releases – Ubuntu Wiki

Ubuntu Server

save some compute/cpu/dram/harddisk-ressources by not installing an Ubuntu-Desktop-Environment inside the Router. Just network services, please!

The whole deployment process can be accomplished in five steps:

build a KVM virtualization-Environment
prepare an ISR-Router (lab: Create a CSR1000v-Instance for nested Virtualization)
create the Service-VM (Ubuntu-Server)
package it inside an OVA
install this OVA at the router

Build the KVM Virtualization Environment

Here I’ll use Ubuntu LTS, but the „Desktop“-Release to get the „Virtual Machine Manager“-GUI.
ubuntu-16.04.3-desktop-amd64.iso

Common vSphere Process „New VM“

Linux Ubuntu 64 Bit, 4GB DRAM, 120GB Disk
mount Ubuntu-Desktop-ISO as DVD

Enable nested virtualization (Hardware-Virtualization for Guest-OS)
Don’t forget: this „KVM Virtualization Environment“ will run on top of an VSphere-ESXi-Host.

nested virtualization: [Linux-Server-VM] on top of [KVM Virtualization] on top of [ESXi Hypervisor]

Hardware virtualization: [x] Expose hardware assisted virtualization to guest-OS

Ubuntu-Desktop VM with KVM-Virtualization
Installation: Defaults are fine

never use the „root“-acccout
use a common user-account instead, i’ll use the username „user“ in the text

Additional Setup
user@KVM-1:~$ sudo apt-get update && sudo apt-get upgrade -y user@KVM-1:~$ sudo apt-get install -y open-vm-tools open-vm-tools-desktop

Add KVM („libvirt“)
user@KVM-1:~$ sudo apt-get install -y qemu-kvm libvirt-bin bridge-utils virt-manager qemu-system
Finalize it
user@KVM-1:~$ sudo reboot

Check the KVM Environment
Should look like this:

user@KVM-1:~/isr-service-container$ virsh -c qemu:///system list

 Id    Name                           State
----------------------------------------------------

Fine-tuning: „Disable Screensaver“

avoid ever and ever re-login to the remote-Desktop
acceptable in an isolated lab environment

all settings => brightness & lock

„never“ turn screen off when inactive
lock „off“
[ ] require password when waking from suspend

Preparation: Mount NFS-Share with „ubuntu-server“-ISO
I store my ISO-Images on an central NFS-Datastore.

the ISO-image has to be available inside the VM

sudo apt-get install -y nfs-common
mkdir /home/user/<NAS-SHARE>
sudo mount.nfs <NAS-IP>:<NAS-SHARE> /home/user/<NAS-SHARE>

2017-12-212017-12-17

Cisco CSR1000V Software Upgrade – Automated

No. There’s no need to export the IOS-Config, deploy another Router-VM using an OVA and import the old IOS-Config to this new router.

even, if finetuned… This strategy might lower the downtime!?
VMware uses this strategy when upgrading NSX-Edge-Gateways very successfully!

But this is a lab environment, i’ll have to upgrade almost ten CSR1000v-Routers and there’s no time to do it manually router-by-router.

The common process is as it has been for the last decades:

copy the new csr1000v-bin-File into the routers bootflash
verify the file
set the boot-variable
reboot

Upload BIN-File into the routers
There might be dozens of valid possibilities to get the bin-file into the router.

I prefer SCP (Secure Copy Protocol) since i uses the same firewall-rules as SSH so it’s unlikely that firewalls will disturb the update process.

i downloaded the bin-file using a windows-machine
i’ll use PSCP from the Putty-software-suite

Basics: Loop over a set of IPs in Windows Command-Shell?
That’s all:

C:> for %i in (235,241,240,239,236,237,238,242,243) do @echo %i
235
241
240
239
236
237
238
242
243

Let’s go


c:\Users\admin\Downloads>dir *.bin

 Verzeichnis von c:\Users\admin\Downloads

16.12.2017  17:44       365.660.728 csr1000v-universalk9.16.03.05.SPA.bin
               1 Datei(en),    365.660.728 Bytes
               0 Verzeichnis(se), 73.892.016.128 Bytes frei

c:\Users\admin\Downloads>for %i in (235,241,240,239,236,237,238,242,243) do @start pscp -2 -scp -l rmond -pw rmondpass csr1000v-universalk916.03.05.SPA.bin 192.168.2.%i:bootflash:csr1000v-universalk9.16.03.05.SPA.bin

This will initiate 9 parallel SCP-Filetransfers:

nobody said, this would improve the transfer speed 😉
i’ll do something else in the meantime

Verify the transferred images

import napalm
from easysnmp import Session
 
#credentials
DEVICE="192.168.2.235"
USER="rmond"
PASS="rmondpass"
SNMPRW="WRITE"
IOSFILE="bootflash:csr1000v-universalk9.16.03.05.SPA.bin"
IOSMD5="49922f08698284312379b4e0a2534bc2"
VERIFIED="Verified"

SNMPOIDReload="1.3.6.1.4.1.9.2.9.9.0"
SNMPOIDReloadVal=2
 
#instanciate NAPALM
iosdriver = napalm.get_network_driver('ios')
 
#connect to device
router = iosdriver(hostname=DEVICE, username=USER,  password=PASS, optional_args={'port': 22, 'dest_file_system': 'bootflash:'})
router.open()

#construct command to verify the integrity 
cliVerify=["verify /md5 "+IOSFILE+" "+IOSMD5]
result=router.cli(cliVerify)[cliVerify[0]]

Set the Bootvar and check, if it’s set

#%Error verifying 
#Verified
if (result.find(VERIFIED)>-1):
    print "(1) uploaded File: OK"
    cmdBootSystem="boot system flash bootflash:csr1000v-universalk9.16.03.05.SPA.bin"
    #push boot-system-command to router
    router.load_merge_candidate(config=cmdBootSystem)
    router.commit_config()

    cliShowBootvar=["show bootvar"]
    result=router.cli(cliShowBootvar)[cliShowBootvar[0]]
    #disconnet
    router.close()
    if (result.find("BOOT variable = "+IOSFILE)>-1):
        print "(2) boot-Variable set"
        print "=> Router "+DEVICE+" ready to reload"

Reload the Router using SNMP

        #snmp-server system-shutdown = 1.3.6.1.4.1.9.2.9.9.0 => Value 2 => Reload
        session = Session(hostname=DEVICE, community=SNMPRW, version=2)
        session.set(SNMPOIDReload,SNMPOIDReloadVal,"INTEGER")
else:
    #disconnet
    router.close()

The Router reboots

***
*** --- SHUTDOWN in 0:00:00 ---
*** Message from network to all terminals:
***
Null Message

Be patient.

W-DCFW#show ver | inc IOS.*Version
Cisco IOS XE Software, Version 16.03.05
Cisco IOS Software [Denali], CSR1000V Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.3.5, RELEASE SOFTWARE (fc1)

The new software-release is up and running.

2017-12-192017-12-16

Pull configs from CSR1000v using NAPALM

I usually use an „wr“-command-alias, an EEM-Applet or the IOS-Archive-Feature to copy the running-config to a central repository when an admin enters the „write“-command [copy running-config startup-config].

Sometimes it’s handy to pull the current running-config from a device.

Following Centralized access to device-configuration and other state-information using NAPALM – NAPALM makes this very easy:

import napalm

#credentials
DEVICE="192.168.2.139"
USER="rmond"
PASS="rmondpass"

#instanciate NAPALM
iosdriver = napalm.get_network_driver('ios')

#connect to device
router = iosdriver(hostname=DEVICE, username=USER,  password=PASS, optional_args={'port': 22, 'dest_file_system': 'bootflash:'})
router.open()

#read the hostname from the device
facts = router.get_facts()
hostname= facts["hostname"]

#read the device-config
clishowrun=['show running-config']
configRun=router.cli(clishowrun)[clishowrun[0]]

#disconnet
router.close()

#construct config-filename
filename = hostname+".cfg"

#write config-text into the file
with open(filename, "w") as f:
  print >>f, configRun

2017-12-062018-10-08

Common OVF Tool (COT) – Automated Lab-Router Deployment

Is this SDN (Software defined Networking)? I think so!
I really love to provision new Cisco CSR1000v lab-routers without the need to do everything manually in the graphical user interface.

It’s possible to provision production-routers, too, of course: Just add your license-information to leave the 100Kbps-limit behind (which is no handicap for normal lab-usage).

The Common OVF Tool (COT) (Documentation) enables me to do this by injecting an initial startup-configuration into the cisco-provided OVA-file before deploying it to an ESXi-host.
Interactive Cisco CLI-Commands possible
And, even better, the Cisco CSR1000V allows interactive CLI-commands within the injected configuration:

building new bootflash-directories,
creating a SSH-key.

Injecting the bootstrap-config

$ cot --version
Common OVF Tool (COT), version 2.0.3
Copyright (C) 2013-2017 the COT project developers.

$ cot inject-config csr1000v-universalk9.03.16.06b.S.155-3.S6b-ext.ova -c ios-napalm.startup.cfg -o csr1000v-universalk9.03.16.06b.S.155-3.S6b-ext.napalm.ova

This is an example-config setting parameters needed for napalm:
Centralized access to device-configuration and other state-information using NAPALM
Automated RMON Alarm/Event-configuration for class-based QoS-Monitoring using NAPALM

!
hostname NAPALM-1
ip domain-name lab.local
!
no ip domain-lookup
!
crypto key generate rsa modulus 2048
!
username rmond privilege 15 secret rmondpass
!
platform console serial
!
vrf definition MGMT
  address-family ipv4
!
int gig 1
  descr mgmt0
  vrf forwarding MGMT
  ip address dhcp
  no shut
!
ip scp server enable
!
ip access-list standard ACL_SNMP
    permit host 192.168.2.89
!
snmp-server community READ ro ACL_SNMP
snmp-server location allones.de
!
file prompt quiet
!
do mkdir bootflash:/ARCHIVE
archive
  path bootflash:/ARCHIVE/bak
!
line vty 0 4
  login local
  transport input ssh
!
end

Deploy the CSR1000V-Router
One command to deploy the router

at an ESXi-Host/vSphere-Datacenter
using a defined Datastore
creating an serial-port (not really needed since SSH is running immedeately)
place the three NICs of this router at the desired vSwitch-Portgroups

Those vSwitch-Portgroups have been created using vSphere-CLI.

$ cot --verbose deploy csr1000v-universalk9.03.16.06b.S.155-3.S6b-ext.napalm.ova esxi //host/ -u -p -n RTR-NAPALM-1 -d -S "telnet://:31001,server" -N GigabitEthernet1="Management" GigabitEthernet2=T24 GigabitEthernet3=T34 -c 1CPU-4GB

SSH-Access to the router

NAPALM-1# show ssh
Connection Version Mode Encryption  Hmac         State                 Username
0          2.0     IN   aes256-ctr  hmac-sha1    Session started       rmond
0          2.0     OUT  aes256-ctr  hmac-sha1    Session started       rmond
%No SSHv1 server connections running.

NAPALM-1#who
    Line       User       Host(s)              Idle       Location
*  1 vty 0     rmond      idle                 00:00:00 192.168.2.312

NAPALM-1#cd bootflash:ARCHIVE
NAPALM-1#pwd
bootflash:/ARCHIVE/

NAPALM-1#show crypto key mypubkey all
Key name: NAPALM-1.lab.local
Key type: RSA KEYS
 Storage Device: private-config
 Usage: General Purpose Key
 Key is not exportable. Redundancy enabled.
 Key Data:
  30820222 300D0609 2A864886 F70D0101 01050003 82020F00 3082020A 02820201
  ...
  70F5FE1C 01BE930D B3C84841 AC46EE0D 451DC530 55F28B9C 82796E8F 1B5F5163
  57020301 0001

NAPALM-1#show ip int brief
Interface              IP-Address      OK? Method Status                Protocol
GigabitEthernet1       192.168.2.135   YES DHCP   up                    up
GigabitEthernet2       unassigned      YES unset  administratively down down
GigabitEthernet3       unassigned      YES unset  administratively down down

NAPALM-1#show ip route vrf MGMT
Routing Table: MGMT
Gateway of last resort is 192.168.2.1 to network 0.0.0.0

S*    0.0.0.0/0 [254/0] via 192.168.2.1
      192.168.2.0/24 is variably subnetted, 3 subnets, 2 masks
C        192.168.2.0/24 is directly connected, GigabitEthernet1
S        192.168.2.1/32 [254/0] via 192.168.2.1, GigabitEthernet1
L        192.168.2.315/32 is directly connected, GigabitEthernet1

2017-12-052017-11-25

Automated RMON Alarm/Event-configuration for class-based QoS-Monitoring using NAPALM

In Configure RMON Alarms&Events by script I’ve shown a short python-algorithm to to discover all Cisco class-based QoS (cbQoS) packet-/drop-counters and to generate RMON-alarms for each. The router monitors these counters every 300s, calculates the delta for the interval and raises RMON-events when there were packets/drops or when the have been before but not anymore.

This RMON-event has been configured as an syslog-message to an syslog-receiver etc.

The existing script just generated a list of cli-commands which had to be entered manually to the router-config.
Not a valid aproach when having hundreds devices to be configured.

Now i want the script to automatically configure the router.

add both „rmon event“-objects for the rising- and the falling-threshold of the monitored alarms
read the existing „rmon alarm“-objects from the device config, which have been configured by this script during a former run
remove these existing alarms
discover all cbQoS-packet/drop-counters
add corresponding „rmon alarm“-objects

I’d like to refer to Centralized access to device-configuration and other state-information using NAPALM for some basic information regarding NAPALM and how to create the „router“-object in python.

NAPALM: Read existing RMON alarms.
I’ll use the following python-logic to

remote-execute the command
immedeately pull the cli-output out of the python-dictionary: the CLI-Command is the dict-key

>>> cligetrmon=['show rmon alarms | inc RMONevent']
>>> rmonalarms = router.cli(cligetrmon)[cligetrmon[0]]
>>> print rmonalarms
Alarm 10001 is active, owned by RMONevent

Generate CLI to delete these RMON alarms

>>> cmdnormon = ""
>>> for alarm in rmonalarms.split('\n'):
...  alarmid = alarm.split(' ')[1]
...  cmdnormon += "no rmon alarm "+alarmid+"\n"
...
>>>
>>> print cmdnormon
no rmon alarm 10001

Static CLI to add required RMON events

>>> cmdrmonevent = "rmon event 10 log owner RMONevent\n"
>>> cmdrmonevent += "rmon event 11 log owner RMONevent\n"

Read Cisco cbQoS-MIB to fetch interesting QoS-counters, generate CLI for RMON-alarms

>>> from easysnmp import Session
>>> hostname = "192.168.2.72"
>>> session = Session(hostname, community='READ', version=2)
>>>
... cbqos = session.walk('1.3.6.1.4.1.9.9.166.1.15.1.1.13')
>>>
... cmdrmon = ""
>>> alarmID = 10001
>>>
... for i in cbqos:
...   oidList=i.oid.split(".")
...   q=oidList.pop()
...   p=oidList.pop()
...   #print p,q
...   ifTypeID=int(session.get("1.3.6.1.4.1.9.9.166.1.1.1.1.2."+p).value)
...   ifDirID=int(session.get("1.3.6.1.4.1.9.9.166.1.1.1.1.3."+p).value)
...   if (ifDirID==2):
...     cmdrmon += "rmon alarm "+str(alarmID)+" "+i.oid+" 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent\n"
...   alarmID += 1
...

Concatenate all commmands

>>> cmd = cmdrmonevent+cmdnormon+cmdrmon
>>> print cmd
rmon event 10 log owner RMONevent
rmon event 11 log owner RMONevent
no rmon alarm 10001
rmon alarm 10001 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10002 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10003 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10004 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10005 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10006 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent

Push the commands to the Router

>>> router.load_merge_candidate(config=cmd)

Check the differences befor apply the changes

>>> print router.compare_config()
-no rmon alarm 10001
+rmon alarm 10001 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
+rmon alarm 10002 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
+rmon alarm 10003 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
+rmon alarm 10004 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
+rmon alarm 10005 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
+rmon alarm 10006 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent

Verify the pushed commands at the router-CLI

IOS-RTR#dir *.txt
Directory of bootflash:/*.txt

Directory of bootflash:/

   21  -rw-         898  Nov 24 2017 15:05:32 +00:00  merge_config.txt
7835619328 bytes total (6613028864 bytes free)

IOS-RTR#more merge_config.txt
rmon event 10 log owner RMONevent
rmon event 11 log owner RMONevent
no rmon alarm 10001
rmon alarm 10001 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10002 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10003 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10004 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10005 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10006 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent

Commit the changes

>>> router.commit_config()

Or discard them

>>> router.discard_config()

It’s possible to rollback committed changes.

>>> router.rollback()

Finally: Disconnect the session with the device

>>> router.close()

Again: A brief look to the router

IOS-RTR#show run | inc rmon
! Last configuration change at 19:08:59 UTC Fri Nov 24 2017 by rmond
! NVRAM config last updated at 19:09:00 UTC Fri Nov 24 2017 by rmond
username rmond privilege 15 secret 5 $1$7VnE$2O18Vfcr4y7eO5gY7l4xx1
rmon event 10 log owner RMONevent
rmon event 11 log owner RMONevent
rmon alarm 10001 cbQosCMStatsEntry.13.18.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10002 cbQosCMStatsEntry.13.18.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10003 cbQosCMStatsEntry.13.18.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10004 cbQosCMStatsEntry.13.34.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10005 cbQosCMStatsEntry.13.34.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10006 cbQosCMStatsEntry.13.34.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
IOS-RTR#

It’s already written to NVRAM

IOS-RTR#show startup-config | inc rmon
! Last configuration change at 19:08:59 UTC Fri Nov 24 2017 by rmond
! NVRAM config last updated at 19:09:00 UTC Fri Nov 24 2017 by rmond
username rmond privilege 15 secret 5 $1$7VnE$2O18Vfcr4y7eO5gY7l4xx1
rmon event 10 log owner RMONevent
rmon event 11 log owner RMONevent
rmon alarm 10001 cbQosCMStatsEntry.13.18.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10002 cbQosCMStatsEntry.13.18.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10003 cbQosCMStatsEntry.13.18.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10004 cbQosCMStatsEntry.13.34.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10005 cbQosCMStatsEntry.13.34.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10006 cbQosCMStatsEntry.13.34.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent

2017-12-042017-11-25

Centralized access to device-configuration and other state-information using NAPALM

Since I still want to build a centralized solution for automated configuration of RMON-alarms to monitor Cisco cbQoS-packet/drop-counters a solution to retrieve and modify device configurations was needed.

The NAPALM (Network Automation and Programmability Abstraction Layer with Multivendor support) Framework seems to provide the required features.
NAPALM Installation
Some NAPALM-IOS dependencies have to be fulfilled first.

sudo apt-get install -y --force-yes libssl-dev libffi-dev python-dev python-cffi

The „partial installation“ seems to be not working anymore, the full installation uses some KB more ressources, not worth to think about the partial installation…

pip install napalm

IOS Preperation
To allow remote-access to the centralized NAPALM server these features need to be enabled in each IOS device:

Remote-Access via SSH,
SCP (Secure Copy),
the IOS „Archive“-feature is the foundation of NAPALM config-operations.

IOS-RTR#conf t
Enter configuration commands, one per line.  End with CNTL/Z.

! AAA preferred for production-systems, of course
IOS-RTR(config)#username rmond privilege 15 secret rmondpass

! required
IOS-RTR(config)#ip scp server enable

! no annoying [yes/no]-prompts for file-operations anymore
IOS-RTR(config)#file prompt quiet

! create the folder in the filesystem for the Archive
IOS-RTR(config)#do mkdir bootflash:/ARCHIVE

IOS-RTR(config)#archive
IOS-RTR(config-archive)#path bootflash:/ARCHIVE/bak-

Explore Napalm

Go the the python shell:

user@snmp-server:~$ python
>>> import napalm
>>> iosdriver = napalm.get_network_driver('ios')

Autodiscovery of the router-filesystem doesn’t work, i know the filesystem of my router and pass it as „optional argument“ to router-object.
*** todo: troubleshoot/fix the autodiscovery ***

>>> router = iosdriver(hostname='192.168.2.72', username='rmond',  password='rmondpass', 
optional_args={'port': 22, 'dest_file_system': 'bootflash:'})
>>> router.open()

Go to the router cli-shell:

user „rmond“ is logged in

IOS-RTR#who
    Line       User       Host(s)              Idle       Location
*  1 vty 0     user       idle                 00:00:00 192.168.2.109
   2 vty 1     rmond      idle                 00:00:02 192.168.2.89

Back to python, try some NAPALM-functions.

>>> print router.get_facts()
{u'os_version': u'CSR1000V Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 15.4(2)S3, RELEASE SOFTWARE (fc2)', 
u'uptime': 657600, u'interface_list': [u'GigabitEthernet1', u'GigabitEthernet2', u'GigabitEthernet3'], u'vendor': u'Cisco', 
u'serial_number': u'afdökjl0123', u'model': u'CSR1000V', u'hostname': u'IOS-RTR', u'fqdn': u'IOS-RTR.lab.local'}

Execute interactive EXEC-Commands.

>>> cliping=['ping 192.168.2.1']
>>> print router.cli(cliping)
{'ping 192.168.2.1': u'Type escape sequence to abort.\n
Sending 5, 100-byte ICMP Echos to 192.168.2.1, timeout is 2 seconds:\n!!!!!\n
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/2 ms'}

Fetch the current running-config.

>>> clishowrun=['show running-config']
>>> print router.cli(clishowrun)
{'show running-config': u'Building configuration...\n\nCurrent configuration : 2411 bytes\n!\n
! Last configuration change at 12:52:32 UTC Fri Nov 24 2017 by user\n! NVRAM config last updated at 10:24:26 UTC Fri Nov 24 2017 by user\n!\n
version 15.4\nservice timestamps debug datetime msec\nservice timestamps log datetime msec\n
...rmon event 10 log owner RMONevent\nrmon event 11 log owner RMONevent\n
rmon alarm 10001 cbQosCMStatsEntry.2.18.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent\n
...
end'}

Read some lines from the running-config, filter using the pipe.

>>> cligetrmon=['show rmon alarms | inc RMONevent']
>>> print router.cli(cligetrmon)
{'show rmon alarms | inc RMONevent': u'Alarm 10001 is active, owned by RMONevent'}

Log-out if you are finished:

>>> router.close()