Cisco CSR1000V Software Upgrade – Automated

No. There’s no need to export the IOS-Config, deploy another Router-VM using an OVA and import the old IOS-Config to this new router.

  • even, if finetuned… This strategy might lower the downtime!?
  • VMware uses this strategy when upgrading NSX-Edge-Gateways very successfully!

But this is a lab environment, i’ll have to upgrade almost ten CSR1000v-Routers and there’s no time to do it manually router-by-router.

The common process is as it has been for the last decades:

  • copy the new csr1000v-bin-File into the routers bootflash
  • verify the file
  • set the boot-variable
  • reboot

Upload BIN-File into the routers
There might be dozens of valid possibilities to get the bin-file into the router.

I prefer SCP (Secure Copy Protocol) since i uses the same firewall-rules as SSH so it’s unlikely that firewalls will disturb the update process.

  • i downloaded the bin-file using a windows-machine
  • i’ll use PSCP from the Putty-software-suite

Basics: Loop over a set of IPs in Windows Command-Shell?
That’s all:

C:> for %i in (235,241,240,239,236,237,238,242,243) do @echo %i
235
241
240
239
236
237
238
242
243

Let’s go


c:\Users\admin\Downloads>dir *.bin

 Verzeichnis von c:\Users\admin\Downloads

16.12.2017  17:44       365.660.728 csr1000v-universalk9.16.03.05.SPA.bin
               1 Datei(en),    365.660.728 Bytes
               0 Verzeichnis(se), 73.892.016.128 Bytes frei

c:\Users\admin\Downloads>for %i in (235,241,240,239,236,237,238,242,243) do @start pscp -2 -scp -l rmond -pw rmondpass csr1000v-universalk916.03.05.SPA.bin 192.168.2.%i:bootflash:csr1000v-universalk9.16.03.05.SPA.bin

This will initiate 9 parallel SCP-Filetransfers:

  • nobody said, this would improve the transfer speed 😉
  • i’ll do something else in the meantime
9x PSCP-File-Transfers

Verify the transferred images

import napalm
from easysnmp import Session
 
#credentials
DEVICE="192.168.2.235"
USER="rmond"
PASS="rmondpass"
SNMPRW="WRITE"
IOSFILE="bootflash:csr1000v-universalk9.16.03.05.SPA.bin"
IOSMD5="49922f08698284312379b4e0a2534bc2"
VERIFIED="Verified"

SNMPOIDReload="1.3.6.1.4.1.9.2.9.9.0"
SNMPOIDReloadVal=2
 
#instanciate NAPALM
iosdriver = napalm.get_network_driver('ios')
 
#connect to device
router = iosdriver(hostname=DEVICE, username=USER,  password=PASS, optional_args={'port': 22, 'dest_file_system': 'bootflash:'})
router.open()

#construct command to verify the integrity 
cliVerify=["verify /md5 "+IOSFILE+" "+IOSMD5]
result=router.cli(cliVerify)[cliVerify[0]]

Set the Bootvar and check, if it’s set

#%Error verifying 
#Verified
if (result.find(VERIFIED)>-1):
    print "(1) uploaded File: OK"
    cmdBootSystem="boot system flash bootflash:csr1000v-universalk9.16.03.05.SPA.bin"
    #push boot-system-command to router
    router.load_merge_candidate(config=cmdBootSystem)
    router.commit_config()

    cliShowBootvar=["show bootvar"]
    result=router.cli(cliShowBootvar)[cliShowBootvar[0]]
    #disconnet
    router.close()
    if (result.find("BOOT variable = "+IOSFILE)>-1):
        print "(2) boot-Variable set"
        print "=> Router "+DEVICE+" ready to reload"

Reload the Router using SNMP

        #snmp-server system-shutdown = 1.3.6.1.4.1.9.2.9.9.0 => Value 2 => Reload
        session = Session(hostname=DEVICE, community=SNMPRW, version=2)
        session.set(SNMPOIDReload,SNMPOIDReloadVal,"INTEGER")
else:
    #disconnet
    router.close()

The Router reboots

***
*** --- SHUTDOWN in 0:00:00 ---
*** Message from network to all terminals:
***
Null Message

Be patient.

W-DCFW#show ver | inc IOS.*Version
Cisco IOS XE Software, Version 16.03.05
Cisco IOS Software [Denali], CSR1000V Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.3.5, RELEASE SOFTWARE (fc1)

The new software-release is up and running.

Cisco CSR1000v – additional Interfaces? Common OVF Tool (COT)!

Routers with more than three interfaces? Not that uncommon!
It might be handy, if several devices are needed with more than three interfaces, to:

  • not:
    • deploy the devices with three interfaces first
    • add the needed number of interfaces to the routers
    • attache tne new interfaces to the correct vSphere-Portgroup
  • but instead:
    • create an OVA-Template with the correct number of interfaces (one time)
    • deploy the routers and attach them during deployment to their portgroup

So using vNIC Hot-Add might be not the favourite way to work.

Common OVF Tool
COT lets you to deploy Cisco CSR1000v routers by easy to use (linux-)commands.
(Common OVF Tool (COT) – Automated Lab-Router Deployment
This is my original OVA-file already containing some basic configurations needed for NAPALM.

  • it will build a virtual router with three GigabitEthernet-Interfaces
user@ubuntu-server:~$ cot info csr1000v-universalk9.16.03.05.napalm.ova | egrep "(Networks|Gigabit)"
Networks:
  GigabitEthernet1  "Data network 1"
  GigabitEthernet2  "Data network 2"
  GigabitEthernet3  "Data network 3"
NICs and Associated Networks:
  GigabitEthernet1 : GigabitEthernet1
  GigabitEthernet2 : GigabitEthernet2
  GigabitEthernet3 : GigabitEthernet3
  Management Interface                                    "GigabitEthernet1"

Add another Interface (long version)
Let’s use the prepared OVA and create another OVA with 4 interfaces:

user@ubuntu-server:~$ cot edit-hardware csr1000v-universalk9.16.03.05.napalm.ova -o csr1000v-universalk9.16.03.05.napalm.4IF.ova -n 4 --nic-type vmxnet3 --nic-networks "GigabitEthernet1" "GigabitEthernet2" "GigabitEthernet3" "GigabitEthernet4" --network-descriptions "Data network 1" "Data network 2" "Data network 3" "Data network 4"
Network GigabitEthernet4 is not currently defined. Create it? [y] y

Result:

user@ubuntu-server:~$ cot info csr1000v-universalk9.16.03.05.napalm.4IF.ova | egrep "(Networks|Gigabit)"
Networks:
  GigabitEthernet1  "Data network 1"
  GigabitEthernet2  "Data network 2"
  GigabitEthernet3  "Data network 3"
  GigabitEthernet4  "Data network 4"
NICs and Associated Networks:
  GigabitEthernet1 : GigabitEthernet1
  GigabitEthernet2 : GigabitEthernet2
  GigabitEthernet3 : GigabitEthernet3
  GigabitEthernet4 : GigabitEthernet4
  Management Interface                                    "GigabitEthernet1"

Add another Interface (short version)
It’s possible to replace the long enumerations („GigabitEthernet1“ „GigabitEthernet2“ „GigabitEthernet3“ „GigabitEthernet4“) by a macro „GigabitEthernet{1}“ => the Variable {1} will get expanded to 1…2…3…4…up to the number needed. This is much more flexible when sometimes 4, sometimes 5 NICs need to be added.

user@ubuntu-server:~$ cot edit-hardware csr1000v-universalk9.16.03.05.napalm.ova -o csr1000v-universalk9.16.03.05.napalm.4IFb.ova -n 4 --nic-type vmxnet3 --nic-networks "GigabitEthernet{1}" --network-descriptions "Data network {1}"
Network GigabitEthernet4 is not currently defined. Create it? [y]

Same result as before:

user@ubuntu-server:~$ cot info csr1000v-universalk9.16.03.05.napalm.4IFb.ova | egrep "(Networks|Gigabit)"
Networks:
  GigabitEthernet1  "Data network 1"
  GigabitEthernet2  "Data network 2"
  GigabitEthernet3  "Data network 3"
  GigabitEthernet4  "Data network 4"
NICs and Associated Networks:
  GigabitEthernet1 : GigabitEthernet1
  GigabitEthernet2 : GigabitEthernet2
  GigabitEthernet3 : GigabitEthernet3
  GigabitEthernet4 : GigabitEthernet4
  Management Interface                                    "GigabitEthernet1"

Deploy a new Router-VM using this OVA

user@ubuntu-server:~$ cot --verbose deploy csr1000v-universalk9.16.03.05.napalm.4IFb.ova esxi VCENTERIP/Datacenter/host/ESXiIP -u VCENTER-USER -p PASS -n CSR-4IF -d "DS-LAB" -S "telnet://:44444,server" -N GigabitEthernet1="Management" GigabitEthernet2=T24 GigabitEthernet3=T34 GigabitEthernet4=TBB -c 1CPU-4GB
INFO    : vm_description  : Loading 'csr1000v-universalk9.16.03.05.napalm.4IFb.ova' as OVF
INFO    : ovf             : OVF version is 1.x
INFO    : ovf             : OVF product class com.cisco.csr1000v --> platform Cisco CSR1000V
INFO    : vm_description  : Successfully loaded OVF from csr1000v-universalk9.16.03.05.napalm.4IFb.ova
WARNING : deploy          : No serial connectivity information is available for 1 serial port(s) - they will not be created or configured.
INFO    : deploy_esxi     : Deploying VM...
NOTICE  : helper          : Calling 'ovftool --deploymentOption=1CPU-4GB --net:GigabitEthernet1=Management --net:GigabitEthernet2=T24 --net:GigabitEthernet3=T34 --net:GigabitEthernet4=TBB --name=CSR-4IF --datastore=DS-LAB csr1000v-universalk9.16.03.05.napalm.4IFb.ova vi://administrator@lab.local:VMware1!@192.168.2.13/Datacenter/host/192.168.2.12'...
Opening OVA source: csr1000v-universalk9.16.03.05.napalm.4IFb.ova
The manifest validates
Opening VI target: vi://administrator%40lab.local@192.168.2.13:443/Datacenter/host/192.168.2.12
Deploying to VI: vi://administrator%40lab.local@192.168.2.13:443/Datacenter/host/192.168.2.12
Transfer Completed
Completed successfully
NOTICE  : helper          : ...done
INFO    : deploy_esxi     : FIXING up serial ports on deployed VM...
INFO    : deploy_esxi     : Serial port will be a telnet server at :44444
INFO    : deploy_esxi     : Done with serial port fixup

Isn’t it beautiful?

edit Hardware“: 4x GigabitEthernet-Interfaces

Test the created CSR1000v-Router
The forth GigabitEthernet-Interface was detected during hardware-setup:

NAPALM-1#show ip int brief
Interface              IP-Address      OK? Method Status                Protocol
GigabitEthernet1       192.168.2.244   YES DHCP   up                    up
GigabitEthernet2       unassigned      YES unset  administratively down down
GigabitEthernet3       unassigned      YES unset  administratively down down
GigabitEthernet4       unassigned      YES unset  administratively down down

The Interface works fine:

NAPALM-1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
NAPALM-1(config)#int gig 4
NAPALM-1(config-if)#ip addr 192.168.64.99 255.255.255.0
NAPALM-1(config-if)#end

NAPALM-1#ping 192.168.64.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.64.1, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 2/3/4 ms

Pull configs from CSR1000v using NAPALM

I usually use an „wr“-command-alias, an EEM-Applet or the IOS-Archive-Feature to copy the running-config to a central repository when an admin enters the „write“-command [copy running-config startup-config].

Sometimes it’s handy to pull the current running-config from a device.

Following Centralized access to device-configuration and other state-information using NAPALM – NAPALM makes this very easy:

import napalm

#credentials
DEVICE="192.168.2.139"
USER="rmond"
PASS="rmondpass"

#instanciate NAPALM
iosdriver = napalm.get_network_driver('ios')

#connect to device
router = iosdriver(hostname=DEVICE, username=USER,  password=PASS, optional_args={'port': 22, 'dest_file_system': 'bootflash:'})
router.open()

#read the hostname from the device
facts = router.get_facts()
hostname= facts["hostname"]

#read the device-config
clishowrun=['show running-config']
configRun=router.cli(clishowrun)[clishowrun[0]]

#disconnet
router.close()

#construct config-filename
filename = hostname+".cfg"

#write config-text into the file
with open(filename, "w") as f:
  print >>f, configRun

Cisco CSR1000v – additional Interfaces? vNIC Hot-Add!

My apprehension – a weird nightmare of having to shutdown the router, add the nic and boot the router again – proved wrong.
vNIC Hot-Add is supported for VMware ESXi, Citrix XenServer and KVM
Not for Hyper-V, Azure, AWS today.
Cisco: Virtual Machine Requirements
…up to 10 vNICs for ESXi btw.

  • Starting with three default-interfaces
IOS-RTR#show ip int brief
Interface              IP-Address      OK? Method Status                Protocol
GigabitEthernet1       192.168.2.13    YES NVRAM  up                    up
GigabitEthernet2       unassigned      YES NVRAM  administratively down down
GigabitEthernet3       unassigned      YES NVRAM  administratively down down
  • select the VM
  • go to „Edit Settings“
vSphere Client: Add Hardware
vSphere Client: Add Hardware
Add Ethernet NIC
Select NIC-Type „VMXNET3“
  • Next, Next, Finish
  • be patient
Dec 16 14:56:10.165: %LINK-5-CHANGED: Interface GigabitEthernet4, changed state to administratively down
Dec 16 14:56:11.166: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4, changed state to down
IOS-RTR#show ip int brief
Interface              IP-Address      OK? Method Status                Protocol
GigabitEthernet1       192.168.2.13    YES NVRAM  up                    up
GigabitEthernet2       unassigned      YES NVRAM  administratively down down
GigabitEthernet3       unassigned      YES NVRAM  administratively down down
GigabitEthernet4       unassigned      YES unset  administratively down down

…Hot-Remove is supported only using a special „two-step“ strategy

Common OVF Tool (COT) – Automated Lab-Router Deployment

Is this SDN (Software defined Networking)? I think so!
I really love to provision new Cisco CSR1000v lab-routers without the need to do everything manually in the graphical user interface.

It’s possible to provision production-routers, too, of course: Just add your license-information to leave the 100Kbps-limit behind (which is no handicap for normal lab-usage).

The Common OVF Tool (COT) (Documentation) enables me to do this by injecting an initial startup-configuration into the cisco-provided OVA-file before deploying it to an ESXi-host.
Interactive Cisco CLI-Commands possible
And, even better, the Cisco CSR1000V allows interactive CLI-commands within the injected configuration:

  • building new bootflash-directories,
  • creating a SSH-key.

Injecting the bootstrap-config

$ cot --version
Common OVF Tool (COT), version 2.0.3
Copyright (C) 2013-2017 the COT project developers.

$ cot inject-config csr1000v-universalk9.03.16.06b.S.155-3.S6b-ext.ova -c ios-napalm.startup.cfg -o csr1000v-universalk9.03.16.06b.S.155-3.S6b-ext.napalm.ova

This is an example-config setting parameters needed for napalm:
Centralized access to device-configuration and other state-information using NAPALM
Automated RMON Alarm/Event-configuration for class-based QoS-Monitoring using NAPALM

!
hostname NAPALM-1
ip domain-name lab.local
!
no ip domain-lookup
!
crypto key generate rsa modulus 2048
!
username rmond privilege 15 secret rmondpass
!
platform console serial
!
vrf definition MGMT
  address-family ipv4
!
int gig 1
  descr mgmt0
  vrf forwarding MGMT
  ip address dhcp
  no shut
!
ip scp server enable
!
ip access-list standard ACL_SNMP
    permit host 192.168.2.89
!
snmp-server community READ ro ACL_SNMP
snmp-server location allones.de
!
file prompt quiet
!
do mkdir bootflash:/ARCHIVE
archive
  path bootflash:/ARCHIVE/bak
!
line vty 0 4
  login local
  transport input ssh
!
end

Deploy the CSR1000V-Router
One command to deploy the router

  • at an ESXi-Host/vSphere-Datacenter
  • using a defined Datastore
  • creating an serial-port (not really needed since SSH is running immedeately)
  • place the three NICs of this router at the desired vSwitch-Portgroups

Those vSwitch-Portgroups have been created using vSphere-CLI.

$ cot --verbose deploy csr1000v-universalk9.03.16.06b.S.155-3.S6b-ext.napalm.ova esxi //host/ -u -p -n RTR-NAPALM-1 -d -S "telnet://:31001,server" -N GigabitEthernet1="Management" GigabitEthernet2=T24 GigabitEthernet3=T34 -c 1CPU-4GB

SSH-Access to the router

NAPALM-1# show ssh
Connection Version Mode Encryption  Hmac         State                 Username
0          2.0     IN   aes256-ctr  hmac-sha1    Session started       rmond
0          2.0     OUT  aes256-ctr  hmac-sha1    Session started       rmond
%No SSHv1 server connections running.

NAPALM-1#who
    Line       User       Host(s)              Idle       Location
*  1 vty 0     rmond      idle                 00:00:00 192.168.2.312

NAPALM-1#cd bootflash:ARCHIVE
NAPALM-1#pwd
bootflash:/ARCHIVE/

NAPALM-1#show crypto key mypubkey all
Key name: NAPALM-1.lab.local
Key type: RSA KEYS
 Storage Device: private-config
 Usage: General Purpose Key
 Key is not exportable. Redundancy enabled.
 Key Data:
  30820222 300D0609 2A864886 F70D0101 01050003 82020F00 3082020A 02820201
  ...
  70F5FE1C 01BE930D B3C84841 AC46EE0D 451DC530 55F28B9C 82796E8F 1B5F5163
  57020301 0001

NAPALM-1#show ip int brief
Interface              IP-Address      OK? Method Status                Protocol
GigabitEthernet1       192.168.2.135   YES DHCP   up                    up
GigabitEthernet2       unassigned      YES unset  administratively down down
GigabitEthernet3       unassigned      YES unset  administratively down down

NAPALM-1#show ip route vrf MGMT
Routing Table: MGMT
Gateway of last resort is 192.168.2.1 to network 0.0.0.0

S*    0.0.0.0/0 [254/0] via 192.168.2.1
      192.168.2.0/24 is variably subnetted, 3 subnets, 2 masks
C        192.168.2.0/24 is directly connected, GigabitEthernet1
S        192.168.2.1/32 [254/0] via 192.168.2.1, GigabitEthernet1
L        192.168.2.315/32 is directly connected, GigabitEthernet1

Automated RMON Alarm/Event-configuration for class-based QoS-Monitoring using NAPALM

In Configure RMON Alarms&Events by script I’ve shown a short python-algorithm to to discover all Cisco class-based QoS (cbQoS) packet-/drop-counters and to generate RMON-alarms for each. The router monitors these counters every 300s, calculates the delta for the interval and raises RMON-events when there were packets/drops or when the have been before but not anymore.

This RMON-event has been configured as an syslog-message to an syslog-receiver etc.

The existing script just generated a list of cli-commands which had to be entered manually to the router-config.
Not a valid aproach when having hundreds devices to be configured.

Now i want the script to automatically configure the router.

  • add both „rmon event“-objects for the rising- and the falling-threshold of the monitored alarms
  • read the existing „rmon alarm“-objects from the device config, which have been configured by this script during a former run
  • remove these existing alarms
  • discover all cbQoS-packet/drop-counters
  • add corresponding „rmon alarm“-objects

I’d like to refer to Centralized access to device-configuration and other state-information using NAPALM for some basic information regarding NAPALM and how to create the „router“-object in python.

NAPALM: Read existing RMON alarms.
I’ll use the following python-logic to

  • remote-execute the command
  • immedeately pull the cli-output out of the python-dictionary: the CLI-Command is the dict-key
>>> cligetrmon=['show rmon alarms | inc RMONevent']
>>> rmonalarms = router.cli(cligetrmon)[cligetrmon[0]]
>>> print rmonalarms
Alarm 10001 is active, owned by RMONevent

Generate CLI to delete these RMON alarms

>>> cmdnormon = ""
>>> for alarm in rmonalarms.split('\n'):
...  alarmid = alarm.split(' ')[1]
...  cmdnormon += "no rmon alarm "+alarmid+"\n"
...
>>>
>>> print cmdnormon
no rmon alarm 10001

Static CLI to add required RMON events

>>> cmdrmonevent = "rmon event 10 log owner RMONevent\n"
>>> cmdrmonevent += "rmon event 11 log owner RMONevent\n"

Read Cisco cbQoS-MIB to fetch interesting QoS-counters, generate CLI for RMON-alarms

>>> from easysnmp import Session
>>> hostname = "192.168.2.72"
>>> session = Session(hostname, community='READ', version=2)
>>>
... cbqos = session.walk('1.3.6.1.4.1.9.9.166.1.15.1.1.13')
>>>
... cmdrmon = ""
>>> alarmID = 10001
>>>
... for i in cbqos:
...   oidList=i.oid.split(".")
...   q=oidList.pop()
...   p=oidList.pop()
...   #print p,q
...   ifTypeID=int(session.get("1.3.6.1.4.1.9.9.166.1.1.1.1.2."+p).value)
...   ifDirID=int(session.get("1.3.6.1.4.1.9.9.166.1.1.1.1.3."+p).value)
...   if (ifDirID==2):
...     cmdrmon += "rmon alarm "+str(alarmID)+" "+i.oid+" 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent\n"
...   alarmID += 1
...

Concatenate all commmands

>>> cmd = cmdrmonevent+cmdnormon+cmdrmon
>>> print cmd
rmon event 10 log owner RMONevent
rmon event 11 log owner RMONevent
no rmon alarm 10001
rmon alarm 10001 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10002 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10003 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10004 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10005 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10006 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent

Push the commands to the Router

>>> router.load_merge_candidate(config=cmd)

Check the differences befor apply the changes

>>> print router.compare_config()
-no rmon alarm 10001
+rmon alarm 10001 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
+rmon alarm 10002 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
+rmon alarm 10003 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
+rmon alarm 10004 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
+rmon alarm 10005 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
+rmon alarm 10006 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent

Verify the pushed commands at the router-CLI

IOS-RTR#dir *.txt
Directory of bootflash:/*.txt

Directory of bootflash:/

   21  -rw-         898  Nov 24 2017 15:05:32 +00:00  merge_config.txt
7835619328 bytes total (6613028864 bytes free)

IOS-RTR#more merge_config.txt
rmon event 10 log owner RMONevent
rmon event 11 log owner RMONevent
no rmon alarm 10001
rmon alarm 10001 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10002 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10003 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10004 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10005 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10006 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent

Commit the changes

>>> router.commit_config()

Or discard them

>>> router.discard_config()

It’s possible to rollback committed changes.

>>> router.rollback()

Finally: Disconnect the session with the device

>>> router.close()

Again: A brief look to the router

IOS-RTR#show run | inc rmon
! Last configuration change at 19:08:59 UTC Fri Nov 24 2017 by rmond
! NVRAM config last updated at 19:09:00 UTC Fri Nov 24 2017 by rmond
username rmond privilege 15 secret 5 $1$7VnE$2O18Vfcr4y7eO5gY7l4xx1
rmon event 10 log owner RMONevent
rmon event 11 log owner RMONevent
rmon alarm 10001 cbQosCMStatsEntry.13.18.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10002 cbQosCMStatsEntry.13.18.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10003 cbQosCMStatsEntry.13.18.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10004 cbQosCMStatsEntry.13.34.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10005 cbQosCMStatsEntry.13.34.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10006 cbQosCMStatsEntry.13.34.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
IOS-RTR#

It’s already written to NVRAM

IOS-RTR#show startup-config | inc rmon
! Last configuration change at 19:08:59 UTC Fri Nov 24 2017 by rmond
! NVRAM config last updated at 19:09:00 UTC Fri Nov 24 2017 by rmond
username rmond privilege 15 secret 5 $1$7VnE$2O18Vfcr4y7eO5gY7l4xx1
rmon event 10 log owner RMONevent
rmon event 11 log owner RMONevent
rmon alarm 10001 cbQosCMStatsEntry.13.18.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10002 cbQosCMStatsEntry.13.18.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10003 cbQosCMStatsEntry.13.18.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10004 cbQosCMStatsEntry.13.34.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10005 cbQosCMStatsEntry.13.34.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10006 cbQosCMStatsEntry.13.34.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent

Centralized access to device-configuration and other state-information using NAPALM

Since I still want to build a centralized solution for automated configuration of RMON-alarms to monitor Cisco cbQoS-packet/drop-counters a solution to retrieve and modify device configurations was needed.

The NAPALM (Network Automation and Programmability Abstraction Layer with Multivendor support) Framework seems to provide the required features.
NAPALM Installation
Some NAPALM-IOS dependencies have to be fulfilled first.

sudo apt-get install -y --force-yes libssl-dev libffi-dev python-dev python-cffi

The „partial installation“ seems to be not working anymore, the full installation uses some KB more ressources, not worth to think about the partial installation…

pip install napalm

IOS Preperation
To allow remote-access to the centralized NAPALM server these features need to be enabled in each IOS device:

  • Remote-Access via SSH,
  • SCP (Secure Copy),
  • the IOS „Archive“-feature is the foundation of NAPALM config-operations.
IOS-RTR#conf t
Enter configuration commands, one per line.  End with CNTL/Z.

! AAA preferred for production-systems, of course
IOS-RTR(config)#username rmond privilege 15 secret rmondpass

! required
IOS-RTR(config)#ip scp server enable

! no annoying [yes/no]-prompts for file-operations anymore
IOS-RTR(config)#file prompt quiet

! create the folder in the filesystem for the Archive
IOS-RTR(config)#do mkdir bootflash:/ARCHIVE

IOS-RTR(config)#archive
IOS-RTR(config-archive)#path bootflash:/ARCHIVE/bak-

Explore Napalm

Go the the python shell:

user@snmp-server:~$ python
>>> import napalm
>>> iosdriver = napalm.get_network_driver('ios')

Autodiscovery of the router-filesystem doesn’t work, i know the filesystem of my router and pass it as „optional argument“ to router-object.
*** todo: troubleshoot/fix the autodiscovery ***

>>> router = iosdriver(hostname='192.168.2.72', username='rmond',  password='rmondpass', 
optional_args={'port': 22, 'dest_file_system': 'bootflash:'})
>>> router.open()

Go to the router cli-shell:

  • user „rmond“ is logged in
IOS-RTR#who
    Line       User       Host(s)              Idle       Location
*  1 vty 0     user       idle                 00:00:00 192.168.2.109
   2 vty 1     rmond      idle                 00:00:02 192.168.2.89

Back to python, try some NAPALM-functions.

>>> print router.get_facts()
{u'os_version': u'CSR1000V Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 15.4(2)S3, RELEASE SOFTWARE (fc2)', 
u'uptime': 657600, u'interface_list': [u'GigabitEthernet1', u'GigabitEthernet2', u'GigabitEthernet3'], u'vendor': u'Cisco', 
u'serial_number': u'afdökjl0123', u'model': u'CSR1000V', u'hostname': u'IOS-RTR', u'fqdn': u'IOS-RTR.lab.local'}

Execute interactive EXEC-Commands.

>>> cliping=['ping 192.168.2.1']
>>> print router.cli(cliping)
{'ping 192.168.2.1': u'Type escape sequence to abort.\n
Sending 5, 100-byte ICMP Echos to 192.168.2.1, timeout is 2 seconds:\n!!!!!\n
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/2 ms'}

Fetch the current running-config.

>>> clishowrun=['show running-config']
>>> print router.cli(clishowrun)
{'show running-config': u'Building configuration...\n\nCurrent configuration : 2411 bytes\n!\n
! Last configuration change at 12:52:32 UTC Fri Nov 24 2017 by user\n! NVRAM config last updated at 10:24:26 UTC Fri Nov 24 2017 by user\n!\n
version 15.4\nservice timestamps debug datetime msec\nservice timestamps log datetime msec\n
...rmon event 10 log owner RMONevent\nrmon event 11 log owner RMONevent\n
rmon alarm 10001 cbQosCMStatsEntry.2.18.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent\n
...
end'}

Read some lines from the running-config, filter using the pipe.

>>> cligetrmon=['show rmon alarms | inc RMONevent']
>>> print router.cli(cligetrmon)
{'show rmon alarms | inc RMONevent': u'Alarm 10001 is active, owned by RMONevent'}

Log-out if you are finished:

>>> router.close()

Linux: SNMPv3 with Python

Works out of the box.

  • if you’re fine with AES128-Encryption.

AES256 might not be a requirement in all cases, but having the opportunity to choose seems to be not absolutely absurd in 2017… Good news: The NetSNMP-AES192/256-patch is on the way.

EasySNMP installation and usage
I’d like to refer to Linux: SNMP with Python for the basics.

IOS-Config: VIEW/GROUP/USER
Take the opportunity and leverage SMP-Views to limit access to several SNMP-OIDs.

snmp-server view SV_EASYSNMP interfaces included
snmp-server view SV_EASYSNMP ciscoCdpMIB included
snmp-server view SV_EASYSNMP ciscoCBQosMIB included

ip access-list standard ACL_SNMP
 permit 192.168.2.89

snmp-server group SG_EASYSNMP v3 auth read SV_EASYSNMP access ACL_SNMP

snmp-server user EASYSNMP SG_EASYSNMP v3 auth sha AUTHPASS priv aes 128 PRIVPASS

EasySNMP: „Session“-Object with SNMPv3-Credentials
Find the official docs here: EasySNMP Session-API

Security level could be:

  • no_auth_or_privacy
    • If you want to use the user-based authentication without need for security
  • auth_without_privacy
    • Authentication only might be „good enough“ when traffic is fully kept within a management network
  • auth_with_privacy

Authentication-Procol:

  • MD5
  • SHA

Privacy-Protocol:

  • AES
    • AES128
  • DES, 3DES
    • For very outdated devices

Create the Session-Object

>>> session3 = Session(hostname='192.168.2.72', version=3,
security_level="auth_with_privacy", security_username="EASYSNMP",
auth_protocol="SHA", auth_password="AUTHPASS",
privacy_protocol="AES", privacy_password="PRIVPASS")

Use this „session“ as before.

>>> session3.walk("1.3.6.1.4.1.9.9.166.1.15.1.1.2")
[<SNMPVariable value='9' (oid='enterprises.9.9.166.1.15.1.1.2.18.65536', oid_index='',
snmp_type='COUNTER')>, <SNMPVariable value='0' (oid='enterprises.9.9.166.1.15.1.1.2.18.131072', oid_index='',
snmp_type='COUNTER')>, <SNMPVariable value='6039' (oid='enterprises.9.9.166.1.15.1.1.2.18.196608', oid_index='', snmp_type='COUNTER')>, <SNMPVariable value='0' (oid='enterprises.9.9.166.1.15.1.1.2.34.65536', oid_index='',
snmp_type='COUNTER')>, <SNMPVariable value='0' (oid='enterprises.9.9.166.1.15.1.1.2.34.131072', oid_index='',
snmp_type='COUNTER')>, <SNMPVariable value='0' (oid='enterprises.9.9.166.1.15.1.1.2.34.196608', oid_index='',
snmp_type='COUNTER')>]