Automation | the broadcast

2017-12-192017-12-16

Pull configs from CSR1000v using NAPALM

I usually use an „wr“-command-alias, an EEM-Applet or the IOS-Archive-Feature to copy the running-config to a central repository when an admin enters the „write“-command [copy running-config startup-config].

Sometimes it’s handy to pull the current running-config from a device.

Following Centralized access to device-configuration and other state-information using NAPALM – NAPALM makes this very easy:

import napalm

#credentials
DEVICE="192.168.2.139"
USER="rmond"
PASS="rmondpass"

#instanciate NAPALM
iosdriver = napalm.get_network_driver('ios')

#connect to device
router = iosdriver(hostname=DEVICE, username=USER,  password=PASS, optional_args={'port': 22, 'dest_file_system': 'bootflash:'})
router.open()

#read the hostname from the device
facts = router.get_facts()
hostname= facts["hostname"]

#read the device-config
clishowrun=['show running-config']
configRun=router.cli(clishowrun)[clishowrun[0]]

#disconnet
router.close()

#construct config-filename
filename = hostname+".cfg"

#write config-text into the file
with open(filename, "w") as f:
  print >>f, configRun

2017-12-062018-10-08

Common OVF Tool (COT) – Automated Lab-Router Deployment

Is this SDN (Software defined Networking)? I think so!
I really love to provision new Cisco CSR1000v lab-routers without the need to do everything manually in the graphical user interface.

It’s possible to provision production-routers, too, of course: Just add your license-information to leave the 100Kbps-limit behind (which is no handicap for normal lab-usage).

The Common OVF Tool (COT) (Documentation) enables me to do this by injecting an initial startup-configuration into the cisco-provided OVA-file before deploying it to an ESXi-host.
Interactive Cisco CLI-Commands possible
And, even better, the Cisco CSR1000V allows interactive CLI-commands within the injected configuration:

building new bootflash-directories,
creating a SSH-key.

Injecting the bootstrap-config

$ cot --version
Common OVF Tool (COT), version 2.0.3
Copyright (C) 2013-2017 the COT project developers.

$ cot inject-config csr1000v-universalk9.03.16.06b.S.155-3.S6b-ext.ova -c ios-napalm.startup.cfg -o csr1000v-universalk9.03.16.06b.S.155-3.S6b-ext.napalm.ova

This is an example-config setting parameters needed for napalm:
Centralized access to device-configuration and other state-information using NAPALM
Automated RMON Alarm/Event-configuration for class-based QoS-Monitoring using NAPALM

!
hostname NAPALM-1
ip domain-name lab.local
!
no ip domain-lookup
!
crypto key generate rsa modulus 2048
!
username rmond privilege 15 secret rmondpass
!
platform console serial
!
vrf definition MGMT
  address-family ipv4
!
int gig 1
  descr mgmt0
  vrf forwarding MGMT
  ip address dhcp
  no shut
!
ip scp server enable
!
ip access-list standard ACL_SNMP
    permit host 192.168.2.89
!
snmp-server community READ ro ACL_SNMP
snmp-server location allones.de
!
file prompt quiet
!
do mkdir bootflash:/ARCHIVE
archive
  path bootflash:/ARCHIVE/bak
!
line vty 0 4
  login local
  transport input ssh
!
end

Deploy the CSR1000V-Router
One command to deploy the router

at an ESXi-Host/vSphere-Datacenter
using a defined Datastore
creating an serial-port (not really needed since SSH is running immedeately)
place the three NICs of this router at the desired vSwitch-Portgroups

Those vSwitch-Portgroups have been created using vSphere-CLI.

$ cot --verbose deploy csr1000v-universalk9.03.16.06b.S.155-3.S6b-ext.napalm.ova esxi //host/ -u -p -n RTR-NAPALM-1 -d -S "telnet://:31001,server" -N GigabitEthernet1="Management" GigabitEthernet2=T24 GigabitEthernet3=T34 -c 1CPU-4GB

SSH-Access to the router

NAPALM-1# show ssh
Connection Version Mode Encryption  Hmac         State                 Username
0          2.0     IN   aes256-ctr  hmac-sha1    Session started       rmond
0          2.0     OUT  aes256-ctr  hmac-sha1    Session started       rmond
%No SSHv1 server connections running.

NAPALM-1#who
    Line       User       Host(s)              Idle       Location
*  1 vty 0     rmond      idle                 00:00:00 192.168.2.312

NAPALM-1#cd bootflash:ARCHIVE
NAPALM-1#pwd
bootflash:/ARCHIVE/

NAPALM-1#show crypto key mypubkey all
Key name: NAPALM-1.lab.local
Key type: RSA KEYS
 Storage Device: private-config
 Usage: General Purpose Key
 Key is not exportable. Redundancy enabled.
 Key Data:
  30820222 300D0609 2A864886 F70D0101 01050003 82020F00 3082020A 02820201
  ...
  70F5FE1C 01BE930D B3C84841 AC46EE0D 451DC530 55F28B9C 82796E8F 1B5F5163
  57020301 0001

NAPALM-1#show ip int brief
Interface              IP-Address      OK? Method Status                Protocol
GigabitEthernet1       192.168.2.135   YES DHCP   up                    up
GigabitEthernet2       unassigned      YES unset  administratively down down
GigabitEthernet3       unassigned      YES unset  administratively down down

NAPALM-1#show ip route vrf MGMT
Routing Table: MGMT
Gateway of last resort is 192.168.2.1 to network 0.0.0.0

S*    0.0.0.0/0 [254/0] via 192.168.2.1
      192.168.2.0/24 is variably subnetted, 3 subnets, 2 masks
C        192.168.2.0/24 is directly connected, GigabitEthernet1
S        192.168.2.1/32 [254/0] via 192.168.2.1, GigabitEthernet1
L        192.168.2.315/32 is directly connected, GigabitEthernet1

2017-12-052017-11-25

Automated RMON Alarm/Event-configuration for class-based QoS-Monitoring using NAPALM

In Configure RMON Alarms&Events by script I’ve shown a short python-algorithm to to discover all Cisco class-based QoS (cbQoS) packet-/drop-counters and to generate RMON-alarms for each. The router monitors these counters every 300s, calculates the delta for the interval and raises RMON-events when there were packets/drops or when the have been before but not anymore.

This RMON-event has been configured as an syslog-message to an syslog-receiver etc.

The existing script just generated a list of cli-commands which had to be entered manually to the router-config.
Not a valid aproach when having hundreds devices to be configured.

Now i want the script to automatically configure the router.

add both „rmon event“-objects for the rising- and the falling-threshold of the monitored alarms
read the existing „rmon alarm“-objects from the device config, which have been configured by this script during a former run
remove these existing alarms
discover all cbQoS-packet/drop-counters
add corresponding „rmon alarm“-objects

I’d like to refer to Centralized access to device-configuration and other state-information using NAPALM for some basic information regarding NAPALM and how to create the „router“-object in python.

NAPALM: Read existing RMON alarms.
I’ll use the following python-logic to

remote-execute the command
immedeately pull the cli-output out of the python-dictionary: the CLI-Command is the dict-key

>>> cligetrmon=['show rmon alarms | inc RMONevent']
>>> rmonalarms = router.cli(cligetrmon)[cligetrmon[0]]
>>> print rmonalarms
Alarm 10001 is active, owned by RMONevent

Generate CLI to delete these RMON alarms

>>> cmdnormon = ""
>>> for alarm in rmonalarms.split('\n'):
...  alarmid = alarm.split(' ')[1]
...  cmdnormon += "no rmon alarm "+alarmid+"\n"
...
>>>
>>> print cmdnormon
no rmon alarm 10001

Static CLI to add required RMON events

>>> cmdrmonevent = "rmon event 10 log owner RMONevent\n"
>>> cmdrmonevent += "rmon event 11 log owner RMONevent\n"

Read Cisco cbQoS-MIB to fetch interesting QoS-counters, generate CLI for RMON-alarms

>>> from easysnmp import Session
>>> hostname = "192.168.2.72"
>>> session = Session(hostname, community='READ', version=2)
>>>
... cbqos = session.walk('1.3.6.1.4.1.9.9.166.1.15.1.1.13')
>>>
... cmdrmon = ""
>>> alarmID = 10001
>>>
... for i in cbqos:
...   oidList=i.oid.split(".")
...   q=oidList.pop()
...   p=oidList.pop()
...   #print p,q
...   ifTypeID=int(session.get("1.3.6.1.4.1.9.9.166.1.1.1.1.2."+p).value)
...   ifDirID=int(session.get("1.3.6.1.4.1.9.9.166.1.1.1.1.3."+p).value)
...   if (ifDirID==2):
...     cmdrmon += "rmon alarm "+str(alarmID)+" "+i.oid+" 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent\n"
...   alarmID += 1
...

Concatenate all commmands

>>> cmd = cmdrmonevent+cmdnormon+cmdrmon
>>> print cmd
rmon event 10 log owner RMONevent
rmon event 11 log owner RMONevent
no rmon alarm 10001
rmon alarm 10001 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10002 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10003 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10004 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10005 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10006 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent

Push the commands to the Router

>>> router.load_merge_candidate(config=cmd)

Check the differences befor apply the changes

>>> print router.compare_config()
-no rmon alarm 10001
+rmon alarm 10001 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
+rmon alarm 10002 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
+rmon alarm 10003 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
+rmon alarm 10004 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
+rmon alarm 10005 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
+rmon alarm 10006 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent

Verify the pushed commands at the router-CLI

IOS-RTR#dir *.txt
Directory of bootflash:/*.txt

Directory of bootflash:/

   21  -rw-         898  Nov 24 2017 15:05:32 +00:00  merge_config.txt
7835619328 bytes total (6613028864 bytes free)

IOS-RTR#more merge_config.txt
rmon event 10 log owner RMONevent
rmon event 11 log owner RMONevent
no rmon alarm 10001
rmon alarm 10001 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10002 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10003 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.18.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10004 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10005 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10006 iso.3.6.1.4.1.9.9.166.1.15.1.1.13.34.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent

Commit the changes

>>> router.commit_config()

Or discard them

>>> router.discard_config()

It’s possible to rollback committed changes.

>>> router.rollback()

Finally: Disconnect the session with the device

>>> router.close()

Again: A brief look to the router

IOS-RTR#show run | inc rmon
! Last configuration change at 19:08:59 UTC Fri Nov 24 2017 by rmond
! NVRAM config last updated at 19:09:00 UTC Fri Nov 24 2017 by rmond
username rmond privilege 15 secret 5 $1$7VnE$2O18Vfcr4y7eO5gY7l4xx1
rmon event 10 log owner RMONevent
rmon event 11 log owner RMONevent
rmon alarm 10001 cbQosCMStatsEntry.13.18.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10002 cbQosCMStatsEntry.13.18.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10003 cbQosCMStatsEntry.13.18.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10004 cbQosCMStatsEntry.13.34.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10005 cbQosCMStatsEntry.13.34.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10006 cbQosCMStatsEntry.13.34.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
IOS-RTR#

It’s already written to NVRAM

IOS-RTR#show startup-config | inc rmon
! Last configuration change at 19:08:59 UTC Fri Nov 24 2017 by rmond
! NVRAM config last updated at 19:09:00 UTC Fri Nov 24 2017 by rmond
username rmond privilege 15 secret 5 $1$7VnE$2O18Vfcr4y7eO5gY7l4xx1
rmon event 10 log owner RMONevent
rmon event 11 log owner RMONevent
rmon alarm 10001 cbQosCMStatsEntry.13.18.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10002 cbQosCMStatsEntry.13.18.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10003 cbQosCMStatsEntry.13.18.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10004 cbQosCMStatsEntry.13.34.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10005 cbQosCMStatsEntry.13.34.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10006 cbQosCMStatsEntry.13.34.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent

2017-12-042017-11-25

Centralized access to device-configuration and other state-information using NAPALM

Since I still want to build a centralized solution for automated configuration of RMON-alarms to monitor Cisco cbQoS-packet/drop-counters a solution to retrieve and modify device configurations was needed.

The NAPALM (Network Automation and Programmability Abstraction Layer with Multivendor support) Framework seems to provide the required features.
NAPALM Installation
Some NAPALM-IOS dependencies have to be fulfilled first.

sudo apt-get install -y --force-yes libssl-dev libffi-dev python-dev python-cffi

The „partial installation“ seems to be not working anymore, the full installation uses some KB more ressources, not worth to think about the partial installation…

pip install napalm

IOS Preperation
To allow remote-access to the centralized NAPALM server these features need to be enabled in each IOS device:

Remote-Access via SSH,
SCP (Secure Copy),
the IOS „Archive“-feature is the foundation of NAPALM config-operations.

IOS-RTR#conf t
Enter configuration commands, one per line.  End with CNTL/Z.

! AAA preferred for production-systems, of course
IOS-RTR(config)#username rmond privilege 15 secret rmondpass

! required
IOS-RTR(config)#ip scp server enable

! no annoying [yes/no]-prompts for file-operations anymore
IOS-RTR(config)#file prompt quiet

! create the folder in the filesystem for the Archive
IOS-RTR(config)#do mkdir bootflash:/ARCHIVE

IOS-RTR(config)#archive
IOS-RTR(config-archive)#path bootflash:/ARCHIVE/bak-

Explore Napalm

Go the the python shell:

user@snmp-server:~$ python
>>> import napalm
>>> iosdriver = napalm.get_network_driver('ios')

Autodiscovery of the router-filesystem doesn’t work, i know the filesystem of my router and pass it as „optional argument“ to router-object.
*** todo: troubleshoot/fix the autodiscovery ***

>>> router = iosdriver(hostname='192.168.2.72', username='rmond',  password='rmondpass', 
optional_args={'port': 22, 'dest_file_system': 'bootflash:'})
>>> router.open()

Go to the router cli-shell:

user „rmond“ is logged in

IOS-RTR#who
    Line       User       Host(s)              Idle       Location
*  1 vty 0     user       idle                 00:00:00 192.168.2.109
   2 vty 1     rmond      idle                 00:00:02 192.168.2.89

Back to python, try some NAPALM-functions.

>>> print router.get_facts()
{u'os_version': u'CSR1000V Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 15.4(2)S3, RELEASE SOFTWARE (fc2)', 
u'uptime': 657600, u'interface_list': [u'GigabitEthernet1', u'GigabitEthernet2', u'GigabitEthernet3'], u'vendor': u'Cisco', 
u'serial_number': u'afdökjl0123', u'model': u'CSR1000V', u'hostname': u'IOS-RTR', u'fqdn': u'IOS-RTR.lab.local'}

Execute interactive EXEC-Commands.

>>> cliping=['ping 192.168.2.1']
>>> print router.cli(cliping)
{'ping 192.168.2.1': u'Type escape sequence to abort.\n
Sending 5, 100-byte ICMP Echos to 192.168.2.1, timeout is 2 seconds:\n!!!!!\n
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/2 ms'}

Fetch the current running-config.

>>> clishowrun=['show running-config']
>>> print router.cli(clishowrun)
{'show running-config': u'Building configuration...\n\nCurrent configuration : 2411 bytes\n!\n
! Last configuration change at 12:52:32 UTC Fri Nov 24 2017 by user\n! NVRAM config last updated at 10:24:26 UTC Fri Nov 24 2017 by user\n!\n
version 15.4\nservice timestamps debug datetime msec\nservice timestamps log datetime msec\n
...rmon event 10 log owner RMONevent\nrmon event 11 log owner RMONevent\n
rmon alarm 10001 cbQosCMStatsEntry.2.18.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent\n
...
end'}

Read some lines from the running-config, filter using the pipe.

>>> cligetrmon=['show rmon alarms | inc RMONevent']
>>> print router.cli(cligetrmon)
{'show rmon alarms | inc RMONevent': u'Alarm 10001 is active, owned by RMONevent'}

Log-out if you are finished:

>>> router.close()

2017-11-302017-11-22

Linux: vSphere CLI Installation

Sometimes I need a lightweight straight-forward toolset to provision, modify or delete vSphere-Objects.

Installation was a nightmare in former times, but vSphere CLI Release 6.5 works with Ubuntu server 16.04.3 LTS „out of the box“.

And – it’s compatible with ESXi 6.0 hosts. (don’t waste time trying to install vSphere CLI Release 6.0 on a current Linux Server)

Consider the docs:
vSphere CLI Documentation
VMware [Code] vSphere-CLI 6.5

Download the 64-bit Archive
Use the vmware-search or Google…
„Download VMware vSphere Command Line Interface 6.5“ might be a promising query.

I downloaded VMware-vSphere-CLI-6.5.0-4566394.x86_64.tar.gz using my Windows Machine.

Transfer the Archive to the Linux-VM using SCP
Since SCP is available at the Linux-VM and this protocol is fast and secure i don’t see the requirement to think about alternative protocols.

I’d suggest to use „PSCP“ from the PuTTY-Suite.

c:\temp>pscp -2 VMware-vSphere-CLI-6.5.0-4566394.x86_64.tar.gz USER_NAME@LINUX_VM_IP:VMware-vSphere-CLI-6.5.0-4566394.x86_64.tar.gz
USER_NAME@LINUX-VM-IP's password: USER_PASS
VMware-vSphere-CLI-6.5.0- | 52264 kB | 3266.5 kB/s | ETA: 00:00:00 | 100%

Disable unneded SSH-Localization
The VMware-Installer seems to not expect non-US-Computers, so avoid automatic special characters from Germany in the SSH-shell.

user@snmp-server:~$ sudo cp /etc/ssh/ssh_config /etc/ssh/ssh_config.bak

Just comment the „SendEnv LANG LC_*“-line:

user@snmp-server:~$ sudo joe /etc/ssh/ssh_config

like this:

user@snmp-server:~$ diff /etc/ssh/ssh_config.bak /etc/ssh/ssh_config
53c53
<     SendEnv LANG LC_*
---
> #    SendEnv LANG LC_*

Install Prerequisites
Taken from Installing Prerequisite Software for Linux Systems with Internet Access „Ubuntu 16.04 64-bit“-Section:

sudo apt-get install lib32z1 lib32ncurses5 build-essential uuid uuid-dev libssl-dev perl-doc libxml-libxml-perl libcrypt-ssleay-perl libsoap-lite-perl libmodule-build-perl

Install vSphere CLI

user@snmp-server:~$ tar xzf VMware-vSphere-CLI-6.5.0-4566394.x86_64.tar.gz

user@snmp-server:~$ sudo vmware-vsphere-cli-distrib/vmware-install.pl
Creating a new vSphere CLI installer database using the tar4 format.

Installing vSphere CLI 6.5.0 build-4566394 for Linux.

You must read and accept the vSphere CLI End User License Agreement to
continue.
Press enter to display it.

VMware® vSphere Software Development Kit License Agreement

…

Do you accept? (yes/no) yes

Thank you.
WARNING: The http_proxy environment variable is not set. If your system is
using a proxy for Internet access, you must set the http_proxy environment
variable .

If your system has direct Internet access, you can ignore this warning .

WARNING: The ftp_proxy environment variable is not set.  If your system is
using a proxy for Internet access, you must set the ftp_proxy environment
variable .

If your system has direct Internet access, you can ignore this warning .

Please wait while configuring CPAN ...

Below mentioned modules with their version needed to be installed,
these modules are available in your system but vCLI need specific
version to run properly

Module: ExtUtils::MakeMaker, Version: 6.96
Module: Module::Build, Version: 0.4205
Module: Net::FTP, Version: 2.77
Module: LWP::Protocol::https, Version: 6.04
Do you want to continue? (yes/no) yes

Be patient, do something else in the meantime…


        Please wait while configuring perl modules using CPAN ...

CPAN is downloading and installing pre-requisite Perl module "Devel::StackTrace" .
CPAN is downloading and installing pre-requisite Perl module "Class::Data::Inheritable" .
CPAN is downloading and installing pre-requisite Perl module "Convert::ASN1" .
CPAN is downloading and installing pre-requisite Perl module "Crypt::OpenSSL::RSA" .
CPAN is downloading and installing pre-requisite Perl module "Crypt::X509" .
CPAN is downloading and installing pre-requisite Perl module "Exception::Class" .
CPAN is downloading and installing pre-requisite Perl module "UUID::Random" .
CPAN is downloading and installing pre-requisite Perl module "Archive::Zip" .
CPAN is downloading and installing pre-requisite Perl module "Path::Class" .
CPAN is downloading and installing pre-requisite Perl module "Class::MethodMaker" .
CPAN is downloading and installing pre-requisite Perl module "UUID" .
CPAN is downloading and installing pre-requisite Perl module "Data::Dump" .
CPAN is downloading and installing pre-requisite Perl module "Socket6 " .
CPAN is downloading and installing pre-requisite Perl module "IO::Socket::INET6" .
CPAN is downloading and installing pre-requisite Perl module "Net::INET6Glue" .

In which directory do you want to install the executable files? [/usr/bin]

Please wait while copying vSphere CLI files...

The installation of vSphere CLI 6.5.0 build-4566394 for Linux completed
successfully. You can decide to remove this software from your system at any
time by invoking the following command:
"/usr/bin/vmware-uninstall-vSphere-CLI.pl".

This installer has successfully installed both vSphere CLI and the vSphere SDK for Perl.

Enjoy,
--the VMware team

Give it a try: Add a vSwitch with 5 Portgroups

user@snmp-server:~$ vicfg-vswitch --help

Synopsis: /usr/bin/vicfg-vswitch OPTIONS [<vswitch>]


Command-specific options:
   --add
    -a
          Add a new virtual switch
   --add-dvp-uplink
    -P
          Add an uplink adapter (pnic) to a DVPort (valid for vSphere 4.0 and later)
   --add-pg
    -A
          Add a portgroup to a virtual switch
...

Define credentials to login into the vSphere-Environment

user@snmp-server:~$ export VI_SERVER=%SERVER_IP_OR_HOSTNAME%
user@snmp-server:~$ export VI_USERNAME=%VSPHERE_USER_NAME%
user@snmp-server:~$ export VI_PASSWORD=%VSPHERE_USER_PASS%

Create the Switch and the portgroups

user@snmp-server:~$ vicfg-vswitch --add "RTR_LAB" -h %ESXi-HOST-IP%

user@snmp-server:~$ vicfg-vswitch -A "T12" "RTR_LAB" -h %ESXi-HOST-IP%
user@snmp-server:~$ vicfg-vswitch -A "T13" "RTR_LAB" -h %ESXi-HOST-IP%
user@snmp-server:~$ vicfg-vswitch -A "T24" "RTR_LAB" -h %ESXi-HOST-IP%
user@snmp-server:~$ vicfg-vswitch -A "T34" "RTR_LAB" -h %ESXi-HOST-IP%
user@snmp-server:~$ vicfg-vswitch -A "T45" "RTR_LAB" -h %ESXi-HOST-IP%

Isn’t it beautiful? 😉

And i think, even with downloading and installing vSphere-CLI it was faster than GUI-clicking to create this vSwitch 😉

2017-11-292017-11-24

Configure RMON Alarms&Events by script

Getting back to the original task..
Use a script on a centralized Controller-VM to figure out for which SNMP-OID RMON-Alarms should get configured

Get all current QoS-Drop-Counters, check the traffic-direction to monitor only outbount-queues, generate RMON-Alarms.

hostname = "192.168.2.72"

session = Session(hostname, community='READ', version=2)

cbqos = session.walk('1.3.6.1.4.1.9.9.166.1.15.1.1.13')

cmds = ["Configure on Host \""+hostname+"\"\n---"]
cmds.append("rmon event 10 log owner RMONevent")
cmds.append("rmon event 11 log owner RMONevent")

alarmID = 10001

for i in cbqos:
  oidList=i.oid.split(".")
  q=oidList.pop()
  p=oidList.pop()
  #print p,q
  ifTypeID=int(session.get("1.3.6.1.4.1.9.9.166.1.1.1.1.2."+p).value)
  ifDirID=int(session.get("1.3.6.1.4.1.9.9.166.1.1.1.1.3."+p).value)
  if (ifDirID==2):
    cmds.append("rmon alarm "+str(alarmID)+" "+i.oid+" 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent")
  alarmID += 1

for cmd in cmds:
  print cmd

Example Output:

Configure on Host "192.168.2.72"
---
rmon event 10 log owner RMONevent
rmon event 11 log owner RMONevent
rmon alarm 10001 enterprises.9.9.166.1.15.1.1.13.18.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10002 enterprises.9.9.166.1.15.1.1.13.18.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10003 enterprises.9.9.166.1.15.1.1.13.18.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10004 enterprises.9.9.166.1.15.1.1.13.34.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10005 enterprises.9.9.166.1.15.1.1.13.34.131072 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent
rmon alarm 10006 enterprises.9.9.166.1.15.1.1.13.34.196608 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent

Todo: Verify existing RMON-Alarm/Event-Configuration at the device
Todo: Push the config automatically to the device

2017-11-272017-11-21

Exploring the SNMP-MIB for Class-based QoS

Discover the OIDs representing the counter-values of all active traffic-classes

Ciscos „SNMP Object Navigator“ (http://mibs.cloudapps.cisco.com/ITDIT/MIBS/servlet/index) is our friend to get the base-OID when you know the name of the MIB:

Object-NAME <=> Object-ID (OID)
„cbQosCMStatsEntry“ <=> „1.3.6.1.4.1.9.9.166.1.15.1.1“

Each object is a set of all counters from „show policy-map interface“-command, the Object Navigator documents the ID of these counters, too.
„Exploring the SNMP-MIB for Class-based QoS“ weiterlesen

2017-11-242017-11-21

Refresher: RMON @ Cisco IOS

RMON Refresher
Think about this given Router-Configuration:

class-map match-all CM_VOIP_CTRL
 match dscp af31
class-map match-all CM_VOIP_RTP
 match dscp ef

policy-map PM_OUT
 class CM_VOIP_RTP
  priority percent 10
 class CM_VOIP_CTRL
  bandwidth percent 1
 class class-default
  fair-queue
!
interface GigabitEthernet1
 ip address 192.168.2.72 255.255.255.0
 service-policy output PM_OUT

Three Queues at interface Gig1:

CM_VOIP_RTP
CM_VOIP_CTRL
class-default

with per-Queue-Statistics:

Packet counters
Drop-counters
etc.

In these first examples, i don’t want to wait for queue-drops, i’ll just generate DSCP=EF-Traffic by the ping-command and watch the Queue-Packet-Counters, not Drops.
Configure RMON Alarms and Events
I’ll add two RMON-Events

rmon event 10 log owner RMONevent
rmon event 11 log owner RMONevent

event #10 = rising-threshold – in my example: >1 Packet has been ~~dropped~~ forwarded
event #11 = the falling-threshold – no packets have been…

Than, instruct the Router to have a look at a QoS-counter:

rmon alarm 10001 enterprises.9.9.166.1.15.1.1.2.18.65536 300 delta rising-threshold 1 11 falling-threshold 0 10 owner RMONevent

In the upcoming post I’ll discover the RMON-MIB to illustrate where the „enterprise.9….65536“-Parameter comes from.

This alarm #10001 monitors:

the value the QoS-counter with OID „enterprises.9.9.166.1.15.1.1.2.18.65536“ (Pkt-Counter of the RTP-Queue).
every 300s
watch for delta-values (not for absolute counters which might be interesting when monitoring temperatures, fan-speed etc…)
define a hysteresis:

rising: if the last counter-delta „was <1" and "is now >=1″ – it raises event#11.
falling: if the last counter-delta „was >=1“ and „is now <1" - it raises event#10.

Both events instruct the router to generate a syslog-message.
In production event 10 will be configured without the „log“-option [to do nothing]. This config is for demonstration purpose.

Forward some Traffic

Generate some Traffic (TOS 184 = DSCP 46 = Expedited Forwarding (EF).

IOS-RTR#ping 192.168.2.1 tos 184
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.2.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/5 ms

So – the Queue wasn’t used before:

the old-counter has been „0“
now 5 Packets have been forwarded

This delta-counter „5“ has exceeded rising-threshold „1“:

event#11 should be raised.

When?

we’ll have to wait between 0..300s for the cyclic 300s-alarm-interval to fire

IOS-RTR#
*Nov 20 14:54:39.015: %RMON-5-RISINGTRAP: Rising threshold has been crossed because the value of cbQosCMStatsEntry.2.18.65536 exceeded the rising-threshold value 1

The current counter is „5“:

whithout rtp-data, the next delta-counter will be 0.

Wait for the next 300s interval:

the falling-event#10 should get raised.

IOS-RTR#
*Nov 20 14:59:38.837: %RMON-5-FALLINGTRAP: Falling threshold has been crossed because the value of cbQosCMStatsEntry.2.18.65536 has fallen below the falling-threshold value 0

Works, perfect!

2017-11-232017-11-21

QoS Monitoring: Watch the Queues!?

Stop frequent polling of everything, please!
Last week I had to troubleshoot a network of a customer which was overwhelmed with SNMP-Queries – it wasn’t the first one.
All Switch- and Router-CPUs have been at high level, since every tiny counter was polled at high rate. To provide real-time graphs to the top-level-management. Which hopefully don’t waste time to watch these colourful pictures all day for entertainment purposes.

Doesn’t anybody remember RMON?
Years ago I’ve been teaching routing&switching-classes as a full-time Cisco/BayNetworks/Fluke-instructor, and in every switching class there was a brief explanation about SNMP.

And about RMON.
RFC2819 – RMON (Remote Network Monitoring) MIB

4 out of 9 RMON-groups are available:

Statistics – Real-Time counters
History – not interesting here 😉
Alarms – how to monitor OIDs (statistics-counters for example) by the device itself, incl. a hysteresis
Events – what to do if hysteresis-thresholds are passed.

Covered in 10 slides, and I’m pretty sure.. I’ve explained the difference between SNMP-GET/Polling and RMON-Alarms&Events/Traps and the negative impact of frequent polling.
Only 15 minutes time given to teach this. Might not been enough.

But people still prefer to poll every second the same error counter value instead of waiting for traps indicating the new counter-value.

Don’t watch the queues: Let the devices watch and notify you if something happens.

Upcoming Project: RMON-QOS Controller
I decided to refresh an old project to help people configuring rmon-alarms for Low-Latency-Queuing(LLQ) packet-drops in an automatic fashion.

Since the old code was TCL-based to run on the routers locally [which had advantages, too] I now want a centralized solution, and I want to take the chance to improve my python skills.

Never start to implement before having a design

Brainstorming:

central controller

orchestrate features

discover outbound QoS-classes/queues
configure alarms&events(SNMP/RMON)

listen for events
provide persistent event-storage

distributed intelligence

watch specified (error-)counters
notify the central snmp-manager if something happens
no dumb devices, please, like in OpenFlow, LAN-Emulation or other failed technologies…

The central controller has to be build.

SNMP-/RMON-Agents will provide the distributed intelligence.

Next step: RMON@IOS Refresher

Tomorrow I’ll start with a „RMON@IOS Refresher“ to visualise why you can’t implement RMON without some kind of automation, intelligence or how you call it.

2017-11-222017-11-20

Standardisation: What and how to standardise?

You have to standardise not only configuration objects (features, parameters) within a device or service – you have to standardise the physical- and logical interworking between several device-types.

But first:

System Architecture: Block-Building
Even if the classical three-tier Network-Topology is outdated since full-featured core-switches have been invented some years ago:

don’t: Build one integrated and complex network
do: Divide the network into independent blocks, with defined interfaces to interconnect them…

…and conquer.

Example:

DataCenter

Core [aka Spine]
Server-Access [aka Leaf]

Enterprise

Core
User-Access

Internet-Access
VPN Termination
DMZ/Perimeter-Access

Logical Structure: Device-classes

Create a (small) set of device-classes depending of placement and required features
assign each device to one class
all devices within a class are configured in an identical manner, but (of course) using differentiating IDs (hostnames, IP-Adressses, etc)

Example:

User-Access-Switch
Server-Access-Switch
Server-Core-Switch
Edge-Services-Gateway
Distributed Logical Router

Topology:Use the same interfaces to connect the same entities.

Example:

ports 1..32 = servers
ports 33..48 = infrastructure

ports 33..40 = fabric-extenders
ports 42/43 = firewall
ports 44/45 = uplinks
ports 47/48 = VPC-Peer-Link

Numbering

Router-IDs – each device-class gets a dedicated ID-range
Subnets – differenc classes again (at least three: Access, Transfer, Loopbacks)

might be useful to split access into several subsets (server-access, DMZ/perimeter, user-access)

Naming
Several configuration-objects needs to be identified by name.

Avoid numbered objects (access-lists for example) if possible as numbers aren’t descriptive at all!

Define rules:

CAPITALIZED Object-Names
Underscore (_) within an Object-Names
dot (.) to concatenate Object-Names

Define common prefixes to identify objects of each object-class:

ACL = Access Control List
CM = Class-Map
PM = Policy-Map
PL = Prefix-List

Example: An Access-List for SNMP-Access:
ACL_SNMP
Example: A Class-Map to create a Traffic-Class for QoS-Purposes:
CM_NMM
Example: An Access-List to identify IP-Sources within the Class-Map „CM_NMM“
ACL_CM_NMM

Devices are named-Objects, too!

the simplest naming-scheme might be a prefix derived from the device-class concetenated with a number.

interesting side aspect: should this number be global-unique for all devices or only unique within the device-class?

Don’t use underscores in device-names, use the hyphen instead.

RFC 1912: Common DNS Operational and Configuration Errors

Example: Prefix for device-classes:

UAS = User-Access-Switch
SAS = Server-Access-Switch
SCS = Server-Core-Switch
ESG = Edge-Services-Gateway
DLR = Distributed Logical Router

Example: the third Server-Core-Switch?
SCS-003
never forget to discuss, if number-padding (as in the example) is „needed“, a hyphen between Prefix and ID is wanted8as in my example), if Device-prefixes have to be in fixed-length(as in my example, too), but these are non-technical issues 😉

General rule
If you want to automate: Use large namespaces!

…max. ten switches are needed? Think abount two digits for numbering, better three…

You don’t want to rewrite your automation-workflows to expand namespeaces