Cloud-Init for VMs in private and public Clouds

Initialize VMs in a vSphere private Cloud using Cloud-Init

Cloud-Init Datasource for VMware GuestInfo is deprecated

The Web is full of explanations, how to use „Cloud-Init Datasource for VMware GuestInfo“ but https://github.com/vmware-archive/cloud-init-vmware-guestinfo it is deprecated.

It is now integrated natively into Cloud-Init

Cloud-Init 21.3 has been released https://discourse.ubuntu.com/t/release-of-cloud-init-21-3/23857 which integrates this software.

New name:

I’d expect it to be in the current Ubuntu 21.10 (Impish Indri) https://cloud-images.ubuntu.com/impish/current/ – the release notes for 21.10 don’t specify the exact version.

Ubuntu 20.04 (Focal Fossa) is still at cloud-init 20.1-10 according to https://wiki.ubuntu.com/FocalFossa/ReleaseNotes which is too old.

Prepare a VM-Template

Unfortunately Ubuntu provides the „cloud“-images in OVA-Format.

Create VM-Template from OVA

Deploy

  • impish-server-cloudimg-amd64.ova

as

  • VM
  • keep all settings set to default

Customize VM

  • disable (or remove) the „Serial“-port
  • disable vApp-Properties
    • those would break the cloud-init process later on
    • VM=>Configure
    • Settings=>vApp-Options
    • disable [ ] vApp-Properties

Convert to Template

  • VM-Template „ubuntu-impish-21.10-cloudimg“.

Clone a VM from this VM-Template

This shouldn’t be done manually, I’d suggest using terraform

Verify that Cloud-Init 21.3 is available and the „VMware“-Datasource is included

Cloud-Init Version

Release 21.3 is available:

ubuntu@ubuntu:~$ cloud-init --version
/usr/bin/cloud-init 21.3-1-g6803368d-0ubuntu3

Check the Cloud-Init Datasource

Datasource „vmware“ is included:

ubuntu@ubuntu:~$ cloud-id
vmware

Enable ESXi-Host as VNC-Server to access a vSphere-VM remotely

I prefer accessing VMs using SSH or RDP directly. Sometimes, the IP-Address of the VM isn’t reachable, or protocols for remoteaccess need to be disabled for security reasons.

In these cases, if an IP-connection to the ESXi-Server is available this could be an option to use the ESXi hypervisor as VNC-Server to provide access to VM keyboard, video, mouse…

Three VM advanced Configuration Parameters need to be set:

  • „password“ is optional, but mRemoteNG as VNC-Client doesn’t work without password set.
RemoteDisplay.vnc.enabled = TRUE
RemoteDisplay.vnc.port = <TCP-Port>
RemoteDisplay.vnc.password = <Passwort>

ESXi 6.7u2 and later – SCAv2 (Side Channel aware Scheduler v2)

While the „old“ SCAv1 built a virtual fence around all virtual processors („Intra VM Security Boundary“), SCAv2 lets processors of one virtual machine (VM) to run within a „common fence“ („Inter VM Security Boundary“) which balances security and performance for most workloads.

See https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/performance/scheduler-options-vsphere67u2-perf.pdf for performance analysis.

Configure

esxcli system settings kernel set -s hyperthreadingMitigation -v TRUE
esxcli system settings kernel set -s hyperthreadingMitigationIntraVM -v FALSE

Verify

esxcli system settings kernel list -o hyperthreadingMitigation
esxcli system settings kernel list -o hyperthreadingMitigationIntraVM
[root@esx:~] esxcli system settings kernel list -o hyperthreadingMitigation
Name                      Type  Configured  Runtime  Default  Description
------------------------  ----  ----------  -------  -------  ----------------------------------------------------------------
hyperthreadingMitigation  Bool  TRUE        TRUE     FALSE    Restrict the simultaneous use of logical processors from the
                                                              same hyperthreaded core as necessary to mitigate a security
                                                              vulnerability.
[root@esx:~] esxcli system settings kernel list -o hyperthreadingMitigationIntraVM
Name                             Type  Configured  Runtime  Default  Description
-------------------------------  ----  ----------  -------  -------  ---------------------------------------------------------
hyperthreadingMitigationIntraVM  Bool  FALSE       FALSE    TRUE     Restrict the simultaneous use of logical processors from
                                                                     the same hyperthreaded core as necessary to mitigate a
                                                                     security vulnerability within a single VM.

Configure SCAv1

esxcli system settings kernel set -s hyperthreadingMitigation -v TRUE
esxcli system settings kernel set -s hyperthreadingMitigationIntraVM -v TRUE

Determine Power-CLI Object-Type

Background

If you needed to pass some „Objects“ to a Power-Shell function you need the name of their Types.

Example

Four Parameters need to be passed to the function:

  • two Objects
    1. VI-Server
    2. Virtual-Machine
  • two Strings:
    1. diskName
    2. diskSize
function resizeVMDisk { Param([VMware.VimAutomation.ViCore.Types.V1.VIServer]$server, [VMware.VimAutomation.ViCore.Types.V1.Inventory.VirtualMachine]$vm, [string]$diskName, [string]$diskSize)

Algorithm

  1. Find an existing PowerCLI-Function which already uses the Object-Parameter you need.

In my case:

Get-VMHost

2) Pull the MetaData of this Command.

3) Display the „Attributes“ of the relevant Object-Parameter

4) Look for „Target Type List“

Result

„VM“ = [VMware.VimAutomation.ViCore.Types.V1.Inventory.VirtualMachine]

„Server“ = [VMware.VimAutomation.ViCore.Types.V1.VIServer]

Example

C:\LAB> $cmd = Get-Command -Module VMware.VimAutomation.Core -Name Get-VMHost
C:\LAB> $meta = New-Object System.Management.Automation.CommandMetadata ($cmd)
C:\LAB> $meta

Name                    : Get-VMHost
CommandType             : VMware.VimAutomation.ViCore.Cmdlets.Commands.GetVMHost
DefaultParameterSetName : Default
SupportsShouldProcess   : False
SupportsPaging          : False
PositionalBinding       : True
SupportsTransactions    : False
HelpUri                 :
RemotingCapability      : PowerShell
ConfirmImpact           : None
Parameters              : {[Name, System.Management.Automation.ParameterMetadata], [NoRecursion, System.Management.Automation.ParameterMetadata], [VM,
                          System.Management.Automation.ParameterMetadata], [ResourcePool, System.Management.Automation.ParameterMetadata]…}


C:\LAB> $meta.Parameters["VM"].Attributes

Position                        : -2147483648
ParameterSetName                : SecondaryParameterSet
ValueFromPipeline               : True
Mandatory                       : False
ExperimentName                  :
ExperimentAction                : None
ValueFromPipelineByPropertyName : False
ValueFromRemainingArguments     : False
HelpMessage                     :
HelpMessageBaseName             :
HelpMessageResourceId           :
DontShow                        : False
TypeId                          : System.Management.Automation.ParameterAttribute

TypeId : System.Management.Automation.ValidateNotNullOrEmptyAttribute

Critical                        : True
ContextPrincipalName            :
ContextOptional                 : False
ContextRelationName             :
TargetTypeList                  : {VMware.VimAutomation.ViCore.Types.V1.Inventory.VirtualMachine}
TransformNullOptionalParameters : True
TypeId                          : VMware.VimAutomation.Sdk.Util10Ps.BaseCmdlet.ObnArgumentTransformationAttribute


C:\LAB> $meta.Parameters["Server"].Attributes

Position                        : -2147483648
ParameterSetName                : Default
ValueFromPipeline               : False
Mandatory                       : False
ExperimentName                  :
ExperimentAction                : None
ValueFromPipelineByPropertyName : False
ValueFromRemainingArguments     : False
HelpMessage                     :
HelpMessageBaseName             :
HelpMessageResourceId           :
DontShow                        : False
TypeId                          : System.Management.Automation.ParameterAttribute

Position                        : -2147483648
ParameterSetName                : ById
ValueFromPipeline               : False
Mandatory                       : False
ExperimentName                  :
ExperimentAction                : None
ValueFromPipelineByPropertyName : False
ValueFromRemainingArguments     : False
HelpMessage                     :
HelpMessageBaseName             :
HelpMessageResourceId           :
DontShow                        : False
TypeId                          : System.Management.Automation.ParameterAttribute

Position                        : -2147483648
ParameterSetName                : DistributedSwitch
ValueFromPipeline               : False
Mandatory                       : False
ExperimentName                  :
ExperimentAction                : None
ValueFromPipelineByPropertyName : False
ValueFromRemainingArguments     : False
HelpMessage                     :
HelpMessageBaseName             :
HelpMessageResourceId           :
DontShow                        : False
TypeId                          : System.Management.Automation.ParameterAttribute

Position                        : -2147483648
ParameterSetName                : SecondaryParameterSet
ValueFromPipeline               : False
Mandatory                       : False
ExperimentName                  :
ExperimentAction                : None
ValueFromPipelineByPropertyName : False
ValueFromRemainingArguments     : False
HelpMessage                     :
HelpMessageBaseName             :
HelpMessageResourceId           :
DontShow                        : False
TypeId                          : System.Management.Automation.ParameterAttribute

Critical                        : True
ContextPrincipalName            :
ContextOptional                 : False
ContextRelationName             :
TargetTypeList                  : {VMware.VimAutomation.ViCore.Types.V1.VIServer}
TransformNullOptionalParameters : True
TypeId                          : VMware.VimAutomation.Sdk.Util10Ps.BaseCmdlet.ObnArgumentTransformationAttribute

VMware ESXi: Generate Self-Signed Certificate for FQDN and retrieve SSL-Thumbprint

Background

VMware wants us to prepopulate an Excel-Sheet with SSH-Keys and SSL-Thumbprints of all ESXi-Hosts, who have been freshly deployed minutes before, which effectivly protects against man-in-the-middle-attacks – maybe a problem in US-datacenters.

Beginning with VMware Cloud Foundation Release VCF 4.2 the Cloud-Builder-App verifies the „CN“ of all ESXi-SSL-Certificates, which is in default-setup set to „localhost“ (to be overwritten when connecting to the vCenter so this is seemed to be no issue) – not accepted, CN has to be set to <server-fqdn>.

VCF PreCheck: SSL Certificate CN Error

Solution

plink.exe

„plink.exe“ from the Putty-Suite allows to be called from PowerShell in automated fashion without interactive Password-Prompt. (In May 2021 there seems to be no other choice for Power-Shell Core 7)

Algorithm

  1. generate „correct“ self-signed certificate with „CN“ set to „fqdn“ not for „localhost“
  2. read the new certificate SSL-sha256-thumbprint
  3. reboot the ESXi-Host to activate the new SSL-Server-Certificate

Result

Correct SSL Server-Certificate

Server SSL-Certificate with correct CN

Log for four ESXi-Hosts

Contains SSL-Thumbprints to get copied into the VCF-Excel-Sheet.

Variables

  • $VMPassword
  • $VMUsername

have to be prepopulated.

PS T:\vmware vcf4> .\esxi_ssl_ssh.ps1
Generate SSL Self-Signed Certificate [ham01-m01-esx01]
Keyboard-interactive authentication prompts from server:
End of keyboard-interactive prompts from server
Fetch SSL-Thumbprint
Generate SSL Self-Signed Certificate [ham01-m01-esx02]
Keyboard-interactive authentication prompts from server:
End of keyboard-interactive prompts from server
Fetch SSL-Thumbprint
Generate SSL Self-Signed Certificate [ham01-m01-esx03]
Keyboard-interactive authentication prompts from server:
End of keyboard-interactive prompts from server
Fetch SSL-Thumbprint
Generate SSL Self-Signed Certificate [ham01-m01-esx04]
Keyboard-interactive authentication prompts from server:
End of keyboard-interactive prompts from server
Fetch SSL-Thumbprint

*** Result ***
172.16.11.101 ham01-m01-esx01
SSL-Thumbprint: D2:6E:01:AD:36:82:3E:D2:AC:F3:66:6E:27:FC:A5:2C:26:99:57:8D:E6:D9:24:E3:42:61:F3:C3:52:65:8C:36
172.16.11.102 ham01-m01-esx02
SSL-Thumbprint: 21:67:3F:11:E4:FE:F3:D2:D9:C6:C2:66:85:7D:3D:3F:02:49:F2:FE:D6:74:86:E1:8E:BE:CC:A2:66:41:72:D2
172.16.11.103 ham01-m01-esx03
SSL-Thumbprint: F6:D3:12:BD:53:36:F0:E5:FD:C9:F9:3C:41:60:80:79:C8:C4:69:30:52:AF:6C:AF:24:C3:C6:DE:2A:75:80:14
172.16.11.104 ham01-m01-esx04
SSL-Thumbprint: AC:0B:D0:E3:6D:03:12:3F:7E:69:5F:0F:75:F0:F5:F2:E1:59:61:46:83:35:1F:AD:2C:15:9D:EB:C1:9D:EF:DE

PowerShell Sourcecode

$NestedESXiHosts = @{
    "ham01-m01-esx01"=@{"vmk0"="172.16.11.101"};
    "ham01-m01-esx02"=@{"vmk0"="172.16.11.102"};
    "ham01-m01-esx03"=@{"vmk0"="172.16.11.103"};
    "ham01-m01-esx04"=@{"vmk0"="172.16.11.104"};
}

$NestedESXiHosts.GetEnumerator() | Sort-Object -Property key | Foreach-Object {
	$VMName = $_.Key
	$VMIPAddress = $_.Value.vmk0
	
	write-host -ForegroundColor Green "Generate SSL Self-Signed Certificate [$VMName]"
	#
	$SSLThumbPrint = echo y | plink -ssh -pw $VMPassword $VMUsername@$VMIPAddress "/sbin/generate-certificates;openssl x509 -in /etc/vmware/ssl/rui.crt -fingerprint -sha256 -noout;reboot;"
    #
	write-host -ForegroundColor Green "Fetch SSL-Thumbprint"
	#
	$SSLThumbPrint = $SSLThumbPrint.split("=")[1]
	$_.Value.SSL = $SSLThumbPrint
}

write-host
write-host -ForegroundColor Green "*** Result ***"

$NestedESXiHosts.GetEnumerator() | Sort-Object -Property key | Foreach-Object {
	$VMName = $_.Key
	$VMIPAddress = $_.Value.vmk0
	$VMSSL = $_.Value.SSL
	write-host -ForegroundColor Green "$VMIPAddress $VMName"
	write-host "SSL-Thumbprint: $VMSSL"
}              

Virtual Machine with direct Storage Access

Business Case

My QNAP-NAS (TS-431X2) is slow

  • this must not be QNAPs fault, maybe the drives (4 disks, RAID5, between 147..167 MB/second for sequential read) are the root cause
  • adding a SSD-Cache didn’t improve performance when I had a 3-disk RAID 5
    • many people around the world are troubleshooting this without success

so I decided to

  1. use the QNAP-NAS as a Virtual-Tape-Library for Backup-Purposes etc
  2. use my ESXi-Host as Storage-Server

„Storage Server“ VM

I regularily use „Nested ESXi“-Hosts for Lab Environments which should access the storage, so I decided to first give NFS a try – ESXi has no built in NFS-Server, so a small VM using Free-NAS, True-NAS, I didn’t decide yet, should provide „feature parity“ to the QNAP System.

RDM is no option

Adding my former „QNAP Cache SSD“ to the ESXi-Host showed that it was impossible (as expected) to passthrough this disk to a VM.

RDM is no option since it doesn’t allow direct hardware access, eg. SMART-Counters or other statistics.

PCIe SATA-Controller: Dell PERC H310

A PERC H310 can get used as a passthrough device for a VM which will get full hardware access for up to 8 disks.

  • as far I remember, this hardware has been released in 2011, but it might still be fast enough to be no bottleneck

IT Firmware

There’s nothing wrong with the original H310-RAID-Firmware, but since I won’t need those features i’d prefer the „IT“-Firmware, which offers much larger buffers to handle bursts easily.

This firmware has to be programmed, a lot of guides for this exist – I had to combine https://www.vladan.fr/flash-dell-perc-h310-with-it-firmware/ providing a link to a nice Dell-Firmware („6gbpsas.fw“) and a precise description what to do – but the „megarec.exe“ found in the archive didn’t work at my Asus mainboard („Error 8192“) so I found another megarec-release https://www.taste-of-it.de/flash-dell-perc-h310-mit-lsi-9211-8i-it-mode/ here.

[root@esx:~] esxcli hardware pci list | egrep -B8 -A26 "Device Name: Dell"
0000:06:00.0
   Address: 0000:06:00.0
   Segment: 0x0000
   Bus: 0x06
   Slot: 0x00
   Function: 0x0
   VMkernel Name: vmhba2
   Vendor Name: LSI Logic / Symbios Logic
   Device Name: Dell 6Gbps SAS HBA Adapter
   Configured Owner: VMkernel
   Current Owner: VMkernel
   Vendor ID: 0x1000
   Device ID: 0x0072
   SubVendor ID: 0x1028
   SubDevice ID: 0x1f1c
   Device Class: 0x0107
   Device Class Name: Serial Attached SCSI controller
   Programming Interface: 0x00
   Revision ID: 0x03
   Interrupt Line: 0x05
   IRQ: 5
   Interrupt Vector: 0x35
   PCI Pin: 0x00
   Spawned Bus: 0x00
   Flags: 0x3201
   Module ID: 4161
   Module Name: mpt2sas
   Chassis: 0
   Physical Slot: 1
   Slot Description: PCIE1
   Passthru Capable: true
   Parent Device: PCI 0:0:28:0
   Dependent Device: PCI 0:6:0:0
   Reset Method: Function reset
   FPT Sharable: true

[root@esx:~] esxcli device driver list
Device   Driver    Status  KB Article
-------  --------  ------  ----------
vmnic0   igbn      normal
vmhba2   mpt2sas   normal
vmnic2   ixgben    normal
vmhba1   vmw_ahci  normal
vmhba32  vmkusb    normal
vmnic1   igbn      normal
vmhba0   vmw_ahci  normal

[root@esx:~] esxtop
 4:48:13pm up 19 min, 852 worlds, 5 VMs, 17 vCPUs; CPU load average: 0.03, 0.03, 0.12

 ADAPTR PATH                 NPTH AQLEN   CMDS/s  READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd
 vmhba0 -                       1   992     0.77     0.19     0.58     0.00     0.00     0.71     0.03     0.74     0.01
 vmhba1 -                       1   992     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00
 vmhba2 -                       1   600     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00
vmhba64 -                       0  1024     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00

PCIe-Passthrough

Passthrough has to be enabled manually, reboot the host and voila:

Dell PERC H310 IT-Firmware: Passthrough active

Now this device can get added to _one_ VM:

Add other device: PCI device
New PCI Device: Controller Passthrough

The (Ubuntu 20.04 Server-)VM seems to be fine with this setup:

Disk as seen by the VM

vSphere vCenter – PowerShell: reliable Connection

From time to time the first try to connect to a vCenter-Server fails, building a simple loop to allow a limited number of retries fixes this possible issue.

$vSphereServer = "vcsa.local"
$vSphereUser = "administrator@vsphere.local"
$vSpherePassword = "********"

$result = @{}

$retries = 6
$viConnection = $null
#
while ($viConnection -eq $null) {
    # My-Logger 'Connecting to Management vCenter Server $vSphereServer ...'
    #
    $error.clear()
    $viConnection = Connect-VIServer $vSphereServer -User $vSphereUser -Password $vSpherePassword -WarningAction SilentlyContinue
    #
    if ($viConnection -eq $null) {
        $retries = $retries - 1
        if ($retries -eq 0) {
            throw ("Connecting to vCenter ($vSphereServer) failed ($vSphereUser): $error")
        }
        Start-Sleep -s 10
    }
}
$result.viConnection = $viConnection

Disclaimer

Since i’m trying to get rid of Evernote, too anoying too often, i’ll start to document non-private-stuff here.

COT – Common OVF-Tool – Import of VCenter HTTPS Certificate

Background: Python-Requests wants to verify https-certificates, which makes sense to me.
– but my VCenter uses a self-signed Certificate

$ python
Python 2.7.15rc1 (default, Apr 15 2018, 21:51:34)
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get("https://<vcenter-ip>")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 520, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 630, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 506, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='192.168.2.13', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)'),))

Don’t think about ignoring certificate errors 😉

Prerequisites

sudo apt-get install unzip

Download the certificate from the VCenter:

wget --no-check-certificate https://<vcenter-ip>/certs/download
mv download download.zip
unzip download.zip

Install the certificate:

cd certs
$ ls -l
total 8
-rw-r--r-- 1 administrator administrator 1294 Apr 29  2016 d2f59c0d.0
-rw-r--r-- 1 administrator administrator  702 Oct 15 14:36 d2f59c0d.r0

$ export REQUESTS_CA_BUNDLE=d2f59c0d.0

Now Python.requests validates the vCenter:

$ python
Python 2.7.15rc1 (default, Apr 15 2018, 21:51:34)
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get("https://<vcenter-ip>")
<Response [200]>

Add the Export to ~/.bashrc

sudo vi ~/.bashrc

at the end of the file

export REQUESTS_CA_BUNDLE=~/certs/d2f59c0d.0

Cisco CSR1000v – additional Interfaces? Common OVF Tool (COT)!

Routers with more than three interfaces? Not that uncommon!
It might be handy, if several devices are needed with more than three interfaces, to:

  • not:
    • deploy the devices with three interfaces first
    • add the needed number of interfaces to the routers
    • attache tne new interfaces to the correct vSphere-Portgroup
  • but instead:
    • create an OVA-Template with the correct number of interfaces (one time)
    • deploy the routers and attach them during deployment to their portgroup

So using vNIC Hot-Add might be not the favourite way to work.

Common OVF Tool
COT lets you to deploy Cisco CSR1000v routers by easy to use (linux-)commands.
(Common OVF Tool (COT) – Automated Lab-Router Deployment
This is my original OVA-file already containing some basic configurations needed for NAPALM.

  • it will build a virtual router with three GigabitEthernet-Interfaces
user@ubuntu-server:~$ cot info csr1000v-universalk9.16.03.05.napalm.ova | egrep "(Networks|Gigabit)"
Networks:
  GigabitEthernet1  "Data network 1"
  GigabitEthernet2  "Data network 2"
  GigabitEthernet3  "Data network 3"
NICs and Associated Networks:
  GigabitEthernet1 : GigabitEthernet1
  GigabitEthernet2 : GigabitEthernet2
  GigabitEthernet3 : GigabitEthernet3
  Management Interface                                    "GigabitEthernet1"

Add another Interface (long version)
Let’s use the prepared OVA and create another OVA with 4 interfaces:

user@ubuntu-server:~$ cot edit-hardware csr1000v-universalk9.16.03.05.napalm.ova -o csr1000v-universalk9.16.03.05.napalm.4IF.ova -n 4 --nic-type vmxnet3 --nic-networks "GigabitEthernet1" "GigabitEthernet2" "GigabitEthernet3" "GigabitEthernet4" --network-descriptions "Data network 1" "Data network 2" "Data network 3" "Data network 4"
Network GigabitEthernet4 is not currently defined. Create it? [y] y

Result:

user@ubuntu-server:~$ cot info csr1000v-universalk9.16.03.05.napalm.4IF.ova | egrep "(Networks|Gigabit)"
Networks:
  GigabitEthernet1  "Data network 1"
  GigabitEthernet2  "Data network 2"
  GigabitEthernet3  "Data network 3"
  GigabitEthernet4  "Data network 4"
NICs and Associated Networks:
  GigabitEthernet1 : GigabitEthernet1
  GigabitEthernet2 : GigabitEthernet2
  GigabitEthernet3 : GigabitEthernet3
  GigabitEthernet4 : GigabitEthernet4
  Management Interface                                    "GigabitEthernet1"

Add another Interface (short version)
It’s possible to replace the long enumerations („GigabitEthernet1“ „GigabitEthernet2“ „GigabitEthernet3“ „GigabitEthernet4“) by a macro „GigabitEthernet{1}“ => the Variable {1} will get expanded to 1…2…3…4…up to the number needed. This is much more flexible when sometimes 4, sometimes 5 NICs need to be added.

user@ubuntu-server:~$ cot edit-hardware csr1000v-universalk9.16.03.05.napalm.ova -o csr1000v-universalk9.16.03.05.napalm.4IFb.ova -n 4 --nic-type vmxnet3 --nic-networks "GigabitEthernet{1}" --network-descriptions "Data network {1}"
Network GigabitEthernet4 is not currently defined. Create it? [y]

Same result as before:

user@ubuntu-server:~$ cot info csr1000v-universalk9.16.03.05.napalm.4IFb.ova | egrep "(Networks|Gigabit)"
Networks:
  GigabitEthernet1  "Data network 1"
  GigabitEthernet2  "Data network 2"
  GigabitEthernet3  "Data network 3"
  GigabitEthernet4  "Data network 4"
NICs and Associated Networks:
  GigabitEthernet1 : GigabitEthernet1
  GigabitEthernet2 : GigabitEthernet2
  GigabitEthernet3 : GigabitEthernet3
  GigabitEthernet4 : GigabitEthernet4
  Management Interface                                    "GigabitEthernet1"

Deploy a new Router-VM using this OVA

user@ubuntu-server:~$ cot --verbose deploy csr1000v-universalk9.16.03.05.napalm.4IFb.ova esxi VCENTERIP/Datacenter/host/ESXiIP -u VCENTER-USER -p PASS -n CSR-4IF -d "DS-LAB" -S "telnet://:44444,server" -N GigabitEthernet1="Management" GigabitEthernet2=T24 GigabitEthernet3=T34 GigabitEthernet4=TBB -c 1CPU-4GB
INFO    : vm_description  : Loading 'csr1000v-universalk9.16.03.05.napalm.4IFb.ova' as OVF
INFO    : ovf             : OVF version is 1.x
INFO    : ovf             : OVF product class com.cisco.csr1000v --> platform Cisco CSR1000V
INFO    : vm_description  : Successfully loaded OVF from csr1000v-universalk9.16.03.05.napalm.4IFb.ova
WARNING : deploy          : No serial connectivity information is available for 1 serial port(s) - they will not be created or configured.
INFO    : deploy_esxi     : Deploying VM...
NOTICE  : helper          : Calling 'ovftool --deploymentOption=1CPU-4GB --net:GigabitEthernet1=Management --net:GigabitEthernet2=T24 --net:GigabitEthernet3=T34 --net:GigabitEthernet4=TBB --name=CSR-4IF --datastore=DS-LAB csr1000v-universalk9.16.03.05.napalm.4IFb.ova vi://administrator@lab.local:VMware1!@192.168.2.13/Datacenter/host/192.168.2.12'...
Opening OVA source: csr1000v-universalk9.16.03.05.napalm.4IFb.ova
The manifest validates
Opening VI target: vi://administrator%40lab.local@192.168.2.13:443/Datacenter/host/192.168.2.12
Deploying to VI: vi://administrator%40lab.local@192.168.2.13:443/Datacenter/host/192.168.2.12
Transfer Completed
Completed successfully
NOTICE  : helper          : ...done
INFO    : deploy_esxi     : FIXING up serial ports on deployed VM...
INFO    : deploy_esxi     : Serial port will be a telnet server at :44444
INFO    : deploy_esxi     : Done with serial port fixup

Isn’t it beautiful?

edit Hardware“: 4x GigabitEthernet-Interfaces

Test the created CSR1000v-Router
The forth GigabitEthernet-Interface was detected during hardware-setup:

NAPALM-1#show ip int brief
Interface              IP-Address      OK? Method Status                Protocol
GigabitEthernet1       192.168.2.244   YES DHCP   up                    up
GigabitEthernet2       unassigned      YES unset  administratively down down
GigabitEthernet3       unassigned      YES unset  administratively down down
GigabitEthernet4       unassigned      YES unset  administratively down down

The Interface works fine:

NAPALM-1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
NAPALM-1(config)#int gig 4
NAPALM-1(config-if)#ip addr 192.168.64.99 255.255.255.0
NAPALM-1(config-if)#end

NAPALM-1#ping 192.168.64.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.64.1, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 2/3/4 ms

Cisco CSR1000v – additional Interfaces? vNIC Hot-Add!

My apprehension – a weird nightmare of having to shutdown the router, add the nic and boot the router again – proved wrong.
vNIC Hot-Add is supported for VMware ESXi, Citrix XenServer and KVM
Not for Hyper-V, Azure, AWS today.
Cisco: Virtual Machine Requirements
…up to 10 vNICs for ESXi btw.

  • Starting with three default-interfaces
IOS-RTR#show ip int brief
Interface              IP-Address      OK? Method Status                Protocol
GigabitEthernet1       192.168.2.13    YES NVRAM  up                    up
GigabitEthernet2       unassigned      YES NVRAM  administratively down down
GigabitEthernet3       unassigned      YES NVRAM  administratively down down
  • select the VM
  • go to „Edit Settings“

vSphere Client: Add Hardware
vSphere Client: Add Hardware

Add Ethernet NIC

Select NIC-Type „VMXNET3“

  • Next, Next, Finish
  • be patient
Dec 16 14:56:10.165: %LINK-5-CHANGED: Interface GigabitEthernet4, changed state to administratively down
Dec 16 14:56:11.166: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4, changed state to down
IOS-RTR#show ip int brief
Interface              IP-Address      OK? Method Status                Protocol
GigabitEthernet1       192.168.2.13    YES NVRAM  up                    up
GigabitEthernet2       unassigned      YES NVRAM  administratively down down
GigabitEthernet3       unassigned      YES NVRAM  administratively down down
GigabitEthernet4       unassigned      YES unset  administratively down down

…Hot-Remove is supported only using a special „two-step“ strategy