I prefer accessing VMs using SSH or RDP directly. Sometimes, the IP-Address of the VM isn’t reachable, or protocols for remoteaccess need to be disabled for security reasons.
In these cases, if an IP-connection to the ESXi-Server is available this could be an option to use the ESXi hypervisor as VNC-Server to provide access to VM keyboard, video, mouse…
Three VM advanced Configuration Parameters need to be set:
„password“ is optional, but mRemoteNG as VNC-Client doesn’t work without password set.
While the „old“ SCAv1 built a virtual fence around all virtual processors („Intra VM Security Boundary“), SCAv2 lets processors of one virtual machine (VM) to run within a „common fence“ („Inter VM Security Boundary“) which balances security and performance for most workloads.
esxcli system settings kernel set -s hyperthreadingMitigation -v TRUE
esxcli system settings kernel set -s hyperthreadingMitigationIntraVM -v FALSE
Verify
esxcli system settings kernel list -o hyperthreadingMitigation
esxcli system settings kernel list -o hyperthreadingMitigationIntraVM
[root@esx:~] esxcli system settings kernel list -o hyperthreadingMitigation
Name Type Configured Runtime Default Description
------------------------ ---- ---------- ------- ------- ----------------------------------------------------------------
hyperthreadingMitigation Bool TRUE TRUE FALSE Restrict the simultaneous use of logical processors from the
same hyperthreaded core as necessary to mitigate a security
vulnerability.
[root@esx:~] esxcli system settings kernel list -o hyperthreadingMitigationIntraVM
Name Type Configured Runtime Default Description
------------------------------- ---- ---------- ------- ------- ---------------------------------------------------------
hyperthreadingMitigationIntraVM Bool FALSE FALSE TRUE Restrict the simultaneous use of logical processors from
the same hyperthreaded core as necessary to mitigate a
security vulnerability within a single VM.
Configure SCAv1
esxcli system settings kernel set -s hyperthreadingMitigation -v TRUE
esxcli system settings kernel set -s hyperthreadingMitigationIntraVM -v TRUE
VMware wants us to prepopulate an Excel-Sheet with SSH-Keys and SSL-Thumbprints of all ESXi-Hosts, who have been freshly deployed minutes before, which effectivly protects against man-in-the-middle-attacks – maybe a problem in US-datacenters.
Beginning with VMware Cloud Foundation Release VCF 4.2 the Cloud-Builder-App verifies the „CN“ of all ESXi-SSL-Certificates, which is in default-setup set to „localhost“ (to be overwritten when connecting to the vCenter so this is seemed to be no issue) – not accepted, CN has to be set to <server-fqdn>.
Solution
plink.exe
„plink.exe“ from the Putty-Suite allows to be called from PowerShell in automated fashion without interactive Password-Prompt. (In May 2021 there seems to be no other choice for Power-Shell Core 7)
Algorithm
generate „correct“ self-signed certificate with „CN“ set to „fqdn“ not for „localhost“
read the new certificate SSL-sha256-thumbprint
reboot the ESXi-Host to activate the new SSL-Server-Certificate
Result
Correct SSL Server-Certificate
Log for four ESXi-Hosts
Contains SSL-Thumbprints to get copied into the VCF-Excel-Sheet.
Variables
$VMPassword
$VMUsername
have to be prepopulated.
PS T:\vmware vcf4> .\esxi_ssl_ssh.ps1
Generate SSL Self-Signed Certificate [ham01-m01-esx01]
Keyboard-interactive authentication prompts from server:
End of keyboard-interactive prompts from server
Fetch SSL-Thumbprint
Generate SSL Self-Signed Certificate [ham01-m01-esx02]
Keyboard-interactive authentication prompts from server:
End of keyboard-interactive prompts from server
Fetch SSL-Thumbprint
Generate SSL Self-Signed Certificate [ham01-m01-esx03]
Keyboard-interactive authentication prompts from server:
End of keyboard-interactive prompts from server
Fetch SSL-Thumbprint
Generate SSL Self-Signed Certificate [ham01-m01-esx04]
Keyboard-interactive authentication prompts from server:
End of keyboard-interactive prompts from server
Fetch SSL-Thumbprint
*** Result ***
172.16.11.101 ham01-m01-esx01
SSL-Thumbprint: D2:6E:01:AD:36:82:3E:D2:AC:F3:66:6E:27:FC:A5:2C:26:99:57:8D:E6:D9:24:E3:42:61:F3:C3:52:65:8C:36
172.16.11.102 ham01-m01-esx02
SSL-Thumbprint: 21:67:3F:11:E4:FE:F3:D2:D9:C6:C2:66:85:7D:3D:3F:02:49:F2:FE:D6:74:86:E1:8E:BE:CC:A2:66:41:72:D2
172.16.11.103 ham01-m01-esx03
SSL-Thumbprint: F6:D3:12:BD:53:36:F0:E5:FD:C9:F9:3C:41:60:80:79:C8:C4:69:30:52:AF:6C:AF:24:C3:C6:DE:2A:75:80:14
172.16.11.104 ham01-m01-esx04
SSL-Thumbprint: AC:0B:D0:E3:6D:03:12:3F:7E:69:5F:0F:75:F0:F5:F2:E1:59:61:46:83:35:1F:AD:2C:15:9D:EB:C1:9D:EF:DE
this must not be QNAPs fault, maybe the drives (4 disks, RAID5, between 147..167 MB/second for sequential read) are the root cause
adding a SSD-Cache didn’t improve performance when I had a 3-disk RAID 5
many people around the world are troubleshooting this without success
so I decided to
use the QNAP-NAS as a Virtual-Tape-Library for Backup-Purposes etc
use my ESXi-Host as Storage-Server
„Storage Server“ VM
I regularily use „Nested ESXi“-Hosts for Lab Environments which should access the storage, so I decided to first give NFS a try – ESXi has no built in NFS-Server, so a small VM using Free-NAS, True-NAS, I didn’t decide yet, should provide „feature parity“ to the QNAP System.
RDM is no option
Adding my former „QNAP Cache SSD“ to the ESXi-Host showed that it was impossible (as expected) to passthrough this disk to a VM.
RDM is no option since it doesn’t allow direct hardware access, eg. SMART-Counters or other statistics.
PCIe SATA-Controller: Dell PERC H310
A PERC H310 can get used as a passthrough device for a VM which will get full hardware access for up to 8 disks.
as far I remember, this hardware has been released in 2011, but it might still be fast enough to be no bottleneck
IT Firmware
There’s nothing wrong with the original H310-RAID-Firmware, but since I won’t need those features i’d prefer the „IT“-Firmware, which offers much larger buffers to handle bursts easily.
From time to time the first try to connect to a vCenter-Server fails, building a simple loop to allow a limited number of retries fixes this possible issue.
Background: Python-Requests wants to verify https-certificates, which makes sense to me. – but my VCenter uses a self-signed Certificate
$ python
Python 2.7.15rc1 (default, Apr 15 2018, 21:51:34)
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get("https://<vcenter-ip>")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/dist-packages/requests/api.py", line 72, in get
return request('get', url, params=params, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 520, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 630, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 506, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='192.168.2.13', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)'),))
Routers with more than three interfaces? Not that uncommon! It might be handy, if several devices are needed with more than three interfaces, to:
not:
deploy the devices with three interfaces first
add the needed number of interfaces to the routers
attache tne new interfaces to the correct vSphere-Portgroup
but instead:
create an OVA-Template with the correct number of interfaces (one time)
deploy the routers and attach them during deployment to their portgroup
So using vNIC Hot-Add might be not the favourite way to work.
Common OVF Tool COT lets you to deploy Cisco CSR1000v routers by easy to use (linux-)commands. (Common OVF Tool (COT) – Automated Lab-Router Deployment This is my original OVA-file already containing some basic configurations needed for NAPALM.
it will build a virtual router with three GigabitEthernet-Interfaces
Add another Interface (short version) It’s possible to replace the long enumerations („GigabitEthernet1“ „GigabitEthernet2“ „GigabitEthernet3“ „GigabitEthernet4“) by a macro „GigabitEthernet{1}“ => the Variable {1} will get expanded to 1…2…3…4…up to the number needed. This is much more flexible when sometimes 4, sometimes 5 NICs need to be added.
user@ubuntu-server:~$ cot edit-hardware csr1000v-universalk9.16.03.05.napalm.ova -o csr1000v-universalk9.16.03.05.napalm.4IFb.ova -n 4 --nic-type vmxnet3 --nic-networks "GigabitEthernet{1}" --network-descriptions "Data network {1}"
Network GigabitEthernet4 is not currently defined. Create it? [y]
user@ubuntu-server:~$ cot --verbose deploy csr1000v-universalk9.16.03.05.napalm.4IFb.ova esxi VCENTERIP/Datacenter/host/ESXiIP -u VCENTER-USER -p PASS -n CSR-4IF -d "DS-LAB" -S "telnet://:44444,server" -N GigabitEthernet1="Management" GigabitEthernet2=T24 GigabitEthernet3=T34 GigabitEthernet4=TBB -c 1CPU-4GB
INFO : vm_description : Loading 'csr1000v-universalk9.16.03.05.napalm.4IFb.ova' as OVF
INFO : ovf : OVF version is 1.x
INFO : ovf : OVF product class com.cisco.csr1000v --> platform Cisco CSR1000V
INFO : vm_description : Successfully loaded OVF from csr1000v-universalk9.16.03.05.napalm.4IFb.ova
WARNING : deploy : No serial connectivity information is available for 1 serial port(s) - they will not be created or configured.
INFO : deploy_esxi : Deploying VM...
NOTICE : helper : Calling 'ovftool --deploymentOption=1CPU-4GB --net:GigabitEthernet1=Management --net:GigabitEthernet2=T24 --net:GigabitEthernet3=T34 --net:GigabitEthernet4=TBB --name=CSR-4IF --datastore=DS-LAB csr1000v-universalk9.16.03.05.napalm.4IFb.ova vi://administrator@lab.local:VMware1!@192.168.2.13/Datacenter/host/192.168.2.12'...
Opening OVA source: csr1000v-universalk9.16.03.05.napalm.4IFb.ova
The manifest validates
Opening VI target: vi://administrator%40lab.local@192.168.2.13:443/Datacenter/host/192.168.2.12
Deploying to VI: vi://administrator%40lab.local@192.168.2.13:443/Datacenter/host/192.168.2.12
Transfer Completed
Completed successfully
NOTICE : helper : ...done
INFO : deploy_esxi : FIXING up serial ports on deployed VM...
INFO : deploy_esxi : Serial port will be a telnet server at :44444
INFO : deploy_esxi : Done with serial port fixup
Isn’t it beautiful?
Test the created CSR1000v-Router The forth GigabitEthernet-Interface was detected during hardware-setup:
NAPALM-1#show ip int brief
Interface IP-Address OK? Method Status Protocol
GigabitEthernet1 192.168.2.244 YES DHCP up up
GigabitEthernet2 unassigned YES unset administratively down down
GigabitEthernet3 unassigned YES unset administratively down down
GigabitEthernet4 unassigned YES unset administratively down down
The Interface works fine:
NAPALM-1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
NAPALM-1(config)#int gig 4
NAPALM-1(config-if)#ip addr 192.168.64.99 255.255.255.0
NAPALM-1(config-if)#end
NAPALM-1#ping 192.168.64.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.64.1, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 2/3/4 ms
My apprehension – a weird nightmare of having to shutdown the router, add the nic and boot the router again – proved wrong. vNIC Hot-Add is supported for VMware ESXi, Citrix XenServer and KVM Not for Hyper-V, Azure, AWS today. Cisco: Virtual Machine Requirements …up to 10 vNICs for ESXi btw.
Starting with three default-interfaces
IOS-RTR#show ip int brief
Interface IP-Address OK? Method Status Protocol
GigabitEthernet1 192.168.2.13 YES NVRAM up up
GigabitEthernet2 unassigned YES NVRAM administratively down down
GigabitEthernet3 unassigned YES NVRAM administratively down down
select the VM
go to „Edit Settings“
Next, Next, Finish
be patient
Dec 16 14:56:10.165: %LINK-5-CHANGED: Interface GigabitEthernet4, changed state to administratively down
Dec 16 14:56:11.166: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4, changed state to down
IOS-RTR#show ip int brief
Interface IP-Address OK? Method Status Protocol
GigabitEthernet1 192.168.2.13 YES NVRAM up up
GigabitEthernet2 unassigned YES NVRAM administratively down down
GigabitEthernet3 unassigned YES NVRAM administratively down down
GigabitEthernet4 unassigned YES unset administratively down down
…Hot-Remove is supported only using a special „two-step“ strategy