Hypervisor Lab

In this section I will review the prep work I did in my lab to get ready to deploy the Nutanix Community Edition into three nested VMs as a HCI based cluster on Proxmox instead of three physical servers. In part 3 I will go over deploying a Nutanix Cluster that uses external storage instead of HCI.

I will be using Nutanix’s Community Edition to deploy the three node HCI cluster since I havent found a way yet to deploy the standard code inside of Nested VMs. The Community Edition is the same code and meant for evaluation purposes, so its a great fit for a nested cluster.

Before we deploy the cluster we need to plan out our AHV VM hardware, logical networking, IPs, DNS entries, and create the three Proxmox VMs.

Virtual Machine Node Hardware Planning

I am going to mock up my three virtual machines to be configured similar to how it would look if I had physical servers. The VM hardware will look like this when we make the VM towards the end of the post:

VM Network Planning

I will be using the 4x 10G ports on each node to mimic a physical server that has 2x dual port 10G NIC cards.

Two of the 10G ports will be placed into a Nutanix Virtual Switch that will support the cluster networking.

The other two 10G ports will be placed into a Nutanix Virtual Switch that will support the VM networking.

The node networking layout will end up looking like this per node:

Logical Network Planning

For the logical networking I will be assigning static IP addresses for AHV, CVM, cluster services, a Nutanix Move VM for V2V migraitons, and a Prism Central VM for multi-cluster management. The VM networks don’t need static IPs since the vNICs will be connected directly to those VLANs.

Here are the IP addresses I will be using for each node:

These will be the cluster IP addresses:

These will be the Nutanix specific management VM IP addresses:

DNS Entries

We need to create DNS entries for all the VLAN 105 hostnames. On my Windows DNS server creating these looked like this:

Download Nutanix CE ISO and Upload to Proxmox

You can download the latest Nutanix CE installer from here with your Nutanix account (make sure to create one if you don’t have one):

Download Community Edition | Nutanix Community

I will be uploading the Phoenix installer ISO to my data1:

Create AHV VMs in Proxmox

Now that we have everything mapped out lets create the three VMs on our Proxmox host.

In order to have the Nutanix CE installer not error out, and LCM updates work after deployment, there are some specific VM configuration settings we need to make sure we set on each VM. You may be able to find a different way to do it, but these are the specific settings that worked for me:

• Legacy BIOS
• Use SATA disks with SSD emulation enabled.
• Manually set Serial Numbers on the SATA disks othewise cluster creation will hang because all disks have the same emulated Serial Number.
• Tag VLAN 105 on the vNIC ports so that we can use the “Native VLAN” on vs0 to avoid any tagging on the Nutanix side for the first virtual switch (vs0). Trying to tag VLAN 105 on vs0 works for the install, but will break LCM updating for some reason in the Community Edition.

Start by creating a new VM. Under General I will be calling my VMs ntxahv0X-hci since after doing HCI I will also have external compute clusters.

Under OS leave it as Linux and select the phoenix ISO:

Under System choose q35 for the machine type, SeaBIOS for the BIOS, and VirtIO SCSI for the SCSI controller:

Under Disks we need to create a 480GB disk for AHV to boto from, a 480GB disk for the CVM to boot from, and a 1T disk for VM storage. Make sure all three are of the SATA type, have SSD emulation enabled, have Discard enabled, and are stored on your data1:

Under CPU give the VM two sockets, 16 cores per socket, and set the CPU type to Host so AHV can see the virtualization extensions. Adjust based on your physical server:

Under Memory give your VM 72GB (or less if you don’t have enough memory in your Proxmox host). I am also going to enable ballooning:

For Network leave the default NIC for now since we will make modifications after the VM creation wizard:

Once the VM is created navigate to its Hardware and add 3x more NICs so that you end up with net0-net3:

Now to make the installer and later LCM updating works edit the first two NICs (net0/net1) and tag VLAN 105 onto them so that we can use the VLAN 105 IPs with the installer ISO:

Now edit the 3rd and 4th NICs (net2/net3) to be disconnected since we dont want them to connect to the network until we are ready to create the VM virtual switch. Leave them with no VLAN tag since we will be tagging the VM VLANs for vs1 from Prism later:

Next we need to use the CLI on our Proxmox host to edit the VMs SATA disks to have a serial number since there currently is no way to do that from the Proxmox GUI. We can use vim to do this from the host shell in Proxmox, or you can SSH to your Proxmox host:

VM configurations are stored under “/etc/pve/qemu-server/XXX.conf” by default with the XXX being the Proxmox VMs unique number. Since my VM is 105 in Proxmox I would use the following commands:

apt-get install vim
vim /etc/pve/qemu-server/135.conf

Inside the VM config file find the lines for the three SATA disks and append onto the end of them “,serial=XXXXX” with a unique number combo in each:

So for my three SATA disks I will just make up a four digit serial number:

Once you have that change saved navigate back to the VM in the Proxmox GUI and you will be able to see each SATA disk has a serial number now:

Next navigate to the VMs Options and make sure sata0 is the first boot device:

Next edit the VM’s SMBIOS settings to give the VM a server identifier and serial number (service tag):

After the SMBIOS is edited that is all thats required for the VM hardware config.

Clone two more copies of the VM so that you can make the three node cluster. After cloning make sure to edit the SATA serial numbers and SMBIOS Serial number to be different, otherwise it can cause issues with the installer seeing duplicate nodes/disks. Here is how I set up my 3x VMs SMBIOS and SATA serial numbers:

When done you should have three identical VMs with slightly different serial numbers:

This concludes all of the planning/VM setup for our three node Nutanix HCI cluster. In the next part we will install, configure, and update AHV.