Recently, we carried out our first expansion of a Nutanix cluster with two virtual switch (VS). During this operation, we encountered an issue related to the connectivity of our new nodes.
First of all, it’s important to note that we are following one of Nutanix’s recommended network best practices for AHV. This specific setup is not illustrated with a diagram, we are following the “4 × 10 GbE (2 + 2) and 2 × 1 GbE separated” scenario. With this configuration, CVM traffic is separated from virtual machine traffic. The setup can be understood as follows :
• vs0 is dedicated to CVM traffic
• vs1 is dedicated to virtual machine traffic
After reinstalling the image on our machines using Nutanix Foundation, we did some tuning to prepare them for integration with vs1:
• br1 was created by AHV
• Two interfaces were removed from vs0 by the CVM and assigned to the newly created bridge. Yes, by default, all four ports are assigned to vs0.
We’re ready, so we launch the cluster expansion through Prism Element. Our node is added to the cluster, but there’s a problem, an inconsistency is detected on vs1. Our node doesn’t come up.
We already know that from the Network view in Prism Element, there’s a button labeled “Convert Bridges to VS”, but once clicked, no bridge is available for conversion.
We continued our investigation and reached out to support. They applied KB13104 and executed the commands listed in the AOS 6.0 and below section.
acli host.enter_maintenance_mode X.X.X.X
acli net.list_virtual_switch
acli net.disable_virtual_switch
acli net.migrate_br_to_virtual_switch br0 vs_name=vs0
acli net.migrate_br_to_virtual_switch br1 vs_name=vs1
acli host.exit_maintenance_mode X.X.X.X
Woohoo! It works! However… we’re a bit cautious about disabling our vSwitches. The KB say « This should not cause a production impact » but we’re still looking for a safety margin.
This CLI-based solution seems to be the equivalent of the “Convert Bridges to VS” option.
So, we kept digging and reached out the Nutanix engineer team. Their response was the following:
« if bridges are created, and bond mode, mtu is matching the nodes. We just need to update virtual switch from the PE to add it on the new node. No need to disable the vs »
Their response encouraged us to update the configuration using the GUI. So. We gave it a try:
We navigated to Network Configuration and proceeded to modify our virtual switch vs1.
We selected the nodes in our cluster and assigned the appropriate interfaces to each.
And there it is. we’re greeted by an API error…
Let’s not lose hope. There’s more to that message (suspense!)…
« when configuration change is done for the VS, the cluster will push the config to hosts. When host is just added to the cluster, only br0 config will be pulled by cluster, and the node will be added to vs0 if config matches. Any other bridge will not be added to the vs until manually configuration from the PE.
kb13104 portal.nutanix.com/kb/13104 – the kb provides an acli command. If the update from PE fails, then we can use acli command AOS 6.5 or above »
OK, so, only br0 is retrieved, and nodes are added to vs0. If its configuration matches the existing one, all other additional bridges must be manually added via Prism Element. We just tried this through the GUI but not through the CLI.
In fact, we were encouraged to test with acli commands, referring to the same KB provided by support. We already knew that br1 was properly configured (we applied the same configuration on all our new nodes during tuning), so that wasn’t the issue since our initial integration with support had worked. We kept one completely clean node, freshly imaged, performed its addition to the cluster, and ran the commands from the AOS 6.5 or above section.
Warning: after running these commands, the new nodes reboot one after the other.
We list all hosts in our cluster to retrieve their UUIDs:
acli host.list
We specify the virtual switch to update; the UUIDs of all our hosts (since the virtual switch is distributed across the nodes in our cluster, we need to update the configuration on all nodes); and the network interfaces to use:
acli net.update_virtual_switch vs1 host_uplink_config=”{host1 UUID:[active interface];host2 UUID:[active interface];host2 UUID:[active interface];host3 UUID:[active interface];host4 UUID:[active interface];hostN UUID:[active interface]}”
This new node was successfully integrated into vs1.
We now have two solutions :
• The first involves converting the bridges into virtual switches. This requires disabling the virtual switch to add our new nodes.
• The second adds our nodes to the existing configuration. We then reconfigure our virtual switch after the expansion to include them.
In both cases, these steps must be performed after the nodes have been added to the cluster.
