esx nic and vswitch and port group configuration

[im822]

    I have a esx host with 4 ethernet ports.

I plan to use a ethernet trunk to my cisco switch with each physical ethernet  port and use vlan tagging.

I have two vlans.

The problem I am trying to solve is running out of ports on my virtual switch.

Should I create one virtual switch with 250+ ports and connect all 4 nic cards to the vswitch. Then create a port group for each vlan ?

vSwitch1:
    4 nics
    port groupA
    port groupB

Or

Should I create two virtual switches and connect two nic cards to each virtual switch.
THen create two port groups in each virtual switch for both vlans.

vSwitch1:
    2 nics
    port group A1
    port group B1

vswitch2:
    2 nics
    port group A2
    port group B2

[Viperman]

You could do it either way.  Why do you think that you will need more than 250 ports?  Each virtual NIC takes up one port.  Do you think that you will have more than 250 VM's attached at one time?  You can modify the vswitch to support more than 4000 ports.

[im822]

I just want to be safe and not need to increase the number of ports ever again. We have many hosts with more that one nic card for NLB and heart beat connections etc. WE have about 50 guests on a host at a time. Thanks for your help.

[Viperman]

What do you mean when you say that you have many host using more than 1 NIC for NLB?  Are you talking about the ESX host?  What do you mean by NLB?  Are you talking about running microsoft NLB inside a VM?

[im822]

Viperman your great, thanks for pointing out my failure to be clear  . I was referring to the VM guests I have in my environment , for example a Microsoft NLB cluster for a pair of share point servers.

[Viperman]

Obviously there are numerous ways to do load balancing in either physical or virtual and they really aren't different in either. 

1.  Obviously the best way to do it is to use a third party load balancer like an F5, but you do have an investment in hardware.
2.  You can use Microsoft NLB with 2 vNICs and use Unicast mode
3.  You can use Microsoft NLB with 1 vNIC and use Multicast mode

The difference between the 2 microsoft solutions is that with 1 vNIC all traffic, both production and communication traffic go over the same NIC. with Unicast, there is a dedicated NIC for production traffic and a dedicated NIC for VM communication traffic.  I personally like the 2 vNIC option and like to separate the traffic.

You also need to separate out the 2 vNICs into separate port groups, and obviously need to take into consideration things like vMotion, DRS, and communication traffic.  There is very little traffic that actually flows over the communication network, and it is usually a waste to have 2 physical NICs dedicated to this network, so I personally like to have the production and communications port groups on the same vSwitch.  I would use a different VLAN on each port group and put them on different subnets and make the communication network non-routable both to and from.

[im822]

Thanks for your help.

[cpdohert]

I would use a different VLAN on each port group and put them on different subnets and make the communication network non-routable both to and from.

I was wondering if you could elaborate a bit on the setup you've described here.  I'm trying to duplicate it (I would use multicast NLB instead, but I don't have complete control over the gateway router) and I'm running into some issues.

Specifically, I have two Server 2008R2 VMs, each with two NICs: One NIC on each is in the VM Network port group on vSwitch1 with VLAN ID 10 and has a 10.x.x.x/24 address.  The other NIC is in a ClusterCommunication port group on vSwitch1 with VLAN ID 444 and has a 192.168.44.x/24 address. vSwitch1 has two physical NICs in the host bound to it.

The two ClusterCommunication NICs can't ping each other, which means NLB can't even be configured.  VLAN ID 10 is defined on the physical switch, but VLAN 444 is not (only on the port group/vSwitch).

My understanding is that when two VM NICs are in the same port group on the same vswitch on the same host, they can talk to each other without touching the physical switch, which is what I want.  If that's not happening (or I am incorrect) then setting the two upstream physical ports up with VLAN ID 444 is possible, but I want to avoid having to mess with the physical switches/routers as much as possible.

I'm also concerned about cluster broadcast traffic leaking onto the physical LAN.

Can you point me in the right direction here?  What am I missing?

[Viperman]

I am going to make the assumption that all of these VMs are running on the same host.  Before you can do anything you need to get the ping.  There are a few things involved to get the ping.

1.  The NICs must be connected on each VM
2.  The NICs must be in the same port group.  The port group must be configured with the correct VLAN, but since you aren't going through the physical switch it isn't hugely important and if the NICs are connected to the same port group, it doesn't matter at all.
3.  Make sure you IP and subnet mask are correct.
4.  Verify that you can ping by IP first, worry about name resolution later.  (potentially a host file problem)
5.  do a basic ping, if that doesn't work, then do a traceroute.
6.  check your routing tables on the VM

Come back to me after you do these things.  Ping first, configuration second.

[cpdohert]

Thank you for your quick response, I appreciate your assistance.

All the VMs are on the same host, and I'm pretty sure I've already got all this set up correctly, but let's walk through it:

1. Two VMs (nt98 and nt99), each has two NICs. Both NICs are Connected and Connect at Power On.

2. On each VM, there's a "VM Network" NIC and a "ClusterCommunication" NIC.  The "VM Network" NICs are in the default VM Network port group.  The ClusterCommunication NICs are in the ClusterCommunication port group.

3. On nt98:
VM Network: 10.118.198.48/255.255.255.0; gateway 10.118.198.1
Cluster Communication: 192.168.44.10/255.255.255.0; no gateway defined
On nt99:
VM Network: 10.118.198.49/255.255.255.0; gateway 10.118.198.1
ClusterCommunication: 192.168.44.20/255.255.255.0; no gateway defined

4.  Either VM can ping their own IP addresses and the other's VM Network IP address.  Neither VM can ping the other's ClusterCommunication IP address  

5. nt98: ping 192.168.44.20 fails with 100% packet loss, tracert times out at the first hop.

6. nt98: route print:

Code:

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0     10.118.198.1    10.118.198.48    261
     10.118.198.0    255.255.255.0         On-link     10.118.198.48    261
    10.118.198.48  255.255.255.255         On-link     10.118.198.48    261
   10.118.198.255  255.255.255.255         On-link     10.118.198.48    261
        127.0.0.0        255.0.0.0         On-link         127.0.0.1    306
        127.0.0.1  255.255.255.255         On-link         127.0.0.1    306
  127.255.255.255  255.255.255.255         On-link         127.0.0.1    306
     192.168.44.0    255.255.255.0         On-link     192.168.44.10    261
    192.168.44.10  255.255.255.255         On-link     192.168.44.10    261
   192.168.44.255  255.255.255.255         On-link     192.168.44.10    261
        224.0.0.0        240.0.0.0         On-link         127.0.0.1    306
        224.0.0.0        240.0.0.0         On-link     10.118.198.48    261
        224.0.0.0        240.0.0.0         On-link     192.168.44.10    261
  255.255.255.255  255.255.255.255         On-link         127.0.0.1    306
  255.255.255.255  255.255.255.255         On-link     10.118.198.48    261
  255.255.255.255  255.255.255.255         On-link     192.168.44.10    261
===========================================================================
Persistent Routes:
  Network Address          Netmask  Gateway Address  Metric
          0.0.0.0          0.0.0.0     10.118.198.1  Default
===========================================================================

Everything looks right to me, and the firewall is off.  Any suggestions?

[Viperman]

none off the top of my head, could we do a remote session so that I could take a look at it?

[cpdohert]

Security policy won't allow it, although I appreciate the offer.  I don't have 100% control over the vSphere instance, so there may be config somewhere I'm not aware of.  Oh well.  We may have to fall back to multicast NLB.