Parallel provisioning

[taliesins]

With these changes I have managed to spin up 10 Windows boxes in parallel. I also make use of a few vagrant plugins like: vagrant-berkshelf, vagrant-triggers and vagrant-windows-domain. So there is lots of crazy things happening like reboots and re-initialization of file shares.

This PR is going to need some help on the test side. I am a bit of a Ruby noob, so I am not sure how we should be changing the tests to suite the code changes. So the tests have been left unchanged. Please send me a PR to fix the tests!

[ghost]

@taliesins I merged your PR #204 (by way of #212 where I fixed tests, etc), but that has caused the parallel-provisioning branch to have some rather large merge conflicts, particularly in lib/vSphere/action/clone.rb. Would you be able to take care of resolving those conflicts?

[taliesins]

@michael-brandt-cu I have resolved the conflicts. It would be wonderful if you could help me with the tests. Would love to get this into the main branch.

I have added a couple more things into this pull request:

• Added support for multiple network cards with full configuration of all features
• Added support for serial ports with full configration of all features
• Selection of management interface for communication between Vagrant and vm
• Changed the wait for windows sysprep to wait for customization, as people can use customization on other oses. Made it optional to wait for customization.

[ghost]

@taliesins I'm working through the rubocop failures right now, and for this line it says:

lib/vSphere/provider.rb:28:9: W: Useless assignment to variable - id.
        id = @machine.id
        ^^

id is not used anywhere in this function. Was this supposed to be @id = @machine.id, setting the id property on an instance of Provider? If not, then the line should be deleted to pass rubocop.

[ghost]

rubocop for this line says:

lib/vSphere/driver.rb:500:9: W: Avoid rescuing the Exception class. Perhaps you meant to rescue StandardError?
        rescue Exception
        ^^^^^^^^^^^^^^^^

We could replace Exception with StandardError like rubocop suggests, but I think it would be even better to use the specific exception or exceptions that we anticipate finding here, eg:

rescue TypeError, NameError

[ghost]

@taliesins Is there a small number of specific exceptions we expect to encounter here?

[taliesins]

I cut and paste this from: https://github.com/nsidc/vagrant-vsphere/blob/master/lib/vSphere/util/vim_helpers.rb

I will see if I can work out why this code was implemented this way.

[ghost]

Ah, I see. It's too bad that it's difficult to tell in git when code is just moved around like that.

Looks like it passed rubocop before because vim_helpers.rb was specifically excluded in the rubocop configuration: https://github.com/nsidc/vagrant-vsphere/blob/master/.rubocop.yml#L7

So, no need to worry about this then! I'll just update the config to point to name the correct file.

[taliesins]

No. I think we do want to run rubocop on this file. Lets see if we can put in something that it will pass with.

This file basically contains all the logic in it :>

[ghost]

Well that the whole file wasn't actually excluded, just the Lint/RescueException cop. So it would still run most of the checks.

[ghost]

@taliesins

I have resolved the conflicts.

Great!

It would be wonderful if you could help me with the tests.

I've created a new branch, parallel-provisioning-tests, starting from what you have here, with changes to get rubocop and the tests passing so that the Travis build can pass. I don't have it all passing yet, but that's where I'm planning to continue work on that.

I have a 3-day weekend this weekend, but I'll be able to get back to this next week.

[ghost]

I took a break from fixing unit tests, and I tried using this branch to destroy and recreate a VM I use for development on my current main project at NSIDC. I got these errors when attempting to destroy it:

There are errors in the configuration of this machine. Please fix
the following errors and try again:

vSphere Provider:
* The following settings shouldn't exist: real_nic_ip, vlan

We use both of these settings in our own infrastructure, and it looks like this branch removes real_nic_ip and changes how vlan works. Considering these breaking changes, and the possibility that other configuration settings are also changed in a backwards-incompatible way, I'm hesitant to pull this in.

It might be possible to release these changes in their current form as vagrant-vsphere-parallel, but then we would quickly have two diverging plugins; given how many changes this PR has, any features/fixes added to vagrant-vsphere could be difficult to apply to vagrant-vsphere-parallel, and vice versa.

[taliesins]

@michael-brandt-cu real_nic_ip, vlan are configuration settings that used to be at machine level. To support multiple nics we need to move them to network adapter level.

Perhaps we could put in a shim property for vlan and ip_address?

It should be trivial to change your vagrant scripts to use the multiple network card approach:

• management_network_adapter_slot to select the management nic
• management_network_adapter_address_family which address family to detect ip for
• vlan: "Env#{environment_number}" is equivalent of vlan at machine level
• ip_address: ipAddress is equivalent of real_nic_ip at machine level

Here is an example snippet from my vagrant file for setting up Juniper FCP for vMX:

      fpc.vm.provider :vsphere do |vsphere, overrides|
        vsphere.host = vsphere_host
        vsphere.insecure = true
        vsphere.user = vsphere_username
        vsphere.password = vsphere_password        
        vsphere.compute_resource_name = vsphere_cluster
        vsphere.resource_pool_name = vsphere_resource_pool   
        vsphere.template_name = vsphere_template_juniperfpc
        vsphere.name = fpc.vm.hostname
        vsphere.vm_base_path = "#{vsphere_vm_base_path}/Env#{environment_number}"
        vsphere.data_store_name = vsphere_datastore
        vsphere.memory_mb = 8192
        vsphere.cpu_count = 3
        vsphere.management_network_adapter_slot = 0
        vsphere.management_network_adapter_address_family = 'ipv4'
        vsphere.network_adapter 0, vlan: "Env#{environment_number}", mac_address: macAddress, ip_address: ipAddress
        vsphere.network_adapter 1, vlan: "Env#{environment_number}-Vmx#{i}"
        vsphere.network_adapter 2, vlan: "Env#{environment_number}-TransportBetweenVmx#{i % 2 == 1 ? i : i-1}AndVmx#{i % 2 == 0 ? i : i+1}"
        vsphere.network_adapter 3, vlan: "Env#{environment_number}-#{i % 2 == 1 ? 'Client' : 'CSP'}#{i % 2 == 1 ? i : i-1}-2"
        vsphere.network_adapter 4, vlan: "Env#{environment_number}-#{i % 2 == 1 ? 'Client' : 'CSP'}#{i % 2 == 1 ? i : i-1}-4"
        vsphere.network_adapter 5, vlan: "Env#{environment_number}-#{i % 2 == 1 ? 'Client' : 'CSP'}#{i % 2 == 1 ? i : i-1}-6"
        vsphere.network_adapter 6, vlan: "Env#{environment_number}-#{i % 2 == 1 ? 'Client' : 'CSP'}#{i % 2 == 1 ? i : i-1}-1"
        vsphere.network_adapter 7, vlan: "Env#{environment_number}-#{i % 2 == 1 ? 'Client' : 'CSP'}#{i % 2 == 1 ? i : i-1}-3"
        vsphere.network_adapter 8, vlan: "Env#{environment_number}-#{i % 2 == 1 ? 'Client' : 'CSP'}#{i % 2 == 1 ? i : i-1}-5"
        vsphere.network_adapter 9, vlan: "Env#{environment_number}-#{i % 2 == 1 ? 'Client' : 'CSP'}#{i % 2 == 1 ? i : i-1}-7"
      end

[macster84]

Hi everyone,

looks like there is lot of potential in this pull request.
Any chance we can get this or parts of it merged?

@michael-brandt-cu: Is backwards compatibility your main concern?

Maybe we can split the pull request in multiple different ones so that risk and merge effort is reduced?
In any case, I would be happy to help.

[ghost]

@michael-brandt-cu: Is backwards compatibility your main concern?

Yes.

Maybe we can split the pull request in multiple different ones so that risk and merge effort is reduced?

Maybe. It's been a while since I've had funding to do much work on vagrant-vsphere, and even longer since I really looked at this PR, so I can't really say how well it could be split up.