In continuation to my last two posts on using Terraform to automate various Cloud Director options, here is another one…In this post we are going to onboard a tenant using vCloud Director Terraform provider , (last post i did 5 steps) there are 12 steps that we are going to automate.Special thanks to my terraform for vCD Product Team who helped me in some of this stuff. 

Here is my old posts on similar topic.

• Automate vCloud Director with Terraform Provider
• Onboard Tenants on Cloud Director in less than 5 Minutes using vCD Terraform Provider
• For More Information refer to official link here

1. Create a new External Network
2. Create a new Organization for the Tenant
3. Create a new Organization Administrator for this Tenant
4. Create a new Organization VDC for the Tenant
5. Deploy a new Edge gateway for the Tenant
6. Create a new Routed Network for the Tenant
7. Create a new Isolated Network for the Tenant
8. Create a new Direct Network for the Tenant
9. Create Organization Catalog
10. Upload OVA/ISO to Catalog
11. Creating vApp
12. Create a VM inside vAPP

Step-1: Code for Creating External Network

As you know External Network is a Tenant connection to the outside world, By adding an external network, you register vSphere network resources for vCloud Director to use. You can create organization VDC networks that connect to an external network. few important parameters to consider:

• Resource Type – “vcd_external_network”
• vsphere_network – This is Required parameter you need to provide a DV_PORTGROUP or Standard port group names that back this external network. Each referenced DV_PORTGROUP or NETWORK must exist on a vCenter server which is registered with vCloud Director.
• Type –
  • For dv Port Group , use Type – DV_PORTGROUP
  • For Standard Port Group , use Type – NETWORK
• retain_net_info_across_deployments – (Optional) Specifies whether the network resources such as IP/MAC of router will be retained across deployments. Default is false.

#Create a new External Network for "tfcloud"
resource "vcd_external_network" "extnet-tfcloud" {
  name        = "extnet-tfcloud"
  description = "external network"
  vsphere_network {
    vcenter = "vcsa.dp-pod.zpod.io" #VC name registered in vCD
    name    = "VM Network"
    type    = "NETWORK"
  }
  ip_scope {
    gateway    = "10.120.30.1"
    netmask    = "255.255.255.0"
    dns1       = "10.120.30.2"
    dns2       = "8.8.4.4"
    dns_suffix = "tfcloud.org"
    static_ip_pool {
      start_address = "10.120.30.3"
      end_address   = "10.120.30.253"
    }
  }
  retain_net_info_across_deployments = "false"
}

Step-2: Code for New Organization

In this section , we are going to create a new organization named “tfcloud” which is enabled to use, This section creates a new vCloud Organisation by specifying the name, full name, description, VM Quota , vApp lease etc… Quota, lease etc..Cloud Provider must need to enter based on the commitment with tenant organization.

#Create a new org name "tfcloud"
resource "vcd_org" "tfcloud" {
  name              = "terraform_cloud"
  full_name         = "Org created by Terraform"
  is_enabled        = "true"
  stored_vm_quota   = 50
  deployed_vm_quota = 50
  delete_force      = "true"
  delete_recursive  = "true"
  vapp_lease {
    maximum_runtime_lease_in_sec          = 0
    power_off_on_runtime_lease_expiration = false
    maximum_storage_lease_in_sec          = 0
    delete_on_storage_lease_expiration    = false
  }
  vapp_template_lease {
    maximum_storage_lease_in_sec       = 0
    delete_on_storage_lease_expiration = false
  }
}

Step-3: Code for Creating Organisation Administrator

Once as a provider you created Org, this org need an admin, below code will create local org admin. In this code everything is self explanatory but few important parameters explained here:

• Resource Type – “vcd_org_user”
• org & name – these are variable, referred in variable file.
• role – role assigned to this user
• password – initial password assigned

#Create a new Organization Admin
resource "vcd_org_user" "tfcloud-admin" {
  org               = vcd_org.tfcloud.name
  name              = "tfcloud-admin"
  password          = "*********"
  role              = "Organization Administrator"
  enabled           = true
  take_ownership    = true
  provider_type     = "INTEGRATED" #INTEGRATED, SAML, OAUTH stored_vm_quota = 50 deployed_vm_quota = 50 }

Step-4: Code for Creating new Organization VDC

So till now we created External Network, Organization and Organization administrator , next is to create a organization virtual data center , so that tenant can provision VMs, Containers and Applications. few important configuration parameters to consider:

• name – vdc-tfcloud
• Resource Type – “vcd_org_vdc”
• Org – referring org name created in previous step
• Allocation Pool – Pay as you go (represented as “AllocationVApp”).
• network_pool_name – Network pool name as defined during provider config.
• provider_vdc_name – Name of Provider VDC name.
• Compute & Storage – Define compute and storage allocation.
• network_quota – Maximum no. of networks can be provisioned in to this VDC

# Create Org VDC for above org
resource "vcd_org_vdc" "vdc-tfcloud" {
  name = "vdc-tfcloud"
  org  = vcd_org.tfcloud.name
  allocation_model  = "AllocationVApp"
  provider_vdc_name = "vCD-A-pVDC-01"
  network_pool_name = "vCD-VXLAN-Network-Pool"
  network_quota     = 50
  compute_capacity {
    cpu {
      limit = 0
    }
    memory {
      limit = 0
    }
  }
  storage_profile {
    name    = "*"
    enabled = true
    limit   = 0
    default = true
  }
  enabled                  = true
  enable_thin_provisioning = true
  enable_fast_provisioning = true
  delete_force             = true
  delete_recursive         = true
}

Step-5: Code for Creating Edge Gateway for Tenant

This next section creates a new vCloud Organization Edge Gateway by specifying the name, full name, and description. Provider configures an edge gateway to provide connectivity to one or more external networks.

• Resource Type – “vcd_edgegateway”
• Configuration – compact
• Advanced – this will be an advance edge
• distributed_routing – distributed routing is enabled
• external_network – uplink information towards DC exit.

# Create Org VDC Edge for above org VDC
resource "vcd_edgegateway" "gw-tfcloud" {
  org                     = vcd_org.tfcloud.name
  vdc                     = vcd_org_vdc.vdc-tfcloud.name
  name                    = "gw-tfcloud"
  description             = "tfcloud edge gateway"
  configuration           = "compact"
  advanced                = true
  external_network {
     name = vcd_external_network.extnet-tfcloud.name
     subnet {
        ip_address            = "10.120.30.11"
        gateway               = "10.120.30.1"
        netmask               = "255.255.255.0"
        use_for_default_route = true
    }
  }
}

Step-6: Code for Creating Organization Routed Network

An organization VDC network with a routed connection provides controlled access to machines and networks outside of the organization VDC. System administrators (Providers) and organization administrators can configure network address translation (NAT) and firewall settings on the network’s Edge Gateway to make specific virtual machines in the VDC accessible from an external network. Things to consider:

• resource -> must be of type “vcd_network_routed”
• Define other networking information

# Create Routed Network for this org
resource "vcd_network_routed" "net-tfcloud-r" {
  name         = "net-tfcloud-r"
  org          = vcd_org.tfcloud.name
  vdc          = vcd_org_vdc.vdc-tfcloud.name
  edge_gateway = vcd_edgegateway.gw-tfcloud.name
  gateway      = "192.168.200.1"
  static_ip_pool {
    start_address = "192.168.200.2"
    end_address   = "192.168.200.100"
  }
}

Step-7: Code for Creating Org Isolated Network

An isolated organization VDC network provides a private network to which virtual machines in the organization VDC can connect. This network provides no connectivity to machines outside this organization VDC. Things to consider:

• resource -> must be of type “vcd_network_isolated”
• Define other networking information like ips etc

# Create Isolated Network for this org
resource "vcd_network_isolated" "net-tfcloud-i" {
  name    = "net-tfcloud-i"
  org     = vcd_org.tfcloud.name
  vdc     = vcd_org_vdc.vdc-tfcloud.name
  gateway = "192.168.201.1"
  static_ip_pool {
    start_address = "192.168.201.2"
    end_address   = "192.168.201.100"
  }
}

Step-8: Code for Creating Organisation Direct Network

This is restricted to System Administrator of vCloud Director Cloud Providers, A System Administrator can create an organization virtual datacenter network that connects directly to an IPv4 or IPv6 external network. VMs on the organization can use the external network to connect to other networks, including the Internet. Things to consider:

• resource -> must be of type “vcd_network_direct”
• Define other networking information
• In this we are connecting directly to external network which created in step-1

# Create Direct Network for this org
resource "vcd_network_direct" "net-tfcloud-d" {
  name             = "net-tfcloud-d"
  org              = vcd_org.tfcloud.name
  vdc              = vcd_org_vdc.vdc-tfcloud.name
  external_network = "extnet-tfcloud"
}

Step-9: Code for Creating Organization Catalog

Catalog allow Tenant to group vApps and media files , in this step provider is providing a private catalog to tenant,Things to consider:

• resource -> must be of type “vcd_catalog”
• Define catalog related information information

# Create Default catalog for this org
resource "vcd_catalog" "cat-tfcloud" {
  org         = vcd_org.tfcloud.name
  name        = "cat-tfcloud"
  description = "tfcloud catalog"
  delete_force     = "true"
  delete_recursive = "true"
  depends_on       = [vcd_org_vdc.vdc-tfcloud]
}

Step-10: Code for Uploading  OVA in to Catalog

It’s up to provider, they can upload few catalog items like .iso and .ova for tenants to consume in to above private catalog or can share with them public catalog, in this case we are uploading few items in to this private catalog.Things to consider:

• resource -> must be of type “vcd_catalog_item”
• ova_path -> it will be a path of your local directory to upload image.
• Define catalog related information information

# Create Default catalog for this org
resource "vcd_catalog_item" "photon-hw11" {
  org     = vcd_org.tfcloud.name
  catalog = vcd_catalog.cat-tfcloud.name
  name                 = "photon-hw11"
  description          = "photon-hw11"
  ova_path             = "/Users/tripathiavni/desktop/Sizing/photon-hw11-3.0-26156e2.ova"
  upload_piece_size    = 5
  show_upload_progress = "true"
}

Step-11: Code for Creating vAPP

In this step as a provider we are creating a vAPP which will hold few client binaries to run Container Service Extension. while creating vAPP things to consider:

• resource -> must be of type “vcd_vapp”
• ova_path -> it will be a path of your local directory to upload image.
• Define catalog related information information

# Create vApp for this org
resource "vcd_vapp" "CSEClientVapp" {
  name             = "CSEClientVapp"
  org              = vcd_org.tfcloud.name
  vdc              = vcd_org_vdc.vdc-tfcloud.name
  # This dependency is must to avoid a lock during destroy
  depends_on = [vcd_network_routed.net-tfcloud-r]
}

Step-12: Code for Creating Virtual Machine

In above vAPP , we will add a VM which is running Photon OS, while creating VM,  Things to consider:

• resource -> must be of type “vcd_vapp_vm”
• catalog_name -> Catalog that we created in above step.
• Define other VM related information.

# Create Default catalog for this org
resource "vcd_vapp_vm" "CSEClientVM" {
  name         = "CSEClientVM"
  org          = vcd_org.tfcloud.name
  vdc          = vcd_org_vdc.vdc-tfcloud.name
  vapp_name    = vcd_vapp.CSEClientVapp.name
  catalog_name = vcd_catalog.cat-tfcloud.name
  template_name = vcd_catalog_item.photon-hw11.name
  cpus = 2
  memory = 1024
  network {
    name               = vcd_network_routed.net-tfcloud-r.name
    type               = "org"
    ip_allocation_mode = "POOL"
  }
}

Putting it all together:

So i have put all this code in to a single file and also created a variable file, which will allow providers to on-board a new Tenant less then “5 minute” , provider admin just need to update few parameters in to the variable file like:

• vcd_user -> Cloud Admin user name
• vcd_pass -> Cloud Admin password
• vcd_url -> Cloud Director provider URL

Once you input the parameters, run terraform plan and Apply the plan, this entire process should not take more than 10 minutes to complete.

• Terraform Plan -out m4.tfplan

  • This slideshow requires JavaScript.

As you can see in above images terraform plan will add “12” items in to my Cloud Director.

• Terraform apply “m4.tfplan”

  • This slideshow requires JavaScript.

Finally terraform created all the 12 resources that we expected it to create.

Result:

As described above all 12 tasks related to a Tenant on-boarding got successfully completed and if you notice highlighted boxes , everything is over in less than around 8 minutes including uploading an OVA isn’t it awesome ?

NOTE: There isn’t need to define org/vdc in every resource if it is defined in provider  unless you working with a few org/VDCs.

Here i am attaching variable and code file , which you can use it in your environment by just changing variable file contents which i explained above. pls try these files in to a non-prod environment and make your self comfortable before doing it in production. here is the full content of above to Download Please share feedback , suggestion any in the comment section…

Edge Firewall Rules:

Tenant can use the edge gateway Firewall tab to add firewall rules for that edge gateway. You can add multiple NSX Edge interfaces and multiple IP address groups as the source and destination for these firewall rules.

The Firewall rules will already have a few entries pre-built in as part of preconfigured services, which you should not need to change in most cases:

To overcome with this issue, Cloud Director offers NSX proxy API , which will help provider/tenant to automate rules. Here is process of creating firewall rule using API:

• Find NSX Edge ID using API
• Get Edge FW Configuration
• Post Firewall rules

Find NSX Edge ID:

To find the NSX Edge ID , you need to fire “Get” to this API “https://<vCD>/network/edges&#8221;, here is “Get” to my API’s headers information , body will be empty.

Here is API Content:

This call will return All the edges and their related information, note on which edge you want to apply firewall rule and its ID.

Get Edge FW Configuration:

Though this step is not required but if you still want to see what all firewall rules has been configured etc..run “get” against “Edge Id”  -“https://<vCD>/network/edges/8417a9fc-c1df-4c03-befd-c79f60d5d0ab/firewall/config&#8221;

Post Firewall Rule:

Header Information:

To add firewall rule, we need to do “Post” against URL “https://<vCD>/network/edges/8417a9fc-c1df-4c03-befd-c79f60d5d0ab/firewall/config/rules&#8221; with following parameters:

• 417a9fc-c1df-4c03-befd-c79f60d5d0ab – this is Edge ID which we got from Step-1
• Content-Type – Application/xml
• Authorization – Bearer Token

Body Content:

Here is body content which is self explanatory. few important items are as below:

• Entire body must be within <firewallRules></firewallRules>
• Within<firewallRules></firewallRules> you can write various rule within <firewallRule></firewallRule> section.
• <service></service>  – In this section you will define “protocol” & “port”.

<firewallRules>
<firewallRule>
<name>New Rule</name>
<ruleType>user</ruleType>
<enabled>true</enabled>
<loggingEnabled>false</loggingEnabled>
<action>accept</action>
<destination>
<exclude>false</exclude>
<vnicGroupId>external</vnicGroupId>
<vnicGroupId>internal</vnicGroupId>
</destination>
<application>
<service>
<protocol>tcp</protocol>
<port>554</port>
<sourcePort>any</sourcePort>
</service>
<service>
<protocol>udp</protocol>
<port>554</port>
<sourcePort>any</sourcePort>
</service>
<service>
<protocol>tcp</protocol>
<port>556</port>
<sourcePort>any</sourcePort>
</service>
<service>
<protocol>udp</protocol>
<port>556</port>
<sourcePort>any</sourcePort>
</service>
</application>
</firewallRule>
</firewallRules>

Screenshot of my API Call body which should be in “application/xml” format.

if you need to write multiple rules in to single API call, then you can create multiple sections of <firewallRule></firewallRule> with single section of <firewallRules></firewallRules>

Result:

Once Header and body is ready, do a post and you should get a valid response of “201 Created”

vCD GUI reflects what you put in to the body of the API call.

I hope this will help Cloud providers/tenants to automate rules which needs to be automated or there are rules which providers need to create by default when onboarding a tenant.

In continuation to my last post, In this post we are going to onboard a tenant using vCloud Director Terraform provider , there are five things that we are going to do:

• Create a new Organisation for the Tenant
• Create a new Organisation Administrator for this Tenant
• Create a new Organisation VDC for the Tenant
• Deploy a new Edge gateway for the Tenant
• Create a new routed Network for the Tenant

Code for New Organisation:

So in this section , we are going to create a new organisation names “T3” which is enabled to use, This section creates a new vCloud Organisation by specifying the name, full name, and description.

#Create a new org names "T3"
resource "vcd_org" "org-name" {
  name             = "T3"
  full_name        = "My organization"
  description      = "The pride of my work"
  is_enabled       = "true"
  delete_recursive = "true"
  delete_force     = "true"
}

Code for Creating Organisation Administrator:

Once as a provider you created Org, this org need an admin, below code will create local org admin. In this code everything is self explanatory but few important parameters explained here:

• Resource Type -> “vcd_org_user”
• org & name -> these are variable, referred in variable file.
• role -> role assigned to this user
• password -> initial password assigned
• depends_on -> Explicit dependencies that this resource has. These dependencies will be created before this resource

#Create a new Organization Admin
resource "vcd_org_user" "org-admin" {
org = var.org_name #variable referred in variable file 
name = var.org_admin #variable referred in variable file
description = "a new org admin"
role = "Organization Administrator"
password = "change-me"
enabled = true
email_address = "avnish@t3company.org"
depends_on = [vcd_org.org-name]
}

Code for Creating new Organisation VDC:

So till now we created Org and Org admin , next is to create a organisation virtual data center , so that tenant can provision VMs, Containers and Applications. few important configuration parameters to consider:

• name -> T3-vdc
• Org -> T3
• Allocation Pool -> Pay as you go (represented as “AllocationVApp”).
• network_pool_name -> Network pool name as defined during provider config.
• provider_vdc_name -> Name of Provider VDC name.
• Compute & Storage -> Define compute and storage allocation.
• VM_quota -> Maximum no. of vms can be provisioned in to this VDC
• network_quota -> Maximum no of networks can be created.

# Create Org VDC for above org
resource "vcd_org_vdc" "vdc-name" {
  name        = var.vdc_name
  description = "The pride of my work"
  org         = var.org_name #variable referred in variable file
  allocation_model = "AllocationVApp"
  network_pool_name = "PVDC-A-VXLAN-NP"
  provider_vdc_name = "PVDC-A"
  compute_capacity {
    cpu {
      limit = 0
    }
    memory {
      limit = 0
    }
  }
  storage_profile {
    name     = "*"
    limit    = 10240
    default  = true    
  }
  metadata = {
    role    = "For Customer T3"
    env     = "staging"
    version = "v1"
  }  
  vm_quota                 = 10 #Max no. of VMs 
  network_quota            =  100
  enabled                  = true
  enable_thin_provisioning = true
  enable_fast_provisioning = true
  delete_force             = true
  delete_recursive         = true
depends_on = [vcd_org.org-name]
}

Code for Creating Edge Gateway for Tenant

This next section creates a new vCloud Organisation Edge Gateway by specifying the name, full name, and description. Provider configures an edge gateway to provide connectivity to one or more external networks.

• Configuration -> compact
• Advanced -> this will be an advance edge
• distributed_routing -> distributed routing is enabled
• external_network ->  uplink information towards DC exit.
• You will notice there is a ‘depends_on’ setting. This means that this resource depends on the resource specified before executing.

resource "vcd_edgegateway" "egw" {
  org = var.org_name #variable referred in variable file
  vdc = var.vdc_name #variable referred in variable file
  name                    = var.edge_name
  description             = "T3 new edge gateway"
  configuration           = "compact"
  advanced                = true
  distributed_routing     = true
  external_network {
    name = "SiteA-ExtNet"
    subnet {
      ip_address            = "192.168.100.20"
      gateway               = "192.168.100.1"
      netmask               = "255.255.255.0"
      use_for_default_route = true
    }
  }
depends_on = [vcd_org_vdc.vdc-name]
}

Code for Creating Organisation Routed Network

An organization VDC network with a routed connection provides controlled access to machines and networks outside of the organization VDC. System administrators (Providers) and organization administrators can configure network address translation (NAT) and firewall settings on the network’s Edge Gateway to make specific virtual machines in the VDC accessible from an external network. Things to consider:

• resource -> must be of type “vcd_network_routed”
• Define other networking information

resource "vcd_network_routed" "net" {
org = var.org_name #variable referred in variable file
vdc = var.vdc_name #variable referred in variable file
name = "T3-Routed-net"
edge_gateway = var.edge_name 
gateway = "10.10.0.1"
dhcp_pool {
start_address = "10.10.0.2"
end_address = "10.10.0.100"
}
static_ip_pool {
start_address = "10.10.0.152"
end_address = "10.10.0.254"
}
depends_on = [vcd_edgegateway.egw]
}

Putting it all together:

So i have put all this code in to a single file and also created a variable file, which will allow providers to on-board a new Tenant less then “5 minute” , provider admin just need to update few parameters in to the variable file like:

• org_name -> Tenant organisation name
• vcd_name -> Tenant Org VDC Name
• edge_name -> Tenant N/S router name
• org_admin -> Org Admin name

Once you input the parameters, run terraform plan and Apply the plan, this enitre process should not take more than 5 minutes to complate.

• Terraform Plan -out f1.tfplan
  • 
• Terraform apply “f1.tfplan”
  • 

Result:

As described above all five tasks related to a Tenant on-boarding got successfully completed and if you notice highlighted boxes , everything is over in less than 2 minutes, isn’t it awesome ?

Here i am attaching variable and code file , which you can use it in your environment by just changing variable file contents like , org_name , vdc_name etc..which i explained above. pls try these files in to a non-prod environment and make your self comfortable before doing it in production.Here is the Code file to download – Terraform.zip. Please share feedback , suggestion any in the comment section…

The new refreshed Terraform vCloud Director provider enables administrators and DevOps engineers to define vCloud Director “infrastructure as code” inside Terraform configuration files. This makes it an efficient automation and integration tool and this project is fully open-source and available on GitHub and also HashiCorp is hosting it in the “terraform-providers” namespace together with all the other official Terraform providers.

If you’d like to contribute with a feature request, report an issue or propose a code improvement please visit the project’s site below. There you can also see current activity and what’s in the works.

• https://github.com/terraform-providers/terraform-provider-vcd

Terraform Installation & Configuration:

Terraform installation is very simple, it is just a single file. if you are running Linux OS system it is “terraform” and if it is windows based systems then it is terraform.exe.You can download the latest version of Terraform from the HashiCorp website using this direct link: https://www.terraform.io/downloads.html

I am using Windows in my Lab so for my Windows based system, I simply downloaded the Terraform Windows x64 file and put into a folder called “c:\tf” then added this folder into my PATHS variable so that I could run terraform.exe from anywhere. This can be done by going to System Properties –> Advanced –> Environment Variable

now go to “System variables” and add :terraform.exe” folder location.

• Select Path and Click on “Edit”
• at the end of the line put symbol “;” and add “terraform.exe” directory path.

Download Terraform vCloud Director Plugin and VS Code:

Download the latest vCloud director terraform plugin from here  and put in to a directory and we will use this during terraform initialization.

Now a days i start using Microsoft Visual Studio Code to write my automation scripts. It has inbuilt terraform plugin to highlight/validate code which makes it way more simple, then working with different text editors and multiple windows. It’s a free download, check it out or you are free to use any other text editor of your choice.

Create Terraform files:

1. Create a new folder
2. Create two terraform files (terraform.tfvars & main.tf) as below and put the below content in to the files
3. Save both the files in to the folder which we created in to step1
4. terraform.tfvars:–
   • This is where we give value to the variables. For example vcd_url = “https://vcd-01a.corp.local/api”, this means that anytime var.vcd_url is referenced in the “main.tf file” it will reference back to . Most variables below are self-explanatory.

   • # vCloud Director Connection Variables
     vcd_user = "administrator"
     vcd_pass = "VMware1!"
     vcd_url = "https://vcd-01a.corp.local/api"
     vcd_max_retry_timeout = "60"
     vcd_allow_unverified_ssl = "true"

   • 
5. main.tf – This will have actual actionable code which will perform action on to vCloud Director.We do this by using a provider and multiple resources. The provider we are using in this demonstration is “vcd”. The resources are then responsible for different parts of vCloud Director. In this example “vcd_org” is going to create, modify or delete an Org. we are created new VCD Organisation.

   • variable "vcd_user" {
     description = "vCloud user"
     }
     variable "vcd_pass" {
     description = "vCloud pass"
     }
     variable "vcd_allow_unverified_ssl" {
     default = true
     }
     variable "vcd_url" {}
     variable "vcd_max_retry_timeout" {
     default = 60
     }
     
     # Connect VMware vCloud Director Provider
     provider "vcd" {
     user = var.vcd_user
     password = var.vcd_pass
     org = "System"
     url = var.vcd_url
     max_retry_timeout = var.vcd_max_retry_timeout
     allow_unverified_ssl = var.vcd_allow_unverified_ssl
     }
     
     #Create a new org names "T3"
     resource "vcd_org" "org-name" {
     name = "T3"
     full_name = "My organization"
     description = "The pride of my work"
     is_enabled = "true"
     delete_recursive = "true"
     delete_force = "true"
     }

   • 

Initialize terraform plugin:

The terraform “init” command is used to initialize a working directory containing Terraform configuration files. This is the first command that you should be run after writing a new Terraform configuration or cloning an existing one from version control. It is safe to run this command multiple times.this command will never delete your existing configuration or state.

During init, Terraform searches the configuration for both direct and indirect references to providers and attempts to load the required plugins. For providers distributed by HashiCorp, init will automatically download and install plugins if necessary.

Since on my windows init didn’t downloaded plugin for some reason , so i have downloaded vCD plugin in above steps and during initialization i am pointing towards my directory and it says terraform will use vCloud Director Plug-in version 2.6.

Terraform Plan:

The terraform “plan” command is a convenient way to check whether the execution plan for a set of changes matches your expectations without making any changes to real resources or to the state.

“Terraform plan” command will check above created files and check what changes it needs to do on the environment and will give you summary of tasks.

Terraform Apply

The “terraform apply” command is used to apply the changes required to reach the desired state of the configuration, or the pre-determined set of actions generated by a terraform plan execution plan.

Org Created Successfully:

above step created an organisation in to my vCloud Director environment and it took only few minutes.

lots of development is happening on terraform  provider for vCloud Director which will help VMware Cloud provider to automate repeated tasks. i will continue to add few more blog articles on this topic, stay tuned….

As you may be aware vCloud Director from its inception initially was relying on vCNS and after that on NSX-V to provide on-demand , self service cloud networking capabilities and now since VMware is moving towards newly re-written networking platform called NSX-T and with every new version , it is getting mature and feature rich , vCloud Director with version 10 brings many of its capabilities in to it to offer more and more self service capabilities to tenant and ease of implementation and operation for providers, in this post i am covering how to integrate NSX-T with vCD from Provider prospective.

Pre-requisite

As you may be aware that NSX-T is no more coupled/dependent on vCenter ,so to integrate NSX-T with vCloud Director you must install and configure NSX-T Data Center. Here are the high level steps:

• Deploy and configure the NSX-T Manager virtual appliances.
• Create transport zones based on your networking requirements.
• Deploy and configure Edge nodes and an Edge cluster.
• Configure the ESXi host transport nodes, these will become PVDC resources of NSX-T based tenants.
• Create a tier-0 gateway , this will work as “External Network” for vCloud Director.

Register NSX-T Manager

Create Network Pool

1. Name it appropriately.
2. Select “Geneve Backed” type Network pool
3. Select Appropriate NSX-T Providers (you can have multiple NSX-T Providers)
4. Select Appropriate Overlay Transport Zone
5. review and submit.

Below is the process to create vCloud Director 10 external network by importing Tier0  router created in side NSX-T.

1. Choose Backing Type as “NSX-T Resources (Tier-0 Router)” and select registered NSX-T
2. Provide Name
3. Select Tier-0 router
4. Add a “Network Pool” with Gateway details.
5. review and complete , which will import T0 in to vCloud Director construct.

Now you can create Provider VDC (PVDC) which is basically mapped to a vSphere cluster or a resource pool. PVDC to successfully work you need to ensure that vSphere cluster has been prepared with NSX-T and part of a transport zone.When creating NSX-T backed PVDC you will have to specify the Geneve Network Pool created in the previous step.

Go to “Cloud Resources” – “Provider VDCs” and Click on “NEW” to create new PVDC backed by NSX-T based networks.

1. Name your PVDC
2. Select vCenter which is having NSX-T backed Cluster
3. Select appropriate Cluster and VM Hardware version
4. Select appropriate Storage policy
5. Select NSX-T manager and Network Pool ( as created above – Geneve backed pool )
6. Review configuration and finish.

This slideshow requires JavaScript.

if everything is configured properly, PVDC get created successfully.

This completes vCloud Director configuration from provider prospective. In the next post i will be covering tenant onboarding process on NSX-T based Network.

In vCloud Director 9.7 compute policies were introduced to offer/manage the T-shirt sizing of the VMs which i have covered in detail in my this post,  in vCloud Director 10 similar concept has been brought in to GUI , which is now easy to implement & manage and in vCD 10 this is being called “Sizing Policy”

So from Cloud Provider prospective VM sizing policy defines the compute resource allocation for virtual machines within an organization VDC. Sizing policy allow provider to control the following aspects of compute resources consumption at the virtual machine level:

Create T-Shirt Sizes:

Let’s create few example T-Shirt sizing policies:

• Policy Name – X1
  • Description: Small-size VM policy Memory: 1024 Number of vCPUs: 1
  • Name: X1
  • Memory: 1024
  • Number of vCPUs: 1
• Policy Name – X2
  • Description: Medium-size VM policy Memory: 2048 Number of vCPUs: 2
  • Name: X2
  • Memory: 2048
  • Number of vCPUs: 2
• Policy Name – X3
  • Description: Large-size VM policy Memory: 4096 Number of vCPUs: 4
  • Name: X3
  • Memory: 4096
  • Number of vCPUs: 4
• Policy Name – X4
  • Description: X-Large-size VM policy Memory: 8192 Number of vCPUs: 8
  • Name: X4
  • Memory: 8192
  • Number of vCPUs: 8

Create T-Shirt Sizing policies:

1. Cloud Provider Administrator, logins to vCloud Director and go to “VM Sizing Policies” and Click on “New” to create new policy
   • 
2. Name and describe the policy as per above example and move to Next.
   • 
3. In next section enter CPU related parameters , in this example i am choosing “vCPU Count” , providers can choose based on their requirement and leave it all blank as none of the fields are mandatory.
   • 
4. In next section enter Memory related parameters , in this example i am choosing only “Memory”, providers can choose based on their requirement and leave it all blank as none of the fields are mandatory.
   • 

that’s it , so simple to create policies , follow the same step to create multiple policies as per above example.

Publish Created Policies:

vCloud Director system administrators create and manage VM sizing policies at a global level and can publish individual policies to one or more organization VDCs.

so above step we have created polices , we need to publish these policies to organisation VDC’s.

1. Select Cloud Resources then click on Organization VDCs and go inside an organization VDC
   • 
2. Inside VDC , go to VM Sizing Policies and click on Add
   • 
3. Select the policies that you want to make available for a Particular oVDC/Tenant
   • 
4. You can set policies as default policies, which will make policy appear as the default choice for the tenants during a VM and vApp creation and VM edit.
   • 

Once polices published to organisation’s VDC, when tenant user logins and try to deploy a new VM , he/she now see options to chose T-Shirt sizes with their descriptions and if user does not choose any policy , it will pickup default policy and i showed you how to setup default policy.

Tag Template with T-Shirt sizes

So while cloud providers can control sizing of new virtual machines, how about Templates ?

vCloud Director helps providers to achieve this by associating  the VMs of a vApp template with specific VM sizing policies, Providers/tenant  can tag individual VMs of a vApp template with the policies you want to assign.

To Tag template to  a particular sizing policy, you need to login to org and then go to Libraries, and select vApp Templates from the left panel.

Click on particular template/highlight the template and select Tag with Compute Policies.

“TAG WITH COMPUTE POLICIES” gives two options to tag with:

• VM Placement Policies – which allows VM to deploy in to particular cluster.
• VM Sizing Policy – As explained in this Post, so when user will try to deploy a VM from template, it will get deployed according to “VM Sizing Policy”

This completes the process , gain control of your cloud offerings.

vCloud Director 10 has introduced a new concept called VM placement policies which helps Cloud Provider to control the virtual machine (VM) placement on a specific cluster or host.VM placement policies give cloud providers various options to allocate resources to various use cases like:

1. Deploy VM’s to specific cluster based on performance requirement
2. Deploy VM’s to Specific cluster based on resource requirements
3. Deploy VM’s based on Licensing requirement as a part of Oracle/SQL licenses optimisation
4. Allocate specific hosts to specific Tenants
5. Deploy container/special use case specific VMs to a specific host/cluster
6. Restrict elastic VDC pools to deploy VMs to a specific cluster

vCD Provider administrator create and manage VM placement policies and placement policies are created and managed for each provider VDC, because a VM placement policy is scoped at the provider VDC level.

Create a VM Placement Policy

Before we create VM Placement policies, provider need to perform few steps on vCenter , so lets go and login to vCenter which is providing resource to vCloud Director and go to Cluster -> Configure -> VM/Host Groups

In this case i want to limit deployment of Oracle and MS SQL VM’s to specific hosts due to licensing, so let’s create Hosts groups and VM Groups:

Host Groups: 

To create Host Groups , Click on Add inside VM/Host Groups:

1. Enter  Host Group Name
2. Select Type as “Host Group”
3. Click on Add to add Host/Hosts of the cluster.

VM Groups

To create VM Groups , Click on Add inside VM/Host Groups

1. Enter  VM Group Name
2. Select Type as “VM Group”
3. Click on Add to add VM/VMs of the cluster. (select any dummy VM as of now)

once both the groups has been created go to VM/Host rules in the cluster and create a rule.

VM/Host Rules

To create VM/Host Rules, Click on Add inside VM/Host Rules

1. Enter  Rule Name
2. Ensure “Enable rule”
3. Select rule type as “Virtual Machine to Hosts”
4. VM Group: Select VM Group that we have created above
5. Here you have four choices: (In my case i have choose Must rule)
   • Must run on host in group
   • Should run on host in group
   • Must not run on host in group
   • Should not run on host in group
6. Host Group: Select Host Group that we have created above

From vCenter prospective we are done, we have multiple choice to create VM to Hosts affinity/anti-affinity rules , once we have created rules , vCloud director picks up only “VM Groups” which provider will expose to tenants.

Create VM Placement Policies in vCloud Director

1. Go to Provider VDCs.
2. Click on a provider VDC from the list , in my case it was “nsxtpvdc”
3. Click on “VM Placement Policies”
4. Click the VM Placement Policies tab and click New.

New Policy Creation Wizard

1. First Page , click on Next
   1. 
2. Enter a name for the VM placement policy and description and click Next
   1. 
3. Select the VM groups or logical VM groups to which you want the VM to be linked and click Next.

   1. 

4. Review the VM placement policy settings and click Finish.
   1. 

Publish VM Placement Policies to Org VDC

When provider creates a VM placement policy, it is not visible to tenants. Provider need to publish a VM placement policy to an org VDC to make it available to tenants and publishing a VM placement policy to an org VDC makes the policy visible to tenants. The tenant can select the policy when they:

• Create a new standalone VM
• Create a VM from a template,
• edit a VM
• add a VM to a vApp
• Create a vApp from a vApp template. 

To publish this newly created policy to tenants , go to:

1. Organization VDCs and Select an organization VDC
   1. 
2. Click the VM Placement Policies tab and Click Add.
   1. 
3. Select the VM placement policies that you want to add to the organization VDC and click OK.
   1. 
4. Provider can make certain policies as “Default” when customer does not choose any policy , system will automatically use “Default”.
   1. 

Policy Usage by Tenant

Once policies has been created and exposed to tenant organisation, tenant can use those policies while provisioning VMs. like here i have created two policies “Oracle” and “SQL” and tenant can choose based on workload requirement.

NOTE –  Placement Policies are optional and a provider can continue to use the default policy that is created during installation and only one policy can be assigned to a VM.

This completes the creation of placement policies and their exposure to tenants. please feel free to share/comment.

VMware vCloud Director vCD-CLI  is a command line interface for vCloud Director using short, easy-to-remember commands to administer vCD. it also allows tenants to perform certain operations for convenience and automation.

vCD-CLI is Python based and fully open source and licensed under the Apache 2.0 license. Installation process is very easy and can be installed on various platforms .  pls check INSTALL.md, which has detailed installation instructions for Mac OS X, Linux, and Windows.

if you are using VMware’s container service extension , you need to add extension to vCD-CLI.

Installation

Here are steps which is followed for the installation on Photon OS v2 , Photon OS minimal installs lack standard tools like pip3 and even ping, so you need to install a number of packages using tdnf.

• #tdnf install -y build-essential python3-setuptools python3-tools python3-pip python3-devel
• 
• #pip3 install –user vcd-cli
• 
• Set PATH  using  #PATH=$PATH:~/.local/bin
• 
• Run #vcd (if everything goes well , you should see as below)
• 

Command Use

• Login to vcd #vcd login cloud.corp.local system administrator –password <********> -w -i  -> this will login to vCD system.
  • 
• Let’s create a PVDC using vcd-cli , to create Provider VDC run this:
  • 
  • 
  • VC NAME and -t NSX-T name should be as per vCD Console
  • -r – Resource Pool – Name should be as per VC cluster name
  • -e to enable the PVDC.
  • -s – storage profile name, i am choosing all.
  • PVDC get created successfully.
  • 
• Let’s now create an organisation
  • to create an organisation run this: #vcd org create T1 Tenant1 -e
  • 

so if you see this is very easy way to create object in vCD using command lines , an script can be written to automate some of the routine tasks and jobs. Refer here for more command syntax.

I am working on a Lab which require messaging queue server , so i setup this and thought of sharing the steps, so here it is..

AMQP is an open standard for message queuing that supports flexible messaging for enterprise systems. vCloud Director uses the RabbitMQ AMQP broker to provide the message bus used by extension services, object extensions, and notifications. we will be setting up this on Ubuntu System , so download Ubuntu and install it on a VM and then follow below steps

Update Ubuntu System

Before starting, you will need to update Ubuntu repository with the latest one.You can do so by running the following commands:

• #sudo apt-get update -y
• #sudo apt-get upgrade -y

Installing Erlang on Ubuntu

Before installing Rabbitmq, we will need to install erlang as a prerequisite of rabbitmq. You can install by running the following commands:

• #wget http://packages.erlang-solutions.com/ubuntu/erlang_solutions.asc
• #sudo apt-key add erlang_solutions.asc
• #sudo apt-get update -y
• #sudo apt-get install erlang -y
• #sudo apt-get install erlang-nox -y
• 

Once we are done with Erlang installation, we can continue with installation of RabbitMQ.

Installing RabbitMQ on Ubuntu

First we will need to add Rabbitmq repository to apt and to do that run the following command:

• #sudo echo “deb http://www.rabbitmq.com/debian/testing main” >> /etc/apt/sources.list

Once the repository has been added, Add the RabbitMQ public key to our trusted key list to avoid any warnings about unsigned packages:

• #wget https://www.rabbitmq.com/rabbitmq-signing-key-public.asc
• #sudo apt-key add rabbitmq-signing-key-public.asc

Next step is to update the apt repository with the following command:

• #sudo apt-get update

Once the repository is updated, go ahead and  install rabbitmq server by running the following command:

• #sudo apt-get install rabbitmq-server

Once installation is complete, start the rabbitmq server and enable it to start on boot by running the following command:

• #sudo systemctl start rabbitmq-server
• #sudo systemctl enable rabbitmq-server

You can check the status of rabbitmq server with the following command:

• #sudo systemctl status rabbitmq-server 

To enable RabbitMQ Management Console, run the following:

• #sudo rabbitmq-plugins enable rabbitmq_management

Login to URL using IP address with port number 1567, here it is you have successfully installed RabbitMQ.

Change default admin user (For security hardening)

By default the admin user for RMQ installation is guest/guest. we can change the default admin account by using below commands

• #rabbitmqctl add_user vmware vmware  (first vmware is admin username and second vmware is password , you change it based on your requirement)
• #rabbitmqctl set_user_tags vmware administrator (taging user with Admin priveledge)
• #rabbitmqctl set_permissions -p / vmware “.*” “.*” “.*”
• Now you can login with new admin user.
• 

Go ahead and configure it in your vCD instance.

This completes the installation and configuration process.

As you know vCloud Director 9.7 comes with a default compute policy for VDC , that does provide options for custom vm sizing and this can go out of control from provides point of view as tenant can try to deploy any size of VM which might impact many things and to control this behaviour we need to limit the VM’s maximum number of vCPU and vRAM of a customer VDC can have and with vCloud Director 9.7, this is now easily can be achieved using few API calls , here is the step by step procedure to set the maximum limits:

NOTE – This gets applied on the policy like default policy which has cpuCount and memory fields as null values.

Step-1 – Create a MAX compute Policy

Let’s suppose we want to setup MAX vCPU = 32 and MAX RAM = 32 GB , so to setup this max , let’s first create a compute policy.

Procedure: Make an API call with below content to create MAX VDC compute policy:

• POST:  https://<vcd-hostname>/cloudapi/1.0.0/vdcComputePolicies
• Payload:  ( i kept payload short , you can create based on sample section)
  • {
    “description”:”Max sized vm policy”,
    “name”:”MAX_SIZE”,
    “memory”:32768,
    “cpuCount”:32
    }
• Header
  • 
• Post to create compute Policy
  • 

Step-2: Create a Default Policy for VDC

Publish MAX policy to VDC.

Procedure

• Get VDC using below API Call
  • GET: https://<vcd>/api/admin/vdc/vdcid
  • 
• Take the entire output of above GET call and put in to body of new call with PUT as below screenshot and inside body add below line after DefaultComputePolicy element
  • <MaxComputePolicy href=”https://vcd-01b.corp.local/cloudapi/1.0.0/vdcComputePolicies/urn:vcloud:vdcComputePolicy:cd38493a-2254-446e-b782-168982c980e1” id=”urn:vcloud:vdcComputePolicy:cd38493a-2254-446e-b782-168982c980e1” name=”max-policy” type=”application/json”/>
  • 
  • Highlighted is policy ID , that we captured in Step-1 and update in your API call.
  • 
  • Header of the above API call is as below:
    • 

Now if you go back and try to provision a virtual machine with more than 32GB memory , it will through the error as below:

Simple two API calls , will complete the much awaited feature now.

Many of my customer with whom i directly interact has been asking this feature from quite some time , few of them says that T-shirt sized based offering matches of what hyper scalars offer, so with the release of vCloud Director 9.7, we can now control the resource allocation and the VM placement much better by using compute policies. As you know traditionally vCloud Director has two type of scope one is Provider VDC and another one Organisation VDC, similarly based on the scope and the function, there are two types of compute policies – provider virtual data center (VDC) compute policies and VDC compute policies.

In this post i will discusses VDC compute Policies and how you can leverage VDC compute policies to offer T-Shirt size option to your Best in class VMware vCloud director based cloud.

Provider VDC Compute Policies

Provider VDC compute policies applies to provider VDC level. A provider VDC compute policy defines VM-host affinity rules that decides the placement of tenant workloads. as you know Provider VDC level configuration is not visible to Tenant users and same applies to PVDC policies.

VDC Compute Policies

VDC compute policies control the compute characteristics of a VM at the organization VDC level and using VDC compute policies. A VDC compute policy groups attributes that define the compute resource allocation for VMs within an organization VDC. The compute resource allocation includes CPU and memory allocation, reservations, limits, and shares. here is the sample configuration:

• {
  “description”:”2vCPU and 2 GB RAM”,
  “name”:”X2 Policy”,
  “cpuSpeed”:1000,
  “memory”:2048,
  “cpuCount”:2,
  “coresPerSocket”:1
  “memoryReservationGuarantee”:0.5,
  “cpuReservationGuarantee”:0.5,
  “cpuLimit”:1000,
  “memoryLimit”:1000,
  “cpuShares”:1000,
  “memoryShares”:1000,
  “extraConfigs”:{
  “config1″:”value1”  – Key Value Pair
  },
  “pvdcComputePolicy”:null
  }

For More detailed description of there parameters , please refer here we will going to create few policies which will reflect your cloud’s T-Shirt sizing options for your tenants/customers.

Step-1: Create VDC Compute Policy

Let’s first Create a VDC compute policy, which should be matching to your T-Shirt Sizes that you want to offer , for example here i am creating four T-shirt sizes as below:

• X1 – 1 vCPU and 1024 MB Memory
• X2 – 2 vCPU and 2048 MB Memory
• X3 – 4 vCPU and 4096 MB Memory
• X4 – 8 vCPU and 8192 MB Memory

Procedure:

Make an API call with below content to create VDC compute policy:

• POST:  https://<vcd-hostname>/cloudapi/1.0.0/vdcComputePolicies
• Payload:  ( i kept payload short , you can create based on sample section)
  • {
    “description”:”8vCPU & 8GB RAM”,
    “name”:”X8″,
    “memory”:8192,
    “cpuCount”:8
    }
• Header:
  • 
• Here is my one of four API call. similarly you make other 3 calls for other three T-Shirt sizes.
  • 
• After each successful API call , you will get a return like above , here note down the “id” of each T-Shirt size policy , which we will use in subsequent steps. you can also see the compatibility of policy for VDC type.
  • X8 – “id”: “urn:vcloud:vdcComputePolicy:b209edac-10fc-455e-8cbc-2d720a67e812”
  • X4 – “id”: “urn:vcloud:vdcComputePolicy:69548b08-c9ff-411a-a7d1-f81996b9a4bf”
  • X2 – “id”: “urn:vcloud:vdcComputePolicy:c71f0a47-d3c5-49fc-9e7e-df6930660817”
  • X1 – “id”: “urn:vcloud:vdcComputePolicy:1c87f0c1-ffa4-41d8-ac5b-9ec3fab211bb”

Step-2: Get VDC Id to Assign  VDC Compute Policies

Make an API call to your vCloud Director with below content to get the VDC ID:

Procedure:

• Get:  https://<vcd-hostname>/api/query?type=adminOrgVdc
• Use Header as in below screenshot:
  • 
• and write down the VDC ID ( as highlighted in above screenshot in return body) , this we will use in other calls. you can also get VDC id from vCD GUI.

Step-3: Get current Compute Policies Applied to the VDC

Using VDC identifier from step2 , Get the current compute policies applied on this VDC using below API Call:

Procedure:

• Get: https://<vcd-hostname>/api/admin/vdc/443e0c43-7a6d-43f0-9d16-9d160e651fa8/computePolicies
  • 443e0c43-7a6d-43f0-9d16-9d160e651fa8 – got from step2
• use Header as per below image
  • 
• Since this is an Get call , so no body.
  • 
• Copy the output of this Get and paste in to a new postman window to make a new API Call as this is going to be body for the our next API call.

Step-4: Publish the T-Shirt Size Compute Policies to VDC

In this step we will publish the policies to VDC , let’s create a new API call with below content:

Procedure:

• PUT: https://<vcd-hostname>/api/admin/vdc/443e0c43-7a6d-43f0-9d16-9d160e651fa8/computePolicies
• Header as below image:  ensure correct “Content-Type” – application/vnd.vmware.vcloud.vdcComputePolicyReferences+xml
  • 
• Payload:
  • paste the output of step3 in the body
  • copy full line starting with <VdcComputePolicyReference ******** /> and paste number of times as your policies. in my case i have four policies , i pasted four times.
  • in each line (underline RED) replace policy identifier with identifier we captured in step1 (compute policy identifier).
  • 
• Here is API call which will associate VDC compute policies to your Tenant’s VDC.

Now go back and login to tenant portal and click on “New VM” and see under compute policy , now you can see all your compute policy which is nothing but your T-shirt size virtual machine offerings..

Once tenant chooses a policy , he can’t choose CPU and Memory parameters..

Step-5: Create a Default Policy for VDC

With Every VDC , there is default policy which is auto generated and  has empty parameters. Now since we have published our four sizing policies to this VDC, we will make one of them default policy of the VDC. This means that if user does not provide any policy during VM creation then the default policy of the vDC would be applied on the VM.

Procedure

• Get VDC using below API Call
  • GET: https://<vcd>/api/admin/vdc/vdcid
• 
• Take the entire output of above GET call and put in to body of new call with PUT as below screenshot and inside body within <DefaultComputePolicy section , change the id of the Policy.
• 

Step-6: Delete System Default Policy

There is “System Default” policy which when selected , give options like “Pre-defined Sizing Options” and “Custom Sizing Options” , and will allow your tenants to define sizes of their choice , to restrict this , we need to un-publish this policy from VDC.

Procedure

To disable this policy , follow the procedure in Step-5

• Query VDC and copy the return Body
• Make a PUT and inside body paste body copied in above step and remove the “system Default” policy , only keep policy , which you want to offer for this particular VDC.
• 
• After above call if you see , there is no “System Default” policy.
• 

NOTE – Ensure that non of the VM and catalogs are associated with this “System Default” policy , ideally after creation of VDC , you must create and assign policy before these policies are consumed by VM/catalogs.

Extra-Step: Update the Policy

if you want to update the policy make an “PUT” api call to policy with updated body content , see below my policy update API call for reference.

Compute policies cannot be edited except for name and description ,I hope this helps providers now offer various T-Shirt size options to their customers.

Much awaited feature for cloud provider to match thier corporate  standards and to create a fully custom cloud experience, now with release on vCloud Director 9.7 you can set the logo and the theme for your vCloud Director Service Provider Admin Portal and also now you can customize the vCloud Director Tenant Portal of each tenants . In addition, you can modify and add custom links to the two upper right menus in the vCloud Director provider and tenant portals.

Provider Portal Branding

vCloud Director 9.7 UI can be modified for the following elements:

• Portal name
• Portal color
• Portal theme (vCloud Director contains two themes – default and dark.)
• Logo & Browser icon

Customize Portal Name ,Portal Color and Portal Theme

To configure the Cloud Provider Portal Branding , make a PUT request to vCloud Director end point as below:

• PUT – https://<vCD Url>/cloudapi/branding
• BODY – {
  “portalName”: “string”,
  “portalColor”: “string”,
  “selectedTheme”: {
  “themeType”: “string”,
  “name”: “string”
  },
  “customLinks”: [
  {
  “name”: “string”,
  “menuItemType”: “link”,
  “url”: “string”
  }
  ]
  }
• Headers
  • 

Here is my API call using Postman client:

Customize Logo

To change the Logo, here is the procedure for API

• Headers
  • 
• PUT
  • https://<vCD Url>/cloudapi/branding/logo
• Body – This is bit tricky since we need to upload an image as a body.
  • In Postman client inside “Body” click on “Binary” which will allow you to choose file as body. select your logo.
  • 

Customize Icon

To customize the icon, follow this API call and procedure.

• Headers
  • 
• PUT
  • https://<vCD Url>/cloudapi/branding/icon
• Body – same as above section , choose a image
  • 

so after running above API calls , here is what my vCloud Director provider portal looks like.

Tenant Portal Branding

As we did above similarly we can now fully customize Tenant Portal

Customize Portal Name ,Portal Color and Portal Theme

To configure the Cloud Provider Portal Branding , make a PUT request to vCloud Director end point in to tenant organisation as below: ( T1 is my org Name)

• PUT – https://<vCD Url>/cloudapi/branding/tenant/T1
• BODY – {
  “portalName”: “string”,
  “portalColor”: “string”,
  “selectedTheme”: {
  “themeType”: “string”,
  “name”: “string”
  },
  “customLinks”: [
  {
  “name”: “string”,
  “menuItemType”: “link”,
  “url”: “string”
  }
  ]
  }
• Headers
  • 

Here is my API call using Postman client:

Customize Logo

To change the Logo, here is the procedure for API

• Headers
  • 
• PUT
  • https://<vCD Url>/cloudapi/branding/tenant/T1/logo
• Body – As said above ,this is bit tricky since we need to upload an image as a body.
  • In Postman client inside “Body” click on “Binary” which will allow you to choose file as body, select your logo.
  • 

Once i have done with above API calls, this is how my Tenant portal look like for “T1” organisation.

For a particular tenant, you can selectively override any combination of the portal name, background color, logo, icon, theme, and custom links. Any value that you do not set uses the corresponding system default value.

This completes feature walk through of Provider and Tenant custom branding options available now with vCD9.7.

Since vCloud Director 9.7 has dropped support for Postgres SQL9.5 , so i had to upgrade my postgres to 10 , then i have updated my vCloud Director to versions 9.7 , i followed below steps to upgrade the DB , basically at High level steps are as below:

• You need to backup the existing database and data directory.
• Uninstall old version of Postgres SQL.
• Install Postgres10
• Restore Backup

Procedure

• Create database backup using:
  • su – postgres
  • pg_dumpall > /tmp/pg9dbbackup
  • exit
  • 
• Check and Stop the service using
  • #chkconfig
  • #service postgresql-9.5 stop
  • 
• Move current data file as .old to /tmp directory using below command.
  • #mv /var/lib/pgsql/9.5/data/ /tmp/data.old
• Uninstall 9.5 version of Postgres SQL using :
  • yum remove postgresql*
• Install  PostgreSQL v10:
  • #yum -y install https://download.postgresql.org/pub/repos/yum/10/redhat/rhel-7-x86_64/pgdg-centos10-10-2.noarch.rpm
  • #yum -y install postgresql10-server
  • enable service autostart using
    • #chkconfig postgresql-10 on
• Initialise the database
  • service postgresql-10 initdb
  • as suggested by my friend miguel if above step is not working then use this (/usr/pgsql-10/bin/postgresql-10-setup initdb)
• Copy the pg_hba.conf and postgresql.conf from old backed up directory to new directory , this will save some time or you can go ahead and edit existing files with required settings.
  • cp /data.old/pg_hba.conf /var/lib/pgsql/10/data/
  • cp /data.old/postgresql.conf /var/lib/pgsql/10/data/
  • service postgresql-10 start
• Restore backup using below commands:
  • su – postgres
  • psql -d postgres -f /tmp/pg9dbbackup

you can run the reconfigure-database command and that’s it. (change your environment variable accordingly)

This will complete the database upgrade and database migration procedure.

With the release of new Container Service Extension (CSE) version 1.2.7 due to vulnerability related to docker (CVE-2019-5736 ) for both Ubuntu and Photon OS templates , it is very important to update the CSE ASAP , here is the procedure to help you to upgrade the CSE easily.

Pre-requisite:

• Check the release notes Here for version compatibility.

Upgrade procedure for Cloud Admins:

• Update CSE to 1.2.7 ( follow procedure below)
• Update the templates (follow procedure below)

Upgrading CSE Server Software

•  Stop CSE Server services gracefully.
  • #vcd cse system stop -y
  • 

• Reinstall container-service-extension using Python Package Index:
  • #pip3 install –user –upgrade container-service-extension
  • 
• Review the configuration file for any new options introduced or deprecated in the new version. cse sample  can be used to generate a new sample config file as well.
  • 
  • Follow the steps listed here , to edit your environment variable for CSE to use.
• If the previously generated templates are no longer supported by the new version, delete the templates and re-generate new ones using below command.
  • cse install -c mysample.yaml –update
  • 
• If running CSE as a service, start the new version of the service with
  • $systemctl start cse
  • 

Upgrade procedure for Tenant Users:

• Delete clusters that were created with older templates. Recreate clusters with new templates
• Alternatively, tenant-users can update docker version manually on their existing clusters.

This completes the upgrade procedure , go ahead and let the customer consume Kubernetes as a Service from your platform.

Trying to upgrade CSE to latest version of CSE 1.2.7 and during upgrade process facing error , like this: Missing keys in config file ‘service’ section: {‘enforce_authorization’}

with this new release there are many new options has been added in to configuration file considering PKS integration , so to resolve this issue , there are two options:

• Create a new sample config.yaml file using command:
  • cse sample > myconfig.yaml  – and reconfigure it.
• If don’t need PKS integration as of now and edit the existing config.yaml file and add “enforce_authorization: false” in to service section
  • 

and once you done the changes , re-run the command and it should now successfully complete the process.

this new process has not been documented properly in the CSE git page 🙂

In continuation of my last post on Kubernetes as a service on vCloud Director , here is the next post on installation of Container Server Extension on vCloud Director.

This post applies to CSE version 1.2.5

CSE Installation

This installation procedure applies to Client VM as well as CSE Server VM. For this installation i will leverage a Photon OS 2.0 VM based on the official OVA which is available here. deploy OVA following the standard OVA deployment procedure.Once deployed, make sure you configure static IP and configure networking correctly based on your environment and ensure that this machine can reach to internet to download necessary binaries.

Configure Static IP on Photon OS

Edit file 99-dhcp-en.network inside directory /etc/systemd/network  and change as below.

By default ping is disabled on this , so open firewall using below commands:

Now Install Python related binaries using below command:

root@photon-machine [ ~ ]# tdnf install -y build-essential python3-setuptools python3-tools python3-pip python3-devel

root@photon-machine [ ~ ]# pip3 install –upgrade pip (double dash –)

Install CSE Software:

Now install and verify the installation CSE:

root@photon-machine [ ~ ]# pip3 install container-service-extension

This completes installation of CSE , now we need to enable CSE client on this VM.

Enable CSE Client:

Go and edit ~/.vcd-cli/profiles.yaml  file to include this section: (exactly like in Image)

vCD Prerequisites:

There are many important requirements that must be fulfilled to install CSE successfully on vCD.

• Catalog Organization creation:

• Create a VDC within the org that has an external org network in which vApps may be instantiated and sufficient storage to create vApps and publish them as templates. The external network connection is required to enable template VMs to download packages during configuration. The process as follows:
  • CSE server will upload base OS image to vCloud Director in a CSE Catalog
  • CSE server will deploy the template as a VM on a Org VDC Network that requires internet access and will download and install required kubernetes and docker binaries.
  • CSE will then validate the VM and capture as vApp template and add it back to the CSE Catalog as a valid item for deploying container hosts.

• Create a user in the org with privileges necessary to perform operations like configuring AMQP, creating public catalog entries, and managing vApps.
• A good network connection from the host running CSE installation to vCD as well as the Internet. This avoids intermittent failures in OVA upload/download operations.

CSE Server Config File:

The CSE server is controlled by a yaml configuration file that must be filled out prior to installation. Once vCD pre-requisites are ready,  You can generate a sample file using below command:

#cse sample > config.yaml  ( cse sample generates sample config yaml)

Run above command on above VM which we have prepared for our CSE server , This file is having five sections , which i am going to cover one by one.

AMQP Section:

• During CSE Server installation, CSE will configure AMQP to ensure communication between vCD and the running CSE server. if vCD has already been configured then skip this section while running install command , if vCD has not been configured with AMQP configuration then enter information in this section which will automatically go and configure this for you in vCD. Configure this section as described below:

vCD Section:

• This section is self explanatory , you need to specify vCD related details (ensure API version is related to vCD version):

vCS Section:

• In this section provide vCenter information like VC name and credential.

 Service Section:

• The service section specifies the number of threads to run in the CSE server process.

Broker Section:

• The broker section contains properties to define resources used by the CSE server including org and VDC as well as template definitions. The following Image summarise key parameters. More Details can be found here

• Sample Config.yaml file can be downloaded from config.

CSE SERVER INSTALLATION:

• Once your are ready with file run CSE install command to start the installation. ( as said earlier we need to create a VM on which CSE server must be installed by the vCloud Director System/Cloud Administrator.The CSE appliance must be reachable to vCenter , vCD and AMQP servers. i am installing on the VM which i have prepared in first section)
• #cse install -c config.yaml –ssh-key=$HOME/.ssh/id_rsa.pub –ext config -amqp skip
• I am skipping amqp configuration as “AMQP” is already configured in my vCD.

• it failed due to some issue , so i have to rerun the command after fixing the issue and same can be done multiple times.

• Once installation is completed , check the installation status using:
• #cse check –config config.yaml –check-install

• Now to validate that CSE has been registered in vCD Use “vcd-cli” command line, check that the extension has been registered in vCD:

Running CSE Server as a Service:

• create a file named “cse.sh”  inside directory /home/vmware with following content:
  • 
• create file name cse.service inside directory /etc/systemd/system with following content:
  • 
• Once installed you can start the CSE service daemon using #systemctl start cse . To enable, disable, and stop the CSE service, use CSE client.
  • 

Setting the API Extension Timeout

• The API extension timeout is the number of seconds that vCD waits for a response from the CSE server extension. The default value is 10 seconds, which may be too short for some environments. To change the time follow the steps :

  • On the vCloud Director cell run:

  • Go to Cd /opt/vmware/vcloud-director/bin and run below commands -l to list -v to Set.

Enable CSE

• Login to vCD and enable the CSE using below commands…

This completes the installation of Container Server Extension and allow providers to offer Kubernetes as a Service to their customers. feel free to share your experience on this installation.

There are lots of partners looking for a solution which can automate the entire deployment of vCD based cloud once racking , stacking and cabling is done for their infrastructure that is where VMware Cloud Provider Pod helps…Basically Cloud Provider Pod automate the deployment of VMware-based clouds. A Cloud Provider Pod-deployed stack adheres to VMware Validated Design principles and is thoroughly tested for interoperability and performance. It is also tested for cloud-scale and is built to handle rigorous Cloud Provider workloads. It deploys technologies with core provider capabilities such as data center extension, cloud migration, and multi-tenancy and chargeback, and helps achieve the fastest path to VMware-based cloud services delivery. Cloud provider POD help cloud providers time to market and help in improving service delivery.

Features:

Cloud Provider Pod 1.1 : Supported Interoperable Version for Deployment

POD Designer walk through

The Cloud Provider Pod Designer offers Providers the choice to start with a VMware Validated Design (CONFIGURE YOUR CLOUD) or the Advanced Design which is custom designer based on your environment specific requirement not as per VMware validated design.

The main difference between the VMware Validated Design and the Advanced Configuration modes is that you can choose to use NFS, iSCSI or Fibre Channel as your storage options. The setup of BGP AS and other options is also not required, but can be done.VVD designer start with asking basic details about your cloud environment that you want to build , Click on “Configure Your Cloud” which will take you to below screen where you need to fill in “General Parameters”

Next will take you the screen where you need to choose optional packages that you want to add/exclude from your deployment.

Next will take you to Resource cluster selection , where you need to choose how many resource cluster your deployment will have and within that resource cluster how many host that you would have.

In the next screen , Enter your environment variable like DNS ,NTP  etc…

Enter your Management Cluster’s Networking and public facing ip addressing in “External/DMZ IP Assignment” and in MAC Addresses , you can add the MAC addresses for the hosts during the Cloud Provider Pod Designer workflow, or later during the deployment workflow. The number of available MAC addresses text boxes depends on how many hosts have been configured on the Sizing page

Enter your resource cluster details like VXLAN Segment etc…

Choose Hypervisor’s NIC allocation.

Enter License Keys now or post deployment also licenses can be assigned.

“Generate all Documentation Files”  –  This is very important and all the providers will like it , it basically automate the creation of design document and configuration work book of your environment , which was the biggest pain where Architects/consultants used to spend lots of time writing design document with visio’s etc.. this is all automatically get generated using CPod.

Once you click on “Generate Configuration” , it will generate your deployment bundle and documentation and email it to you then you can use “Cloud Provider Pod Deployer” to start the deployment. here is overall flow of the entire process

Cloud Provider Pod Deployer

Use Cloud provider deployer to deploy entire infrastructure on a click of a button. Detailed documentation and step-by-step instructions on how to use the Cloud Provider Pod Deployer to create a new environment are available in the Cloud Provider Operations guide. This guide is delivered by the Cloud Provider Pod Designer as part of generated documentation by an Email , which you registered at the start of designer.

Deployer Video is here for your reference – https://www.youtube.com/watch?v=5xOiToL2o94&feature=youtu.be&list=PLunwH0gjkUBi7Mu18nNXxUl6FgzpU3iyd

VMware’s Container Service Extension (CSE)  on vCloud Director is a VMware vCloud Director extension that helps Cloud Providers to Offer Kubernetes-as-a-Service to their tenants , who can easily create and work with Kubernetes clusters. basically it means  using CSE a Service Provider can offer compute resources to tenants secured through a multi-tenant IaaS vCloud Director deployment , and tenants/end users will have the ability to deploy & manage their kubernetes clusters from a self service portal

CSE brings Kubernetes-as-a-service to vCD by creating customized VM templates and enabling tenant/organization administrators to deploy fully functional Kubernetes clusters in self-contained vApps.

CSE Components:

• CSE Client

  • Tenant Organization Administrators and users can use CSE client to handle Kubernetes cluster management. This includes deploying clusters, adding worker nodes, configuring NFS storage etc…
  • CSE client running on a Virtual Machine runs as an extension of vcd-cli which leverages CSE/vCD public API to manage and administer the service.
  • CSE Client which is extension of  vcd-cli offers easy way to manage life cycle of the kubernetes cluster by the Tenant.
  • From this VM CSE commands are getting issued to vCloud Director , which takes these instructions using AMQP message bus to CSE server.

• vCloud Director Based Cloud

  • Service Provider’s cloud administrators will setup vCD, Org Network , catalog etc.
  • vCD will be the platform which will provider compute , network , security and multi-tenancy on which kubernetes clusters will be deployed.
  • CSE will use vCloud Directors Extensibility framework to deploy Kubernetes cluster , kubernetes cluster scaling operations like scale up/down , scale In/out etc..

• CSE Server

  • Service Provider’s cloud administrators will setup CSE config file, CSE Server, and VM templates.
  • You install CSE Server on a new VM and it works in conjunction with vCD extensibility framework.
  • CSE automatically downloads and installs required binaries like Kubernetes , docker , weave etc on a template.
  • Handles CSE Client request for creation and deletion of K8s Cluster and nodes.

User Accessibility of Kubernetes cluster

• Kubectl

  • Developers and other Kubernetes users interact with CSE Kubernetes clusters using kubernetes native “Kubectl” command line tool, For any tenant  users, Kubernetes clusters work like any other Kubernetes cluster implementation. No special knowledge of vCloud Director or CSE administration is required. Such users do not even need a vCloud Director account.

Below figure clearly lists out the required component and their owners , this picture and more details can be accessed from here

Installation Type:

Installation Type dependent on the type of the user as stated in above figure:

Kubernetes User – Install Kubectl on your laptop/desktop.

Tenant Administrator – Install CSE and configure CSE Client on a VM.

Service Provider – Install CSE , Install Messaging Bus , configure and register with vCloud Director.

In the Next series of posts i will be covering installation and configuration of CSE.

vCloud Architecture Toolkit white paper that I have written now has been  published on the cloudsolutions.vmware.com website – this design and deploy guide helps cloud providers to design and deploy vCloud Availability Cloud-to-Cloud DR solution.  This guide is based on real life example and helps cloud providers to successfully plan , design and deploy vCloud Availability Cloud-to-Cloud DR based on version 1.5.

White Paper Download Link

This white paper includes the following chapters to plan your deployment:

• Introduction
•  Use Cases
• vCloud Availability Cloud-to-Cloud DR Components
• vCloud Availability Cloud-to-Cloud DR Node Types and Sizing
• vCloud Availability Cloud-to-Cloud DR Deployment Requirements
• vCloud Availability Cloud-to-Cloud DR Architecture Design
• Physical Design
• Certificate
• Network Communication and Firewalls
• Deployment
• Replication Policy
• Services Management Interface Addresses
• Log Files
• Configuration Files
•  References

I hope this helps in your plan, design  and deployment of vCloud Availability Cloud-to-Cloud DR version 1.5. please feel free to share the feedback to make this white paper more effective and helpful.