With the release of new Container Service Extension (CSE) version 1.2.7 due to vulnerability related to docker (CVE-2019-5736 ) for both Ubuntu and Photon OS templates , it is very important to update the CSE ASAP , here is the procedure to help you to upgrade the CSE easily.

Pre-requisite:

• Check the release notes Here for version compatibility.

Upgrade procedure for Cloud Admins:

• Update CSE to 1.2.7 ( follow procedure below)
• Update the templates (follow procedure below)

Upgrading CSE Server Software

•  Stop CSE Server services gracefully.
  • #vcd cse system stop -y
  • 

• Reinstall container-service-extension using Python Package Index:
  • #pip3 install –user –upgrade container-service-extension
  • 
• Review the configuration file for any new options introduced or deprecated in the new version. cse sample  can be used to generate a new sample config file as well.
  • 
  • Follow the steps listed here , to edit your environment variable for CSE to use.
• If the previously generated templates are no longer supported by the new version, delete the templates and re-generate new ones using below command.
  • cse install -c mysample.yaml –update
  • 
• If running CSE as a service, start the new version of the service with
  • $systemctl start cse
  • 

Upgrade procedure for Tenant Users:

• Delete clusters that were created with older templates. Recreate clusters with new templates
• Alternatively, tenant-users can update docker version manually on their existing clusters.

This completes the upgrade procedure , go ahead and let the customer consume Kubernetes as a Service from your platform.

Trying to upgrade CSE to latest version of CSE 1.2.7 and during upgrade process facing error , like this: Missing keys in config file ‘service’ section: {‘enforce_authorization’}

with this new release there are many new options has been added in to configuration file considering PKS integration , so to resolve this issue , there are two options:

• Create a new sample config.yaml file using command:
  • cse sample > myconfig.yaml  – and reconfigure it.
• If don’t need PKS integration as of now and edit the existing config.yaml file and add “enforce_authorization: false” in to service section
  • 

and once you done the changes , re-run the command and it should now successfully complete the process.

this new process has not been documented properly in the CSE git page 🙂

In continuation of my last post on Kubernetes as a service on vCloud Director , here is the next post on installation of Container Server Extension on vCloud Director.

This post applies to CSE version 1.2.5

CSE Installation

This installation procedure applies to Client VM as well as CSE Server VM. For this installation i will leverage a Photon OS 2.0 VM based on the official OVA which is available here. deploy OVA following the standard OVA deployment procedure.Once deployed, make sure you configure static IP and configure networking correctly based on your environment and ensure that this machine can reach to internet to download necessary binaries.

Configure Static IP on Photon OS

Edit file 99-dhcp-en.network inside directory /etc/systemd/network  and change as below.

By default ping is disabled on this , so open firewall using below commands:

Now Install Python related binaries using below command:

root@photon-machine [ ~ ]# tdnf install -y build-essential python3-setuptools python3-tools python3-pip python3-devel

root@photon-machine [ ~ ]# pip3 install –upgrade pip (double dash –)

Install CSE Software:

Now install and verify the installation CSE:

root@photon-machine [ ~ ]# pip3 install container-service-extension

This completes installation of CSE , now we need to enable CSE client on this VM.

Enable CSE Client:

Go and edit ~/.vcd-cli/profiles.yaml  file to include this section: (exactly like in Image)

vCD Prerequisites:

There are many important requirements that must be fulfilled to install CSE successfully on vCD.

• Catalog Organization creation:

• Create a VDC within the org that has an external org network in which vApps may be instantiated and sufficient storage to create vApps and publish them as templates. The external network connection is required to enable template VMs to download packages during configuration. The process as follows:
  • CSE server will upload base OS image to vCloud Director in a CSE Catalog
  • CSE server will deploy the template as a VM on a Org VDC Network that requires internet access and will download and install required kubernetes and docker binaries.
  • CSE will then validate the VM and capture as vApp template and add it back to the CSE Catalog as a valid item for deploying container hosts.

• Create a user in the org with privileges necessary to perform operations like configuring AMQP, creating public catalog entries, and managing vApps.
• A good network connection from the host running CSE installation to vCD as well as the Internet. This avoids intermittent failures in OVA upload/download operations.

CSE Server Config File:

The CSE server is controlled by a yaml configuration file that must be filled out prior to installation. Once vCD pre-requisites are ready,  You can generate a sample file using below command:

#cse sample > config.yaml  ( cse sample generates sample config yaml)

Run above command on above VM which we have prepared for our CSE server , This file is having five sections , which i am going to cover one by one.

AMQP Section:

• During CSE Server installation, CSE will configure AMQP to ensure communication between vCD and the running CSE server. if vCD has already been configured then skip this section while running install command , if vCD has not been configured with AMQP configuration then enter information in this section which will automatically go and configure this for you in vCD. Configure this section as described below:

vCD Section:

• This section is self explanatory , you need to specify vCD related details (ensure API version is related to vCD version):

vCS Section:

• In this section provide vCenter information like VC name and credential.

 Service Section:

• The service section specifies the number of threads to run in the CSE server process.

Broker Section:

• The broker section contains properties to define resources used by the CSE server including org and VDC as well as template definitions. The following Image summarise key parameters. More Details can be found here

• Sample Config.yaml file can be downloaded from config.

CSE SERVER INSTALLATION:

• Once your are ready with file run CSE install command to start the installation. ( as said earlier we need to create a VM on which CSE server must be installed by the vCloud Director System/Cloud Administrator.The CSE appliance must be reachable to vCenter , vCD and AMQP servers. i am installing on the VM which i have prepared in first section)
• #cse install -c config.yaml –ssh-key=$HOME/.ssh/id_rsa.pub –ext config -amqp skip
• I am skipping amqp configuration as “AMQP” is already configured in my vCD.

• it failed due to some issue , so i have to rerun the command after fixing the issue and same can be done multiple times.

• Once installation is completed , check the installation status using:
• #cse check –config config.yaml –check-install

• Now to validate that CSE has been registered in vCD Use “vcd-cli” command line, check that the extension has been registered in vCD:

Running CSE Server as a Service:

• create a file named “cse.sh”  inside directory /home/vmware with following content:
  • 
• create file name cse.service inside directory /etc/systemd/system with following content:
  • 
• Once installed you can start the CSE service daemon using #systemctl start cse . To enable, disable, and stop the CSE service, use CSE client.
  • 

Setting the API Extension Timeout

• The API extension timeout is the number of seconds that vCD waits for a response from the CSE server extension. The default value is 10 seconds, which may be too short for some environments. To change the time follow the steps :

  • On the vCloud Director cell run:

  • Go to Cd /opt/vmware/vcloud-director/bin and run below commands -l to list -v to Set.

Enable CSE

• Login to vCD and enable the CSE using below commands…

This completes the installation of Container Server Extension and allow providers to offer Kubernetes as a Service to their customers. feel free to share your experience on this installation.

There are lots of partners looking for a solution which can automate the entire deployment of vCD based cloud once racking , stacking and cabling is done for their infrastructure that is where VMware Cloud Provider Pod helps…Basically Cloud Provider Pod automate the deployment of VMware-based clouds. A Cloud Provider Pod-deployed stack adheres to VMware Validated Design principles and is thoroughly tested for interoperability and performance. It is also tested for cloud-scale and is built to handle rigorous Cloud Provider workloads. It deploys technologies with core provider capabilities such as data center extension, cloud migration, and multi-tenancy and chargeback, and helps achieve the fastest path to VMware-based cloud services delivery. Cloud provider POD help cloud providers time to market and help in improving service delivery.

Features:

Cloud Provider Pod 1.1 : Supported Interoperable Version for Deployment

POD Designer walk through

The Cloud Provider Pod Designer offers Providers the choice to start with a VMware Validated Design (CONFIGURE YOUR CLOUD) or the Advanced Design which is custom designer based on your environment specific requirement not as per VMware validated design.

The main difference between the VMware Validated Design and the Advanced Configuration modes is that you can choose to use NFS, iSCSI or Fibre Channel as your storage options. The setup of BGP AS and other options is also not required, but can be done.VVD designer start with asking basic details about your cloud environment that you want to build , Click on “Configure Your Cloud” which will take you to below screen where you need to fill in “General Parameters”

Next will take you the screen where you need to choose optional packages that you want to add/exclude from your deployment.

Next will take you to Resource cluster selection , where you need to choose how many resource cluster your deployment will have and within that resource cluster how many host that you would have.

In the next screen , Enter your environment variable like DNS ,NTP  etc…

Enter your Management Cluster’s Networking and public facing ip addressing in “External/DMZ IP Assignment” and in MAC Addresses , you can add the MAC addresses for the hosts during the Cloud Provider Pod Designer workflow, or later during the deployment workflow. The number of available MAC addresses text boxes depends on how many hosts have been configured on the Sizing page

Enter your resource cluster details like VXLAN Segment etc…

Choose Hypervisor’s NIC allocation.

Enter License Keys now or post deployment also licenses can be assigned.

“Generate all Documentation Files”  –  This is very important and all the providers will like it , it basically automate the creation of design document and configuration work book of your environment , which was the biggest pain where Architects/consultants used to spend lots of time writing design document with visio’s etc.. this is all automatically get generated using CPod.

Once you click on “Generate Configuration” , it will generate your deployment bundle and documentation and email it to you then you can use “Cloud Provider Pod Deployer” to start the deployment. here is overall flow of the entire process

Cloud Provider Pod Deployer

Use Cloud provider deployer to deploy entire infrastructure on a click of a button. Detailed documentation and step-by-step instructions on how to use the Cloud Provider Pod Deployer to create a new environment are available in the Cloud Provider Operations guide. This guide is delivered by the Cloud Provider Pod Designer as part of generated documentation by an Email , which you registered at the start of designer.

Deployer Video is here for your reference – https://www.youtube.com/watch?v=5xOiToL2o94&feature=youtu.be&list=PLunwH0gjkUBi7Mu18nNXxUl6FgzpU3iyd