[Intro] VMware Edge Compute Stack
VMware Edge Cloud Compute Stack is a Solution for running workloads on the Edge - "Edge" in this case means running a modified Version of ESXi on really small compute footprint. Management is done from a central Appliane called VECO (VMware Edge Cloud Orchestrator).
ECS vs VCF
ECS has nothing todo with VMware Cloud Foundation Edge (VCF-Edge). VCF Edge is just a SKU. VCF has different Edge Reference Architectures. ECS is a different Product, generally meat for smaller compute needs.
Management - Edge Cloud Orchestrator
Unlinke vSphere you won't deploy a vCenter to manage your ECS Hosts. In a classic vSphere Infrastructure the vCenter is Control- and Management Plane. In ECS your Management Plane is the VMware Edge Cloud Orchestrator (VECO). This Appliance runs either as a SaaS Service (on Google Cloud) or can be deployed on-premises.
Control Plane Components are always residing on the Edge Site, so you do not need constant connectivity to VECO.
Compute - ESC Hosts
ECS Hosts (a modified Version of ESXi, but with full ESXi Capabilities) can be even consumer Hardware with very small energy consumption. Like Barebones, NUCs, etc. You do not need beefy enterprise servers.
You can even have USB to RJ45 Network Adapters with the VMware USB Network Driver Fling!
Just follow the VMware VGC for supported Hardware.
Each ESC Host has to be booted from a custom ISO, which can be downloaded from VECO. The ISO is a little bit over 2GB in Size.
GitOps Approach
The unique thing about ECS is you manage the ECS Hosts and the Workloads with a GitOps approach. The configuration of the ECS Hosts like Networking, vSwitch Config is defined via yaml and stored in a Git Repository. The ECS Hosts has FluxCD installed on will check within the GitRepo for new yaml Manifests. FluxCD has a pull based approach - the destination (the ECS Host) will pull the config (yaml Files) themselves. With this approach it isn't a big Problem if the Edge Site does not have constant connectivity.
You cannot modify anything manually, the ECS Hosts will only have the GitRepo as a Configuration Source.
The ESC Hosts must have Network Connectivity to the desired GitRepo you add within VECO.
Example
1#Example host configuration adding a new vSwitch and portgroup and assigning them to vmnic1
2apiVersion: esx.vmware.com/v1alpha1
3kind: HostConfiguration
4metadata:
5 name: vswitch-config
6 namespace: esx-system
7spec:
8 layertype: Incremental
9 profile: |
10 {
11 "esx": {
12 "network_vss": {
13 "switches": [
14 {
15 "name": "vSwitch1",
16 "policy": {
17 "nic_teaming": {
18 "policy": "LOADBALANCE_SRCID",
19 "notify_switches": true,
20 "rolling_order": false,
21 "link_criteria_beacon": "IGNORE",
22 "active_nics": [
23 "vmnic1"
24 ],
25 "standby_nics": []
26 },
27 "security": {
28 "allow_promiscuous": false,
29 "mac_changes": false,
30 "forged_transmits": false
31 },
32 "traffic_shaping": {
33 "enabled": false
34 }
35 },
36 "bridge": {
37 "link_discovery_protocol": {
38 "protocol": "CDP",
39 "operation": "LISTEN"
40 },
41 "nics": [
42 "vmnic1"
43 ]
44 },
45 "num_ports": 1024,
46 "mtu": 1500,
47 "port_groups": [
48 {
49 "name": "VM Network 16",
50 "vlan_id": 0
51 }
52 ]
53 }
54 ]
55 }
56 }
57 }
Workloads
Workloads on ECS can either be classical VMs or Kubernetes Workloads. For Kubernetes Workload a single Kubernetes Worker Node will be deployed on the ESC Host. ESC Kubernetes has nothing todo with Tanzu or vSphere Kubernetes Services. It's a much simpler approach with less footprint.
VM Deployment are done via Virtual Machine Classes (same approach as VMClasses in vSphere IaaS Controle Plane). You can either deploy Linux or Windows Workloads.
Windows Workloads are also deployed with yaml Manifest. You can even use sysprep or Powershell within the yaml. Thats quite a new way to deploy Windows!
Example:
1apiVersion: vmoperator.vmware.com/v1alpha1
2kind: VirtualMachine
3metadata:
4 name: windows-server
5 namespace: default
6spec:
7 className: best-effort-large
8 imageName: windows-server-2022.ova
9 powerState: poweredOn
10 networkInterfaces:
11 - network: workload-network
12 vmMetadata:
13 configMapName: windows-sysprep-config
14 transport: ExtraConfig
15---
16apiVersion: v1
17kind: ConfigMap
18metadata:
19 name: windows-sysprep-config
20 namespace: default
21data:
22 userdata: |
23 #cloud-config
24 hostname: win-server
25 users:
26 - name: administrator
27 passwd: VMware1!
28 runcmd:
29 - powershell -Command "Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))"
30 - choco install -y googlechrome
Kubernetes
To deploy a Kubernetes Worker Node on the ESC Host use the following example yaml File:
1# Example of how to change the default kubernetes worker node configuration to provide more memory or CPUs
2apiVersion: vmoperator.vmware.com/v1alpha1
3kind: VirtualMachineClass
4metadata:
5 name: best-effort-ec-small
6 # No resource requests/reservations.
7 # Memory overcommit possible.
8spec:
9 # VM operator controller name for VirtualMachine class
10 # is different so addition of controllerName attr with invalid
11 # value makes vm operator ignore this VM CRD instance.
12 # This is majorly done because vm operator doesn't have all the
13 # needed capabilities to deploy ec worker vm.
14 controllerName: vmoperator.vmware.com/ecworker
15 hardware:
16 cpus: 4
17 memory: 8Gi
18---
19apiVersion: v1
20kind: Namespace
21metadata:
22 name: ec-system
23---
24apiVersion: vmoperator.vmware.com/v1alpha1
25kind: VirtualMachine
26metadata:
27 name: ec-worker
28 namespace: ec-system
29spec:
30 className: best-effort-ec-small
31 imageName: ec-wrkr.ova
32 powerState: poweredOff
33 vmMetadata:
34 transport: CloudInit
Note
After the worker virtual machine is created and booted, changes to desired state are not applied to it until the next Edge Compute Stack Host reboot.
Kubernetes Networking
Kubernetes Networking on the ESC Hosts is very simple. No need for Ingress Controllers. Workloads can be directly exposed with Nodeport.
Networking
LAN Connectivity will rely on vSwitches, NSX isn't designed for ECS as the footprint is too big. But for the WAN Connectivty, well there is a really good solution: VeloCloud SD-WAN
SD-WAN
VeloCloud SD-WAN can either be deployed as a Virtual Appliance or Hardware. VeloCloud allows for SD-WAN Connectivity, L3+L4+L7 Firewall and VPN Services as well as a lot of other features. VeloCloud SD-WAN deploys so called "Edges", the Appliance that will handle the traffic (Data Plane). The Management will be done from the SD-WAN Orchestrator Appliance, either running onpremises or as a SaaS (running on Google Cloud).
Clusters
As of now two topologies are supported in a Edge Site:
- One Node
- Two Node
I heard some rumors that three Nodes ESC Setup (event with vSAN) is comming ;)