SPDX-License-Identifier: Apache-2.0       
Copyright (c) 2020-2021 Intel Corporation

Edge Multi-Cluster Orchestrator (EMCO)

• Background
• EMCO Introduction
  • EMCO Terminology
  • EMCO Architecture
    • Cluster Registration
    • Distributed Application Scheduler
      • Lifecycle Operations
    • Network Configuration Management
      • Lifecycle Operations
    • Distributed Cloud Manager
      • Lifecycle Operations
      • Level-1 Logical Clouds
      • Level-0 Logical Clouds
    • Hardware Platform Awareness
    • OVN Action Controller
    • Traffic Controller
    • Generic Action Controller
    • Resource Synchronizer
    • Placement and Action Controllers in EMCO
    • Status Monitoring and Queries in EMCO
  • EMCO Terminology
  • EMCO API
  • EMCO Authentication and Authorization
  • EMCO Installation With Smart Edge Open Flavor
• EMCO Example: SmartCity Deployment
  • Cluster Setup
  • Project Setup
  • Logical Cloud Setup
  • Deploy SmartCity Application
  • SmartCity Termination
  • Deploy SmartCity Application With HPA Intent
    • HPA intent based on alloctable resource requirements - CPU
    • HPA intent based on non-alloctable resource requirements - VCAC-A

Background

Edge Multi-Cluster Orchestration(EMCO), an Smart Edge Open Building Block, is a Geo-distributed application orchestrator for Kubernetes*. EMCO operates at a higher level than Kubernetes* and interacts with multiple of edges and clouds running Kubernetes. The main objective of EMCO is automation of the deployment of applications and services across multiple clusters. It acts as a central orchestrator that can manage edge services and network functions across geographically distributed edge clusters from different third parties.

Increasingly we see a requirement of deploying ‘composite applications’ in multiple geographical locations. Some of the catalysts for this change are:

• Latency - requirements for new low latency application use cases such as AR/VR. Need for ultra low latency response needed in IIOT and other cases. This requires running some parts of the applications on edges close to the user
• Bandwidth - processing data on edges to avoid costs associated with transporting the data to clouds for processing,
• Context/Promixity - running some part of the applications on edges near the user that require local context
• Privacy/Legal - some data can have legal requirements to not leave a geographic location

Figure 1 - Orchestrate GeoDitributed Edge Applications

NOTE: A ‘composite application’ is a combination of multiple applications with each application packaged as a Helm chart. Based on the deployment intent, various applications of the composite application get deployed at various locations, and get replicated in multiple locations.

Life cycle management of composite applications is complex. Instantiation and terminations of the complex application across multiple K8s clusters (Edges and Clouds), monitoring the status of the complex application deployment, Day 2 operations (Modification of the deployment intent, upgrades etc..) are few complex operations.

Number of K8s clusters (Edges or clouds) could be in tens of thousands, number of complex applications that need to be managed could be in hundreds, number of applications in a complex application could be in tens and number of micro-services in each application of the complex application can be in tens. Moreover, there can be multiple deployments of the same complex applications for different purposes. To reduce the complexity, all these operations are to be automated. There shall be one-click deployment of the complex applications and one simple dashboard to know the status of the complex application deployment at any time. Hence, there is a need for Multi-Edge and Multi-Cloud distributed application orchestrator.

Compared with other multiple-clusters orchestration, EMCO focuses on the following functionalities:

• Enrolling multiple geographically distributed Smart Edge Open clusters and third party cloud clusters.
• Orchestrating composite applications (composed of multiple individual applications) across different clusters.
• Deploying edge services and network functions on to different nodes spread across different clusters.
• Monitoring the health of the deployed edge services/network functions across different clusters.
• Orchestrating edge services and network functions with deployment intents based on compute, acceleration, and storage requirements.
• Supporting multiple tenants from different enterprises while ensuring confidentiality and full isolation between the tenants.

The following figure shows the topology overview for the Smart Edge Open EMCO orchestration with edge and multiple clusters. It also shows an example of deploying SmartCity with EMCO.

Figure 2 - Topology Overview with Smart Edge Open EMCO

All the managed edge clusters and cloud clusters are connected with the EMCO cluster through the WAN network.

• The central orchestration (EMCO) cluster can be installed and provisioned by using the Smart Edge Open Central Orchestrator Flavor.
• The edge clusters and the cloud cluster can be installed and provisioned by using the Smart Edge Open Flavor.
• The composite application - SmartCity is composed of two parts: edge application and cloud (web) application.
  • The edge application executes media processing and analytics on multiple edge clusters to reduce latency.
  • The cloud application is like a web application for additional post-processing, such as calculating statistics and display/visualization on the cloud cluster side.
  • The EMCO user can deploy the SmartCity applications across the clusters. Besides that, EMCO allows the operator to override configurations and profiles to satisfy deployment needs.

This document aims to familiarize the user with EMCO and Smart Edge Open deployment flavor for EMCO installation and provision, and provide instructions accordingly.

EMCO Introduction

EMCO Terminology

EMCO Architecture

The following diagram depicts a high level overview of the EMCO architecture.

Figure 3 - EMCO Architecture

• Cluster Registration Controller registers clusters by cluster owners.
• Distributed Application Scheduler provides a simplified and extensible placement.
• Network Configuration Management handles creation/management of virtual and provider networks.
• Hardware Platform Aware Controller enables scheduling with auto-discovery of platform features/ capabilities.
• Distributed Cloud Manager presents a single logical cloud from multiple edges.
• Secure Mesh Controller auto-configures both service mesh (ISTIO) and security policy (NAT, firewall).
• Secure WAN Controller automates secure overlays across edge groups.
• Resource Syncronizer manages instantiation of resources to clusters.
• Monitoring covers distributed application.

Cluster Registration

A microservice exposes RESTful API. User can register cluster providers and clusters of those providers via these APIs. After preparing edge clusters and cloud clusters, which can be any Kubernetes* cluster, user can onboard those clusters to EMCO by creating a cluster provider and then adding clusters to the cluster provider. After cluster providers are created, the KubeConfig files of edge and cloud clusters should be provided to EMCO as part of the multi-part POST call to the Cluster API.

Additionally, after a cluster is created, labels and key value pairs can be added to the cluster via the EMCO API. Clusters can be specified by label when preparing placement intents.

NOTE: The cluster provider is someone who owns clusters and registers them to EMCO. If an Enterprise has clusters, for example from AWS, then the cluster provider for those clusters from AWS is still considered as from that Enterprise. AWS is not the provider. Here, the provider is someone who owns clusters and registers them here. Since AWS does not register their clusters here, AWS is not considered cluster provider in this context.

Distributed Application Scheduler

The distributed application scheduler microservice:

• Project Management provides multi-tenancy in the application from a user perspective.
• Composite App Management manages composite apps that are collections of Helm Charts, one per application.
• Composite Profile Management manages composite profiles that are collections of profile, one per application.
• Deployment Intent Group Management manages Intents for composite applications.
• Controller Registration manages placement and action controller registration, priorities etc.
• Status Notifier framework allows user to get on-demand status updates or notifications on status updates.
• Scheduler:
  • Placement Controllers: Generic Placement Controller.
  • Action Controllers.

Lifecycle Operations

The Distributed Application Scheduler supports operations on a deployment intent group resource to instantiate the associated composite application with any placement, and action intents performed by the registered placement and action controllers. The basic flow of lifecycle operations on a deployment intent group after all the supporting resources have been created via the APIs are:

• approve: marks that the deployment intent group has been approved and is ready for instantiation.
• instantiate: the Distributed Application Scheduler prepares the application resourcs for deployment, and applies placement and action intents before invoking the Resource Synchronizer to deploy them to the intended remote clusters.
• status: (may be invoked at any step) provides information on the status of the deployment intent group.
• terminate: terminates the application resources of an instantiated application from all of the clusters to which it was deployed. In some cases, if a remote cluster is intermittently unreachable, the instantiate operation may still retry the instantiate operation for that cluster. The terminate operation will cause the instantiate operation to complete (i.e. fail), before the termination operation is performed.
• stop: In some cases, if the remote cluster is intermittently unreachable, the Resource Synchronizer will continue retrying an instantiate or terminate operation. The stop operation can be used to force the retry operation to stop, and the instantiate or terminate operation will complete (with a failed status). In the case of terminate, this allows the deployment intent group resource to be deleted via the API, since deletion is prevented until a deployment intent group resource has reached a completed terminate operation status. Refer to EMCO Resource Lifecycle Operations for more details.

Network Configuration Management

The network configuration management (NCM) microservice:

• Provider Network Management to create provider networks.
• Virtual Network Management to create dynamic virtual networks.
• Controller Registration manages network plugin controllers, priorities etc.
• Status Notifier framework allows user to get on-demand status updates or notifications on status updates.
• Scheduler with Built in Controller - OVN-for-K8s-NFV Plugin Controller.

Lifecycle Operations

The Network Configuration Management microservice supports operations on the network intents of a cluster resource to instantiate the associated provider and virtual networks that have been defined via the API for the cluster. The basic flow of lifecycle operations on a cluster, after all the supporting network resources have been created via the APIs are:

• apply: the Network Configuration Management microservice prepares the network resources and invokes the Resource Synchronizer to deploy them to the designated cluster.
• status: (may be invoked at any step) provides information on the status of the cluster networks.
• terminate: terminates the network resources from the cluster to which they were deployed. In some cases, if a remote cluster is intermittently unreachable, the Resource Synchronizer may still retry the instantiate operation for that cluster. The terminate operation will cause the instantiate operation to complete (i.e. fail), before the termination operation is performed.
• stop: In some cases, if the remote cluster is intermittently unreachable, the Resource Synchronizer will continue retrying an instantiate or terminate operation. The stop operation can be used to force the retry operation to stop, and the instantate or terminate operation will be completed (with a failed status). In the case of terminate, this allows the deployment intent group resource to be deleted via the API, since deletion is prevented until a deployment intent group resource has reached a completed terminate operation status.

Distributed Cloud Manager

The Distributed Cloud Manager (DCM) provides the Logical Cloud abstraction and effectively completes the concept of “multi-cloud”. One Logical Cloud is a grouping of one or many clusters, each with their own control plane, specific configurations and geo-location, which get partitioned for a particular EMCO project. This partitioning is made via the creation of distinct, isolated namespaces in each of the Kubernetes* clusters that thus make up the Logical Cloud.

A Logical Cloud is the overall target of a Deployment Intent Group and is a mandatory parameter (the specific applications under it further refine what gets run and in which location). A Logical Cloud must be explicitly created and instantiated before a Deployment Intent Group can be instantiated.

Due to the close relationship with Clusters, which are provided by Cluster Registration (clm) above, it is important to understand the mapping between the two. A Logical Cloud groups many Clusters together but a Cluster may also be grouped by multiple Logical Clouds, effectively turning the cluster multi-tenant. The partitioning/multi-tenancy of a particular Cluster, via the different Logical Clouds, is done today at the namespace level (different Logical Clouds access different namespace names, and the name is consistent across the multiple clusters of the Logical Cloud).

Figure 4 - Mapping between Logical Clouds and Clusters

Lifecycle Operations

Prerequisites to using Logical Clouds:

• with the project-less Cluster Registration API, create the cluster providers, clusters and optionally cluster labels.
• with the Distributed Application Scheduler API, create a project which acts as a tenant in EMCO.

The basic flow of lifecycle operations to get a Logical Cloud up and running via the Distributed Cloud Manager API is:

• Create a Logical Cloud specifying the following attributes:
  • Level: either 1 or 0, depending on whether an admin or a custom/user cloud is sought - more on the differences below.
  • (for Level-1 only) Namespace name - the namespace to use in all of the Clusters of the Logical Cloud.
  • (for Level-1 only) User name - the name of the user that will be authenticating to the Kubernetes* APIs to access the namespaces created.
  • (for Level-1 only) User permissions - permissions that the user specified will have in the namespace specified, in all of the clusters.
• (for Level-1 only) Create resource quotas and assign them to the Logical Cloud created: this specifies what quotas/limits the user will face in the Logical Cloud, for each of the Clusters.
• Assign the Clusters previously created with the project-less Cluster Registration API to the newly-created Logical Cloud.
• Instantiate the Logical Cloud. All of the clusters assigned to the Logical Cloud are automatically set up to join the Logical Cloud. Once this operation is complete, the Distributed Application Scheduler’s lifecycle operations can be followed to deploy applications on top of the Logical Cloud.

Apart from the creation/instantiation of Logical Clouds, the following operations are also available:

• Terminate a Logical Cloud - this removes all of the Logical Cloud -related resources from all of the respective Clusters.
• Delete a Logical Cloud - this eliminates all traces of the Logical Cloud in EMCO.

Level-1 Logical Clouds

Logical Clouds were introduced to group and partition clusters in a multi-tenant way and across boundaries, improving flexibility and scalability. A Level-1 Logical Cloud is the default type of Logical Cloud providing just that much. When projects request a Logical Cloud to be created, they provide what permissions are available, resource quotas and clusters that compose it. The Distributed Cloud Manager, alongside the Resource Synchronizer, sets up all the clusters accordingly, with the necessary credentials, namespace/resources, and finally generating the kubeconfig files used to authenticate/reach each of those clusters in the context of the Logical Cloud.

Level-0 Logical Clouds

In some use cases, and in the administrative domains where it makes sense, a project may want to access raw, unmodified, administrator-level clusters. For such cases, no namespaces need to be created and no new users need to be created or authenticated in the API. To solve this, the Distributed Cloud Manager introduces Level-0 Logical Clouds, which offer the same consistent interface as Level-1 Logical Clouds to the Distributed Application Scheduler. Being of type Level-0 means “the lowest-level”, or the administrator level. As such, no changes will be made to the clusters themselves. Instead, the only operation that takes place is the reuse of credentials already provided via the Cluster Registration API for the clusters assigned to the Logical Cloud (instead of generating new credentials, namespace/resources and kubeconfig files).

Hardware Platform Awareness

The Hardware Platform Awareness (HPA) is a feature that enables placement of workloads in different Kubernetes clusters based on availability of hardware resources in those clusters. Some examples of hardware resources are CPU, memory, devices such as GPUs, and PCI Virtual Functions (VFs) in SR-IOV capable PCI devices. HPA Intents can be added to the deployment intent group to express hardware resource requirements for individual microservices within an application.

To elaborate, HPA tracks two kinds of resources:

A. Capabilities, also called Non-Allocatable Resources: A workload may need CPUs with specific instruction sets such as AVX512, or a node in which Huge Pages are enabled for memory. Such capabilities are expressed in Kubernetes as a label on the node. Since capabilities are properties rather than quantities, HPA models them as resources for which one cannot specify how many of them are needed: they are not allocatable.

B. Capacities, also called Allocatable Resources: A workload may need, say, 2 CPUs, 4 GB RAM and 1 GPU. HPA Intents for such quantifiable resources state how many of each resource type is needed. So they are called allocatable resources.

Every HPA resource has a name and one or more values. The name is exactly the same as the one used by Kubernetes. For example, the name feature.node.kubernetes.io/cpu-cpuid.AVX512BW identifies nodes with CPUs that have the AVX512 instruction set. A resource specification for it would look like this:

 resource: {"key":"feature.node.kubernetes.io/cpu-cpuid.AVX512BW", "value":"true"}

For non-allocatable resources, the key is the resource name as reported by the Node Feature Discovery feature in Kubernetes. The value would be the same as what one would use in the nodeSelector field of a Kubernetes pod manifest for that resource. For the example above, the value would be true.

Allocatable resources fall into two categories: (a) those treated by Kubernetes as distinct types, namely, cpu and memory, and (b) generic resources, such as devices reported by device plugins in the cluster nodes. For each of these, as per the Kubernetes model, one can assign a requests parameter, which is the minimum resource amount that needs to be guaranteed for the workload to function. Optionally, one can also assign a limits parameter, which is the maximum amount of that resource that can be assigned. Both parameters in the HPA intent get added to the pod manifest of the microservice specified in the HPA intent, so that the scheduler of the Kubernetes cluster on which the microservice gets placed can act on them for node-level placement.

Only the requests field is used for placement decisions; the limits parameter (if present) is passed transparently to Kubernetes but otherwise ignored. The HPA placement tracks the total capacity of each resource in each cluster, and subtracts the number guaranteed to each microservice (i.e. requests) to determine the free number of each resource in each cluster. If the application’s Helm chart specifies default resources, the HPA intent values will override them.

Resource specifications in Kubernetes are made at the level of containers. HPA intents therefore require the container name to be specified. However, non-allocatable resources often correspond to node-level properties or capabilities, and they would be common to all containers within a pod.

The intent author should note that Kubernetes has many implicit semantics for CPU management policy based on requests and limits fields for cpu and memory. In particular, these fields can be used to decide the QoS class of the pod and its CPU affinity. Specifically, to get exclusive CPUs for a pod, the following need to be done:

• In each node of the relevant Kubernetes clusters, set the kubelet option --cpu-manager-policy=static. This enables the static CPU manager policy in those nodes.
• In the HPA intent, specify both requests and limits for cpu and ensure they are equal. Do the same for memory. This puts the pod in Guaranteed QoS class.
• In the HPA intent, ensure the CPU counts are integers. This enables exclusive CPU access.

To arrange for a microservice to get access to a specific PCI device or PCI Virtual Functions (VFs) from an SR-IOV device, it is assumed that the necessary system prerequisites, such as installing device plugins, have been addressed in each relevant cluster. Often, the hardware requirements of a microservice has two parts: (a) the type and count of needed devices and (b) a specific version of the device driver to operate those devices. HPA expects that the appropriate driver version has been published as a node label (non-allocatable resource in HPA terms). Then the HPA intent would have two parts:

• A non-allocatable resource requirement, for the driver/software version. Example: resource: {"key": "foo.driver.version", "value": "10.1.0"}
• An allocatable resource requirement, specifying the device resource name, requests and limits. Example: resource: {"name": "myvendor.com/foo", "requests": "2", "limits": "2"}

In general, the HPA resource specifications and semantics are based on the corresponding Kubernetes concepts, consistent with the principle that EMCO automates Kubernetes deployments rather than pose yet another layer for the user to learn. So, please consult the Kubernetes documentation for further details.

Examples of HPA Intents can be seen in the repository within the folder src/placement-controllers/hpa/examples.

In the context of EMCO architecture, HPA provides a placement controller and an action controller. The HPA placement controller always runs after the generic placement controller.

Please read the release notes regarding caveats and known limitations.

OVN Action Controller

The OVN Action Controller (ovnaction) microservice is an action controller which may be registered and added to a deployment intent group to apply specific network intents to resources in the composite application. It provides the following functionalities:

• Network intent APIs which allow specification of network connection intents for resources within applications.
• On instantiation of a deployment intent group configured to utilize the ovnaction controller, network interface annotations will be added to the pod template of the identified application resources.
• ovnaction supports specifying interfaces which attach to networks created by the Network Configuration Management microservice.

Traffic Controller

The traffic controller microservice provides a way to create network policy resources across edge clusters. It provides inbound RESTful APIs to create intents to open the traffic from clients, and provides change and delete APIs for update and deletion of traffic intents. Using the information provided through intents, it also creates a network policy resource for each of the application servers on the corresponding edge cluster.

NOTE:For network policy to work, edge cluster must have network policy support using CNI such as calico.

Generic Action Controller

The generic action controller microservice is an action controller which may be registered with the central orchestrator. It can achieve the following usecases:

• Create a new Kubernetes* object and deploy that along with a specific application which is part of the composite Application. There are two variations here:

  • Default : Apply the new object to every instance of the app in every cluster where the app is deployed.
  • Cluster-Specific : Apply the new object only where the app is deployed to a specific cluster, denoted by a cluster-name or a list of clusters denoted by a cluster-label.

• Modify an existing Kubernetes* object which may have been deployed using the Helm chart for an app, or may have been newly created by the above mentioned usecase. Modification may correspond to specific fields in the YAML definition of the object.

To achieve both the usecases, the controller exposes RESTful APIs to create, update and delete the following:

• Resource - Specifies the newly defined object or an existing object.
• Customization - Specifies the modifications (using JSON Patching) to be applied on the objects.

Resource Synchronizer

This microservice is the one which deploys the resources in edge/cloud clusters. ‘Resource contexts’ created by various microservices are used by this microservice. It takes care of retrying, in case the remote clusters are not reachable temporarily.

Placement and Action Controllers in EMCO

This section illustrates some key aspects of the EMCO controller architecture. Depending on the needs of a composite application, intents that handle specific operations for application resources (e.g. addition, modification, etc.) can be created via the APIs provided by the corresponding controller API. The following diagram shows the sequence of interactions to register controllers with EMCO.

Figure 5 - Register placement and action controllers with EMCO

This diagram illustrates the sequence of operations taken to prepare a Deployment Intent Group that utilizes some intents supported by controllers. The desired set of controllers and associated intents are included in the definition of a Deployment Intent Group to satisfy the requirements of a specific deployed instance of a composite application.

Figure 6 - Create a Deployment Intent Group

When the Deployment Intent Group is instantiated, the identified set of controllers are invoked in order to perform their specific operations.

Figure 7 - Instantiate a Deployment Intent Group

In this initial release of EMCO, a built-in generic placement controller is provided in the orchestrator. As described above, the three provided action controllers are the OVN Action, Traffic and Generic Action controllers.

Status Monitoring and Queries in EMCO

When a resource like a Deployment Intent Group is instantiated, status information about both the deployment and the deployed resources in the cluster are collected and made available for query by the API. The following diagram illustrates the key components involved. For more information about status queries see EMCO Resource Lifecycle Operations.

Figure 8 - Status Monitoring and Query Sequence

EMCO Terminology

EMCO API

For user interaction, EMCO provides RESTful API. Apart from that, EMCO also provides CLI. For the detailed usage, refer to EMCO CLI

NOTE: The EMCO RESTful API is the foundation for the other interaction facilities like the EMCO CLI, EMCO GUI (available in the future) and other orchestrators.

EMCO Authentication and Authorization

EMCO uses Istio and other open source solutions to provide Multi-tenancy solution leveraging Istio Authorization and Authentication frameworks. This is achieved without adding any logic to EMCO microservices.

• Authentication and Authorization for EMCO users is done at the Istio Ingress Gateway, where all the traffic enters the cluster.

• Istio along with autherservice (Istio ecosystem project) enables request-level authentication with JSON Web Token (JWT) validation. Authservice is an entity that works along side with Envoy proxy. It is used to work with external IAM systems (OAUTH2). Many Enterprises have their own OAUTH2 server for authenticating users and provide roles.

• Authservice and ISTIO can be configured to talk to multiple OAUTH2 servers. Using this capability EMCO can support multiple tenants, for example one tenant belonging to one project.

• Using Istio AuthorizationPolicy access to different EMCO resources can be controlled based on roles defined for the users.

The following figure shows various EMCO services running in a cluster with Istio.

Figure 9 - EMCO setup with Istio and Authservice

The following figure shows the authentication flow with EMCO, Istio and Authservice

Figure 10 - EMCO Authenication with external OATH2 Server

Detailed steps for configuring EMCO with Istio can be found in EMCO Integrity and Access Management document.

Steps for EMCO Authentication and Authorization Setup:

• Install and Configure Keycloak Server to be used in the setup. This server runs outside the EMCO cluster
  • Create a new realm, add users and roles to Keycloak
• Install Istio in the Kubernetes* cluster where EMCO is running
• Enable Sidecar Injection in EMCO namesapce
• Install EMCO in EMCO namespace (with Istio sidecars)
• Configure Istio Ingress gateway resources for EMCO Services
• Configure Istio Ingress gateway to enable running along with Authservice
• Apply EnvoyFilter for Authservice
• Apply Authentication and Authorization Policies

EMCO Installation With Smart Edge Open Flavor

EMCO supports multiple deployment options. Converged Edge Experience Kits offers the central_orchestrator flavor to automate EMCO build and deployment as mentioned below.

• The first step is to prepare one server environment which needs to fulfill the preconditions.
• Place the EMCO server hostname in controller_group/hosts/ctrl.openness.org: dictionary in inventory.yml file of converged-edge-experience-kit.
• Update the inventory.yaml file by setting the deployment flavor as central_orchestrator

    ---
    all:
      vars:
        cluster_name: central_orchestrator_cluster
        flavor: central_orchestrator
        single_node_deployment: false
        limit: controller_group
    controller_group:
      hosts:
        ctrl.openness.org:
          ansible_host: <controller ip address>
          ansible_user: openness
    edgenode_group:
      hosts:
    edgenode_vca_group:
      hosts:
    ptp_master:
      hosts:
    ptp_slave_group:
      hosts:
    ... 
  

  NOTE: edgenode_group: and edgenode_vca_group: are not required for configuration, since EMCO micro services just need to be deployed on the Kubernetes* control plane node.

NOTE: for more details about deployment and defining inventory please refer to CEEK getting started page.

• Run script python3 deploy.py. Deployment should complete successfully. In the flavor, harbor registry is deployed to provide images services as well.

# kubectl get pods -n emco
NAMESPACE NAME                           READY   STATUS   RESTARTS AGE
emco      clm-6979f6c886-tjfrv           1/1     Running  0        14m
emco      dcm-549974b6fc-42fbm           1/1     Running  0        14m
emco      dtc-948874b6fc-p2fbx           1/1     Running  0        14m
emco      etcd-5f646586cb-p7ctj          1/1     Running  0        14m
emco      gac-788874b6fc-p1kjx           1/1     Running  0        14m
emco      mongo-5f7d44fbc5-n74lm         1/1     Running  0        14m
emco      ncm-58b85b4688-tshmc           1/1     Running  0        14m
emco      orchestrator-78b76cb547-xrvz5  1/1     Running  0        14m
emco      ovnaction-5d8d4447f9-nn7l6     1/1     Running  0        14m
emco      rsync-99b85b4x88-ashmc         1/1     Running  0        14m

Besides that, Smart Edge Open EMCO also provides Azure templates and supports deployment automation for EMCO cluster on Azure public cloud. More details refer to Smart Edge Open Development Kit for Microsoft Azure.

EMCO Example: SmartCity Deployment

• The SmartCity application is a sample application that is built on top of the OpenVINO™ and Open Visual Cloud software stacks for media processing and analytics. The composite application is composed of two parts: EdgeApp + WebApp (cloud application for additional post-processing such as calculating statistics and display/visualization)
• The edge cluster (representing regional office), the cloud cluster and the EMCO are connected with each other.
• The whole deployment architecture diagram is shown as below:

Figure 11 - SmartCity Deployment Architecture Overview

The example steps are shown as follows:

• Prerequisites
  • Make one edge cluster and one cloud cluster ready by using Smart Edge Open Flavor.
    • If testing with HPA intent, need to prepare two edge clusters.
  • Prepare one server with a vanilla CentOS* 7.9.2009 for EMCO installation.
• EMCO installation
• Cluster setup
• Project setup
• Logical cloud Setup
• Deploy SmartCity application

EMCO installation

Follow the guidance as EMCO Installation With Smart Edge Open Flavor, logon to the EMCO host server and maker sure that Harbor and EMCO microservices are in running status.

Cluster Setup

The step includes:

• Prepare edge and cloud clusters kubeconfig files, SmartCity helm charts and relevant artifacts.
• Register clusters provider by EMCO CLI.
• Register provider’s clusters by EMCO CLI.
• Register EMCO controllers and resource synchroizer by EMCO CLI.

1. On the edge and cloud cluster, run the following command to make Docker logon to the Harbor deployed on the EMCO server, thus the clusters can pull SmartCity images from the Harbor:

    HARBORRHOST=<emco_harbor_registry_host_ip>:30003
   
    cd /etc/docker/certs.d/
    mkdir ${HARBORRHOST}
    cd ${HARBORRHOST}
    curl -sk -X GET https://${HARBORRHOST}/api/v2.0/systeminfo/getcert -H "accept: application/json" -o harbor.crt 
    HARBORRPW=Harbor12345
    docker login ${HARBORRHOST} -u admin -p ${HARBORRPW}
   

   NOTE: should be `:30003`.

2. On the EMCO server, download the scripts,profiles and configmap JSON files.

3. Artifacts Preparation for clusters’s kubeconfig, smartcity helm charts and other relevant artifacts Run the command for the environment setup with success return as below:

    # cd cli-scripts/
    # ./setup_env.sh -e <EMCO_IP> -d <EDGE_HOST_IP> -c <CLOUD_HOST_IP> -r
   

   NOTE: EMCO CLI is used in the setup script, and the steps include SmartCity github repo clone, docker images building, helm charts prepration and clusters configuration information preparation…etc.

4. Run the command for the clusters setup with expected result as below:

    # cd cli-scripts/
    # ./01_apply.sh
    ....
    URL: cluster-providers/smartcity-cluster-provider/clusters/edge01/labels Response Code: 201 Response: {"label-name":"LabelSmartCityEdge"}
    URL: cluster-providers/smartcity-cluster-provider/clusters/cloud01/labels Response Code: 201 Response: {"label-name":"LabelSmartCityCloud"}
   

   NOTE: The cluster setup steps include clusters providers registration, clusters registration, adding labels for the clusters, EMCO controller creation and registration.

   NOTE: The 01_apply.sh script invokes EMCO CLI tool - emcoctl and applies resource template file - 01_clusters_template.yaml which contains the clusters related resources to create in EMCO. For example: Cluster Providers, Labels…etc.

Project Setup

The step invokes EMCO CLI and registers a project which groups SmartCity application under a common tenant.

Run the command for the project setup with expected result as below:

 ```shell
 # cd cli-scripts/
 # ./02_apply.sh

 Using config file: emco_cfg.yaml
 http://localhost:31298/v2
 URL: projects Response Code: 201 Response: {"metadata":{"name":"project_smtc","description":"","UserData1":"","UserData2":""}}
 ``` The `02_apply.sh` script invokes EMCO CLI tool - `emcoctl` and applies resource template file - `02_project_template.yaml` which contains the projects related resources to create in EMCO.

Logical Cloud Setup

The step invokes EMCO CLI and registers a logical cloud associated with the physical clusters.

Run the command for the logical cloud setup with expected result as below:

 ```shell
 # cd cli-scripts/
 # ./03_apply.sh

 Using config file: emco_cfg.yaml
 http://localhost:31877/v2
 URL: projects/project_smtc/logical-clouds Response Code: 201 Response: {"metadata":{"name":"default","description":"","userData1":"","userData2":""},"spec":{"namespace":"","level":"0","user":{"user-name":"","type":"","user-permissions":null}}}
 http://localhost:31877/v2
 URL: projects/project_smtc/logical-clouds/default/cluster-references Response Code: 201 Response: {"metadata":{"name":"lc-edge01","description":"","userData1":"","userData2":""},"spec":{"cluster-provider":"smartcity-cluster-provider","cluster-name":"edge01","loadbalancer-ip":"0.0.0.0","certificate":""}}
 http://localhost:31877/v2
 URL: projects/project_smtc/logical-clouds/default/instantiate Response Code: 200 Response:
 ``` The `03_apply.sh` script invokes EMCO CLI tool - `emcoctl` and applies resource template file - `03_logical_cloud_template.yaml` which contains the logical cloud related resources to create in EMCO.

Deploy SmartCity Application

The setup includes:

• Onboard SmartCity Application helm charts and profiles
• Create generic placement intent to specify the edge/cloud cluster locations for each applicaiton of SmartCity
• Create deployment intent references of the generic placement intent and generic actions intent for SmartCity generic kuberenetes resource: configmap, secret…etc.
• Approve and Instantiate SmartCityp deployment

1. Run the command for the SmartCity application deployment with expected result as below:

    # cd cli-scripts/
    # ./04_apply.sh
   
    http://localhost:31298/v2
    URL: projects/project_smtc/composite-apps/composite_smtc/v1/deployment-intent-groups/smtc-deployment-intent-group/approve Response Code: 202 Response:
    http://localhost:31298/v2
    URL: projects/project_smtc/composite-apps/composite_smtc/v1/deployment-intent-groups/smtc-deployment-intent-group/instantiate Response Code: 202 Response:
   

   NOTE: EMCO supports generic K8S resource configuration including configmap, secret,etc. The example offers the usage about configmap configuration to the clusters.

   NOTE: The 04_apply.sh script invokes EMCO CLI tool - emcoctl and applies resource template file - 04_apps_template.yaml which contains the application related resources to create in EMCO, for example deployment-intent, application helm chart entries, override profiles, configmap…etc. The placement intent for the use case is cluster label name and provider name.

2. Verify SmartCity Application Deployment Information. The pods on the edge cluster are in the running status as shown as below:

    # kubectl get pods
    NAME                                              READY STATUS  RESTARTS AGE
    traffic-office1-alert-5b56f5464c-ldwrf            1/1   Running 0        20h
    traffic-office1-analytics-traffic-6b995d4d6-nhf2p 1/1   Running 0        20h
    traffic-office1-camera-discovery-78bccbdb44-k2ffx 1/1   Running 0        20h
    traffic-office1-cameras-6cb67ccc84-8zkjg          1/1   Running 0        20h
    traffic-office1-db-84bcfd54cd-ht52s               1/1   Running 1        20h
    traffic-office1-db-init-64fb9db988-jwjv9          1/1   Running 0        20h
    traffic-office1-mqtt-f9449d49c-dwv6l              1/1   Running 0        20h
    traffic-office1-mqtt2db-5649c4778f-vpxhq          1/1   Running 0        20h
    traffic-office1-smart-upload-588d95f78d-8x6dt     1/1   Running 1        19h
    traffic-office1-storage-7889c67c57-kbkjd          1/1   Running 1        19h
   

   The pods on the cloud cluster are in the running status as shown as below:

    # kubectl get pods
    NAME                             READY   STATUS    RESTARTS   AGE
    cloud-db-5d6b57f947-qhjz6        1/1     Running   0          20h
    cloud-storage-5658847d79-66bxz   1/1     Running   0          96m
    cloud-web-64fb95884f-m9fns       1/1     Running   0          20h
   

3. Verify Smart City GUI From a web browser, launch the Smart City web UI at URL https://<cloudcluster-controller-node-ip>. The GUI shows like:
   

Figure 12 - SmartCity UI

SmartCity Termination

Run the command for the SmartCity termination with expected result as below: ```shell # cd cli-scripts/ # ./88_terminate.sh

Using config file: emco_cfg.yaml
http://localhost:31298/v2
URL: projects/project_smtc/composite-apps/composite_smtc/v1/deployment-intent-groups/smtc-deployment-intent-group/terminate Response Code: 202 Response:
```

After termination, the SmartCity application will be deleted from the clusters.

Deploy SmartCity Application With HPA Intent

Smart Edge Open EMCO supports Hardware Platform Awareness (HPA) based Placement Intent.

• Application developer such as SmartCity can state that a certain microservice needs a specific list of resources.
• EMCO can pass that requirement to each appropriate K8s cluster so that the K8s scheduler can place the microservice on a node that has that specific list of resources.
  • There are two kinds of resources:
    • Allocatable resources which can be quantified and allocated to containers in specific quantities, such as cpu and memory.
    • Non-allocatable resources which are properties of CPUs, hosts, etc. such as the availablity of a specifc instruction set like AVX512.
  • Each resource requirement in the intent shall be stated using the same name as in Kubernetes, such as cpu, memory, and intel.com/gpu, for both allocatable and non-allocatable resources.
• More details about EMCO HPA can refer to EMCO HPA Design.

Smart Edge Open EMCO offers an example for HPA based SmartCity application deployment. To obtain all the deployment related scripts, contact your Intel representative. Below will give overview about how to enable HPA intent based on EMCO CLI tool - emcoctl’s resource template files.

The overall setup topology looks like:

Figure 12 - SmartCity HPA Setup

HPA intent based on alloctable resource requirements - CPU

• Two edge clusters and one cloud cluster need to be prepared beforehand.
• HPA related controller registeration section as below example ```yaml — #creating placement controller entries for determining a suitable cluster based on the hardware requirements for each microservice version: emco/v2 resourceContext: anchor: controllers metadata : name: hpa-placement-controller-1 description: test userData1: test1 userData2: test2 spec: host: port: type: placement priority: 1

#creating action controller entries for modifying the Kubernetes objects corresponding to the app or microservice, so that the Kubernetes controller in the target cluster can satisfy those requirements. version: emco/v2 resourceContext: anchor: controllers metadata : name: hpa-action-controller-1 spec: host: port: type: action priority: 1

#creating clm controller entries version: emco/v2 resourceContext: anchor: clm-controllers metadata : name: hpa-placement-controller-1 description: test userData1: test1 userData2: test2 spec: host: port: priority: 1

  
> **NOTE**: To test with multiple edge clusters, can add more edge clusters registration in `01_clusters_template.yaml` and add the new reference edge cluster to logical cloud in `03_logical_cloud_template.yaml`.
 

- Create HPA intent creation and consumer application context section as below example:    
    
```yaml
---
#create app hpa placement intent
version: emco/v2
resourceContext:
  anchor: projects//composite-apps//v1/deployment-intent-groups//hpa-intents
metadata:
  name: hpa-placement-intent-1
  description: "smtc app hpa placement intent"
  userData1: test1
  userData2: test2
spec:
  app-name: 

---
#add consumer 1 to app hpa placement intent. A resource consumer for an allocatable resource is a container within a pod and resource consumer is expressed in terms of any of these Kubernetes objects.
version: emco/v2
resourceContext:
  anchor: projects//composite-apps//v1/deployment-intent-groups//hpa-intents/hpa-placement-intent-1/hpa-resource-consumers
metadata:
  name: hpa-placement-consumer-1
spec:
  api-version: apps/v1
  kind: Deployment
  name: traffic-office1-analytics-traffic
  container-name:  traffic-office1-analytics-traffic

---
#add allocatable-resource to app hpa placement consumer
version: emco/v2
resourceContext:
  anchor: projects//composite-apps//v1/deployment-intent-groups//hpa-intents/hpa-placement-intent-1/hpa-resource-consumers/hpa-placement-consumer-1/resource-requirements
metadata:
  name: hpa-placement-allocatable-resource-1
  description: "resource requirements"
spec:
    allocatable : true
    mandatory : true
    weight : 1
    resource : {"name":"cpu", "requests":8, "limits":9}

NOTE: traffic-office1-analytics-traffic is SmartCity analytics micro service kubernetes deployment name and container name.

• After deployment with SmartCity application instantiation, the expected result is: edge application will be deployed on the edge cluster which satisfies the CPU resource requirements intent.

HPA intent based on non-alloctable resource requirements - VCAC-A

The Visual Cloud Accelerator Card - Analytics (VCAC-A) equips 2nd Generation Intel® Xeon® processor- based platforms with Iris® Pro Graphics and Intel® Movidius™ VPUs to enhance video codec, computer vision, and inference capabilities. Refer to details in Smart Edge Open VCAC-A

During the VCAC-A installation, the VCA nodes are labeled with vcac-zone=yes and features with NFD. For the non-allocatable resource requirement intent, can refer to below example:

---
# add non-allocatable-resource to app hpa placement consumer
version: emco/v2
resourceContext:
 anchor: projects//composite-apps//v1/deployment-intent-groups//hpa-intents/hpa-placement-intent-1/hpa-resource-consumers/hpa-placement-consumer-1/resource-requirements
metadata:
 name: hpa-placement-nonallocatable-resource-1
 description: description of hpa placement_nonallocatable_resource
spec:
   allocatable: false
   mandatory: true
   weight: 1
   resource: {"key":"vcac-zone", "value":"yes"}

After SmartCity application instantiation, the expected result is: edge application will be only deployed on the edge cluster which contains VACA-A accelerator.