SPDX-License-Identifier: Apache-2.0
Copyright (c) 2019-2020 Intel Corporation

Network Edge Applications Onboarding

• Introduction
• Installing Smart Edge Open
• Building applications
  • Building sample application images
  • Building the OpenVINO application images
• Onboarding sample application
  • Prerequisites
  • Verifying image availability
  • Applying Kubernetes network policies
  • Deploying consumer and producer sample application
• Onboarding OpenVINO application
  • Prerequisites
  • Setting up networking interfaces
  • Deploying the Application
  • Applying Kubernetes network policies
  • Setting up Edge DNS
  • Starting traffic from Client Simulator
• Onboarding Smart City sample application
  • Setting up networking interfaces
  • Building Smart City ingredients
  • Running Smart City
  • Inter application communication
• Enhanced Platform Awareness
• VM support for Network Edge
• Troubleshooting
  • Useful Commands:

Introduction

This document aims to familiarize users with the Smart Edge Open application on-boarding process for the Network Edge. This document provides instructions on how to deploy an application from the Edge Controller to Edge Nodes in the cluster; it also provides sample deployment scenarios and traffic configuration for the application. The applications will be deployed from the Edge Controller via the Kubernetes kubectl command-line utility. Sample specification files for application onboarding are also provided.

Installing Smart Edge Open

The following application onboarding steps assume that Smart Edge Open was installed through Smart Edge Open playbooks.

Building applications

Users must provide the application to be deployed on the Smart Edge Open platform for Network Edge. The application must be provided in a Docker* image format that is available either from an external Docker repository (Docker Hub) or a locally built Docker image. The image must be available on the Edge Node, which the application will be deployed on.

Note: The Harbor registry setup is out of scope for this document. If users already have a docker container image file and would like to copy it to the node manually, they can use the docker load command to add the image. The success of using a pre-built Docker image depends on the application dependencies that users must know.

The Smart Edge Open edgeapps repository provides images for Smart Edge Open supported applications. Pull the repository to your Edge Node to build the images.

This document explains the build and deployment of two applications:

1. Sample application: a simple “Hello, World!” reference application for Smart Edge Open
2. OpenVINO™ application: A close to real-world inference application

Building sample application images

The sample application is available in the edgeapps repository; further information about the application is contained within the Readme.md file.

The following steps are required to build the sample application Docker images for testing the Smart Edge Open Edge Application Agent (EAA) with consumer and producer applications:

1. To build the application binaries and Docker images run make:

   make
   make build-docker
   

2. Check that the images are built successfully and available in the local Docker image registry:

   docker images | grep producer
   docker images | grep consumer
   

   Building the OpenVINO application images

   The OpenVINO application is available in the EdgeApps repository; further information about the application is contained within Readme.md file.

The following steps are required to build the sample application Docker images for testing OpenVINO consumer and producer applications:

1. To build the producer application image from the application directory, navigate to the ./producer directory and run:

   ./build-image.sh
   

   Note: Only CPU inference support is currently available for OpenVINO application on Smart Edge Open Network Edge. The environmental variable OPENVINO_ACCL must be set to CPU within the Dockerfile available in the directory.

2. To build the consumer application image from the application directory, navigate to the ./consumer directory and run:

   ./build-image.sh
   

3. Check that the image builds are successful and available in the local Docker image registry:

   docker images | grep openvino-prod-app
   docker images | grep openvino-cons-app
   

Additionally, an application to generate sample traffic is provided. The application should be built on a separate host, which generates the traffic.

1. To build the client simulator application image from the application directory, navigate to the ./clientsim directory and run:

   ./build-image.sh
   

2. Check that the image build is successful and available in the local Docker image registry:

   docker images | grep client-sim
   

Onboarding sample application

This section guides users through the complete process of onboarding a sample application and testing the EAA functionality of Smart Edge Open for the Network Edge. This process outlines how to start the application, setup network policies, and verify functionality.

Prerequisites

• Smart Edge Open for Network Edge is fully installed and set up.
• Docker images for the sample application consumer and producer are available on Edge Node.

Verifying image availability

To verify that the images for sample application consumer and producer are built and available on the Edge Node run:

   docker image list | grep producer
   docker image list | grep consumer

Applying Kubernetes network policies

Kubernetes NetworkPolicy is a mechanism that enables control over how pods are allowed to communicate with each other and other network endpoints. By default, in the Network Edge environment, all ingress traffic is blocked (services running inside of deployed applications are not reachable) and all egress traffic is enabled (pods can reach the internet).

1. To apply a network policy for the sample application allowing ingress traffic, create a sample_policy.yml file that specifies the network policy (in the example network policy cidr field contains Calico CNI cidr; for other CNI use specific CNI cidr, e.g. for Kube-ovn CNI use 10.16.0.0/16): ```yml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: eaa-prod-cons-policy namespace: default spec: podSelector: {} policyTypes:
   • Ingress ingress:
   • from:
     • ipBlock: cidr: 10.245.0.0/16 ports:
     • protocol: TCP port: 80
     • protocol: TCP port: 443 ```
2. Apply the network policy:

   kubectl apply -f sample_policy.yml
   

   Deploying crd for sriov cni

   apiVersion: "k8s.cni.cncf.io/v1"
   kind: NetworkAttachmentDefinition
   metadata:
     name: sriov-openvino
     annotations:
       k8s.v1.cni.cncf.io/resourceName: intel.com/intel_sriov_netdevice
   spec:
     config: '{
     "type": "sriov",
     "cniVersion": "0.3.1",
     "name": "sriov-network",
     "ipam": {
       "type": "host-local",
       "subnet": "192.168.2.0/24",
       "routes": [{
         "dst": "0.0.0.0/0"
       }],
       "gateway": "192.168.2.1"
     }
   }'
   

Deploying consumer and producer sample application

NOTE: The producer application must be deployed before the consumer application. Also, the applications must be deployed within a short time of each other as they have a limited lifespan.

1. To deploy a sample producer application, create the following sample_producer.yml pod specification file. ```yml

   SPDX-License-Identifier: Apache-2.0

   Copyright (c) 2019 Intel Corporation

apiVersion: v1 kind: ServiceAccount metadata: name: openvino-prod-app

kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: openvino-prod-app-csr-requester roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: csr-requester subjects: - kind: ServiceAccount name: openvino-prod-app namespace: default

apiVersion: v1 kind: ConfigMap metadata: name: openvino-prod-app-csr-config data: certrequest.json: | { “CSR”: { “Name”: “openvino-prod-app”, “Subject”: { “CommonName”: “openvino:producer”, “Organization”: [“Intel Corporation”] }, “DNSSANs”: [], “IPSANs”: [], “KeyUsages”: [ “digital signature”, “key encipherment”, “client auth” ] }, “Signer”: “openness.org/certsigner”, “WaitTimeout”: “5m” } — apiVersion: apps/v1 kind: Deployment metadata: name: openvino-prod-app spec: replicas: 1 selector: matchLabels: app: openvino-prod-app template: metadata: labels: app: openvino-prod-app spec: serviceAccountName: openvino-prod-app initContainers: - name: alpine image: alpine:3.12.0 command: [“/bin/sh”] args: - “-c” - “cp /root/ca-certrequester/cert.pem /root/certs/root.pem && chmod 0777 /root/certs/root.pem” imagePullPolicy: IfNotPresent resources: requests: cpu: “0.1” limits: cpu: “0.1” memory: “128Mi” volumeMounts: - name: ca-certrequester mountPath: /root/ca-certrequester - name: certs mountPath: /root/certs - name: certrequester image: certrequester:1.0 imagePullPolicy: Never args: [”–cfg”, “/home/certrequester/config/certrequest.json”] imagePullPolicy: IfNotPresent resources: requests: cpu: “0.1” limits: cpu: “0.1” memory: “128Mi” volumeMounts: - name: config mountPath: /home/certrequester/config - name: certs mountPath: /home/certrequester/certs containers: - name: openvino-prod-app image: openvino-prod-app:1.0 imagePullPolicy: Never ports: - containerPort: 443 volumeMounts: - name: tmp mountPath: /tmp - name: certs mountPath: /var/user/certs env: - name: OPENVINO_ACCL value: “CPU” volumes: - name: tmp hostPath: path: /tmp type: Directory - name: config configMap: name: openvino-prod-app-csr-config - name: ca-certrequester secret: secretName: ca-certrequester - name: certs emptyDir: {}

2. Deploy the pod:

kubectl create -f sample_producer.yml

3. Accept the producer's CSR:

kubectl certificate approve producer

4. Check that the pod is running:

kubectl get pods | grep producer

5. Verify logs of the sample application producer:

kubectl logs -f

Expected output: The Example Producer eaa.openness [{ExampleNotification 1.0.0 Description for Event #1 by Example Producer}]}]} Sending notification

6. Verify logs of EAA

kubectl logs -f -n openness

Expected output: RequestCredentials request from CN: ExampleNamespace:ExampleProducerAppID, from IP: properly handled

7. To deploy a sample consumer application, create the following `sample_consumer.yml` pod specification file.
   ```yml
   ---
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: producer

   ---
   kind: ClusterRoleBinding
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
     name: producer-csr-requester
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: csr-requester
   subjects:
     - kind: ServiceAccount
       name: producer
       namespace: default

   ---
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: producer-csr-config
   data:
     certrequest.json: |
       {
           "CSR": {
               "Name": "producer",
               "Subject": {
                   "CommonName": "ExampleNamespace:ExampleProducerAppID"
               },
               "DNSSANs": [],
               "IPSANs": [],
               "KeyUsages": [
                   "digital signature", "key encipherment", "client auth"
               ]
           },
           "Signer": "openness.org/certsigner",
           "WaitTimeout": "5m"
       }

   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: openvino-cons-app
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: openvino-cons-app
     template:
       metadata:
         labels:
           app: openvino-cons-app
         annotations:
           k8s.v1.cni.cncf.io/networks: sriov-openvino
       spec:
         serviceAccountName: openvino-cons-app
         initContainers:
           - name: alpine
             image: alpine:3.12.0
             command: ["/bin/sh"]
             args:
               - "-c"
               - "cp /ca-certrequester/cert.pem /root/certs/root.pem"
             imagePullPolicy: IfNotPresent
             resources:
               requests:
                 cpu: "0.1"
               limits:
                 cpu: "0.1"
                 memory: "128Mi"
             volumeMounts:
               - name: ca-certrequester
                 mountPath: /ca-certrequester
               - name: certs
                 mountPath: /root/certs
           - name: certrequester
             image: certrequester:1.0
             imagePullPolicy: IfNotPresent
             args: ["--cfg", "/home/certrequester/config/certrequest.json"]
             imagePullPolicy: IfNotPresent
             resources:
               requests:
                 cpu: "0.1"
               limits:
                 cpu: "0.1"
                 memory: "128Mi"
             volumeMounts:
               - name: config
                 mountPath: /home/certrequester/config
               - name: certs
                 mountPath: /home/certrequester/certs
           - name: alpine2
             image: alpine:3.12.0
             command: ["/bin/sh"]
             args:
               - "-c"
               - "chmod -R 0777 /root/certs"
             imagePullPolicy: IfNotPresent
             resources:
               requests:
                 cpu: "0.1"
               limits:
                 cpu: "0.1"
                 memory: "128Mi"
             volumeMounts:
               - name: certs
                 mountPath: /root/certs
         containers:
           - name: openvino-cons-app
             image: openvino-cons-app:1.0
             imagePullPolicy: IfNotPresent
             ports:
             - containerPort: 443
             - containerPort: 5000
               protocol: TCP
             volumeMounts:
               - name: certs
                 mountPath: /home/openvino/certs
               - name: tmp
                 mountPath: /var/tmp
               - name: shm
                 mountPath: /dev/shm
              resources:
                requests:
                  intel.com/intel_sriov_netdevice: '1'
                limits:
                  intel.com/intel_sriov_netdevice: '1'
         volumes:
           - name: config
             configMap:
               name: openvino-cons-app-csr-config
           - name: ca-certrequester
             secret:
               secretName: ca-certrequester
           - name: certs
             emptyDir: {}
           - name: tmp
             hostPath:
               path: /var/tmp
               type: Directory
           - name: shm
             hostPath:
               path: /dev/shm
               type: Directory

1. Accept the consumer’s CSR:

   kubectl certificate approve consumer
   

2. Deploy the pod:

   kubectl create -f sample_consumer.yml
   

3. Check that the pod is running:

      kubectl get pods | grep consumer
   

4. Verify logs of the sample application consumer: ``` kubectl logs -f

Expected output: Received notification

12. Verify logs of EAA
    ```
    kubectl logs <eaa_pod_name> -f

    Expected output:
    RequestCredentials  request from CN: ExampleNamespace:ExampleConsumerAppID, from IP: <IP_ADDRESS> properly handled
    ```
# Onboarding OpenVINO application
This section guides users through the complete process of onboarding the OpenVINO producer and consumer applications. This process will also guide the user on setting up a network connection between Client Simulator (Traffic Generator), setting up network policies, and testing the application. The following sub-sections should be executed step by step.

If you use kube-ovn as your primary CNI, please click [here](https://github.com/smart-edge-open/ido-specs/blob/openvino_spec/doc/applications-onboard/network-edge-applications-onboarding.md#onboarding-openvino-application-with-kube-ovn).

## Prerequisites

* Smart Edge Open for Network Edge is fully installed and set up (calico as default cni and sriov as cni to support Interfaceservice which is openness developed kubectl plugin.).
* The Docker images for OpenVINO are available on the Edge Node.
* A separate host used for generating traffic via Client Simulator is set up.
* The Edge Node host and traffic generating host are connected point to point via unused physical network interfaces.
* The Docker image for the Client Simulator application is available on the traffic generating host.

## Setting up networking interfaces
1. Make sure the sriov CNI is setup on your cluster. Commands on the master node are below.

[root@controller ~]# kubectl get pod -o custom-columns=NAME:.metadata.name -n kube-system | grep sriov sriov-release-kube-sriov-cni-ds-amd64-vmsl5 sriov-release-kube-sriov-device-plugin-amd64-2l5pq

2. On the traffic generating host setup to run Client Simulator, configure the network interface connected to Edge Node host. External client traffic in the Smart Edge Open Network Edge configuration is routed via 192.168.2.1, the IP address of traffic generating host must be one from the same subnet. Configure the routing accordingly:

ifconfig 192.168.2.10/24

## Deploying the Application

1. An application `yaml` specification file for the OpenVINO producer that is used to deploy the K8s pod can be found in the Edge Apps repository at [./applications/openvino/producer/openvino-prod-app.yaml](https://github.com/smart-edge-open/edgeapps/blob/master/applications/openvino/producer/openvino-prod-app.yaml). The pod will use the Docker image, which must be [built](#building-openvino-application-images) and available on the platform. Deploy the producer application by running:

kubectl apply -f openvino-prod-app.yaml kubectl certificate approve openvino-prod-app

2. An application `yaml` specification file for the OpenVINO consumer that is used to deploy K8s pod can be found in the Edge Apps repository at [./applications/openvino/consumer/openvino-cons-app.yaml](https://github.com/smart-edge-open/edgeapps/blob/master/applications/openvino/consumer/openvino-cons-app.yaml). The pod will use the Docker image, which must be [built](#building-openvino-application-images) and available on the platform. Deploy the consumer application by running:

kubectl apply -f openvino-cons-app.yaml kubectl certificate approve openvino-cons-app

3. Verify that no errors show up in the logs of the OpenVINO consumer application:

kubectl logs openvino-cons-app kubectl get po -o custom-columns=NAME:.metadata.name,IP:.status.podIP | grep cons-app | awk ‘{print $2}’