Ingress for Anthos architecture

Ingress for Anthos uses a centralized Kubernetes API server to deploy Ingress across multiple clusters. This centralized API server is called the config cluster. Any GKE cluster can act as the config cluster. The config cluster uses two custom resource types: MultiClusterIngress and MultiClusterService. By deploying these resources on the config cluster, the Anthos Ingress Controller deploys load balancers across multiple clusters.

The following concepts and components make up Ingress for Anthos:

Configure Ingress for Anthos to route traffic across multiple clusters in different regions.

It’s a Google-hosted service that supports deploying shared load balancing resources across clusters and across regions.

Requirements for Ingress for Anthos

Ingress for Anthos is supported on:

Enable the required google api’s


gcloud services enable \
 --project=`project-id` \
 container.googleapis.com \
 gkeconnect.googleapis.com \
 anthos.googleapis.com \
 multiclusteringress.googleapis.com \
 gkehub.googleapis.com \
 cloudresourcemanager.googleapis.com

Registering your clusters

Prerequisites for registering a cluster


MEMBERSHIP_NAME=[MEMBERSHIP_NAME]
HUB_PROJECT_ID=[HUB_PROJECT_ID]
SERVICE_ACCOUNT_NAME=[SERVICE_ACCOUNT_NAME]
gcloud projects add-iam-policy-binding ${HUB_PROJECT_ID} \
 --member="serviceAccount:${SERVICE_ACCOUNT_NAME}@${HUB_PROJECT_ID}.iam.gserviceaccount.com" \
 --role="roles/gkehub.connect" \
 --condition "expression=resource.name == \
'projects/${HUB_PROJECT_ID}/locations/global/memberships/${MEMBERSHIP_NAME}',\
title=bind-${SERVICE_ACCOUNT_NAME}-to-${MEMBERSHIP_NAME}"

Find the URIs for your clusters:

gcloud container clusters list --uri

Register your first cluster:


gcloud container hub memberships register first-cluster-name \
    --project=project-id \
    --gke-uri=uri \
    --service-account-key-file=service-account-key-path

Register your second cluster:


gcloud container hub memberships register second-cluster-name \
    --project=project-id \
    --gke-uri=uri \
    --service-account-key-file=service-account-key-path

where:

Verify your clusters are registered:

gcloud container hub memberships list

Specifying a config cluster:

The config cluster is a GKE cluster you choose to be the central point of control for Ingress across the member clusters. Unlike GKE Ingress, the Anthos Ingress controller does not live in a single cluster but is a Google-managed service that watches resources in the config cluster. This GKE cluster is used as a multi-cluster API server to store resources such as MultiClusterIngress and MultiClusterService. Any member cluster can become a config cluster, but there can only be one config cluster at a time.

If the config cluster is down or inaccessible, then MultiClusterIngress and MultiClusterService objects cannot update across the member clusters. Load balancers and traffic can continue to function independently of the config cluster in the case of an outage.

Enabling Ingress for Anthos and selecting the config cluster occurs in the same step. The GKE cluster you choose as the config cluster must already be registered to an environ.

Identify the URI of the cluster you want to specify as the config cluster:

gcloud container hub memberships list

Enable Ingress for Anthos and select first-cluster as the config cluster:

gcloud alpha container hub ingress enable --config-membership=projects/project_id/locations/global/memberships/first-cluster

The output is similar to this:

Waiting for Feature to be created...done.

Note that this process can take a few minutes while the controller is bootstrapping. If successful, the output is similar to this:


Waiting for Feature to be created...done.
Waiting for controller to start...done.

If unsuccessful, the command will timeout like below:


Waiting for controller to start...failed.
ERROR: (gcloud.alpha.container.hub.ingress.enable) Controller did not start in 2 minutes. Please use the `describe` command to check Feature state for debugging information.

If no failure occurred in the previous step, you may proceed with next steps. If a failure occurred in the previous step, then check the feature state. It should indicate what exactly went wrong:

gcloud alpha container hub ingress describe

Now deploying a sample app in your multi clusters as an example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
  namespace: sample-app
  labels:
    app: sample-app
spec:
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
      - name: frontend
        image: gcr.io/google-samples/sample-app:0.1
        ports:
        - containerPort: 8080

Deploying through the config cluster

Now that the application is deployed across gke-us and gke-eu, you will deploy a load balancer by deploying MultiClusterIngress (MCI) and MultiClusterService (MCS) resources in the config cluster. MCI and MCS are custom resources (CRDs) that are the multi-cluster equivalents of Ingress and Service resources.

configuring the first-cluster as the config cluster. The config cluster is used to deploy and configure Ingress across all clusters.

kubectl config use-context first-cluster

Note: Only one cluster can be the active config cluster at any time. If you deploy MultiClusterIngress and MultiClusterService resources to other clusters, but they will not be seen or processed by the Anthos Ingress Controller.

Backendconfig

apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: sample-app-health-check-cfg
  namespace: sample-app
spec:
  healthCheck:
    checkIntervalSec: 60
    timeoutSec: 30
    healthyThreshold: 1
    unhealthyThreshold: 10
    type: HTTP
    port: 8080
    requestPath: /ping
  timeoutSec: 120
  cdn:
    enabled: false
  securityPolicy:
    name: sample-app-security-rules
  logging:
    enable: true
    sampleRate: 1.0 

MultiClusterService

Create a file named mcs.yaml from the following manifest:


apiVersion: networking.gke.io/v1
kind: MultiClusterService
metadata:
  name: sample-app-mcs
  namespace: sample-app
  annotations:
    beta.cloud.google.com/backend-config: '{"ports": {"8080":"sample-app-health-check-cfg"}}'
spec:
  template:
    spec:
      selector:
        app: sample-app
      ports:
      - name: web
        protocol: TCP
        port: 8080
        targetPort: 8080

To view the multiclusterservice:

kubectl get mcs -n sample-app-mcs

MultiClusterIngress

Create a file named mci.yaml from the following manifest:


apiVersion: networking.gke.io/v1
kind: MultiClusterIngress
metadata:
  name: sample-app-ingress
  namespace: sample-app
  annotations:
    networking.gke.io/static-ip: <your static ip>
    networking.gke.io/pre-shared-certs: "cert1, cert2"
spec:
  template:
    spec:
      backend:
        serviceName: sample-app-mcs
        servicePort: 8080
      rules:
        - host: sample-app.com
          http:
            paths:
            - path: /
              backend:
                serviceName: sample-app-mcs
                servicePort: 8080
  clusters:
  - link: "us-east1/first-cluster"
  - link: "us-central1/second-cluster"

Deploy the MultiClusterIngress resource that references sample-app-mcs as a backend:

kubectl apply -f mci.yaml

The output is similar to this:

multiclusteringress.networking.gke.io/sample-app-mci created

Note that MultiClusterIngress has the same schema as the Kubernetes Ingress. The Ingress resource semantics are also the same with the exception of the backend.serviceName field.

The backend.serviceName field in a MultiClusterIngress references a MultiClusterService in the environ API rather than a Service in a Kubernetes cluster. This means that any of the settings for Ingress, such as TLS termination, settings can be configured in the same way.

Validating a successful deployment status:

Google Cloud Load Balancer deployment may take several minutes to deploy for new load balancers.

kubectl describe mci sample-app-ingress -n sample-app

Managing clusters

The set of clusters targeted by the load balancer can be changed by adding or removing a Membership.

For example, to remove the second-cluster as a backend for an ingress, run:

gcloud container hub memberships unregister second-cluster --gke-uri=uri

To add a cluster in Europe, run:

gcloud container hub memberships register europe-cluster --context=europe-cluster --service-account-key-file=/path/to/service-account-key-file