Monitoring

Two monitoring tools are presented on the Gateway Device using an NGINX Reverse Proxy. Each of them is also exposed by the NGINX Ingress Controller running in the k8s cluster.

sequenceDiagram
  autonumber
  Workstation->>Gateway Device Proxy: request
  Gateway Device Proxy->>Cluster LB IP: request
  Cluster LB IP->>NGINX Ingress Controller Path: request
  NGINX Ingress Controller Path->>Prometheus Svc: request

In this project:

Prometheus and Grafana web console applications use Root/Base URL that match the NGINX Reverse Proxy URIs. (eg: http://cluster.home/prometheus)
NGINX Reverse Proxy uses URI path filtering (eg: http://cluster.home/prometheus/)
NGINX Ingress Controller uses host routing (eg: http://prometheus.cluster.home/)

Prometheus

Prometheus path prefix support and is a simple web-console (not websockets) or regex/re-write rules.

Example: helm values for prometheus

extract from k8s-control-node-install playbook

- name: Deploy kube-prometheus-stack chart
  kubernetes.core.helm:
    kubeconfig: /home/kube/.kube/config
    wait: true
    name: kube-prometheus-stack
    chart_ref: prometheus-community/kube-prometheus-stack
    release_namespace: monitoring
    values:
      prometheus:
        ingress:
          ingressClassName: nginx
          enabled: true
          pathType: ImplementationSpecific
          paths: ["/prometheus"]
          hosts: ["prometheus.cluster.home"]
        prometheusSpec:
          externalUrl: "http://prometheus.cluster.home/prometheus"
          routePrefix: /prometheus

Example: NGINX Reverse Proxy location for prometheus

extract from /etc/nginx/sites-enabled/default

    location = /prometheus {
        proxy_pass http://prometheus.cluster.home/prometheus;
    }

Example

extract from NGINX Ingress Controller for Prometheus

- name: Deploy kube-prometheus-stack chart
  kubernetes.core.helm:
    kubeconfig: /home/kube/.kube/config
    wait: true
    name: kube-prometheus-stack
    chart_ref: prometheus-community/kube-prometheus-stack
    release_namespace: monitoring
    values:
      prometheus:
        ingress:
          ingressClassName: nginx
          enabled: true
          pathType: ImplementationSpecific
          paths: ["/prometheus"]
          hosts: ["prometheus.cluster.home", "cloud.dtmc.ca"]
        prometheusSpec:
          externalUrl: "http://cluster.home/prometheus"
          routePrefix: /prometheus

Example k8s Ingress for prometheus

kube@k8s-control-plane-node:~$ kubectl -n monitoring describe ingress kube-prometheus-stack-prometheus
Name:             kube-prometheus-stack-prometheus
Labels:           app=kube-prometheus-stack-prometheus
                  app.kubernetes.io/instance=kube-prometheus-stack
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/part-of=kube-prometheus-stack
                  app.kubernetes.io/version=45.0.0
                  chart=kube-prometheus-stack-45.0.0
                  heritage=Helm
                  release=kube-prometheus-stack
Namespace:        monitoring
Address:          192.168.57.200
Ingress Class:    nginx
Default backend:  <default>
Rules:
  Host                   Path  Backends
  ----                   ----  --------
  prometheus.cluster.home  
                         /prometheus   kube-prometheus-stack-prometheus:9090 (10.244.107.234:9090)
Annotations:             meta.helm.sh/release-name: kube-prometheus-stack
                         meta.helm.sh/release-namespace: monitoring
Events:
  Type    Reason          Age                 From                      Message
  ----    ------          ----                ----                      -------
  Normal  AddedOrUpdated  59m (x13 over 37h)  nginx-ingress-controller  Configuration for monitoring/kube-prometheus-stack-prometheus was added or updated

Grafana

When presenting Grafana through the NGINX Ingress Controller it is important to use a hostname (eg: grafana.cluster.home).

The NGINX Ingress will not assign an “Address” if the hostname is the same for other services.

Example: Grafana k8s ingress

Note, the NGINX Ingress for Grafana has no path-prefix. NGINX Ingress performs hostname path routing.

Namespace:        monitoring
Address:          192.168.57.200
Ingress Class:    nginx
Default backend:  <default>
Rules:
  Host                    Path  Backends
  ----                    ----  --------
  grafana.cluster.home    
                          /   kube-prometheus-stack-grafana:80 (10.244.122.84:3000)
  web-proxy.cluster.home  
                          /   kube-prometheus-stack-grafana:80 (10.244.122.84:3000)
Annotations:              meta.helm.sh/release-name: kube-prometheus-stack
                          meta.helm.sh/release-namespace: monitoring
Events:
  Type    Reason          Age                 From                      Message
  ----    ------          ----                ----                      -------
  Normal  AddedOrUpdated  25s (x9 over 171m)  nginx-ingress-controller  Configuration for monitoring/kube-prometheus-stack-grafana was added or updated

Example: NGINX Reverse Proxy location for Grafana

extract from /etc/nginx/sites-enabled/default

 location /grafana/ {
    rewrite  ^/grafana/(.*)  /$1 break;
    proxy_set_header Host $http_host;
    proxy_pass http://grafana.cluster.home;
  }

  # Proxy Grafana Live WebSocket connections.
  location /grafana/api/live/ {
    rewrite  ^/grafana/(.*)  /$1 break;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header Host $http_host;
    proxy_pass http://grafana.cluster.home;
  }

Troubleshooting

Unique HOSTS for NGINX Ingress Controller.

If only one of the prometheus and grafana ingresses actually have an ADDRESS value then it is most likely the Host name is the same.

NGINX Ingress Controller expects unique HOSTS (eg: prometheus.cluster.home, grafana.cluster.home) to ingress paths.

In the following example, grafana and prometheus both have the same hostname (services.cluster.home). The first service registered with the NGINX Ingress Controller gets the ADDRESS.

The other one does not and is not active/live.

kube@k8s-control-plane-node:~$ kubectl -n monitoring get ingress
NAME                               CLASS   HOSTS                   ADDRESS          PORTS   AGE
kube-prometheus-stack-grafana      nginx   services.cluster.home                    80      36h
kube-prometheus-stack-prometheus   nginx   services.cluster.home   192.168.57.200   80      3d9h

Grafana Origin Not Allowed

Grafana websocket connection (to datasources) will not accept IP Addresses or unknown Hosts in the incoming HTTP Request. The NGINX Reverse Proxy has to forward requests to Grafana with a known FQDN such as grafana.cluster.home.

The proxy_set_header must be set to the incoming http_host and Grafana k8s Service must be provisioned to accept this host name.

extract from NGINX Reverse Proxy

  location /grafana/api/live/ {
    ...
    proxy_set_header Host $http_host;
    proxy_pass http://web-proxy.cluster.home;
  }

extract from kube-prometheus-stack grafana ingress section

      grafana:
        ingress:
          enabled: true         
          ingressClassName: nginx   
          hosts: ["grafana.cluster.home", "web-proxy.cluster.home"]
          path: "/"
          pathType: ImplementationSpecific