Unable to deploy prometheus to the test cluster with make target

[somaritane]

Steps to reproduce:

1. Deploy test clusters with make deploy-full-local-setup target
2. Deploy prometheus to the test clusters with make deploy-prometheus target

Expected behavior:

2. Prometheus should be deployed to the test clusters

Actual behavior:

2. make deploy-prometheus target fails to deploy prometheus to the test clusters:

   > make deploy-prometheus
   ...
   Release "prometheus" does not exist. Installing it now.
   Error: timed out waiting for the condition
   make: *** [deploy-prometheus] Error 1

   prometheus-server-... pod is always in pending state with the following events:

   > k describe --context k3d-test-gslb1 -n k8gb po prometheus-server-6b868bd6c4-q7r86
   ...
   Events:
   Type     Reason            Age    From               Message
   ----     ------            ----   ----               -------
   Warning  FailedScheduling  2m37s  default-scheduler  0/2 nodes are available: 2 pod has unbound immediate PersistentVolumeClaims.
   Warning  FailedScheduling  2m35s  default-scheduler  0/2 nodes are available: 2 pod has unbound immediate PersistentVolumeClaims.
   

   pvc for prometheus-server stays in pending state with the following events:

   > k describe --context k3d-test-gslb1 -n k8gb persistentvolumeclaims prometheus-server
   ...
   Events:
   Type    Reason         Age                  From                         Message
   ----    ------         ----                 ----                         -------
   Normal  FailedBinding  1s (x14 over 3m12s)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set
   

Assignees:
Labels: bug

[k0da]

Why just not to reuse capabily of k3d and let it handle p8s installation?

[k0da]

And yes, but happening due to removal of pv controller

[somaritane]

Why just not to reuse capabily of k3d and let it handle p8s installation?

@k0da can you pls elaborate more on that? disabled k3s metrics-server capability is not a p8s server

[somaritane]

Just checked, I'm able to install Prometheus with make deploy-prometheus target once I enabled the local-storage component: in k3d configs
https://github.com/k8gb-io/k8gb/blob/master/k3d/test-gslb1.yaml#L28

[jkremser]

right, metric-server != prometheus. I think, it's the old heapster + cadvisor combo. The PV + PVCs can be disabled in Prom's helm chart. Then it is totally ephemeral and the data will be lost if pod restarts, but we can mention that in the docs.

I am on it: #798

[k0da]

@jkremser this is not inteded to be prod ready so ephemeral is 200% fine