Blogs

Monitoring with Grafana and Prometheus

Monitoring with Grafana and Prometheus

Monitoring With CloudWatch Observability

  • Policy Required: CloudWatchAgentServerPolicy

Monitoring With Grafana and Prometheus

  1. Helm Install
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-4
chmod 700 get_helm.sh
./get_helm.sh
  1. Create an IAM OIDC provider for your cluster:
eksctl utils associate-iam-oidc-provider --region <region> --cluster <cluster-name> --approve
  1. Create an IAM policy for monitoring (for example CloudWatch).
  2. Create a Kubernetes service account and associate it with the IAM role:
eksctl create iamserviceaccount \
  --name <service-account-name> \
  --namespace <namespace> \
  --cluster <cluster-name> \
  --attach-policy-arn arn:aws:iam::<account-id>:policy/<policy-name> \
  --approve

1. Create a Prometheus Namespace

kubectl create namespace prometheus

2. Add the Prometheus Community Chart Repository

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

3. Deploy Prometheus ( Adds Prometheus by default )

For ephemeral storage: data lost when pods are recreated: prometheus.prometheusSpec.storageSpec.emptyDir={}

helm install prometheus \
>   prometheus-community/kube-prometheus-stack \
>   -n prometheus \
>   --create-namespace \
>   --set crds.enabled=true
  • avoid
    • --set alertmanager.persistence.storageClass="gp2"
    • --set server.persistentVolume.storageClass="gp2"
# Check its status
kubectl --namespace prometheus get pods -l "release=prometheus"

# Get Grafana 'admin'
kubectl --namespace prometheus get secrets prometheus-grafana -o jsonpath="{.data.admin-password}" | base64 -d ; echo

# Access Grafana on local instance
export POD_NAME=$(kubectl --namespace prometheus get pod -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=prometheus" -oname)
  kubectl --namespace prometheus port-forward $POD_NAME 3000 # 9000:3000
  
# Get Grafana 'admin'
kubectl get secret --namespace prometheus -l app.kubernetes.io/component=admin-secret -o jsonpath="{.items[0].data.admin-password}" | base64 --decode ; echo

4. Verify the Deployment

kubectl get pods -n prometheus

5. Port Forward the Prometheus Console

Use kubectl to port forward the Prometheus console to your local machine:

kubectl --namespace=prometheus port-forward deploy/prometheus-server 9090

You can then access the Prometheus console at:

http://localhost:9090

Add Grafana

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install grafana-k8s-monitoring grafana/k8s-monitoring -n <namespace> --create-namespace

debug

kubectl get events -n prometheus --sort-by=.lastTimestamp

# Check Alertmanager
kubectl describe pod prometheus-alertmanager-0 -n prometheus
# remove everything
kubectl delete namespace prometheus

Related Posts