❗ This is not a production setup, it’s only for testing and demo purpose
Envrionments
- minikube v1.32.0
- Kubernetes v1.24
- ECK v2.12
- Elasticsearch + Kibana v8.13.0
- Fluentd v1.16
Introduction
This is a simple infra level logging system setup, Fluentd as a daemonset running on each Kubernetes node as an agent to collect all containers’ logs to elasticseach. Again, this is not for production setup, just meet basic log collection requirement, production setup colud be more conprehensive.
Step By Step Setup
1. Prepare Kubernetes
minikube start --cpus=3 --memory=6G --disk-size=25G --kubernetes-version=v1.24
2. Deploy ECK
Refer to official documentation Deploy ECK in your Kubernetes cluster
2.1 Deploy Elastic Operator
1.Create CRD
kubectl create -f https://download.elastic.co/downloads/eck/2.12.1/crds.yaml
2.Create Operator
kubectl apply -f https://download.elastic.co/downloads/eck/2.12.1/operator.yaml
2.2 Deploy Elasticsearch
1.Create namespace eck for ECK
2.Deploy Elaisticsearch
kubectl apply -f elasticsearch.yaml
elasticsearch.yaml can be found here.
Make sure HEALTH is green, and PHASE is Ready.
2.3 Deploy Kibana
kubectl apply -f kibana.yaml
kibana.yaml can be found here.
2.4 Access Kibana
1.Get password
kubectl get secret quickstart-es-elastic-user -n eck -o go-template='{{.data.elastic | base64decode}}'
2.Expose kibana service access from outside Kubernetes
kubectl port-forward service/quickstart-kb-http -n eck 5601 --address=0.0.0.0
3.Login Access Kibana URL https://{your host}:5601/
3. Deploy Fluentd
kubectl apply -f fluentd-daemonset-elasticsearch-rbac.yaml
fluentd-daemonset-elasticsearch-rbac.yaml can be found here.
Replace the environment varibles accordingly, especically FLUENT_ELASTICSEARCH_PASSWORD
4. Create Data View in Kibana
5. View K8S Logs in Kibana
Enjoy EFK!!!
