❗ This is not a production setup, it’s only for testing and demo purpose
Introduction
Since TiDB components deployed by TiDB Operator output the logs in the stdout and stderr of the container by default, as long as you have EFK deployed in your Kubernetes cluster, all the TiDB components logs will be collected by Fluentd running on each Kubernetes node, you can find TiDB logs same as other Kubernetes serviceslogs.
This article demonstrates how to collect TiDB logs using Fluent-bit in Kubernetes based on EFK, especially for slow logs which need to be properly formated before streaming to EFK, for the EFK deployment please refer to my previous blog Kubernetes Logging EFK Deployment
Enviornments
- Fluent-bit v1.8 (needed by multiline parser)
- TiDB operator v1.5.2
- TiDB v7.5
Step by Step Setup
1. Deploy TiDB Operator
kubectl create -f https://raw.githubusercontent.com/pingcap/tidb-operator/v1.5.2/manifests/crd.yaml
kubectl create namespace tidb-admin
helm repo add pingcap https://charts.pingcap.org/
helm install --namespace tidb-admin tidb-operator pingcap/tidb-operator --version v1.5.2
2. Deploy TiDB Cluster with Fluent-bit
Fluent bit running as sidecar in TiDB pod, you can use Fluentd as well. I choose fluent-bit as it’s light weight.
2.1 Prepare Fluent bit config
For demo purpose, I will manually mount Fluent-bit config. You may customize your image for a clean deployment, as slow query log is multiline format, we need to handle it properly, mutiline format parser is included in the config.
1.Create fluent-bit ConfigMap
kubectl apply -f tidb-fluentbit-cm-slowlog.yaml
tidb-fluentbit-cm-slowlog.yaml can be found here.
2.2 Deploy TiDB Cluster with fluent-bit as sidecar container
Option 1: fluent-bit in additional container
This deployment will keep the default slowlog container, fluent-bit as an additional container, tidb pod in total 3 containers.
kubectl apply -f tidb-advanced-cluster-additional-container.yaml
tidb-advanced-cluster-additional-container.yaml can be found here.
Option2: Fluent-bit Merged with default slowlog container
We customize the default slowlog container to run Fluent bit by merging the default slowlog container and additional container. tidb pod only 2 containers
kubectl apply -f tidb-advanced-cluster-fluentbit-merged-with-slowlog.yaml
tidb-advanced-cluster-fluentbit-merged-with-slowlog.yaml can be found here.
3. View TiDB Slow Query Logs in Kibana
1.You will find tidb-yyyy.mm.dd index
2.Create data view for tidb index
3.View TiDB Slow Logs
4.Formated TiDB Slow Log
