Kubernetes cheat sheet
Offering: Self-managed
This is a list of useful information regarding Kubernetes that the GitLab Support Team sometimes uses while troubleshooting. GitLab is making this public, so that anyone can make use of the Support team’s collected knowledge
If you are on a paid tier and are not sure how to use these commands, it is best to contact Support and they will assist you with any issues you are having.
Generic Kubernetes commands
-
How to authorize to your GCP project (can be especially useful if you have projects under different GCP accounts):
gcloud auth login
-
How to access Kubernetes dashboard:
# for minikube: minikube dashboard —url # for non-local installations if access via Kubectl is configured: kubectl proxy
-
How to SSH to a Kubernetes node and enter the container as root https://github.com/kubernetes/kubernetes/issues/30656:
- For GCP, you may find the node name and run
gcloud compute ssh node-name
. - List containers using
docker ps
. - Enter container using
docker exec --user root -ti container-id bash
.
- For GCP, you may find the node name and run
-
How to copy a file from local machine to a pod:
kubectl cp file-name pod-name:./destination-path
-
What to do with pods in
CrashLoopBackoff
status:- Check logs via Kubernetes dashboard.
-
Check logs via Kubectl:
kubectl logs <webservice pod> -c dependencies
-
How to tail all Kubernetes cluster events in real time:
kubectl get events -w --all-namespaces
-
How to get logs of the previously terminated pod instance:
kubectl logs <pod-name> --previous
No logs are kept in the containers/pods themselves. Everything is written to
stdout
. This is the principle of Kubernetes, read Twelve-factor app for details. -
How to get cron jobs configured on a cluster
kubectl get cronjobs
When one configures cron-based backups, you will be able to see the new schedule here. Some details about the schedules can be found in Running Automated Tasks with a CronJob
GitLab-specific Kubernetes information
-
Minimal configuration that can be used to test a Kubernetes Helm chart.
-
Tailing logs of a separate pod. An example for a
webservice
pod:kubectl logs gitlab-webservice-54fbf6698b-hpckq -c webservice
-
Tail and follow all pods that share a label (in this case,
webservice
):# all containers in the webservice pods kubectl logs -f -l app=webservice --all-containers=true --max-log-requests=50 # only the webservice containers in all webservice pods kubectl logs -f -l app=webservice -c webservice --max-log-requests=50
-
One can stream logs from all containers at once, similar to the command
gitlab-ctl tail
in a Linux package installation:kubectl logs -f -l release=gitlab --all-containers=true --max-log-requests=100
-
Check all events in the
gitlab
namespace (the namespace name can be different if you specified a different one when deploying the Helm chart):kubectl get events -w --namespace=gitlab
-
Most of the useful GitLab tools (console, Rake tasks, etc) are found in the toolbox pod. You may enter it and run commands inside or run them from the outside.
Thetask-runner
pod was renamed totoolbox
in GitLab 14.2 (charts 5.2).# find the pod kubectl --namespace gitlab get pods -lapp=toolbox # enter it kubectl --namespace gitlab exec -it <toolbox-pod-name> -- bash # open rails console # rails console can be also called from other GitLab pods /srv/gitlab/bin/rails console # source-style commands should also work cd /srv/gitlab && bundle exec rake gitlab:check RAILS_ENV=production # run GitLab check. The output can be confusing and invalid because of the specific structure of GitLab installed via helm chart /usr/local/bin/gitlab-rake gitlab:check # open console without entering pod kubectl exec -it <toolbox-pod-name> -- /srv/gitlab/bin/rails console # check the status of DB migrations kubectl exec -it <toolbox-pod-name> -- /usr/local/bin/gitlab-rake db:migrate:status
You can also use
gitlab-rake
, instead of/usr/local/bin/gitlab-rake
. -
Troubleshooting Infrastructure > Kubernetes clusters integration:
- Check the output of
kubectl get events -w --all-namespaces
. - Check the logs of pods within
gitlab-managed-apps
namespace. - On the side of GitLab check Sidekiq log and Kubernetes log. When GitLab is installed
via Helm chart,
kubernetes.log
can be found inside the Sidekiq pod.
- Check the output of
-
How to get your initial administrator password https://docs.gitlab.com/charts/installation/deployment.html#initial-login:
# find the name of the secret containing the password kubectl get secrets | grep initial-root # decode it kubectl get secret <secret-name> -ojsonpath={.data.password} | base64 --decode ; echo
-
How to connect to a GitLab PostgreSQL database.
Thetask-runner
pod was renamed totoolbox
in GitLab 14.2 (charts 5.2).In GitLab 14.2 (chart 5.2) and later:
kubectl exec -it <toolbox-pod-name> -- /srv/gitlab/bin/rails dbconsole --include-password --database main
In GitLab 14.1 (chart 5.1) and earlier:
kubectl exec -it <task-runner-pod-name> -- /srv/gitlab/bin/rails dbconsole --include-password
-
How to get information about Helm installation status:
helm status name-of-installation
-
How to update GitLab installed using Helm chart:
helm repo update # get current values and redirect them to yaml file (analogue of gitlab.rb values) helm get values <release name> > gitlab.yaml # run upgrade itself helm upgrade <release name> <chart path> -f gitlab.yaml
See also Updating GitLab by using the Helm chart.
-
How to apply changes to GitLab configuration:
- Modify the
gitlab.yaml
file. -
Run the following command to apply changes:
helm upgrade <release name> <chart path> -f gitlab.yaml
- Modify the
-
How to get the manifest for a release. It can be useful because it contains the information about all Kubernetes resources and dependent charts:
helm get manifest <release name>
Fast-Stats for KubeSOS reports
KubeSOS is a tool that gathers the GitLab cluster configuration and logs from GitLab Cloud Native chart deployments. You can use fast-stats, a tool with minimal memory use, to quickly create and compare performance statistics from and between GitLab logs.
-
Run
fast-stats
:cat <file-name> | cut -d ' ' -f2- | grep ^{ | fast-stats
-
List errors:
cat <file-name> | cut -d ' ' -f2- | grep ^{ | fast-stats errors
-
Run
fast-stats
top:cat <file-name> | cut -d ' ' -f2- | grep ^{ | fast-stats top
-
Change the number of rows printed. By default, 10 rows are printed.
cat <file-name> | cut -d ' ' -f2- | grep ^{ | fast-stats -l <number of rows>
Installation of minimal GitLab configuration via minikube on macOS
This section is based on Developing for Kubernetes with minikube and Helm. Refer to those documents for details.
-
Install Kubectl via Homebrew:
brew install kubernetes-cli
-
Install minikube via Homebrew:
brew cask install minikube
-
Start minikube and configure it. If minikube cannot start, try running
minikube delete && minikube start
and repeat the steps:minikube start --cpus 3 --memory 8192 # minimum amount for GitLab to work minikube addons enable ingress
-
Install Helm via Homebrew and initialize it:
brew install helm
-
Copy the minikube minimum values YAML file to your workstation:
curl --output values.yaml "https://gitlab.com/gitlab-org/charts/gitlab/raw/master/examples/values-minikube-minimum.yaml"
-
Find the IP address in the output of
minikube ip
and update the YAML file with this IP address. -
Install the GitLab Helm chart:
helm repo add gitlab https://charts.gitlab.io helm install gitlab -f <path-to-yaml-file> gitlab/gitlab
If you want to modify some GitLab settings, you can use the above-mentioned configuration as a base and create your own YAML file.
-
Monitor the installation progress via
helm status gitlab
andminikube dashboard
. The installation could take up to 20-30 minutes depending on the amount of resources on your workstation. -
When all the pods show either a
Running
orCompleted
status, get the GitLab password as described in Initial login, and log in to GitLab via the UI. It will be accessible viahttps://gitlab.domain
wheredomain
is the value provided in the YAML file.
Patching the Rails code in the toolbox
pod
Patching operational GitLab service pods requires building new images, with the modified source code inside. These can not be directly patched.
The toolbox
/ task-runner
pod has everything needed to operate as a Rails-based pod, without interfering with other normal service operations. You can use it to run independent tasks, and to modify the source code temporarily to perform some tasks.
toolbox
pod, those will not be persisted if the pod is restarted. They’re only present for the life of the container’s operation.To patch the source code in the toolbox
pod:
-
Fetch the desired
.patch
file to be applied:- Either download the diff of a merge request directly as a patch file.
-
Or, fetch the diff directly using
curl
. Replace<mr_iid>
below with the IID of the merge request, or change the URL to point to a raw snippet:curl --output ~/<mr_iid>.patch "https://gitlab.com/gitlab-org/gitlab/-/merge_requests/<mr_iid>.patch"
-
Patch the local files on the
toolbox
pod:cd /srv/gitlab busybox patch -p1 -f < ~/<mr_iid>.patch