Helm으로 LMA 구축하기¶
LMA를 구축하기에 앞서 어떤식으로 구현할건지 간략하게 설명한다
EFK + prometheus를 조합해
log : fluentbit -> elasticsearch ->kibana metric : fluentbit -> prometheus -> elasticsearch -> kibana
위의 방식으로 LM을 수집하고 Alert는 kibana에서 기본으로 제공하는걸 가져다 쓸 계획이다
LMA그런건 모르겠고 이미지나 긁어와주쇼
VM생성은 파티션부분만 보고 나머진 스킵해도된다.
실제 LMA구축시작은 5. helm 설치부터 보면 된다
1. LMA용 VM 생성¶
1.1 LMA용 worker-Node 생성¶
[yjh-lma01]
qm create 575 \
--net0 virtio,bridge=vmbr0,tag=2188 \
--name yjh-lma01 \
--serial0 socket \
--boot order=virtio0 \
--memory 8192 \
--cicustom "user=local:snippets/debian12-lab.yaml" \
--ipconfig0 gw=10.21.248.1,ip=10.21.248.175/24 \
--socket 1 \
--cores 4 \
--agent 1 \
--ostype l26 \
--virtio0 hdd-2m-data:0,import-from=/var/lib/vz/template/iso/jammy-server-cloudimg-amd64.img,format=raw \
--ide2 local-zfs:cloudinit
[yjh-lma02]
qm create 576 \
--net0 virtio,bridge=vmbr0,tag=2188 \
--name yjh-lma02 \
--serial0 socket \
--boot order=virtio0 \
--memory 8192 \
--cicustom "user=local:snippets/debian12-lab.yaml" \
--ipconfig0 gw=10.21.248.1,ip=10.21.248.176/24 \
--socket 1 \
--cores 4 \
--agent 1 \
--ostype l26 \
--virtio0 hdd-2m-data:0,import-from=/var/lib/vz/template/iso/jammy-server-cloudimg-amd64.img,format=raw \
--ide2 local-zfs:cloudinit
[yjh-lma03]
qm create 577 \
--net0 virtio,bridge=vmbr0,tag=2188 \
--name yjh-lma03 \
--serial0 socket \
--boot order=virtio0 \
--memory 4096 \
--cicustom "user=local:snippets/debian12-lab.yaml" \
--ipconfig0 gw=10.21.248.1,ip=10.21.248.177/24 \
--socket 1 \
--cores 4 \
--agent 1 \
--ostype l26 \
--virtio0 hdd-2m-data:0,import-from=/var/lib/vz/template/iso/jammy-server-cloudimg-amd64.img,format=raw \
--ide2 local-zfs:cloudinit
[yjh-lma04]
qm create 578 \
--net0 virtio,bridge=vmbr0,tag=2188 \
--name yjh-lma04 \
--serial0 socket \
--boot order=virtio0 \
--memory 4096 \
--cicustom "user=local:snippets/debian12-lab.yaml" \
--ipconfig0 gw=10.21.248.1,ip=10.21.248.178/24 \
--socket 1 \
--cores 4 \
--agent 1 \
--ostype l26 \
--virtio0 hdd-2m-data:0,import-from=/var/lib/vz/template/iso/jammy-server-cloudimg-amd64.img,format=raw \
--ide2 local-zfs:cloudinit
[yjh-lma05]
qm create 579 \
--net0 virtio,bridge=vmbr0,tag=2188 \
--name yjh-lma05 \
--serial0 socket \
--boot order=virtio0 \
--memory 4096 \
--cicustom "user=local:snippets/debian12-lab.yaml" \
--ipconfig0 gw=10.21.248.1,ip=10.21.248.179/24 \
--socket 1 \
--cores 4 \
--agent 1 \
--ostype l26 \
--virtio0 hdd-2m-data:0,import-from=/var/lib/vz/template/iso/jammy-server-cloudimg-amd64.img,format=raw \
--ide2 local-zfs:cloudinit
elasticsearch전용 disk 추가 증설
qm set 575 -virtio1 hdd-2m-data:100
qm set 576 -virtio1 hdd-2m-data:100
qm set 577 -virtio1 hdd-2m-data:100
확인
disk resize
qm resize 575 virtio0 50G
qm resize 576 virtio0 50G
qm resize 577 virtio0 50G
qm resize 578 virtio0 50G
qm resize 579 virtio0 50G
vm start
ssh 한번씩 접속 (VM별 접속해서 hostname 설정)
ssh adun@10.21.248.175
ssh adun@10.21.248.176
ssh adun@10.21.248.177
ssh adun@10.21.248.178
ssh adun@10.21.248.179
hostname 설정
sudo hostnamectl set-hostname yjh-lma01
sudo hostnamectl set-hostname yjh-lma02
sudo hostnamectl set-hostname yjh-lma03
sudo hostnamectl set-hostname yjh-lma04
sudo hostnamectl set-hostname yjh-lma05
OS에서 디스크 추가 설정을 진행 lma01, lma02, lma03 진행
추가한 디스크 파티션생성
파일시스템 생성
fstab 등록 (추후 만드는 storageclass가 /opt/local-path-provisioner경로를 사용함으로 미리 설정만 해둔다)
디스크 마운트 확인
2. kubernetes 기본 구성¶
2.1 swap off¶
kubelet이 제대로 동작하려면 노드들은 swap disable 해야 하기 때문에 각 노드별로 다음 명령을 진행한다.
2.2 bridge-iptables module 설정¶
bridge 네트워크를 통해 송수신되는 컨테이너 패킷이 iptables 설정에 따라 제어되도록 설정한다.
overlay 커널 모듈을 즉시 로드한다. 컨테이너 오버레이 파일 시스템에 사용한다.
br_netfilter 커널 모듈을 즉시 로드한다. linux bridge network와 관련된 네트워크 필터링에 사용한다.
2.3 sysctl.conf 설정.¶
net.bridge.bridge-nf-call-iptables는 iptables가 브릿지 트래픽을 처리할 수 있도록 하는 설정입니다.
sudo cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
변경 내용 저장
2.4 방화벽 off¶
testbed니 방화벽 끔
방화벽 올리고 쓸거면 kube가 쓰는 port 찾아서 다 열어줘야함
3. Kubernetes CRE기본 구축¶
CRE를 설치한다.
3.1 Containerd 설치¶
containerd 1.7.21 설치
curl -L -O https://github.com/containerd/containerd/releases/download/v1.7.21/containerd-1.7.21-linux-amd64.tar.gz
systemd 등록
sudo curl -L -O --output-dir /etc/systemd/system/ https://raw.githubusercontent.com/containerd/containerd/main/containerd.service
systemd 수정
sudo vi /etc/systemd/system/containerd.service
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target dbus.service
[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/bin/containerd ## /usr/local/bin/containerd -> /usr/bin/containerd 로 변경
/etc/containerd/config.toml 파일 생성
config.toml 수정
containerd systemd daemon-reload, enable, 시작
sudo systemctl daemon-reload
sudo systemctl enable --now containerd
sudo systemctl restart containerd
sudo systemctl status containerd
확인
3.2 runc 설치¶
runc 1.1.13 설치
압축풀기
확인
4. kebernetes 설치¶
4.1 kube 설치 사전 패키지 설치¶
4.2 kubernetes 전용 apt 키링 생성¶
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
4.3 kubernetes repo apt 추가¶
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
4.4 kubeadm, kubelet, kubectl 설치¶
저장소 업데이트 후 kubeadm, kubelet, kubectl 설치하고 버전을 고정합니다.
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
쿠버네티스 서비스 등록 및 재시작
4.5 worker join¶
worker join
sudo kubeadm join 10.21.248.171:6443 --token px7ijl.cc6gvq0s4nf3sjq5 \
--discovery-token-ca-cert-hash sha256:b460439d004d758e0b504cd70c931e0f73543fd98b812908e298dd096045d9c8
5. helm 설치하기¶
5.1 helm 설치¶
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod +x get_helm.sh
./get_helm.sh
5.2 helm repo 추가 및 확인¶
elastic repo 추가
prometheus repo 추가
fluentbit repo 추가
repo update
확인
adun@yjh-bastion:~$ helm repo list
NAME URL
elastic https://helm.elastic.co
prometheus-community https://prometheus-community.github.io/helm-charts
fluent https://fluent.github.io/helm-charts
6. resource 생성¶
6.1 namespace 생성¶
lma namespace 생성
확인
6.2 StorageClass 생성¶
동적 프로비저닝을 쓸건데 rancher project에서 쓰는 stroageclass를 훔쳐다 쓰기로함
kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml
kubectl apply로 storageclass 생성
확인
adun@yjh-bastion:~/elasticsearch/values$ kubectl get sc local-path -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"local-path"},"provisioner":"rancher.io/local-path","reclaimPolicy":"Retain","volumeBindingMode":"WaitForFirstConsumer"}
creationTimestamp: "2025-03-07T05:00:39Z"
name: local-path
resourceVersion: "4578082"
uid: 80a298c0-aed8-485d-a875-464859872ee2
provisioner: rancher.io/local-path
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
7. elasticsearch 설치¶
elasticsearch node는 master, data, ingest 등등 node에 여러 role을 부여할 수 있음
master + data + ingest 3가지 role을 모두 소화하는 node를 만들 수도 있으나
상용환경에선 master 와 data는 분리하는걸 권장하기때문에
master-node, data + ingest node 2개의 노드로 분리할거다
7.1 nodeselctor를 위한 설정¶
elasticsearch를 lma01, lma02, lma03에 statefulset으로 띄우기 위해 라벨을 지정해준다
kubectl label nodes yjh-lma01 elasticsearch=true
kubectl label nodes yjh-lma02 elasticsearch=true
kubectl label nodes yjh-lma03 elasticsearch=true
7.2 values.yaml 생성¶
master-values.yaml 파일 생성
data-values.yaml 파일 생성
7.3 values.yaml 수정¶
7.3.1 matser-value.yaml 작성¶
helm chart에서 가져온 values 파일을 아래처럼 수정해서 사용한다
adun@yjh-bastion:~/elasticsearch/values$ vi master-values.yaml
---
clusterName: "elasticsearch"
nodeGroup: "master"
masterService: "elasticsearch-master"
roles:
- master ## master role 부여
replicas: 3 ## 3개의 master-node 운용
minimumMasterNodes: 2 ## master-node가 1대까지는 뻗어도 ㄱㅊ 최소 2대는 돌고있어야한다는 얘기
esMajorVersion: "" ## elasticsearch 버전을 쓰는건데 아래 image버전으로 명시해서 그냥 비워둠
esConfig: ## https 사용시 certificate를 어떻게 처리할지에 대한 구현, 모니터링 tool 사용 유무 등등 여기에 넣음
elasticsearch.yml: |
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.key: /usr/share/elasticsearch/config/certs/tls.key
xpack.security.transport.ssl.certificate: /usr/share/elasticsearch/config/certs/tls.crt
xpack.security.transport.ssl.certificate_authorities: "/usr/share/elasticsearch/config/certs/ca.crt"
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /usr/share/elasticsearch/config/certs/tls.key
xpack.security.http.ssl.certificate: /usr/share/elasticsearch/config/certs/tls.crt
xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.certificate_authorities: [ "/usr/share/elasticsearch/config/certs/ca.crt" ]
xpack.monitoring.collection.enabled: true
discovery.seed_hosts: ["elasticsearch-master-headless"]
cluster.initial_master_nodes: ["elasticsearch-master-0", "elasticsearch-master-1", "elasticsearch-master-2"]
createCert: true ## true면 helm install시 secret을 새로 생성해줌(tls.key, tls,crt ca.crt를 만들어줌)
esJvmOptions: {}
extraEnvs: []
envFrom: []
# Disable it to use your own elastic-credential Secret.
secret:
enabled: true
password: "dkensxhflektm"
env:
- name: ELASTIC_PASSWORD
value: "dkensxhflektm"
# A list of secrets and their paths to mount inside the pod
# This is useful for mounting certificates for security and for mounting
# the X-Pack license
hostAliases: []
#- ip: "127.0.0.1"
# hostnames:
# - "foo.local"
# - "bar.local"
image: "docker.elastic.co/elasticsearch/elasticsearch"
imageTag: "8.5.1"
imagePullPolicy: "IfNotPresent"
podAnnotations: {}
# iam.amazonaws.com/role: es-cluster
# additionals labels
labels: {}
esJavaOpts: "-Xmx1g -Xms1g" # example: "-Xmx1g -Xms1g"
resources:
requests:
cpu: "1000m"
memory: "2Gi"
limits:
cpu: "2000m"
memory: "4Gi"
initResources: {}
# limits:
# cpu: "25m"
# # memory: "128Mi"
# requests:
# cpu: "25m"
# memory: "128Mi"
networkHost: "0.0.0.0"
volumeClaimTemplate:
storageClassName: local-path
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 20Gi
rbac:
create: false
serviceAccountAnnotations: {}
serviceAccountName: ""
automountToken: true
podSecurityPolicy:
create: false
name: ""
spec:
privileged: true
fsGroup:
rule: RunAsAny
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- secret
- configMap
- persistentVolumeClaim
- emptyDir
persistence:
enabled: true
labels:
# Add default labels for the volumeClaimTemplate of the StatefulSet
enabled: false
annotations: {}
storageClassName: local-path
accessMode: ReadWriteOnce
size: 20Gi
extraVolumes: []
# - name: extras
# emptyDir: {}
extraVolumeMounts: []
# - name: extras
# mountPath: /usr/share/extras
# readOnly: true
extraContainers: []
# - name: do-something
# image: busybox
# command: ['do', 'something']
extraInitContainers: []
# - name: do-something
# image: busybox
# command: ['do', 'something']
# This is the PriorityClass settings as defined in
# https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass
priorityClassName: ""
# By default this will make sure two pods don't end up on the same node
# Changing this to a region would allow you to spread pods across regions
antiAffinityTopologyKey: "kubernetes.io/hostname"
# Hard means that by default pods will only be scheduled if there are enough nodes for them
# and that they will never end up on the same node. Setting this to soft will do this "best effort"
antiAffinity: "hard"
# This is the node affinity settings as defined in
# https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: elasticsearch
operator: In
values:
- "true"
weight: 100
# The default is to deploy all pods serially. By setting this to parallel all pods are started at
# the same time when bootstrapping the cluster
podManagementPolicy: "Parallel"
# The environment variables injected by service links are not used, but can lead to slow Elasticsearch boot times when
# there are many services in the current namespace.
# If you experience slow pod startups you probably want to set this to `false`.
enableServiceLinks: true
protocol: https ## 기본값 http임 https로 바꿔줘야함
httpPort: 9200
transportPort: 9300
service:
enabled: true
labels: {}
labelsHeadless: {}
type: NodePort
#type: ClusterIP
# Consider that all endpoints are considered "ready" even if the Pods themselves are not
# https://kubernetes.io/docs/reference/kubernetes-api/service-resources/service-v1/#ServiceSpec
publishNotReadyAddresses: false
nodePort: "30001"
annotations: {}
httpPortName: http
transportPortName: transport
loadBalancerIP: ""
loadBalancerSourceRanges: []
externalTrafficPolicy: ""
updateStrategy: RollingUpdate
# This is the max unavailable setting for the pod disruption budget
# The default value of 1 will make sure that kubernetes won't allow more than 1
# of your pods to be unavailable during maintenance
maxUnavailable: 1
podSecurityContext:
fsGroup: 1000
runAsUser: 1000
securityContext:
capabilities:
drop:
- ALL
# readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
# How long to wait for elasticsearch to stop gracefully
terminationGracePeriod: 120
sysctlVmMaxMapCount: 262144
readinessProbe:
failureThreshold: 3
initialDelaySeconds: 120
periodSeconds: 10
successThreshold: 3
timeoutSeconds: 10
# https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html#request-params wait_for_status
clusterHealthCheckParams: "wait_for_status=yellow&timeout=30s"
## Use an alternate scheduler.
## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
##
schedulerName: ""
imagePullSecrets: []
nodeSelector:
elasticsearch: "true"
tolerations: []
# Enabling this will publicly expose your Elasticsearch instance.
# Only enable this if you have security enabled on your cluster
ingress:
enabled: false
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
className: "nginx"
pathtype: ImplementationSpecific
hosts:
- host: chart-example.local
paths:
- path: /
tls: []
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
nameOverride: ""
fullnameOverride: ""
healthNameOverride: ""
lifecycle: {}
# preStop:
# exec:
# command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
# postStart:
# exec:
# command:
# - bash
# - -c
# - |
# #!/bin/bash
# # Add a template to adjust number of shards/replicas
# TEMPLATE_NAME=my_template
# INDEX_PATTERN="logstash-*"
# SHARD_COUNT=8
# REPLICA_COUNT=1
# ES_URL=http://localhost:9200
# while [[ "$(curl -s -o /dev/null -w '%{http_code}\n' $ES_URL)" != "200" ]]; do sleep 1; done
# curl -XPUT "$ES_URL/_template/$TEMPLATE_NAME" -H 'Content-Type: application/json' -d'{"index_patterns":['\""$INDEX_PATTERN"\"'],"settings":{"number_of_shards":'$SHARD_COUNT',"number_of_replicas":'$REPLICA_COUNT'}}'
sysctlInitContainer:
enabled: true
keystore: []
networkPolicy:
## Enable creation of NetworkPolicy resources. Only Ingress traffic is filtered for now.
## In order for a Pod to access Elasticsearch, it needs to have the following label:
## {{ template "uname" . }}-client: "true"
## Example for default configuration to access HTTP port:
## elasticsearch-master-http-client: "true"
## Example for default configuration to access transport port:
## elasticsearch-master-transport-client: "true"
http:
##enabled: true
enabled: false
## if explicitNamespacesSelector is not set or set to {}, only client Pods being in the networkPolicy's namespace
## and matching all criteria can reach the DB.
## But sometimes, we want the Pods to be accessible to clients from other namespaces, in this case, we can use this
## parameter to select these namespaces
##
# explicitNamespacesSelector:
# # Accept from namespaces with all those different rules (only from whitelisted Pods)
# matchLabels:
# role: frontend
# matchExpressions:
# - {key: role, operator: In, values: [frontend]}
## Additional NetworkPolicy Ingress "from" rules to set. Note that all rules are OR-ed.
##
# additionalRules:
# - podSelector:
# matchLabels:
# role: frontend
# - podSelector:
# matchExpressions:
# - key: role
# operator: In
# values:
# - frontend
transport:
## Note that all Elasticsearch Pods can talk to themselves using transport port even if enabled.
enabled: false
# explicitNamespacesSelector:
# matchLabels:
# role: frontend
# matchExpressions:
# - {key: role, operator: In, values: [frontend]}
# additionalRules:
# - podSelector:
# matchLabels:
# role: frontend
# - podSelector:
# matchExpressions:
# - key: role
# operator: In
# values:
# - frontend
tests:
enabled: true
7.3.2 data-values.yaml 작성¶
master와 yaml파일 내용이 좀 다르기 때문에 잘 작성한다.
adun@yjh-bastion:~/elasticsearch/values$ vi data-values.yaml
---
clusterName: "elasticsearch"
nodeGroup: "data" ## nodegroup만 변경
masterService: "elasticsearch-master"
roles: ## data, ingest role을 부여함
- data
- ingest
replicas: 3 ## data는 replica만 설정하고 minimum 어쩌고가 안들어감
esMajorVersion: ""
esConfig: ## master에서 생성한 certificate를 참조해서 https 통신을 하게끔 설정하기때문에 master와 설정이 쪼금 다름
elasticsearch.yml: |
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.key: /usr/share/elasticsearch/config/certs/tls.key
xpack.security.transport.ssl.certificate: /usr/share/elasticsearch/config/certs/tls.crt
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.certificate_authorities: [ "/usr/share/elasticsearch/config/certs/ca.crt" ]
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /usr/share/elasticsearch/config/certs/tls.key
xpack.security.http.ssl.certificate: /usr/share/elasticsearch/config/certs/tls.crt
xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.certificate_authorities: [ "/usr/share/elasticsearch/config/certs/ca.crt" ]
xpack.monitoring.collection.enabled: true
discovery.seed_hosts: ["elasticsearch-master-headless"]
createCert: false ## master와는 다르게 false로 설정하고 esConfig에서 master secret을 참조해서 연동하게끔 설정함
secretMounts: ## master가 만든 secret인 elasticsearch-master-certs를 참조해서 data-node pod 내부에 인증관련 파일을 복사해서 넣게함
- name: elastic-certificates
secretName: elasticsearch-master-certs
path: /usr/share/elasticsearch/config/certs
defaultMode: 0755
esJvmOptions: {}
extraEnvs: []
envFrom: []
# Disable it to use your own elastic-credential Secret.
secret:
name: elasticsearch-master-certs
enabled: true
password: "dkensxhflektm" # generated randomly if not defined
env:
- name: ELASTIC_PASSWORD
value: "dkensxhflektm"
# A list of secrets and their paths to mount inside the pod
# This is useful for mounting certificates for security and for mounting
# the X-Pack license
hostAliases: []
#- ip: "127.0.0.1"
# hostnames:
# - "foo.local"
# - "bar.local"
image: "docker.elastic.co/elasticsearch/elasticsearch"
imageTag: "8.5.1"
imagePullPolicy: "IfNotPresent"
podAnnotations: {}
# iam.amazonaws.com/role: es-cluster
# additionals labels
labels: {}
esJavaOpts: "-Xmx1g -Xms1g"
resources:
requests:
cpu: "1000m"
memory: "2Gi"
limits:
cpu: "2000m"
memory: "4Gi"
initResources: {}
# limits:
# cpu: "25m"
# # memory: "128Mi"
# requests:
# cpu: "25m"
# memory: "128Mi"
networkHost: "0.0.0.0"
volumeClaimTemplate:
storageClassName: local-path
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 60Gi ## 실제로 data가 들어가는 node이기 때문에 master보다 용량을 많이 줘야함
rbac:
create: false
serviceAccountAnnotations: {}
serviceAccountName: ""
automountToken: true
podSecurityPolicy:
create: false
name: ""
spec:
privileged: true
fsGroup:
rule: RunAsAny
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- secret
- configMap
- persistentVolumeClaim
- emptyDir
persistence:
enabled: true
labels:
# Add default labels for the volumeClaimTemplate of the StatefulSet
enabled: false
annotations: {}
storageClassName: local-path
accessMode: ReadWriteOnce
size: 60Gi ## 역시 마찬가지로 master보다 용량많게넣음 (물론 volumeClaimTemplate의 용량이랑 똑같이 작성해야함)
extraVolumes: []
# - name: extras
# emptyDir: {}
extraVolumeMounts: []
# - name: extras
# mountPath: /usr/share/extras
# readOnly: true
extraContainers: []
# - name: do-something
# image: busybox
# command: ['do', 'something']
extraInitContainers: []
# - name: do-something
# image: busybox
# command: ['do', 'something']
# This is the PriorityClass settings as defined in
# https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass
priorityClassName: ""
# By default this will make sure two pods don't end up on the same node
# Changing this to a region would allow you to spread pods across regions
antiAffinityTopologyKey: "kubernetes.io/hostname"
# Hard means that by default pods will only be scheduled if there are enough nodes for them
# and that they will never end up on the same node. Setting this to soft will do this "best effort"
antiAffinity: "hard"
# This is the node affinity settings as defined in
# https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: elasticsearch
operator: In
values:
- "true"
# The default is to deploy all pods serially. By setting this to parallel all pods are started at
# the same time when bootstrapping the cluster
podManagementPolicy: "Parallel"
# The environment variables injected by service links are not used, but can lead to slow Elasticsearch boot times when
# there are many services in the current namespace.
# If you experience slow pod startups you probably want to set this to `false`.
enableServiceLinks: true
protocol: https ## 역시 https로 변경해서 작성
httpPort: 9200
transportPort: 9300
service:
enabled: true
labels: {}
labelsHeadless: {}
type: NodePort
#type: ClusterIP
# Consider that all endpoints are considered "ready" even if the Pods themselves are not
# https://kubernetes.io/docs/reference/kubernetes-api/service-resources/service-v1/#ServiceSpec
publishNotReadyAddresses: false
nodePort: "30002"
annotations: {}
httpPortName: http
transportPortName: transport
loadBalancerIP: ""
loadBalancerSourceRanges: []
externalTrafficPolicy: ""
updateStrategy: RollingUpdate
# This is the max unavailable setting for the pod disruption budget
# The default value of 1 will make sure that kubernetes won't allow more than 1
# of your pods to be unavailable during maintenance
maxUnavailable: 1
podSecurityContext:
fsGroup: 1000
runAsUser: 1000
securityContext:
capabilities:
drop:
- ALL
# readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
# How long to wait for elasticsearch to stop gracefully
terminationGracePeriod: 120
sysctlVmMaxMapCount: 262144
readinessProbe:
failureThreshold: 3
initialDelaySeconds: 60
periodSeconds: 20
successThreshold: 1
timeoutSeconds: 15
# https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html#request-params wait_for_status
## 기본값은 green인데 green상태까지 충족하는게 꽤 오래걸려서 성격이 급한 우리는 yellow정도로 낮추고 진행하는게 정신건강에 좋다
## green은 primary shard + replica shard가 전부 배포된 상태
## yellow는 primary shard는 잘 배포됐으나 일부 replica shard는 배포가 덜 된 상태
## red는 primary도 배포가 잘안된 상태 (똥망)
clusterHealthCheckParams: "wait_for_status=yellow&timeout=60s"
## Use an alternate scheduler.
## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
##
schedulerName: ""
imagePullSecrets: []
nodeSelector:
elasticsearch: "true"
tolerations: []
# Enabling this will publicly expose your Elasticsearch instance.
# Only enable this if you have security enabled on your cluster
ingress:
enabled: false
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
className: "nginx"
pathtype: ImplementationSpecific
hosts:
- host: chart-example.local
paths:
- path: /
tls: []
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
nameOverride: ""
fullnameOverride: ""
healthNameOverride: ""
lifecycle: {}
# preStop:
# exec:
# command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
# postStart:
# exec:
# command:
# - bash
# - -c
# - |
# #!/bin/bash
# # Add a template to adjust number of shards/replicas
# TEMPLATE_NAME=my_template
# INDEX_PATTERN="logstash-*"
# SHARD_COUNT=8
# REPLICA_COUNT=1
# ES_URL=http://localhost:9200
# while [[ "$(curl -s -o /dev/null -w '%{http_code}\n' $ES_URL)" != "200" ]]; do sleep 1; done
# curl -XPUT "$ES_URL/_template/$TEMPLATE_NAME" -H 'Content-Type: application/json' -d'{"index_patterns":['\""$INDEX_PATTERN"\"'],"settings":{"number_of_shards":'$SHARD_COUNT',"number_of_replicas":'$REPLICA_COUNT'}}'
sysctlInitContainer:
enabled: true
keystore: []
networkPolicy:
## Enable creation of NetworkPolicy resources. Only Ingress traffic is filtered for now.
## In order for a Pod to access Elasticsearch, it needs to have the following label:
## {{ template "uname" . }}-client: "true"
## Example for default configuration to access HTTP port:
## elasticsearch-master-http-client: "true"
## Example for default configuration to access transport port:
## elasticsearch-master-transport-client: "true"
http:
##enabled: true
enabled: false
## if explicitNamespacesSelector is not set or set to {}, only client Pods being in the networkPolicy's namespace
## and matching all criteria can reach the DB.
## But sometimes, we want the Pods to be accessible to clients from other namespaces, in this case, we can use this
## parameter to select these namespaces
##
# explicitNamespacesSelector:
# # Accept from namespaces with all those different rules (only from whitelisted Pods)
# matchLabels:
# role: frontend
# matchExpressions:
# - {key: role, operator: In, values: [frontend]}
## Additional NetworkPolicy Ingress "from" rules to set. Note that all rules are OR-ed.
##
# additionalRules:
# - podSelector:
# matchLabels:
# role: frontend
# - podSelector:
# matchExpressions:
# - key: role
# operator: In
# values:
# - frontend
transport:
## Note that all Elasticsearch Pods can talk to themselves using transport port even if enabled.
enabled: false
# explicitNamespacesSelector:
# matchLabels:
# role: frontend
# matchExpressions:
# - {key: role, operator: In, values: [frontend]}
# additionalRules:
# - podSelector:
# matchLabels:
# role: frontend
# - podSelector:
# matchExpressions:
# - key: role
# operator: In
# values:
# - frontend
tests:
enabled: true
7.4 elasticserach 배포¶
작성한 values.yaml 내용을 반영하여 elasticsearch를 배포하게끔 명령어를 입력해서 설치한다
7.4.1 master-node 배포¶
master를 우선 배포해본다
master,data를 한꺼번에 배포하면 나중에 알아서 연동하는지는 모르겠음
es-node가 생성뜨는데 꽤 오래걸리고 이때 describe pod 해보면 readiness probe faile이 계속 떠있음
그냥 여유롭게 10분정도까진 기다려보는 인내심이 필요함
추가로 pod가 잘뜨는것도 중요하지만
내가 지정한 node-selctor에 맞게 들어갔는지도 확인해야한다
master-0 -> lma01
master-1 -> lma02
master-2 -> lma03 이렇게 생성되게끔 이쁘게 하고싶으나 그건 나중에 알아볼 예정
adun@yjh-bastion:~/elasticsearch/values$ kubectl get pod -n lma -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
elasticsearch-master-0 1/1 Running 0 5h38m 10.1.92.97 yjh-lma03 <none> <none>
elasticsearch-master-1 1/1 Running 0 5h38m 10.1.39.174 yjh-lma01 <none> <none>
elasticsearch-master-2 1/1 Running 0 5h37m 10.1.224.112 yjh-lma02 <none> <none>
7.4.2 data-node 배포¶
data를 배포해보자
역시 마찬가지로 좀 기다려야하고 node-selector에 맞게 떴는지 확인하자
adun@yjh-bastion:~/elasticsearch/values$ kubectl get pod -n lma -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
elasticsearch-data-0 1/1 Running 0 5h46m 10.1.92.94 yjh-lma03 <none> <none>
elasticsearch-data-1 1/1 Running 0 5h44m 10.1.39.172 yjh-lma01 <none> <none>
elasticsearch-data-2 1/1 Running 0 5h46m 10.1.224.111 yjh-lma02 <none> <none>
elasticsearch-master-0 1/1 Running 0 5h38m 10.1.92.97 yjh-lma03 <none> <none>
elasticsearch-master-1 1/1 Running 0 5h38m 10.1.39.174 yjh-lma01 <none> <none>
elasticsearch-master-2 1/1 Running 0 5h37m 10.1.224.112 yjh-lma02 <none> <none>
8. kibana 설치¶
8.1 kibana yaml생성¶
elastic helm chart에 kibana가 따로 있어서 kibana용 helm repo를 따로 추가할 필요 없음
그냥 바로 아래명령어로 땡겨온다
8.2 kibana-values.yaml 수정¶
adun@yjh-bastion:~/elasticsearch/values$ adun@yjh-bastion:~/elasticsearch/values$ cat kibana-values.yaml
---
elasticsearchHosts: "https://elasticsearch-master.lma.svc:9200" ## master-node내에서 쓰는 DNS이름 쓰면 됌
elasticsearchCertificateSecret: elasticsearch-master-certs ## master-node가 만든 secert
elasticsearchCertificateAuthoritiesFile: ca.crt ## master-node가 만든 ca.crt파일
elasticsearchCredentialSecret: elasticsearch-master-credentials ## master-node가 만든 secert
replicas: 1
# Extra environment variables to append to this nodeGroup
# This will be appended to the current 'env:' key. You can use any of the kubernetes env
# syntax here
extraEnvs: ## 7.x 버전은 elasticsearch의 계정,계정 패스워드를 넣어서 인증할 수 있었으나 8.x버전 부터는
- name: ELASTICSEARCH_SERVICEACCOUNTTOKEN ## token방식을 쓰기때문에 이거 넣어줘야함
valueFrom:
secretKeyRef:
name: kibana-kibana-es-token
key: token
# valueFrom:
# secretKeyRef:
# name: elasticsearch-master-credentials
# key: password
- name: "NODE_OPTIONS"
value: "--max-old-space-size=1800"
# - name: MY_ENVIRONMENT_VAR
# value: the_value_goes_here
# Allows you to load environment variables from kubernetes secret or config map
envFrom: []
# - secretRef:
# name: env-secret
# - configMapRef:
# name: config-map
# A list of secrets and their paths to mount inside the pod
# This is useful for mounting certificates for security and for mounting
# the X-Pack license
secretMounts: ## master-node가 만든 secret 참조
- name: elastic-certificates
secretName: elasticsearch-master-certs
path: /usr/share/kibana/config/elastic-certs
# - name: kibana-keystore
# secretName: kibana-keystore
# path: /usr/share/kibana/data/kibana.keystore
# subPath: kibana.keystore # optional
hostAliases: []
#- ip: "127.0.0.1"
# hostnames:
# - "foo.local"
# - "bar.local"
image: "docker.elastic.co/kibana/kibana"
imageTag: "8.5.1"
imagePullPolicy: "IfNotPresent"
# additionals labels
labels: {}
annotations: {}
podAnnotations: {}
# iam.amazonaws.com/role: es-cluster
resources:
requests:
cpu: "1000m"
memory: "2Gi"
limits:
cpu: "1000m"
memory: "2Gi"
protocol: https
serverHost: "0.0.0.0"
healthCheckPath: "/app/kibana"
# Allows you to add any config files in /usr/share/kibana/config/
# such as kibana.yml
kibanaConfig: ## elasticsearch처럼 https 어떻게 처리할지 config 넣어준거임
kibana.yml: |
server.ssl.enabled: true
server.ssl.certificate: /usr/share/kibana/config/elastic-certs/tls.crt
server.ssl.key: /usr/share/kibana/config/elastic-certs/tls.key
monitoring.ui.ccs.enabled: false
elasticsearch.ssl.certificateAuthorities: ["/usr/share/kibana/config/elastic-certs/ca.crt"]
# kibana.yml: |
# key:
# nestedkey: value
# If Pod Security Policy in use it may be required to specify security context as well as service account
podSecurityContext:
fsGroup: 1000
securityContext:
capabilities:
drop:
- ALL
# readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
serviceAccount: ""
# Whether or not to automount the service account token in the pod. Normally, Kibana does not need this
automountToken: true
# This is the PriorityClass settings as defined in
# https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass
priorityClassName: ""
httpPort: 5601
extraVolumes:
[]
# - name: extras
# emptyDir: {}
extraVolumeMounts:
[]
# - name: extras
# mountPath: /usr/share/extras
# readOnly: true
#
extraContainers: []
# - name: dummy-init
# image: busybox
# command: ['echo', 'hey']
extraInitContainers: []
# - name: dummy-init
# image: busybox
# command: ['echo', 'hey']
updateStrategy:
type: "Recreate"
service:
type: NodePort
loadBalancerIP: ""
port: 5601
nodePort: "30003"
labels: {}
annotations: {}
# cloud.google.com/load-balancer-type: "Internal"
# service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
# service.beta.kubernetes.io/azure-load-balancer-internal: "true"
# service.beta.kubernetes.io/openstack-internal-load-balancer: "true"
# service.beta.kubernetes.io/cce-load-balancer-internal-vpc: "true"
loadBalancerSourceRanges: []
# 0.0.0.0/0
httpPortName: http
ingress:
enabled: false
className: "nginx"
pathtype: ImplementationSpecific
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
hosts:
- host: kibana-example.local
paths:
- path: /
#tls: []
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
readinessProbe:
failureThreshold: 5
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
imagePullSecrets: []
nodeSelector: {}
tolerations: []
affinity: {}
nameOverride: ""
fullnameOverride: ""
lifecycle: {}
# preStop:
# exec:
# command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
# postStart:
# exec:
# command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
8.3 kibana 배포¶
아래 명령어로 배포한다
개인적으론 배포가 오래걸렸음 명령어를 치고서 출력이 안나오더라도 인내심을 가지고 기다리자
adun@yjh-bastion:~/elasticsearch/values$ kubectl get pod -n lma -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
elasticsearch-data-0 1/1 Running 0 6h25m 10.1.92.94 yjh-lma03 <none> <none>
elasticsearch-data-1 1/1 Running 0 6h23m 10.1.39.172 yjh-lma01 <none> <none>
elasticsearch-data-2 1/1 Running 0 6h25m 10.1.224.111 yjh-lma02 <none> <none>
elasticsearch-master-0 1/1 Running 0 6h17m 10.1.92.97 yjh-lma03 <none> <none>
elasticsearch-master-1 1/1 Running 0 6h17m 10.1.39.174 yjh-lma01 <none> <none>
elasticsearch-master-2 1/1 Running 0 6h16m 10.1.224.112 yjh-lma02 <none> <none>
kibana-kibana-c5f5fc479-9ddcx 1/1 Running 0 6h41m 10.1.115.115 yjh-lma04 <none> <none>
9. fluentbit 설치¶
9.1 fluentbit yaml 생성¶
아래 명령어로 values.yaml을 뽑아낸다
9.2 fluent-bit-values.yaml 수정¶
내용은 좀 긴데 수정할거는 많지 않다
adun@yjh-bastion:~/elasticsearch/values$ # Default values for fluent-bit.
# kind -- DaemonSet or Deployment
kind: DaemonSet
# replicaCount -- Only applicable if kind=Deployment
replicaCount: 1
image:
repository: cr.fluentbit.io/fluent/fluent-bit
# Overrides the image tag whose default is {{ .Chart.AppVersion }}
# Set to "-" to not use the default value
tag:
digest:
pullPolicy: IfNotPresent
testFramework:
enabled: true
namespace:
image:
repository: busybox
pullPolicy: Always
tag: latest
digest:
imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""
serviceAccount:
create: true
annotations: {}
name:
rbac:
create: true
nodeAccess: false
eventsAccess: false
# Configure podsecuritypolicy
# Ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/
# from Kubernetes 1.25, PSP is deprecated
# See: https://kubernetes.io/blog/2022/08/23/kubernetes-v1-25-release/#pod-security-changes
# We automatically disable PSP if Kubernetes version is 1.25 or higher
podSecurityPolicy:
create: false
annotations: {}
runAsUser:
rule: RunAsAny
seLinux:
# This policy assumes the nodes are using AppArmor rather than SELinux.
rule: RunAsAny
# OpenShift-specific configuration
openShift:
enabled: false
securityContextConstraints:
# Create SCC for Fluent-bit and allow use it
create: true
name: ""
annotations: {}
runAsUser:
type: RunAsAny
seLinuxContext:
type: MustRunAs
# Use existing SCC in cluster, rather then create new one
existingName: ""
podSecurityContext:
fsGroup: 0
hostNetwork: false
dnsPolicy: ClusterFirst
dnsConfig: {}
# nameservers:
# - 1.2.3.4
# searches:
# - ns1.svc.cluster-domain.example
# - my.dns.search.suffix
# options:
# - name: ndots
# value: "2"
# - name: edns0
hostAliases: []
# - ip: "1.2.3.4"
# hostnames:
# - "foo.local"
# - "bar.local"
securityContext:
# capabilities:
# drop:
# - ALL
# readOnlyRootFilesystem: true
# runAsNonRoot: true
runAsUser: 0
runAsGroup: 0
service:
type: ClusterIP
port: 2020
internalTrafficPolicy:
loadBalancerClass:
loadBalancerSourceRanges: []
labels: {}
# nodePort: 30020
# clusterIP: 172.16.10.1
annotations: {}
# prometheus.io/path: "/api/v1/metrics/prometheus"
# prometheus.io/port: "2020"
# prometheus.io/scrape: "true"
externalIPs: []
# externalIPs:
# - 2.2.2.2
serviceMonitor:
enabled: false
# namespace: monitoring
# interval: 10s
# scrapeTimeout: 10s
# selector:
# prometheus: my-prometheus
# ## metric relabel configs to apply to samples before ingestion.
# ##
# metricRelabelings:
# - sourceLabels: [__meta_kubernetes_service_label_cluster]
# targetLabel: cluster
# regex: (.*)
# replacement: ${1}
# action: replace
# ## relabel configs to apply to samples after ingestion.
# ##
# relabelings:
# - sourceLabels: [__meta_kubernetes_pod_node_name]
# separator: ;
# regex: ^(.*)$
# targetLabel: nodename
# replacement: $1
# action: replace
# scheme: ""
# tlsConfig: {}
## Bear in mind if you want to collect metrics from a different port
## you will need to configure the new ports on the extraPorts property.
additionalEndpoints: []
# - port: metrics
# path: /metrics
# interval: 10s
# scrapeTimeout: 10s
# scheme: ""
# tlsConfig: {}
# # metric relabel configs to apply to samples before ingestion.
# #
# metricRelabelings:
# - sourceLabels: [__meta_kubernetes_service_label_cluster]
# targetLabel: cluster
# regex: (.*)
# replacement: ${1}
# action: replace
# # relabel configs to apply to samples after ingestion.
# #
# relabelings:
# - sourceLabels: [__meta_kubernetes_pod_node_name]
# separator: ;
# regex: ^(.*)$
# targetLabel: nodename
# replacement: $1
# action: replace
prometheusRule:
enabled: false
# namespace: ""
# additionalLabels: {}
# rules:
# - alert: NoOutputBytesProcessed
# expr: rate(fluentbit_output_proc_bytes_total[5m]) == 0
# annotations:
# message: |
# Fluent Bit instance {{ $labels.instance }}'s output plugin {{ $labels.name }} has not processed any
# bytes for at least 15 minutes.
# summary: No Output Bytes Processed
# for: 15m
# labels:
# severity: critical
dashboards:
enabled: false
labelKey: grafana_dashboard
labelValue: 1
annotations: {}
namespace: ""
deterministicUid: false
lifecycle: {}
# preStop:
# exec:
# command: ["/bin/sh", "-c", "sleep 20"]
livenessProbe:
httpGet:
path: /
port: http
readinessProbe:
httpGet:
path: /api/v1/health
port: http
resources:
requests:
cpu: "100m"
memory: "200Mi"
limits:
cpu: "500m"
memory: "500Mi"
## only available if kind is Deployment
ingress:
enabled: false
ingressClassName: ""
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
hosts: []
# - host: fluent-bit.example.tld
extraHosts: []
# - host: fluent-bit-extra.example.tld
## specify extraPort number
# port: 5170
tls: []
# - secretName: fluent-bit-example-tld
# hosts:
# - fluent-bit.example.tld
## only available if kind is Deployment
autoscaling:
vpa:
enabled: false
annotations: {}
# List of resources that the vertical pod autoscaler can control. Defaults to cpu and memory
controlledResources: []
# Define the max allowed resources for the pod
maxAllowed: {}
# cpu: 200m
# memory: 100Mi
# Define the min allowed resources for the pod
minAllowed: {}
# cpu: 200m
# memory: 100Mi
updatePolicy:
# Specifies whether recommended updates are applied when a Pod is started and whether recommended updates
# are applied during the life of a Pod. Possible values are "Off", "Initial", "Recreate", and "Auto".
updateMode: Auto
enabled: false
minReplicas: 1
maxReplicas: 3
targetCPUUtilizationPercentage: 75
# targetMemoryUtilizationPercentage: 75
## see https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-multiple-metrics-and-custom-metrics
customRules: []
# - type: Pods
# pods:
# metric:
# name: packets-per-second
# target:
# type: AverageValue
# averageValue: 1k
## see https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-configurable-scaling-behavior
behavior: {}
# scaleDown:
# policies:
# - type: Pods
# value: 4
# periodSeconds: 60
# - type: Percent
# value: 10
# periodSeconds: 60
## only available if kind is Deployment
podDisruptionBudget:
enabled: false
annotations: {}
maxUnavailable: "30%"
nodeSelector: {}
tolerations: []
affinity: {}
labels: {}
annotations: {}
podAnnotations: {}
podLabels: {}
## How long (in seconds) a pods needs to be stable before progressing the deployment
##
minReadySeconds:
## How long (in seconds) a pod may take to exit (useful with lifecycle hooks to ensure lb deregistration is done)
##
terminationGracePeriodSeconds:
priorityClassName: ""
env: []
# - name: FOO
# value: "bar"
# The envWithTpl array below has the same usage as "env", but is using the tpl function to support templatable string.
# This can be useful when you want to pass dynamic values to the Chart using the helm argument "--set <variable>=<value>"
# https://helm.sh/docs/howto/charts_tips_and_tricks/#using-the-tpl-function
envWithTpl: []
# - name: FOO_2
# value: "{{ .Values.foo2 }}"
#
# foo2: bar2
envFrom: []
# This supports either a structured array or a templatable string
extraContainers: []
# Array mode
# extraContainers:
# - name: do-something
# image: busybox
# command: ['do', 'something']
# String mode
# extraContainers: |-
# - name: do-something
# image: bitnami/kubectl:{{ .Capabilities.KubeVersion.Major }}.{{ .Capabilities.KubeVersion.Minor }}
# command: ['kubectl', 'version']
flush: 1
metricsPort: 2020
extraPorts: []
# - port: 5170
# containerPort: 5170
# protocol: TCP
# name: tcp
# nodePort: 30517
extraVolumes:
- name: elasticsearch-master-certs
secret:
secretName: elasticsearch-master-certs
extraVolumeMounts:
- name: elasticsearch-master-certs
mountPath: /etc/fluentbit/
readOnly: true
#extraContainers:
#- name: debug-container
#image: nicolaka/netshoot
#command:
#- sleep
#- "3600"
#volumeMounts:
#- name: elasticsearch-master-certs
#mountPath: /etc/fluentbit/
updateStrategy: {}
# type: RollingUpdate
# rollingUpdate:
# maxUnavailable: 1
# Make use of a pre-defined configmap instead of the one templated here
existingConfigMap: ""
networkPolicy:
enabled: false
# ingress:
# from: []
# See Lua script configuration example in README.md
luaScripts: {}
## https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/configuration-file
config:
service: |
[SERVICE]
Daemon Off
Flush {{ .Values.flush }}
Log_Level {{ .Values.logLevel }}
Parsers_File /fluent-bit/etc/parsers.conf
Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
Health_Check On
## https://docs.fluentbit.io/manual/pipeline/inputs
inputs: |
[INPUT]
Name tail
Path /var/log/containers/*.log
multiline.parser docker, cri
Tag kube.*
Mem_Buf_Limit 5MB
Skip_Long_Lines On
[INPUT]
Name systemd
Tag host.*
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
Read_From_Tail On
[INPUT]
Name node_exporter_metrics
Tag node_metrics
Scrape_Interval 2
[INPUT]
Name fluentbit_metrics
Tag fluentbit_metrics
scrape_interval 2
## https://docs.fluentbit.io/manual/pipeline/filters
filters: |
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Keep_Log Off
K8S-Logging.Parser On
K8S-Logging.Exclude On
## https://docs.fluentbit.io/manual/pipeline/outputs
outputs: |
[OUTPUT]
Name es
Match kube.*
Host elasticsearch-master.lma.svc
Port 9200
HTTP_User elastic
HTTP_Passwd dkensxhflektm
tls On
tls.verify Off
tls.ca_file /etc/fluentbit/..data/ca.crt
tls.debug 4
Logstash_Format Off
Retry_Limit False
suppress_type_name On
[OUTPUT]
Name es
Match host.*
Host elasticsearch-master.lma.svc
Port 9200
HTTP_User elastic
HTTP_Passwd dkensxhflektm
tls On
tls.verify Off
tls.ca_file /etc/fluentbit/..data/ca.crt
tls.debug 4
Logstash_Format Off
Retry_Limit False
suppress_type_name On
[OUTPUT]
Name prometheus_exporter
Match node_metrics
Host 0.0.0.0
Port 2020
[OUTPUT]
Name prometheus_exporter
Match fluentbit_metrics
Host 0.0.0.0
Port 2020
## https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/upstream-servers
## This configuration is deprecated, please use `extraFiles` instead.
upstream: {}
## https://docs.fluentbit.io/manual/pipeline/parsers
customParsers: |
[PARSER]
Name docker_no_time
Format json
Time_Keep Off
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
# This allows adding more files with arbitrary filenames to /fluent-bit/etc/conf by providing key/value pairs.
# The key becomes the filename, the value becomes the file content.
extraFiles: {}
# upstream.conf: |
# [UPSTREAM]
# upstream1
#
# [NODE]
# name node-1
# host 127.0.0.1
# port 43000
# example.conf: |
# [OUTPUT]
# Name example
# Match foo.*
# Host bar
# The config volume is mounted by default, either to the existingConfigMap value, or the default of "fluent-bit.fullname"
volumeMounts:
- name: config
mountPath: /fluent-bit/etc/conf
daemonSetVolumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: etcmachineid
hostPath:
path: /etc/machine-id
type: File
daemonSetVolumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: etcmachineid
mountPath: /etc/machine-id
readOnly: true
command:
- /fluent-bit/bin/fluent-bit
args:
- --workdir=/fluent-bit/etc
- --config=/fluent-bit/etc/conf/fluent-bit.conf
# This supports either a structured array or a templatable string
initContainers: []
# Array mode
# initContainers:
# - name: do-something
# image: bitnami/kubectl:1.22
# command: ['kubectl', 'version']
# String mode
# initContainers: |-
# - name: do-something
# image: bitnami/kubectl:{{ .Capabilities.KubeVersion.Major }}.{{ .Capabilities.KubeVersion.Minor }}
# command: ['kubectl', 'version']
logLevel: info
hotReload:
enabled: false
image:
repository: ghcr.io/jimmidyson/configmap-reload
tag: v0.14.0
digest:
pullPolicy: IfNotPresent
resources: {}
extraWatchVolumes: []
10.prometheus 설치¶
10.1 prometheus.yaml 생성¶
10.2 prometheus.yaml 배포¶
11.metricbeat 설치¶
11. ETC¶
트러블 슈팅하면서 썼던 명령어들 기록함
elasticsearch master-node의 cluster status 확인
cluster status(green, yellow, red) 보고싶으면 이렇게 보면 됌
adun@yjh-bastion:~/elasticsearch/values$ kubectl exec -it elasticsearch-master-0 -n lma -- curl -u elastic:dkensxhflektm -k "https://localhost:9200/_cluster/health?pretty"
Defaulted container "elasticsearch" out of: elasticsearch, configure-sysctl (init)
{
"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 6,
"number_of_data_nodes" : 3,
"active_primary_shards" : 22,
"active_shards" : 44,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
각 project사용시 elasticsearch의 dns를 넣어서 사용하는데 master-node에 어떤 DNS가 등록되었는지 확인할 수 있음
adun@yjh-bastion:~/ kubectl exec -it elasticsearch-master-0 -n lma -- openssl x509 -in //usr/share/elasticsearch/config/certs/tls.crt -text -noout | grep "DNS:"
Defaulted container "elasticsearch" out of: elasticsearch, configure-sysctl (init)
DNS:elasticsearch-master, DNS:elasticsearch-master.lma, DNS:elasticsearch-master.lma.svc
인증키 가지고 삽질할때 썼던 명령어 secret에서 키 파일형태로 빼낼때 썼음
kubectl get secret elasticsearch-master-certs -n lma -o jsonpath='{.data.tls\.crt}' | base64 -d > tls.crt
kubectl get secret elasticsearch-master-certs -n lma -o jsonpath='{.data.tls\.key}' | base64 -d > tls.key
elasticsearch master pod에서 elasticsearch-master.lma.svc:9200 + ca.crt 값으로 들어오는 log 종류(?) 를 볼 수 있는 명령어인거같음
promemetheus -> elastcisearch 연결할때 metric값이 잘 들어오나 안들어오나 확인할 때 사용함
elasticsearch@elasticsearch-master-0:~$ curl -X GET "https://elasticsearch-master.lma.svc:9200/_cat/indices?v" -u elastic:dkensxhflektm --cacert /usr/share/elasticsearch/config/certs/ca.crt
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .monitoring-es-7-2025.03.14 VIVjgsYaTnaUGqjRTW-rpw 1 1 97861 203100 162.1mb 81mb
green open .monitoring-es-7-2025.03.21 wGGsfHrnRe2qHxVz7tVMWg 1 1 39797 81960 63mb 31.7mb
green open .monitoring-es-7-2025.03.20 wUcSSYQCSQWnwwEUPBxoqw 1 1 233555 138208 300.2mb 149.6mb
green open .monitoring-kibana-7-2025.03.21 7hAg7bEzQiaYPhAMdP542w 1 1 6 0 638.1kb 310.1kb
green open .monitoring-kibana-7-2025.03.20 Y94Q4npiQzWwzYuBCBH6AQ 1 1 17280 0 11.2mb 5.6mb
green open .monitoring-kibana-7-2025.03.16 fPbLWh5PRga0H1o_Gr6lXQ 1 1 6884 0 2.9mb 1.4mb
green open .monitoring-kibana-7-2025.03.15 muH58ORXQkiOlVj8wdiumA 1 1 18640 0 7.8mb 3.9mb
green open .monitoring-kibana-7-2025.03.14 ONxxKAodQhiKER-jSeDolw 1 1 14098 0 6mb 3mb
green open .monitoring-kibana-7-2025.03.19 oVgFuVZlQ0KuxHgebIIDPg 1 1 17276 0 6.8mb 3.3mb
green open fluent-bit C98QByuTTXuUwOF3H-msRw 1 1 1248797 0 1.4gb 925.9mb
green open .monitoring-kibana-7-2025.03.18 zp9H2g1fR3yFeP9_lSALcQ 1 1 17280 0 6.6mb 3.3mb
green open .monitoring-kibana-7-2025.03.17 yYZL_pVpRq2hzObb3ztRdQ 1 1 16136 0 6.7mb 3.3mb
green open .monitoring-es-7-2025.03.19 3iCyQODLSNKUqIh9DqPyWg 1 1 216236 102388 270.2mb 133.6mb
green open .monitoring-es-7-2025.03.16 veDZ0HzoRoS3Q9Q4QTetjg 1 1 59802 125640 97.3mb 48.7mb
green open .monitoring-es-7-2025.03.15 BAFgzKHrRne5Ct7EL3j33A 1 1 142003 300456 231.9mb 115.9mb
green open .monitoring-es-7-2025.03.18 vN4ni3l0R0m2VIpl8l3knw 1 1 198813 66912 247.5mb 123.7mb
green open .monitoring-es-7-2025.03.17 yc10AuvDQEmt0LyPOAwVKw 1 1 171290 6688 201.7mb 99.7mb