【微服务部署】06-日志集成

1. EFK日志三件套集成

1.1 核心组件

• Elasticsearch（存储）
• Fluentd（收集器）
• Kibana（数据看板）

每个Kubernetes节点上可以运行一个Fluentd实例，这个实例会去收集所有Pod打印出来的日志流。Fluentd可以做到无侵入的收集日志；
多个Kubernetes节点组成了Kubernetes集群，每个Fluentd收集到的日志都推送到Elasticsearch中。Elasticsearch可以单独部署，也可以部署在Kuberbetes集群中；
Kibana负责管理Elasticsearch，并作为数据看板

1.2 部署

需要部署Elasticsearch、Fluentd、Kibana

Elasticsearch

apiVersion: apps/v1beta1
kind: Deployment
metadata:name: elasticsearchnamespace: defaultlabels:tag: elasticsearch
spec:replicas: 1template:metadata:labels:app: elasticsearchspec:containers:- name: elasticsearchimage: "elasticsearch:7.5.2"imagePullPolicy: IfNotPresentenv:- name: discovery.typevalue: "single-node"volumeMounts:- mountPath: "/usr/share/elasticsearch/data"name: elasticsearch-storageports:- containerPort: 9200- containerPort: 9300resources:limits:cpu: 1000mmemory: 2048Mirequests:cpu: 100mmemory: 512MiterminationGracePeriodSeconds: 10volumes:- name: elasticsearch-storagehostPath:path: "/d/k8s/volumes/elasticsearch/data"type: DirectoryOrCreaterestartPolicy: Always--- apiVersion: v1
kind: Service
metadata:name: elasticsearchnamespace: defaultlabels:tag: "elasticsearch"
spec:type: NodePortports:- nodePort: 30007port: 9200targetPort: 9200name: "9200"protocol: TCP- nodePort: 30008port: 9300targetPort: 9300name: "9300"protocol: TCPselector:app: elasticsearch

Fluentd

apiVersion: apps/v1beta1
kind: Deployment
metadata:name: fluentdnamespace: defaultlabels:tag: fluentd
spec:replicas: 1template:metadata:labels:app: fluentdspec:containers:- name: fluentdimage: "witskeeper/fluentd-es:68"imagePullPolicy: IfNotPresentenv:- name: discovery.typevalue: "single-node"volumeMounts:- mountPath: "/fluentd/etc/fluent.conf"name: fluentd-configsubPath: fluent.conf- mountPath: "/var/log"name: containers-logs- mountPath: "/var/lib/docker/containers"name: docker-logsports:- containerPort: 24224resources:limits:cpu: 1000mmemory: 2048Mirequests:cpu: 100mmemory: 512MiterminationGracePeriodSeconds: 10volumes:- name: fluentd-configconfigMap:name: fluentd-config- name: containers-logshostPath:path: "/var/log"type: DirectoryOrCreate- name: docker-logshostPath:path: /var/lib/docker/containersrestartPolicy: Always--- apiVersion: v1
kind: Service
metadata:name: fluentdnamespace: defaultlabels:tag: "fluentd"
spec:type: NodePortports:- nodePort: 30011port: 24224targetPort: 24224protocol: TCPselector:app: fluentd

Kibana

apiVersion: apps/v1beta1
kind: Deployment
metadata:name: kibananamespace: defaultlabels:tag: kibana
spec:replicas: 1template:metadata:labels:app: kibanaspec:containers:- name: kibanaimage: "kibana:6.8.6"imagePullPolicy: IfNotPresentenv:- name: discovery.typevalue: "single-node"- name: XPACK_MONITORING_ENABLEDvalue: "true"ports:- containerPort: 5601resources:limits:cpu: 1000mmemory: 2048Mirequests:cpu: 100mmemory: 512MiterminationGracePeriodSeconds: 10restartPolicy: Always--- apiVersion: v1
kind: Service
metadata:name: kibananamespace: defaultlabels:tag: "kibana"
spec:type: NodePortports:- nodePort: 30009port: 5601targetPort: 5601protocol: TCPselector:app: kibana

2. Exceptionless日志系统

2.1 Exceptionless核心特性

• 友好的界面
• 内置日志分类看板
• 按团队和项目管理应用
• 支持Web hook发送异常通知
• 基于.Net Core开源

Exceptionless由UI、Job、Api三个镜像组成。背后依赖了Redis存储和文件存储；
文件存储是用来存储应用提交的日志文件，然后通过Job将这些文件推送到Elasticsearch中，用户可以通过UI和Api从dashboard上查询异常日志；
应用程序通过Api推送异常日志

2.2 Exceptionless部署文件

apiVersion: apps/v1beta1
kind: Deployment
metadata:name: exceptionless-apinamespace: defaultlabels:tag: exceptionless-api
spec:replicas: 1template:metadata:labels:app: exceptionless-apispec:containers:- name: exceptionless-apiimage: "exceptionless/api:6.0.0"imagePullPolicy: IfNotPresentenv:- name: EX_AppModevalue: "Production"- name: EX_BaseURLvalue: http://localhost:30013- name: EX_ConnectionStrings__Cachevalue: provider=redis- name: EX_ConnectionStrings__Elasticsearchvalue: server=http://elasticsearch:9200;enable-size-plugin=false#- name: EX_ConnectionStrings__Email#  value: smtps://user:password@smtp.host.com:587- name: EX_ConnectionStrings__MessageBusvalue: provider=redis- name: EX_ConnectionStrings__Queuevalue: provider=redis- name: EX_ConnectionStrings__Redisvalue: server=redis,abortConnect=false- name: EX_ConnectionStrings__Storagevalue: provider=folder;path=/app/storage- name: EX_RunJobsInProcessvalue: 'false'volumeMounts:- mountPath: "/app/storage"name: exceptionless-storageports:- containerPort: 80resources:limits:cpu: 1000mmemory: 2048Mirequests:cpu: 100mmemory: 128MiterminationGracePeriodSeconds: 10volumes:- name: exceptionless-storagehostPath:path: "/d/k8s/volumes/exceptionless" // 本地存储目录type: DirectoryOrCreaterestartPolicy: Always---apiVersion: v1
kind: Service
metadata:name: exceptionless-apinamespace: defaultlabels:tag: "exceptionless-api"
spec:type: NodePortports:- nodePort: 30012port: 80targetPort: 80protocol: TCPselector:app: exceptionless-api---apiVersion: apps/v1beta1
kind: Deployment
metadata:name: exceptionless-jobnamespace: defaultlabels:tag: exceptionless-job
spec:replicas: 1template:metadata:labels:app: exceptionless-jobspec:containers:- name: exceptionless-jobimage: "exceptionless/job:6.0.0"imagePullPolicy: IfNotPresentenv:- name: EX_AppModevalue: "Production"- name: EX_BaseURLvalue: http://localhost:30013- name: EX_ConnectionStrings__Cachevalue: provider=redis- name: EX_ConnectionStrings__Elasticsearchvalue: server=http://elasticsearch:9200;enable-size-plugin=false#- name: EX_ConnectionStrings__Email#  value: smtps://user:password@smtp.host.com:587- name: EX_ConnectionStrings__MessageBusvalue: provider=redis- name: EX_ConnectionStrings__Queuevalue: provider=redis- name: EX_ConnectionStrings__Redisvalue: server=redis,abortConnect=false- name: EX_ConnectionStrings__Storagevalue: provider=folder;path=/app/storage- name: EX_RunJobsInProcessvalue: 'false'volumeMounts:- mountPath: "/app/storage"name: exceptionless-job-storageports:- containerPort: 80resources:limits:cpu: 1000mmemory: 2048Mirequests:cpu: 100mmemory: 128MiterminationGracePeriodSeconds: 10volumes:- name: exceptionless-job-storagehostPath:path: "/d/k8s/volumes/exceptionless"type: DirectoryOrCreaterestartPolicy: Always---apiVersion: v1
kind: Service
metadata:name: exceptionless-jobnamespace: defaultlabels:tag: "exceptionless-job"
spec:type: NodePortports:- nodePort: 30014port: 80targetPort: 80protocol: TCP---
apiVersion: apps/v1beta1
kind: Deployment
metadata:name: exceptionless-uinamespace: defaultlabels:tag: exceptionless-ui
spec:replicas: 1template:metadata:labels:app: exceptionless-uispec:containers:- name: exceptionless-uiimage: "exceptionless/ui:2.8.1497"imagePullPolicy: IfNotPresentenv:- name: AppModevalue: "Production"- name: EX_ApiUrlvalue: http://localhost:30012ports:- containerPort: 80resources:limits:cpu: 1000mmemory: 2048Mirequests:cpu: 100mmemory: 128MiterminationGracePeriodSeconds: 10restartPolicy: Always---apiVersion: v1
kind: Service
metadata:name: exceptionless-uinamespace: defaultlabels:tag: "exceptionless-ui"
spec:type: NodePortports:- nodePort: 30013port: 80targetPort: 80protocol: TCPselector:app: exceptionless-ui

// Program
public static int Main(string[] args)
{Log.Logger = new LoggerConfiguration().ReadFrom.Configuration(Configuration).MinimumLevel.Debug().Enrich.FromLogContext().WriteTo.Console(new RenderedCompactJsonFormatter())//.WriteTo.Fluentd("localhost", 30011, tag: "geektime-ordering-api", restrictedToMinimumLevel: Serilog.Events.LogEventLevel.Debug).CreateLogger();// 将Configuration中的exceptionless读取出来,并绑定到它默认的配置上去Configuration.GetSection("exceptionless").Bind(Exceptionless.ExceptionlessClient.Default.Configuration);try{Log.Information("Starting web host");CreateHostBuilder(args).Build().Run();return 0;}catch (Exception ex){Log.Fatal(ex, "Host terminated unexpectedly");return 1;}finally{Log.CloseAndFlush();}
}// 配置文件
"exceptionless": {"ApiKey": "PSaokqmXC8T4xgWal9I0atA8TlYG7Ytz65JvxAL3","ServerUrl": "http://localhost:30012"},// startup
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{app.UseExceptionless();// 放在所有中间件最前面.为了能捕获到所有异常
}

2.3 K8s中使用Exceptionless

配置文件

 "exceptionless": {"ApiKey": "PSaokqmXC8T4xgWal9I0atA8TlYG7Ytz65JvxAL3" // 需要是当前项目的秘钥},

因为serverUrl地址是共享的，地址被配置在env.txt中

要将配置引用进来,需要在deployment.yaml中注册serverUrl为环境变量

通过EFK收集应用的全量日志,Exceptionless收集异常日志