【微服务部署】06-日志集成

news/2024/5/20 0:33:43 标签: 微服务, 架构, .netcore, 后端, Exceptionless, EFK

文章目录

EFK_1">1. EFK日志三件套集成

1.1 核心组件

  • Elasticsearch（存储）
  • Fluentd（收集器）
  • Kibana（数据看板）

在这里插入图片描述

每个Kubernetes节点上可以运行一个Fluentd实例，这个实例会去收集所有Pod打印出来的日志流。Fluentd可以做到无侵入的收集日志；
多个Kubernetes节点组成了Kubernetes集群，每个Fluentd收集到的日志都推送到Elasticsearch中。Elasticsearch可以单独部署，也可以部署在Kuberbetes集群中；
Kibana负责管理Elasticsearch，并作为数据看板

1.2 部署

需要部署Elasticsearch、Fluentd、Kibana

Elasticsearch

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: elasticsearch
  namespace: default
  labels:
    tag: elasticsearch
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: "elasticsearch:7.5.2"
        imagePullPolicy: IfNotPresent
        env:
        - name: discovery.type
          value: "single-node"
        volumeMounts:
          - mountPath: "/usr/share/elasticsearch/data"
            name: elasticsearch-storage
        ports:
        - containerPort: 9200
        - containerPort: 9300
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 512Mi
      terminationGracePeriodSeconds: 10
      volumes:
        - name: elasticsearch-storage
          hostPath:
              path: "/d/k8s/volumes/elasticsearch/data"
              type: DirectoryOrCreate
      restartPolicy: Always

--- 

  
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: default
  labels:
    tag: "elasticsearch"
spec:
  type: NodePort
  ports:
  - nodePort: 30007
    port: 9200
    targetPort: 9200
    name: "9200"
    protocol: TCP
  - nodePort: 30008
    port: 9300
    targetPort: 9300
    name: "9300"
    protocol: TCP
  selector:
    app: elasticsearch

Fluentd

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: fluentd
  namespace: default
  labels:
    tag: fluentd
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      containers:
      - name: fluentd
        image: "witskeeper/fluentd-es:68"
        imagePullPolicy: IfNotPresent
        env:
        - name: discovery.type
          value: "single-node"
        volumeMounts:
          - mountPath: "/fluentd/etc/fluent.conf"
            name: fluentd-config
            subPath: fluent.conf
          - mountPath: "/var/log"
            name: containers-logs
          - mountPath: "/var/lib/docker/containers"
            name: docker-logs
        ports:
        - containerPort: 24224
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 512Mi
      terminationGracePeriodSeconds: 10
      volumes:
        - name: fluentd-config
          configMap:
              name: fluentd-config
        - name: containers-logs
          hostPath:
              path: "/var/log"
              type: DirectoryOrCreate
        - name: docker-logs
          hostPath:
            path: /var/lib/docker/containers
      restartPolicy: Always

--- 

  
apiVersion: v1
kind: Service
metadata:
  name: fluentd
  namespace: default
  labels:
    tag: "fluentd"
spec:
  type: NodePort
  ports:
  - nodePort: 30011
    port: 24224
    targetPort: 24224
    protocol: TCP
  selector:
    app: fluentd

Kibana

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: kibana
  namespace: default
  labels:
    tag: kibana
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - name: kibana
        image: "kibana:6.8.6"
        imagePullPolicy: IfNotPresent
        env:
        - name: discovery.type
          value: "single-node"
        - name: XPACK_MONITORING_ENABLED
          value: "true"
        ports:
        - containerPort: 5601
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 512Mi
      terminationGracePeriodSeconds: 10
      restartPolicy: Always

--- 

  
apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: default
  labels:
    tag: "kibana"
spec:
  type: NodePort
  ports:
  - nodePort: 30009
    port: 5601
    targetPort: 5601
    protocol: TCP
  selector:
    app: kibana

Exceptionless_221">2. Exceptionless日志系统

Exceptionless_222">2.1 Exceptionless核心特性

  • 友好的界面
  • 内置日志分类看板
  • 按团队和项目管理应用
  • 支持Web hook发送异常通知
  • 基于.Net Core开源

在这里插入图片描述

Exceptionless由UI、Job、Api三个镜像组成。背后依赖了Redis存储和文件存储；
文件存储是用来存储应用提交的日志文件，然后通过Job将这些文件推送到Elasticsearch中，用户可以通过UI和Api从dashboard上查询异常日志；
应用程序通过Api推送异常日志

Exceptionless_234">2.2 Exceptionless部署文件

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: exceptionless-api
  namespace: default
  labels:
    tag: exceptionless-api
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: exceptionless-api
    spec:
      containers:
      - name: exceptionless-api
        image: "exceptionless/api:6.0.0"
        imagePullPolicy: IfNotPresent
        env:
        - name: EX_AppMode
          value: "Production"
        - name: EX_BaseURL
          value: http://localhost:30013
        - name: EX_ConnectionStrings__Cache
          value: provider=redis
        - name: EX_ConnectionStrings__Elasticsearch
          value: server=http://elasticsearch:9200;enable-size-plugin=false
        #- name: EX_ConnectionStrings__Email
        #  value: smtps://user:password@smtp.host.com:587
        - name: EX_ConnectionStrings__MessageBus
          value: provider=redis
        - name: EX_ConnectionStrings__Queue
          value: provider=redis
        - name: EX_ConnectionStrings__Redis
          value: server=redis,abortConnect=false
        - name: EX_ConnectionStrings__Storage
          value: provider=folder;path=/app/storage
        - name: EX_RunJobsInProcess
          value: 'false'
        volumeMounts:
          - mountPath: "/app/storage"
            name: exceptionless-storage
        ports:
        - containerPort: 80
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 128Mi
      terminationGracePeriodSeconds: 10
      volumes:
        - name: exceptionless-storage
          hostPath:
              path: "/d/k8s/volumes/exceptionless" // 本地存储目录
              type: DirectoryOrCreate
      restartPolicy: Always

---

apiVersion: v1
kind: Service
metadata:
  name: exceptionless-api
  namespace: default
  labels:
    tag: "exceptionless-api"
spec:
  type: NodePort
  ports:
  - nodePort: 30012
    port: 80
    targetPort: 80
    protocol: TCP
  selector:
    app: exceptionless-api

---

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: exceptionless-job
  namespace: default
  labels:
    tag: exceptionless-job
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: exceptionless-job
    spec:
      containers:
      - name: exceptionless-job
        image: "exceptionless/job:6.0.0"
        imagePullPolicy: IfNotPresent
        env:
        - name: EX_AppMode
          value: "Production"
        - name: EX_BaseURL
          value: http://localhost:30013
        - name: EX_ConnectionStrings__Cache
          value: provider=redis
        - name: EX_ConnectionStrings__Elasticsearch
          value: server=http://elasticsearch:9200;enable-size-plugin=false
        #- name: EX_ConnectionStrings__Email
        #  value: smtps://user:password@smtp.host.com:587
        - name: EX_ConnectionStrings__MessageBus
          value: provider=redis
        - name: EX_ConnectionStrings__Queue
          value: provider=redis
        - name: EX_ConnectionStrings__Redis
          value: server=redis,abortConnect=false
        - name: EX_ConnectionStrings__Storage
          value: provider=folder;path=/app/storage
        - name: EX_RunJobsInProcess
          value: 'false'
        volumeMounts:
          - mountPath: "/app/storage"
            name: exceptionless-job-storage
        ports:
        - containerPort: 80
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 128Mi
      terminationGracePeriodSeconds: 10
      volumes:
        - name: exceptionless-job-storage
          hostPath:
              path: "/d/k8s/volumes/exceptionless"
              type: DirectoryOrCreate
      restartPolicy: Always

---

apiVersion: v1
kind: Service
metadata:
  name: exceptionless-job
  namespace: default
  labels:
    tag: "exceptionless-job"
spec:
  type: NodePort
  ports:
  - nodePort: 30014
    port: 80
    targetPort: 80
    protocol: TCP

---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: exceptionless-ui
  namespace: default
  labels:
    tag: exceptionless-ui
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: exceptionless-ui
    spec:
      containers:
      - name: exceptionless-ui
        image: "exceptionless/ui:2.8.1497"
        imagePullPolicy: IfNotPresent
        env:
        - name: AppMode
          value: "Production"
        - name: EX_ApiUrl
          value: http://localhost:30012
        ports:
        - containerPort: 80
        resources:
            limits:
              cpu: 1000m
              memory: 2048Mi
            requests:
              cpu: 100m
              memory: 128Mi
      terminationGracePeriodSeconds: 10
      restartPolicy: Always

---

apiVersion: v1
kind: Service
metadata:
  name: exceptionless-ui
  namespace: default
  labels:
    tag: "exceptionless-ui"
spec:
  type: NodePort
  ports:
  - nodePort: 30013
    port: 80
    targetPort: 80
    protocol: TCP
  selector:
    app: exceptionless-ui

// Program
public static int Main(string[] args)
{
    Log.Logger = new LoggerConfiguration().ReadFrom.Configuration(Configuration)
   .MinimumLevel.Debug()
   .Enrich.FromLogContext()
   .WriteTo.Console(new RenderedCompactJsonFormatter())
   //.WriteTo.Fluentd("localhost", 30011, tag: "geektime-ordering-api", restrictedToMinimumLevel: Serilog.Events.LogEventLevel.Debug)
   .CreateLogger();
   // 将Configuration中的exceptionless读取出来,并绑定到它默认的配置上去
    Configuration.GetSection("exceptionless").Bind(Exceptionless.ExceptionlessClient.Default.Configuration);
    try
    {
        Log.Information("Starting web host");
        CreateHostBuilder(args).Build().Run();
        return 0;
    }
    catch (Exception ex)
    {
        Log.Fatal(ex, "Host terminated unexpectedly");
        return 1;
    }
    finally
    {
        Log.CloseAndFlush();
    }
}


// 配置文件
"exceptionless": {
    "ApiKey": "PSaokqmXC8T4xgWal9I0atA8TlYG7Ytz65JvxAL3",
    "ServerUrl": "http://localhost:30012"
  },

// startup
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
	app.UseExceptionless();// 放在所有中间件最前面.为了能捕获到所有异常
}

Exceptionless_492">2.3 K8s中使用Exceptionless

配置文件

 "exceptionless": {
    "ApiKey": "PSaokqmXC8T4xgWal9I0atA8TlYG7Ytz65JvxAL3" // 需要是当前项目的秘钥
  },

因为serverUrl地址是共享的，地址被配置在env.txt中

要将配置引用进来,需要在deployment.yaml中注册serverUrl为环境变量

通过EFK收集应用的全量日志,Exceptionless收集异常日志


http://www.niftyadmin.cn/n/4985740.html

相关文章

Centos7源码编译安装Redis

Centos7源码编译安装Redis

详情见官网：https://redis.io/docs/getting-started/installation/install-redis-from-source/ 1、下载源码包 wget https://download.redis.io/redis-stable.tar.gz2、解压 tar -xzvf redis-stable.tar.gz cd redis-stable编译 make安装 make install PREFIX/…
阅读更多...
RabbitMQ工作模式-发布订阅模式

RabbitMQ工作模式-发布订阅模式

Publish/Subscribe（发布订阅模式） 官方文档： https://www.rabbitmq.com/tutorials/tutorial-three-python.html 使用fanout类型类型的交换器，routingKey忽略。每个消费者定义生成一个队列关绑定到同一个Exchange，每个…
阅读更多...
浙大陈越何钦铭数据结构07-图6 旅游规划【最小堆实现】

浙大陈越何钦铭数据结构07-图6 旅游规划【最小堆实现】

题目： 题目和浙大陈越何钦铭数据结构07-图6 旅游规划是一样的，不同的是用最小堆实现函数【FindMinDist】。 时间复杂度对比： 浙大陈越何钦铭数据结构07-图6 旅游规划： 创建图（CreateGraph）：时…
阅读更多...
三、原型模式

三、原型模式

一、什么是原型模式 原型（Prototype）模式的定义如下：用一个已经创建的实例作为原型，通过复制该原型对象来创建一个和原型相同或相似的新对象。在这里，原型实例指定了要创建的对象的种类。用这种方式创建对象非常高效&a…
阅读更多...
MES管理系统数据建模有哪些注意事项

MES管理系统数据建模有哪些注意事项

在进行MES管理系统数据建模时，需要注意以下几个方面，以确保建立高效、可靠且适应性强的数据模型，为企业的生产管理提供有力的支持。 首先，精确理解业务需求是进行数据建模的前提。与相关部门和人员进行充分的沟通，了解…
阅读更多...
3A算法——自动曝光

3A算法——自动曝光

目录 前沿一. 自动曝光算法AE1.1. 自动曝光1.2. 18%灰1.3. 测光区域1.4. 摄影曝光加法系统1.5. AE算法1.5.1. 考虑事项1.5.2. AE实现过程 1.4. 自动曝光方法1.4.1. 均值法AE1.4.2. 直方图改进的均值法AE1.4.3. N段式统计自动曝光 二. 自动对焦 前沿 最近在学习3A技术&#xff…
阅读更多...
nvm 安装nodejs

nvm 安装nodejs

1. 下载nvm 地址：Releases coreybutler/nvm-windows GitHub 2. 按要求一步步进行 3. 安装完成后配置nvm 的环境变量 找到nvm文件的路径，选中path，点击编辑讲nvm的路径放进去确定保存即可
阅读更多...
2023_Spark_实验三：基于IDEA开发Scala例子

2023_Spark_实验三：基于IDEA开发Scala例子

一、创建一个空项目&#xff0c;作为整个项目的基本框架 二、创建SparkStudy模块&#xff0c;用于学习基本的Spark基础 三、创建项目结构 1、在SparkStudy模块下的pom.xml文件中加入对应的依赖&#xff0c;并等待依赖包下载完毕。 在pom.xml文件中加入对应的依赖 ​<!-- S…
阅读更多...
最新文章

Copyright @ 2022~2023