Kubernetes运维-Skywalking微服务链路追踪 {#CrawlerTitle}

王先森 2024-01-26 2024-01-26

概述 {#概述}

前面我给大家分享了关于分布式链路追踪的基本原理和 SkyWalking 的k8s部署玩法，如果还没来得及看的朋友可以看我上俩篇文章。

今天要给大家分享是我们日常工作中最常见的一种场景，那就是部署在k8s环境下的 Java微服务 ，要接入 SkyWalking 的具体玩法，通过这个过程咱们可以更深入的理解 SkyWalking 进行数据采集的逻辑，也能更深刻地从运维角度理解日常工作中所写的Java微服务被无侵入的方式接入分布式链路追踪系统的过程！

Java微服务接入SkyWalking的方式 {#Java微服务接入SkyWalking的方式}

SkyWalking 的数据采集主要是通过**业务探针(Agent)**来实现的，针对不同的编程语言 SkyWalking 提供了对应的 Agent 实现。Java微服务接入SkyWalking可以使用 SkyWalking Java Agent 来上报监控数据。

这就需要Java微服务在部署启动的过程中需要获取 SkyWalking Java Agent 探针包，并在启动参数中通过 --javaagent:xxx 进行参数指定。而具体的集成方式大致有以下三种：

使用官方提供的基础镜像；
将agent包构建到已存在的基础镜像中；
通过 sidecar 模式挂载agent；

其中前两种方式主要是通过在构建 Docker镜像 的过程中将Agent依赖打包集成到Java服务的Docker镜像中，而 sidecar 模式则是利用k8s的相关特性来实现在 容器启动时挂载Agent 相关依赖。

如果微服务是直接部署在Kubernetes集群，那么采用sidecar模式来使用SkyWalking Agent会更加方便，因为这种方式 不需要修改原来的基础镜像，也不需要重新构建新的服务镜像 ，而是会以sidecar模式，通过共享的volume将agent所需的相关文件直接挂载到已经存在的服务镜像中。

构建SkyWalking-Agent镜像 {#构建SkyWalking-Agent镜像}

在开始以sidecar方式，将一个用Spring Cloud框架编写的Java微服务接入SkyWalking之前，我们需要构建 SkyWalking Java Agent 的公共镜像，具体步骤如下：

|---------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | 1 2 3 4 5 6 | FROM busybox:latest LABEL www.boysec.cn wangxiansen ENV LANG=C.UTF- 8 # 通过网络下载解压 RUN wget -q https://dlcdn.apache.org/skywalking/java-agent/9.1.0/apache-skywalking-java-agent-9.1.0.tgz -O -|tar -xz -C /opt/ |

在上述Dockefile文件中使用是的 bosybox 镜像，而不是SkyWalking的发行镜像，这样可以确保构建出来的sidecar镜像保持最小。

完成Docker文件编写后，执行镜像构建命令

|-------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | 1 2 3 4 5 | # 执行镜像构建命令 $ docker build . -t wangxiansen/skywalking-agent:sidecar-9.1.0 # 为了验证构建的镜像是否成功，可以通过命令查看本地构建的镜像 $ docker images|grep sidecar wangxiansen/skywalking-agent sidecar-9.1.0 f4b8b30bd947 4 hours ago 38.7MB |

我已经将打包好的镜像推送至 hub.docker.com 仓库中，如果需要可直接下载使用。

微服务接入SkyWalking服务 {#微服务接入SkyWalking服务}

上面我们通过手工构建的方式构建了SkyWalking Java Agent的公共Docker镜像，并将其Push到了镜像仓库，接下来我们将演示如何通过编写Kubernetes服务发布文件，来将Java服务发布到K8s集群的过程中自动以SideCar的形式集成Agent、并接入SkyWalking服务。

本次演示还是使用的 Kubernetes之交付dubbo微服务这篇文章内容，这里需要你安装 java8+ 和 maven 环境。

编译dubbo-demo-service {#编译dubbo-demo-service}

|-------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | 1 2 3 4 5 | git clone https://gitee.com/dabou/dubbo-demo-service.git cd dubbo-demo-service mvn clean package -Dmaven.test.skip=true mkdir project_dir mv /dubbo-demo-service/dubbo-server/target/dubbo-server.jar dubbo-demo-service/project_dir |

创建Dockefile

|---------------|--------------------------------------------------------------------------------------------| | 1 2 3 | cat Dockerfile FROM wangxiansen/base_jre8: 8 u112 ADD project_dir /opt/project_dir |

编译dubbo-demo-web {#编译dubbo-demo-web}

|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | 1 2 3 4 5 | git clone https://gitee.com/dabou/dubbo-demo-web.git cd dubbo-demo-web mvn clean package -Dmaven.test.skip=true mkdir project_dir mv /dubbo-demo-web/dubbo-client/target/dubbo-client.jar dubbo-demo-web/project_dir |

创建Dockefile

微服务发布进行埋点 {#微服务发布进行埋点}

到这里你并没有发现为了将Java服务接入SkyWalking，你需要在Java微服务本身做任何动作，而接下来在k8s部署文件中的将演示，为什么要将这种方式称之为 SideCar。

其主要原理是通过Kubernetes的初始化容器 initContainers 来实现的，initContainers是一种专用容器，可以在应用容器启动之前运行，可以用于完成应用启动前的必要初始化工作。

编辑 dubbo-demo-service 资源配置清单

|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 | # vim dubbo-demo-service.yaml kind: Deployment apiVersion: apps/v1 metadata: name: dubbo-demo-service namespace: app labels: name: dubbo-demo-service spec: replicas: 1 selector: matchLabels: name: dubbo-demo-service template: metadata: labels: app: dubbo-demo-service name: dubbo-demo-service spec: volumes: # 创建公共挂载点 - name: sw-agent emptyDir: {} initContainers: # 构建初始化镜像 (通过初始化镜像的方式集成SkyWalking Agent) - name: sw-agent-sidecar image: wangxiansen/skywalking-agent:sidecar-9.1.0 imagePullPolicy: IfNotPresent command: - sh - -c - >- mkdir -p /skywalking/agent; # 创建挂载目录 cp -r /opt/skywalking-agent/* /skywalking/agent # 复制skywalking-agent目录内容 volumeMounts: - name: sw-agent mountPath: "/skywalking/agent" containers: - name: dubbo-demo-service image: wangxiansen/dubbo-demo-consumer:test imagePullPolicy: IfNotPresent ports: - containerPort: 20880 protocol: TCP env: #这里通过JAR_BALL，而不是JAVA_OPTS可以实现不通过将agent命令加入到java应用jvm参数而实现agent的集成 - name: JAR_BALL value: "-javaagent:/skywalking/agent/skywalking-agent.jar dubbo-server.jar" - name: ZK_ADDRES # 这个环境依赖zookeeper value: "zk.od.com:2181" - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES # OAP 地址 value: "oap-svc.skywalking:11800" - name: SW_AGENT_NAME # 监控服务名 value: "dubbo" resources: limits: cpu: 300m memory: 400Mi requests: cpu: 200m memory: 400Mi volumeMounts: - name: sw-agent mountPath: "/skywalking/agent" restartPolicy: Always terminationGracePeriodSeconds: 30 securityContext: runAsUser: 0 schedulerName: default-scheduler strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 revisionHistoryLimit: 7 progressDeadlineSeconds: 600 |

编辑 dubbo-demo-web 资源配置清单

|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 | # vim dubbo-demo-web.yaml kind: Deployment apiVersion: apps/v1 metadata: name: dubbo-demo-consumer namespace: app labels: name: dubbo-demo-consumer spec: replicas: 1 selector: matchLabels: name: dubbo-demo-consumer template: metadata: labels: app: dubbo-demo-consumer name: dubbo-demo-consumer spec: volumes: - name: sw-agent emptyDir: {} initContainers: - name: sw-agent-sidecar image: wangxiansen/skywalking-agent:sidecar-9.1.0 imagePullPolicy: IfNotPresent command: - sh - -c - >- mkdir -p /skywalking/agent; cp -r /opt/skywalking-agent/* /skywalking/agent volumeMounts: - name: sw-agent mountPath: "/skywalking/agent" containers: - name: dubbo-demo-consumer image: wangxiansen/dubbo-demo-service:test imagePullPolicy: IfNotPresent ports: - containerPort: 8080 protocol: TCP - containerPort: 20880 protocol: TCP env: - name: JAR_BALL value: "-javaagent:/skywalking/agent/skywalking-agent.jar dubbo-client.jar" - name: ZK_ADDRES value: "zk.od.com:2181" - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES value: "oap-svc.skywalking:11800" - name: SW_AGENT_NAME value: "dubbo" resources: limits: cpu: 400m memory: 500Mi requests: cpu: 300m memory: 500Mi volumeMounts: - name: sw-agent mountPath: "/skywalking/agent" restartPolicy: Always terminationGracePeriodSeconds: 30 securityContext: runAsUser: 0 schedulerName: default-scheduler strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 revisionHistoryLimit: 7 progressDeadlineSeconds: 600 --- kind: Service apiVersion: v1 metadata: name: dubbo-demo-consumer namespace: app spec: ports: - protocol: TCP port: 8080 targetPort: 8080 selector: app: dubbo-demo-consumer clusterIP: None type: ClusterIP sessionAffinity: None --- apiVersion: traefik.containo.us/v1alpha1 kind: IngressRoute metadata: name: dubbo-demo-consumer namespace: app spec: entryPoints: - web routes: - match: Host(`dome.od.com`) kind: Rule services: - name: dubbo-demo-consumer port: 8080 |

以上是挂载sidecar的k8s发布文件，以微服务 dubbo 为例，主要是通过共享 volume 的方式挂载agent。其中 initContainers 通过 sw-agent 卷挂载了 skywalking-agent 镜像中的 /skywalking/agent ，并将上面构建好的镜像中的 agent 目录 cp 到 /skywalking/agent 目录，完成之后微服务容器启动时也挂载了skywalking-agent卷，并将其挂载到容器的 /skywalking/agent 目录，这样就完成了共享过程。

部署启动微服务,并验证其是否已经正常接入SkyWalking监控

|-------------|---------------------------------------------------------------------------------------| | 1 2 | kubectl apply -f dubbo-demo-service.yaml kubectl apply -f dubbo-demo-web.yaml |

查看相关Pod是否运行成功：

|-----------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | 1 2 3 4 | $ kubectl get pods -n app NAME READY STATUS RESTARTS AGE dubbo-demo-consumer-647855d4f6-v7j64 1/1 Running 0 14m dubbo-demo-service-6fd4f5685c-n2jbx 1/1 Running 0 14m |

运行成功了！此时可以访问下服务的测试接口，多刷几次，也可以制作一些错误，之后通过SkyWalking UI查看是否有监控数据。

dubbo

SkyWalking-ui

requests

如上图所示，在访问微服务测试接口后可以看到SpringCloud微服务已经通过Agent像SkyWalking上报了APM监控数据！