{"id":1296,"date":"2022-02-19T16:50:36","date_gmt":"2022-02-19T23:50:36","guid":{"rendered":"https:\/\/bullyrooks.com\/?p=1296"},"modified":"2022-02-21T11:08:33","modified_gmt":"2022-02-21T18:08:33","slug":"kube-cloud-pt4-tracing","status":"publish","type":"post","link":"https:\/\/bullyrooks.com\/index.php\/2022\/02\/19\/kube-cloud-pt4-tracing\/","title":{"rendered":"Kube Cloud Pt4 | Tracing"},"content":{"rendered":"\n

For this portion we’re going to use OpenTelemetry<\/a> for tracing. OpenTelemetry projects intent is to solve all of the observability space in an opensource way. Unfortunately, at this time the logging and metrics portions are still in development, so we won’t be able to use them. However the tracing piece works well. In order to get our opentelementry integration with logz.io working we’ll have to install a service into kubernetes. This service is the tracing collector which the services will send their tracing information to. The collector then pushes the traces over to logz.io. This takes the burden off of the microservices so they can focus on their core functionality.<\/p>\n\n\n\n

Installing the Collector<\/h2>\n\n\n\n

Navigate to the tracing setup in logz.io<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Choose the OT installation<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Select the Kubernetes tab and you should see that they’re using helm to install the service. Very convenient.<\/p>\n\n\n\n

Go ahead and install their repo and get the latest charts<\/p>\n\n\n\n

helm repo add logzio-helm https:\/\/logzio.github.io\/logzio-helm\nhelm repo update<\/code><\/pre>\n\n\n\n
and then install the chart per their instructions<\/p>\n\n\n\n
helm install  \\\n--set<\/strong> config.exporters.logzio.region=us \\\n--set<\/strong> config.exporters.logzio.account_token=... \\\nlogzio-otel-traces logzio-helm\/logzio-otel-traces<\/pre>\n\n\n\n
You should see it startup in okteto and pumping data over (there’s no data to pump yet)<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Install the OpenTelemetry Agent<\/h2>\n\n\n\n
Update your dockerfile to pull down the opentelemetry agent and add it to the java options<\/p>\n\n\n\n
FROM openjdk:15.0.2-slim-buster\nCOPY --from=build target\/*.jar app.jar\nADD https:\/\/github.com\/open-telemetry\/opentelemetry-java-instrumentation\/releases\/download\/v1.11.0\/opentelemetry-javaagent.jar \/opt\/opentelemetry-javaagent.jar\nENV JAVA_TOOL_OPTIONS=-javaagent:\/opt\/opentelemetry-javaagent.jar\nENTRYPOINT [\"java\", \"-jar\", \"app.jar\"]<\/code><\/pre>\n\n\n\n
Update Kubernetes Deployment<\/h2>\n\n\n\n
We also need to update the kubernetes deployment to point the agent to our collector.  Add this environment configuration to the deployment.yaml<\/code> in helm\/cloud-application-templates<\/code><\/p>\n\n\n\n
           env:\n          - name: OTEL_SERVICE_NAME\n            value: {{ include \"cloud-application.fullname\" . }}\n          - name: OTEL_EXPORTER_OTLP_ENDPOINT\n            value: \"http:\/\/logzio-otel-traces:4317\"\n          - name: SPRING_DATA_MONGODB_URI\n            valueFrom:\n              secretKeyRef:\n                name: mongo-secrets\n                key: SPRING_DATA_MONGODB_URI<\/code><\/pre>\n\n\n\n
That should be it.  You can’t test locally (unless you’re deploying into kubernetes locally with something like minikube) so you’ve got to build and deploy into okteto.  Might as well push to main<\/code><\/p>\n\n\n\n
git add .\ngit commit -m \"opentelemetry\"\ngit push<\/code><\/pre>\n\n\n\n
You should be able to find traces in jaeger now<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
You should be able to see the flame graphs that show you the interactions between the services.  Make a request to cloud application without a message so that it calls through to message generator and locate that request in jaeger<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Select that request and you should see the flame graph populate<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
This allows you to see the entire call trace and the duration of the calls.<\/p>\n\n\n\n
Add Tracing Fields to Logs<\/h2>\n\n\n\n
One more thing we can do is add the trace data to the logs.  Although this isn’t necessarily important because logz.io links the logs to the traces, we may want to see this data and it doesn’t take much effort.<\/p>\n\n\n\n
Modify the pattern layout to add the trace and span ids in the logback-spring.xml<\/p>\n\n\n\n
            <Pattern>\n                %black(%d{ISO8601}) %highlight(%-5level) [%blue(%t)] [%cyan(%X{trace_id}),%cyan(%X{span_id})] %yellow(%C{1.}): %msg%n%throwable\n            <\/Pattern><\/code><\/pre>\n\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
\nFor this portion we’re going to use OpenTelemetry for tracing. OpenTelemetry projects…\n<\/div>\n
Continue reading “Kube Cloud Pt4 | Tracing”<\/span>…<\/a><\/div>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1,41],"tags":[181,165,166,182,65],"course":[164],"class_list":["post-1296","post","type-post","status-publish","format-standard","hentry","category-general","category-software-development","tag-jaeger","tag-logz-io","tag-observability","tag-opentelemetry","tag-tracing","course-kube-cloud-4-observability","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":1276,"url":"https:\/\/bullyrooks.com\/index.php\/2022\/02\/19\/kube-cloud-pt4-observability\/","url_meta":{"origin":1296,"position":0},"title":"Kube Cloud Pt4 | Observability","author":"Bullyrook","date":"February 19, 2022","format":false,"excerpt":"In this course, I'm going to walk through how to implement the 3 pillars of observability on our example application using the logz.io platform. Before we do that, I'm going to explain the 3 pillars of observability, which are: LoggingTracingMetrics Logging Logz.io uses ELK for their log management platform. There's\u2026","rel":"","context":"In "Software Development"","block_context":{"text":"Software Development","link":"https:\/\/bullyrooks.com\/index.php\/category\/software-development\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1141,"url":"https:\/\/bullyrooks.com\/index.php\/2022\/01\/02\/kubernetes-application-hosted-in-the-cloud\/","url_meta":{"origin":1296,"position":1},"title":"Kubernetes Application Hosted in the Cloud","author":"Bullyrook","date":"January 2, 2022","format":false,"excerpt":"It's been a few years since my last spring boot based kubernetes application. That course ended with a microservice deployed into kubernetes via minikube locally. I'm using this course to expand on that effort. In this course I'm going to show a few ways to do the same things in\u2026","rel":"","context":"In "Software Development"","block_context":{"text":"Software Development","link":"https:\/\/bullyrooks.com\/index.php\/category\/software-development\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1278,"url":"https:\/\/bullyrooks.com\/index.php\/2022\/02\/19\/kube-cloud-pt4-logging\/","url_meta":{"origin":1296,"position":2},"title":"Kube Cloud Pt4 | Logging","author":"Bullyrook","date":"February 19, 2022","format":false,"excerpt":"Add Structured Logging Structured logging means that we're going to log our output in json format. This is extremely useful because it allows tools to easily parse the json to make the individual log message attributes available for searching and allows you to control the presentation (highlight signal, reduce noise).\u2026","rel":"","context":"In "Software Development"","block_context":{"text":"Software Development","link":"https:\/\/bullyrooks.com\/index.php\/category\/software-development\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/bullyrooks.com\/wp-content\/uploads\/2022\/02\/image-7.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":1356,"url":"https:\/\/bullyrooks.com\/index.php\/2022\/02\/27\/kube-cloud-pt5-create-an-event-consumer\/","url_meta":{"origin":1296,"position":3},"title":"Kube Cloud Pt5 | Create an Event Consumer","author":"Bullyrook","date":"February 27, 2022","format":false,"excerpt":"Now that we've got messages being published to kafka, we are going to need to build our consumer that receives those events and stores them into the mongo database. Go ahead and create a new repository called message-repository according to the microservice startup course here. This is the pom.xml that\u2026","rel":"","context":"In "Software Development"","block_context":{"text":"Software Development","link":"https:\/\/bullyrooks.com\/index.php\/category\/software-development\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/bullyrooks.com\/wp-content\/uploads\/2022\/02\/image-49.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/bullyrooks.com\/wp-content\/uploads\/2022\/02\/image-49.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/bullyrooks.com\/wp-content\/uploads\/2022\/02\/image-49.png?resize=700%2C400&ssl=1 2x"},"classes":[]},{"id":831,"url":"https:\/\/bullyrooks.com\/index.php\/2020\/03\/30\/simple-spring-boot-service-to-kubernetes-application-step-5-a2d164711554\/","url_meta":{"origin":1296,"position":4},"title":"Logging, Tracing and Error Handling","author":"Bullyrook","date":"March 30, 2020","format":false,"excerpt":"Now that we\u2019ve got an almost fully functional webservice we\u2019re going to make sure that we can debug it properly before moving forward to deployment. In this article we\u2019re going to describe how to setup logging. Logging We need to start by actually logging something. Spring boot comes packaged with\u2026","rel":"","context":"In "Software Development"","block_context":{"text":"Software Development","link":"https:\/\/bullyrooks.com\/index.php\/category\/software-development\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1301,"url":"https:\/\/bullyrooks.com\/index.php\/2022\/02\/20\/kube-cloud-pt4-metrics\/","url_meta":{"origin":1296,"position":5},"title":"Kube Cloud Pt4 | Metrics","author":"Bullyrook","date":"February 20, 2022","format":false,"excerpt":"Logz.io uses prometheus for metrics. Go ahead and enable metrics collection in logz.io. Navigate to send your metrics Search for java Choose java custom metrics via micrometer. We're using micrometer because its integrated into spring boot and available via actuator, which we're already using. The important part here is that\u2026","rel":"","context":"In "General"","block_context":{"text":"General","link":"https:\/\/bullyrooks.com\/index.php\/category\/general\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/bullyrooks.com\/wp-content\/uploads\/2022\/02\/image-22.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]}],"_links":{"self":[{"href":"https:\/\/bullyrooks.com\/index.php\/wp-json\/wp\/v2\/posts\/1296","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/bullyrooks.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bullyrooks.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bullyrooks.com\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/bullyrooks.com\/index.php\/wp-json\/wp\/v2\/comments?post=1296"}],"version-history":[{"count":3,"href":"https:\/\/bullyrooks.com\/index.php\/wp-json\/wp\/v2\/posts\/1296\/revisions"}],"predecessor-version":[{"id":1326,"href":"https:\/\/bullyrooks.com\/index.php\/wp-json\/wp\/v2\/posts\/1296\/revisions\/1326"}],"wp:attachment":[{"href":"https:\/\/bullyrooks.com\/index.php\/wp-json\/wp\/v2\/media?parent=1296"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bullyrooks.com\/index.php\/wp-json\/wp\/v2\/categories?post=1296"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bullyrooks.com\/index.php\/wp-json\/wp\/v2\/tags?post=1296"},{"taxonomy":"course","embeddable":true,"href":"https:\/\/bullyrooks.com\/index.php\/wp-json\/wp\/v2\/course?post=1296"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}