{"id":1276,"date":"2022-02-19T11:14:33","date_gmt":"2022-02-19T18:14:33","guid":{"rendered":"https:\/\/bullyrooks.com\/?p=1276"},"modified":"2022-02-21T11:09:10","modified_gmt":"2022-02-21T18:09:10","slug":"kube-cloud-pt4-observability","status":"publish","type":"post","link":"https:\/\/bullyrooks.com\/index.php\/2022\/02\/19\/kube-cloud-pt4-observability\/","title":{"rendered":"Kube Cloud Pt4 | Observability"},"content":{"rendered":"\n<p>In this course, I&#8217;m going to walk through how to implement the 3 pillars of observability on our example application using the <a href=\"https:\/\/logz.io\/\">logz.io<\/a> platform.  Before we do that, I&#8217;m going to explain the 3 pillars of observability, which are:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Logging<\/li><li>Tracing<\/li><li>Metrics<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"logging\">Logging<\/h2>\n\n\n\n<p><a href=\"https:\/\/logz.io\/platform\/log-management\/#collect\">Logz.io uses ELK for their log management platform<\/a>.  There&#8217;s not much more to say here that logz.io won&#8217;t say better and in more detail.  But I will touch on my philosophy here:<\/p>\n\n\n\n<p>Logging is the foundation of your application observability though and it should be very familiar to most developers since your logs are usually immediately visible in your console and you&#8217;re using them to see if your application is behaving the way you expect and debug if it isn&#8217;t.  You can do so much from the logs though so they are very important.  For example, you can scrape together a makeshift tracing if you need to (but tracing tools are better), you can parse logs to generate metrics if you need to (but metrics frameworks are better).  You face a balancing act with logs:  you can&#8217;t see what isn&#8217;t logged, so you almost should log everything possible, but storing logs over time can be expensive and sifting through noisy logs is tedious.  My recommendation is to log so much that you feel like you might overdo it with the logs and use log level configuration to hide what isn&#8217;t needed.<\/p>\n\n\n\n<p>Ideally, you should log<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Every input to every method<\/li><li>Every output from every method<\/li><li>Every decision outcome<\/li><li>Every exception<ul><li>Including as much information as you can about the inputs that lead to the exception<\/li><li>The full stacktrace<\/li><\/ul><\/li><\/ul>\n\n\n\n<p>This is a lot, but again, if you have the logs you can bootstrap another observability solution as needed.  If you don&#8217;t have the logs, you&#8217;re pretty much guessing.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"tracing\">Tracing<\/h2>\n\n\n\n<p>Tracing is often overlooked but is a <strong>critical<\/strong> component to debugging a cloud based distributed application.  You <strong>must<\/strong> trace every request across all of your services (IMHO).  Your tracing has to propagate across REST calls, message consumers, stateless functions, etc&#8230; so that you have full visibility of the calls through your system.<\/p>\n\n\n\n<p>Tracing not only allows you to debug, but it can be very useful in performance analysis.  Tracing can show you where slow calls are being made, where unnecessary or duplicate calls are being made and missing traces can show you where your observability picture is not complete.<\/p>\n\n\n\n<p><a href=\"https:\/\/logz.io\/platform\/distributed-tracing\/\">Logz.io uses jaeger for their tracing solution<\/a>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"metrics\">Metrics<\/h2>\n\n\n\n<p>Metrics is often misunderstood, but <a href=\"https:\/\/www.datadoghq.com\/blog\/monitoring-101-collecting-data\/\">datadog has an excellent series of articles on monitoring<\/a> and they clearly explain how metrics are a key piece of any observability platform.<\/p>\n\n\n\n<p>That being said, engineers often focus on the &#8216;resource&#8217; metrics: errors, latency, throughput, etc&#8230; and while these are very important, the &#8216;work&#8217; (or business metrics) are just as important to tell you how your system is actually doing the work it was built to do.  Because they take some extra work to implement and careful thought about what is important to collect, they can be neglected. <\/p>\n\n\n\n<p><a href=\"https:\/\/logz.io\/platform\/distributed-tracing\/\">Logz.io uses prometheus and grafana for their metrics solution<\/a>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"conclusion\">Conclusion<\/h2>\n\n\n\n<p>We&#8217;re going to implement the 3 pillars of observability in our application so that you know their value and how to leverage them in your projects.  Let&#8217;s get started.<\/p>\n","protected":false},"excerpt":{"rendered":"<div class=\"entry-summary\">\nIn this course, I&#8217;m going to walk through how to implement the&hellip;\n<\/div>\n<div class=\"link-more\"><a href=\"https:\/\/bullyrooks.com\/index.php\/2022\/02\/19\/kube-cloud-pt4-observability\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &ldquo;Kube Cloud Pt4 | Observability&rdquo;<\/span>&hellip;<\/a><\/div>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[41],"tags":[135,168,169,61,165,167,55,166,65],"course":[164],"class_list":["post-1276","post","type-post","status-publish","format-standard","hentry","category-software-development","tag-cloud","tag-datadog","tag-distributed-computing","tag-logging","tag-logz-io","tag-metrics","tag-microservice","tag-observability","tag-tracing","course-kube-cloud-4-observability","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":1296,"url":"https:\/\/bullyrooks.com\/index.php\/2022\/02\/19\/kube-cloud-pt4-tracing\/","url_meta":{"origin":1276,"position":0},"title":"Kube Cloud Pt4 | Tracing","author":"Bullyrook","date":"February 19, 2022","format":false,"excerpt":"For this portion we're going to use OpenTelemetry for tracing. OpenTelemetry projects intent is to solve all of the observability space in an opensource way. Unfortunately, at this time the logging and metrics portions are still in development, so we won't be able to use them. However the tracing piece\u2026","rel":"","context":"In &quot;General&quot;","block_context":{"text":"General","link":"https:\/\/bullyrooks.com\/index.php\/category\/general\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/bullyrooks.com\/wp-content\/uploads\/2022\/02\/image-35.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/bullyrooks.com\/wp-content\/uploads\/2022\/02\/image-35.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/bullyrooks.com\/wp-content\/uploads\/2022\/02\/image-35.png?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/bullyrooks.com\/wp-content\/uploads\/2022\/02\/image-35.png?resize=1050%2C600&ssl=1 3x, https:\/\/i0.wp.com\/bullyrooks.com\/wp-content\/uploads\/2022\/02\/image-35.png?resize=1400%2C800&ssl=1 4x"},"classes":[]},{"id":1278,"url":"https:\/\/bullyrooks.com\/index.php\/2022\/02\/19\/kube-cloud-pt4-logging\/","url_meta":{"origin":1276,"position":1},"title":"Kube Cloud Pt4 | Logging","author":"Bullyrook","date":"February 19, 2022","format":false,"excerpt":"Add Structured Logging Structured logging means that we're going to log our output in json format. This is extremely useful because it allows tools to easily parse the json to make the individual log message attributes available for searching and allows you to control the presentation (highlight signal, reduce noise).\u2026","rel":"","context":"In &quot;Software Development&quot;","block_context":{"text":"Software Development","link":"https:\/\/bullyrooks.com\/index.php\/category\/software-development\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/bullyrooks.com\/wp-content\/uploads\/2022\/02\/image-7.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":1301,"url":"https:\/\/bullyrooks.com\/index.php\/2022\/02\/20\/kube-cloud-pt4-metrics\/","url_meta":{"origin":1276,"position":2},"title":"Kube Cloud Pt4 | Metrics","author":"Bullyrook","date":"February 20, 2022","format":false,"excerpt":"Logz.io uses prometheus for metrics. Go ahead and enable metrics collection in logz.io. Navigate to send your metrics Search for java Choose java custom metrics via micrometer. We're using micrometer because its integrated into spring boot and available via actuator, which we're already using. The important part here is that\u2026","rel":"","context":"In &quot;General&quot;","block_context":{"text":"General","link":"https:\/\/bullyrooks.com\/index.php\/category\/general\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/bullyrooks.com\/wp-content\/uploads\/2022\/02\/image-22.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":1356,"url":"https:\/\/bullyrooks.com\/index.php\/2022\/02\/27\/kube-cloud-pt5-create-an-event-consumer\/","url_meta":{"origin":1276,"position":3},"title":"Kube Cloud Pt5 | Create an Event Consumer","author":"Bullyrook","date":"February 27, 2022","format":false,"excerpt":"Now that we've got messages being published to kafka, we are going to need to build our consumer that receives those events and stores them into the mongo database. Go ahead and create a new repository called message-repository according to the microservice startup course here. This is the pom.xml that\u2026","rel":"","context":"In &quot;Software Development&quot;","block_context":{"text":"Software Development","link":"https:\/\/bullyrooks.com\/index.php\/category\/software-development\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/bullyrooks.com\/wp-content\/uploads\/2022\/02\/image-49.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/bullyrooks.com\/wp-content\/uploads\/2022\/02\/image-49.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/bullyrooks.com\/wp-content\/uploads\/2022\/02\/image-49.png?resize=700%2C400&ssl=1 2x"},"classes":[]},{"id":1333,"url":"https:\/\/bullyrooks.com\/index.php\/2022\/02\/27\/kube-cloud-pt5-create-a-consumer-service\/","url_meta":{"origin":1276,"position":4},"title":"Kube Cloud Pt5 | Create a Consumer Service","author":"Bullyrook","date":"February 27, 2022","format":false,"excerpt":"In this course we're going to migrate the repository storage function out of cloud-application and migrate it into a new service called message-repository. Although this is a relatively minor change in behavior (in exchange for a significant effort), there are some real world benefits that this example will provide cloud-application\u2026","rel":"","context":"In &quot;Software Development&quot;","block_context":{"text":"Software Development","link":"https:\/\/bullyrooks.com\/index.php\/category\/software-development\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/bullyrooks.com\/wp-content\/uploads\/2022\/02\/pg1.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/bullyrooks.com\/wp-content\/uploads\/2022\/02\/pg1.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/bullyrooks.com\/wp-content\/uploads\/2022\/02\/pg1.png?resize=700%2C400&ssl=1 2x"},"classes":[]},{"id":1395,"url":"https:\/\/bullyrooks.com\/index.php\/2022\/03\/12\/kube-cloud-pt6-update-to-consumer-contract\/","url_meta":{"origin":1276,"position":5},"title":"Kube Cloud Pt6 | Break the Contract from a Consumer Change","author":"Bullyrook","date":"March 12, 2022","format":false,"excerpt":"We're going to go back to cloud-application at this point and update our client to expect a field that isn't being currently provided (message generated date). This should break our contract test and prevent us from deploying, but there's some updates we need to make to support this. Add the\u2026","rel":"","context":"In &quot;Software Development&quot;","block_context":{"text":"Software Development","link":"https:\/\/bullyrooks.com\/index.php\/category\/software-development\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/bullyrooks.com\/wp-content\/uploads\/2022\/03\/image-25.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/bullyrooks.com\/wp-content\/uploads\/2022\/03\/image-25.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/bullyrooks.com\/wp-content\/uploads\/2022\/03\/image-25.png?resize=700%2C400&ssl=1 2x"},"classes":[]}],"_links":{"self":[{"href":"https:\/\/bullyrooks.com\/index.php\/wp-json\/wp\/v2\/posts\/1276","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/bullyrooks.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bullyrooks.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bullyrooks.com\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/bullyrooks.com\/index.php\/wp-json\/wp\/v2\/comments?post=1276"}],"version-history":[{"count":2,"href":"https:\/\/bullyrooks.com\/index.php\/wp-json\/wp\/v2\/posts\/1276\/revisions"}],"predecessor-version":[{"id":1330,"href":"https:\/\/bullyrooks.com\/index.php\/wp-json\/wp\/v2\/posts\/1276\/revisions\/1330"}],"wp:attachment":[{"href":"https:\/\/bullyrooks.com\/index.php\/wp-json\/wp\/v2\/media?parent=1276"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bullyrooks.com\/index.php\/wp-json\/wp\/v2\/categories?post=1276"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bullyrooks.com\/index.php\/wp-json\/wp\/v2\/tags?post=1276"},{"taxonomy":"course","embeddable":true,"href":"https:\/\/bullyrooks.com\/index.php\/wp-json\/wp\/v2\/course?post=1276"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}