{"id":1301,"date":"2022-02-20T16:23:39","date_gmt":"2022-02-20T23:23:39","guid":{"rendered":"https:\/\/bullyrooks.com\/?p=1301"},"modified":"2022-02-21T10:36:31","modified_gmt":"2022-02-21T17:36:31","slug":"kube-cloud-pt4-metrics","status":"publish","type":"post","link":"https:\/\/bullyrooks.com\/index.php\/2022\/02\/20\/kube-cloud-pt4-metrics\/","title":{"rendered":"Kube Cloud Pt4 | Metrics"},"content":{"rendered":"\n

Logz.io uses prometheus for metrics<\/a>. Go ahead and enable metrics collection in logz.io. Navigate to send your metrics
<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Search for java<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Choose java custom metrics via micrometer. We’re using micrometer because its integrated into spring boot and available via actuator, which we’re already using.<\/p>\n\n\n\n

The important part here is that you grab the metrics token from the code example. We’re not going to follow their example very closely, because spring boot offers some tools to make it easier. The example is not very good either.<\/p>\n\n\n\n

Securing our Tokens<\/h2>\n\n\n\n

At this point we’ve got a new token to deal with and our logging token is hardcoded into a config file. This is not good from a security aspect because now our logging token is available in our git repo. So, let’s get this cleaned up and have good practices going forward. We’re going to start with the cloud-application <\/code>project.<\/p>\n\n\n\n

First, load our secrets up to kubernetes (replace with your logging and metric token from logz.io)<\/p>\n\n\n\n

$ kubectl create secret generic logzio-secrets --from-literal=LOGZIO_LOGGER_TOKEN=... --from-literal=LOGZIO_METRICS_TOKEN=...\nsecret\/logzio-secrets created\n<\/code><\/pre>\n\n\n\n
Now update your deployment.yaml file in the helm templates to inject the secrets into the environment<\/p>\n\n\n\n
          env:\n          ...\n          - name: LOGZIO_LOGGER_TOKEN\n            valueFrom:\n              secretKeyRef:\n                name: logzio-secrets\n                key: LOGZIO_LOGGER_TOKEN\n          - name: LOGZIO_METRICS_TOKEN\n            valueFrom:\n              secretKeyRef:\n                name: logzio-secrets\n                key: LOGZIO_METRICS_TOKEN<\/code><\/pre>\n\n\n\n
Now reference the environment variable in logback-spring.yaml<\/p>\n\n\n\n
<token>${LOGZIO_LOGGER_TOKEN}<\/token><\/code><\/pre>\n\n\n\n
Activating Micrometer<\/h2>\n\n\n\n
Add the logzio micrometer library to your pom.xml<\/p>\n\n\n\n
        <micrometer-registry-logzio.version>1.0.2<\/micrometer-registry-logzio.version>\n    <\/properties>\n...\n        <dependency>\n            <groupId>io.logz.micrometer<\/groupId>\n            <artifactId>micrometer-registry-logzio<\/artifactId>\n            <version>${micrometer-registry-logzio.version}<\/version>\n        <\/dependency><\/code><\/pre>\n\n\n\n
Just this configuration alone should get you metrics on the actuator metrics endpoint.  However, we want to create some custom metrics as well in addition to pumping the metrics over to logz.io for visualization.<\/p>\n\n\n\n
Metrics Registry Configuration<\/h2>\n\n\n\n
Micrometer relies on a metrics registry to know what metrics are being collected and what it should present to the prometheus backend service.  Additionally, this registry depends on a configuration which tells it how to authenticate and the location of prometheus.  You can set them up by creating a configuration class in the config <\/code>package called LogzioMicrometerConfiguration <\/code>with this content.<\/p>\n\n\n\n
package com.bullyrooks.cloud_application.config;\r\n\r\nimport io.micrometer.core.instrument.Clock;\r\nimport io.micrometer.core.instrument.MeterRegistry;\r\nimport io.micrometer.core.instrument.Tag;\r\nimport io.micrometer.core.instrument.simple.SimpleMeterRegistry;\r\nimport io.micrometer.logzio.LogzioConfig;\r\nimport io.micrometer.logzio.LogzioMeterRegistry;\r\nimport org.springframework.beans.factory.annotation.Value;\r\nimport org.springframework.boot.actuate.autoconfigure.metrics.CompositeMeterRegistryAutoConfiguration;\r\nimport org.springframework.boot.actuate.autoconfigure.metrics.MetricsAutoConfiguration;\r\nimport org.springframework.boot.actuate.autoconfigure.metrics.export.simple.SimpleMetricsExportAutoConfiguration;\r\nimport org.springframework.boot.autoconfigure.AutoConfigureAfter;\r\nimport org.springframework.boot.autoconfigure.AutoConfigureBefore;\r\nimport org.springframework.boot.autoconfigure.condition.ConditionalOnClass;\r\nimport org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;\r\nimport org.springframework.context.annotation.Bean;\r\nimport org.springframework.context.annotation.Configuration;\r\n\r\nimport java.time.Duration;\r\nimport java.util.ArrayList;\r\nimport java.util.Hashtable;\r\n\r\n@Configuration\r\n@AutoConfigureBefore({\r\n        CompositeMeterRegistryAutoConfiguration.class,\r\n        SimpleMetricsExportAutoConfiguration.class\r\n})\r\n@AutoConfigureAfter(MetricsAutoConfiguration.class)\r\n@ConditionalOnClass(LogzioMeterRegistry.class)\r\npublic class LogzioMicrometerConfiguration {\r\n\r\n    @Value(\"${logzio.metrics.url:http:\/\/test.com}\")\r\n    String logzioMetricsUrl;\r\n    @Value(\"${logzio.metrics.token:TEST}\")\r\n    String logzioMetricsToken;\r\n\r\n    \/* real service instances connect to logzio *\/\r\n    @Bean\r\n    @ConditionalOnProperty(name=\"logzio.metrics.registry.mock\", havingValue=\"false\")\r\n    public LogzioConfig newLogzioConfig() {\r\n        return new LogzioConfig() {\r\n            @Override\r\n            public String get(String key) {\r\n                return null;\r\n            }\r\n\r\n            @Override\r\n            public String uri() {\r\n                return logzioMetricsUrl;\r\n                \/\/ example:\r\n                \/\/ return \"https:\/\/listener.logz.io:8053\";\r\n            }\r\n\r\n            @Override\r\n            public String token() {\r\n                return logzioMetricsToken;\r\n            }\r\n\r\n            @Override\r\n            public Duration step() {\r\n                return Duration.ofSeconds(30);\r\n                \/\/ example:\r\n                \/\/ return Duration.ofSeconds(30);\r\n            }\r\n\r\n            @Override\r\n            public Hashtable<String, String> includeLabels() {\r\n                return new Hashtable<>();\r\n            }\r\n\r\n            @Override\r\n            public Hashtable<String, String> excludeLabels() {\r\n                return new Hashtable<>();\r\n            }\r\n        };\r\n    }\r\n\r\n    @Bean\r\n    @ConditionalOnProperty(name=\"logzio.metrics.registry.mock\", havingValue=\"false\")\r\n    public MeterRegistry logzioMeterRegistry(LogzioConfig config) {\r\n        LogzioMeterRegistry logzioMeterRegistry =\r\n                new LogzioMeterRegistry(config, Clock.SYSTEM);\r\n        ArrayList<Tag> tags = new ArrayList<>();\r\n        tags.add(Tag.of(\"env\", \"dev\"));\r\n        tags.add(Tag.of(\"service\", \"cloud-application\"));\r\n        return logzioMeterRegistry;\r\n    }\r\n\r\n    \/* mock configuration used locally and in tests *\/\r\n\r\n    @Bean\r\n    @ConditionalOnProperty(name=\"logzio.metrics.registry.mock\", havingValue=\"true\")\r\n    public MeterRegistry testRegistry() {\r\n        return new SimpleMeterRegistry();\r\n    }\r\n    @Bean\r\n    @ConditionalOnProperty(name=\"logzio.metrics.registry.mock\", havingValue=\"true\")\r\n    public LogzioConfig logzioConfig(){\r\n            \/\/this is not needed\r\n            return null;\r\n        }\r\n}\r\n\n<\/code><\/pre>\n\n\n\n
The first section describes the live configuration.  The important parts in the configuration are the url and token.  The second bean depends on the configuration to setup the registry.  There’s not much here besides global tokens.  I don’t want metrics to connect to logz.io during local testing (and pipeline testing), so the second section describes how to setup mock instances of the needed metrics components.  We’re using a conditional property to control which beans are created.  So lets create those flags.<\/p>\n\n\n\n
application.yaml<\/code> in src\/main\/resources<\/code><\/p>\n\n\n\n
logzio:\r\n  metrics:\r\n    url: https:\/\/listener.logz.io:8053\r\n    registry:\r\n      mock: false<\/code><\/pre>\n\n\n\n
application-dev.yaml<\/code> in src\/main\/resources<\/code>.  This is probably a new file for your setup.  It will inheirit from application.yaml and overwrite whatever is duplicated, so it only has this content.<\/p>\n\n\n\n
logzio:
  metrics:
    registry:
      mock: true<\/pre>\n\n\n\n
also application-test.yaml<\/code> in src\/test\/resources<\/code><\/p>\n\n\n\n
logzio:\r\n  metrics:\r\n    registry:\r\n      mock: true<\/code><\/pre>\n\n\n\n
When we create the metrics later this will allow our tests to inject a mock registry instead of the logzio configured one.<\/p>\n\n\n\n
Lets go make sure that the configuration is set.  We’ve already added the metrics token to the kubernetes environment configuration so that will be taken care of at deploy time.  Update application.yaml with this configuration<\/p>\n\n\n\n
logging:\n  level:\n    root: INFO\n    com.bullyrooks: DEBUG\n    io.micrometer.logzio: WARN\nlogzio:\n  metrics:\n    url: https:\/\/listener.logz.io:8053<\/code><\/pre>\n\n\n\n
Logz.io’s micrometer logging is really chatty, so we’re turning that down and setting that metrics url.<\/p>\n\n\n\n
Custom Metrics<\/h2>\n\n\n\n
Let’s go ahead and create the custom metrics.  For this case I’m tracking four metrics in cloud-application<\/p>\n\n\n\n