Kube Cloud Pt4 | Metrics

Logz.io uses prometheus for metrics. Go ahead and enable metrics collection in logz.io. Navigate to send your metrics

Search for java

Choose java custom metrics via micrometer. We’re using micrometer because its integrated into spring boot and available via actuator, which we’re already using.

The important part here is that you grab the metrics token from the code example. We’re not going to follow their example very closely, because spring boot offers some tools to make it easier. The example is not very good either.

Securing our Tokens

At this point we’ve got a new token to deal with and our logging token is hardcoded into a config file. This is not good from a security aspect because now our logging token is available in our git repo. So, let’s get this cleaned up and have good practices going forward. We’re going to start with the cloud-application project.

First, load our secrets up to kubernetes (replace with your logging and metric token from logz.io)

$ kubectl create secret generic logzio-secrets --from-literal=LOGZIO_LOGGER_TOKEN=... --from-literal=LOGZIO_METRICS_TOKEN=...
secret/logzio-secrets created

Now update your deployment.yaml file in the helm templates to inject the secrets into the environment

          env:
          ...
          - name: LOGZIO_LOGGER_TOKEN
            valueFrom:
              secretKeyRef:
                name: logzio-secrets
                key: LOGZIO_LOGGER_TOKEN
          - name: LOGZIO_METRICS_TOKEN
            valueFrom:
              secretKeyRef:
                name: logzio-secrets
                key: LOGZIO_METRICS_TOKEN

Now reference the environment variable in logback-spring.yaml

<token>${LOGZIO_LOGGER_TOKEN}</token>

Activating Micrometer

Add the logzio micrometer library to your pom.xml

        <micrometer-registry-logzio.version>1.0.2</micrometer-registry-logzio.version>
    </properties>
...
        <dependency>
            <groupId>io.logz.micrometer</groupId>
            <artifactId>micrometer-registry-logzio</artifactId>
            <version>${micrometer-registry-logzio.version}</version>
        </dependency>

Just this configuration alone should get you metrics on the actuator metrics endpoint. However, we want to create some custom metrics as well in addition to pumping the metrics over to logz.io for visualization.

Metrics Registry Configuration

Micrometer relies on a metrics registry to know what metrics are being collected and what it should present to the prometheus backend service. Additionally, this registry depends on a configuration which tells it how to authenticate and the location of prometheus. You can set them up by creating a configuration class in the config package called LogzioMicrometerConfiguration with this content.

package com.bullyrooks.cloud_application.config;

import io.micrometer.core.instrument.Clock;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Tag;
import io.micrometer.core.instrument.simple.SimpleMeterRegistry;
import io.micrometer.logzio.LogzioConfig;
import io.micrometer.logzio.LogzioMeterRegistry;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.actuate.autoconfigure.metrics.CompositeMeterRegistryAutoConfiguration;
import org.springframework.boot.actuate.autoconfigure.metrics.MetricsAutoConfiguration;
import org.springframework.boot.actuate.autoconfigure.metrics.export.simple.SimpleMetricsExportAutoConfiguration;
import org.springframework.boot.autoconfigure.AutoConfigureAfter;
import org.springframework.boot.autoconfigure.AutoConfigureBefore;
import org.springframework.boot.autoconfigure.condition.ConditionalOnClass;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.time.Duration;
import java.util.ArrayList;
import java.util.Hashtable;

@Configuration
@AutoConfigureBefore({
        CompositeMeterRegistryAutoConfiguration.class,
        SimpleMetricsExportAutoConfiguration.class
})
@AutoConfigureAfter(MetricsAutoConfiguration.class)
@ConditionalOnClass(LogzioMeterRegistry.class)
public class LogzioMicrometerConfiguration {

    @Value("${logzio.metrics.url:http://test.com}")
    String logzioMetricsUrl;
    @Value("${logzio.metrics.token:TEST}")
    String logzioMetricsToken;

    /* real service instances connect to logzio */
    @Bean
    @ConditionalOnProperty(name="logzio.metrics.registry.mock", havingValue="false")
    public LogzioConfig newLogzioConfig() {
        return new LogzioConfig() {
            @Override
            public String get(String key) {
                return null;
            }

            @Override
            public String uri() {
                return logzioMetricsUrl;
                // example:
                // return "https://listener.logz.io:8053";
            }

            @Override
            public String token() {
                return logzioMetricsToken;
            }

            @Override
            public Duration step() {
                return Duration.ofSeconds(30);
                // example:
                // return Duration.ofSeconds(30);
            }

            @Override
            public Hashtable<String, String> includeLabels() {
                return new Hashtable<>();
            }

            @Override
            public Hashtable<String, String> excludeLabels() {
                return new Hashtable<>();
            }
        };
    }

    @Bean
    @ConditionalOnProperty(name="logzio.metrics.registry.mock", havingValue="false")
    public MeterRegistry logzioMeterRegistry(LogzioConfig config) {
        LogzioMeterRegistry logzioMeterRegistry =
                new LogzioMeterRegistry(config, Clock.SYSTEM);
        ArrayList<Tag> tags = new ArrayList<>();
        tags.add(Tag.of("env", "dev"));
        tags.add(Tag.of("service", "cloud-application"));
        return logzioMeterRegistry;
    }

    /* mock configuration used locally and in tests */

    @Bean
    @ConditionalOnProperty(name="logzio.metrics.registry.mock", havingValue="true")
    public MeterRegistry testRegistry() {
        return new SimpleMeterRegistry();
    }
    @Bean
    @ConditionalOnProperty(name="logzio.metrics.registry.mock", havingValue="true")
    public LogzioConfig logzioConfig(){
            //this is not needed
            return null;
        }
}

The first section describes the live configuration. The important parts in the configuration are the url and token. The second bean depends on the configuration to setup the registry. There’s not much here besides global tokens. I don’t want metrics to connect to logz.io during local testing (and pipeline testing), so the second section describes how to setup mock instances of the needed metrics components. We’re using a conditional property to control which beans are created. So lets create those flags.

application.yaml in src/main/resources

logzio:
  metrics:
    url: https://listener.logz.io:8053
    registry:
      mock: false

application-dev.yaml in src/main/resources. This is probably a new file for your setup. It will inheirit from application.yaml and overwrite whatever is duplicated, so it only has this content.

logzio:
  metrics:
    registry:
      mock: true

also application-test.yaml in src/test/resources

logzio:
  metrics:
    registry:
      mock: true

When we create the metrics later this will allow our tests to inject a mock registry instead of the logzio configured one.

Lets go make sure that the configuration is set. We’ve already added the metrics token to the kubernetes environment configuration so that will be taken care of at deploy time. Update application.yaml with this configuration

logging:
  level:
    root: INFO
    com.bullyrooks: DEBUG
    io.micrometer.logzio: WARN
logzio:
  metrics:
    url: https://listener.logz.io:8053

Logz.io’s micrometer logging is really chatty, so we’re turning that down and setting that metrics url.

Custom Metrics

Let’s go ahead and create the custom metrics. For this case I’m tracking four metrics in cloud-application

• Number of message requests
• Number of requests that didn’t have a message
• Number of successfully generated messages
• Number of successfully stored messages

Edit MessageService class. First set some new attributes of the class

    MeterRegistry logzioMeterRegistry;

    Counter msgCount;
    Counter genMsgCount;
    Counter genMsgSuccess;
    Counter messageSaved;

This is a reference to our registry and the new metrics that we’re going to capture.

Now we’re going to change up the autowiring scheme to autowire by constructor so remove the autowire annotations and add this constructor

    @Autowired
    public MessageService(MessageRepository messageRepository,
                          MessageGeneratorClient messageGeneratorClient,
                          MeterRegistry logzioMeterRegistry){
        this.messageRepository = messageRepository;
        this.messageGeneratorClient = messageGeneratorClient;
        this.logzioMeterRegistry = logzioMeterRegistry;
        initCounters();
    }

You can see we’ve got a new method to initialize the counters. That logic is

    private void initCounters() {
        msgCount= Counter.builder("message.request.count")
                .description("Number of message requests received by the service")
                .register(logzioMeterRegistry);
        genMsgCount = Counter.builder("message.generated.request.count")
                .description("Number of message generated requests to message-generator")
                .register(logzioMeterRegistry);
        genMsgSuccess = Counter.builder("message.generated.request.success")
                .description("Number of message generated success responses from message-generator")
                .register(logzioMeterRegistry);
        messageSaved = Counter.builder("message.stored.count")
                .description("Number of messages successfully stored in the repository")
                .register(logzioMeterRegistry);
    }

There are several key points here. Use a readable and maintainable key name, make sure to add a description for future developers and register the metric with the registry.

Now instrument the service

    public Message saveMessage(Message message){

        msgCount.increment();
        if (StringUtils.isEmpty(message.getMessage())){

            genMsgCount.increment();
            log.info("No message, retrieve from message generator");
            MessageResponseDTO dto = messageGeneratorClient.getMessage();
            message.setMessage(dto.getMessage());
            genMsgSuccess.increment();
            log.info("retrieved message: {}", message.getMessage());
        }

        MessageDocument msgDoc = MessageDocumentMapper.INSTANCE.modelToDocument(message);

        log.info("saving document: {}", msgDoc);
        MessageDocument returnDoc = messageRepository.save(msgDoc);
        messageSaved.increment();
        return MessageDocumentMapper.INSTANCE.documentToModel(returnDoc);
    }

Its pretty basic… just increment the counter when the relevant event occurs. Its very similar to logging but the structure comes into play later.

Validate the Metrics Endpoint

Start up the application, hit the post message endpoint a few times and navigate to /actuator/metrics in postman. You should see something like this

There’s a lot of metrics that come preloaded. Ours should be in here too. This is essentially all of the existing metrics that have been registered with the micrometer registry.

And you can see the actual metrics if you navigate to the metric (actuator/metrics/message.request.count)

$ git add .
$ git commit -m "metrics"
[main 6ce8574] metrics
 7 files changed, 152 insertions(+), 5 deletions(-)
 create mode 100644 src/main/java/com/bullyrooks/cloud_application/config/LogzioMicrometerConfiguration.java
$ git push
Enumerating objects: 40, done.
Counting objects: 100% (40/40), done.
Delta compression using up to 4 threads
Compressing objects: 100% (17/17), done.
Writing objects: 100% (21/21), 3.24 KiB | 663.00 KiB/s, done.
Total 21 (delta 10), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (10/10), completed with 10 local objects.
To github.com-bullyrook:bullyrooks/cloud_application.git
   c8542f4..6ce8574  main -> main

While this is building and deploying, lets go build the visualizations in grafana

Visualization with Grafana

Navigate to the grafana instance

Create a new dashboard

Add an empty panel

In the Edit Panel page, add sum(increase(message_request_count_total[1m])) to the query and change the name to Number of Message Requests

And hit Apply. You should see your dashboard with the new panel. Make sure to save and name the dashboard.

Go ahead and repeat for all of the metrics. Remember that the metric format is the name of your metric, substitute the “.” with “_” and append with “_total“.

When you’re done you should see something like this:

Visualizing Resource Metrics

At this point, we’re just visualizing our work metrics, but our resource metrics are just as important. We’re going to want to see error rate and latency, which are provided by the default metrics.

There are a few ways to do this. You can create them all by hand (you have the metrics). However, I think grafana is kind of difficult to work with. An alternative is to find, download and import a dashboard that is already configured for spring boot service visualization.

You can search grafana’s dashboard repository here.

Here’s a sample:

You may have to do a little extra configuration, which I won’t dive into. There’s usually some documentation that comes with these pre configured dashboards.