message-generator<\/code><\/h2>\n\n\n\nSwitch over to your message-generator project and start a new branch<\/p>\n\n\n\n
$ git checkout -b health\r\nSwitched to a new branch 'health'\r<\/code><\/pre>\n\n\n\nUpdate pom.xml<\/code> to add the actuator dependency<\/p>\n\n\n\n <!-- Health Checks and Metrics -->\r\n <dependency>\r\n <groupId>org.springframework.boot<\/groupId>\r\n <artifactId>spring-boot-starter-actuator<\/artifactId>\r\n <\/dependency><\/code><\/pre>\n\n\n\nYou may need to reload your maven dependencies to make sure that this new library is being brought in correctly.<\/p>\n\n\n\n
Next, create an application.yaml<\/code> in src\/main\/resource<\/code> (remove application.properties if it exists)<\/p>\n\n\n\nspring:\r\n application:\r\n name: message-generator\r\nmanagement:\r\n endpoints:\r\n web:\r\n exposure:\r\n include: \"*\"\r\n endpoint:\r\n health:\r\n probes:\r\n enabled: true\r\n show-details: always\r\n health:\r\n livenessstate:\r\n enabled: true\r\n readinessstate:\r\n enabled: true<\/code><\/pre>\n\n\n\nHere’s what’s happening: <\/p>\n\n\n\n
- setting our spring application name (which is just good practice)<\/li>
- enabling all actuator endpoints. This might not be generally a safe practice as there may be some useful information here which you will probably want to secure.<\/li>
- Turn on the liveness and readiness health probes and allow the details to be shown on the health endpoint<\/li><\/ul>\n\n\n\n
Some really good information about liveness and readiness are here<\/a>. Some additional information about actuator and the various endpoints we enabled is here<\/a>.<\/p>\n\n\n\nNow start up your application and hit the health endpoint (\/actuator\/health<\/code>). You should see something like this:<\/p>\n\n\n\n![\"\"](\"https:\/\/i0.wp.com\/bullyrooks.com\/wp-content\/uploads\/2022\/02\/image-4.png?resize=857%2C732&ssl=1\")
<\/figure>\n\n\n\nNow let’s go back to our helm configuration and enable those liveness and readiness probe endpoints. Modify your deployment.yaml<\/code> in helm\/message-generator\/templates<\/code> to reflect this<\/p>\n\n\n\n livenessProbe:\r\n httpGet:\r\n path: \/actuator\/health\/liveness\r\n port: 8080\r\n initialDelaySeconds: 15\r\n readinessProbe:\r\n httpGet:\r\n path: \/actuator\/health\/readiness\r\n port: 8080\r\n initialDelaySeconds: 15<\/code><\/pre>\n\n\n\nCommit and Push<\/h2>\n\n\n\n
Now push up to build and test<\/p>\n\n\n\n
$ git add .\n\n$ git commit -m \"added health checks\"\r\n[health f69d658] added health checks\r\n 4 files changed, 32 insertions(+), 9 deletions(-)\r\n delete mode 100644 src\/main\/resources\/application.properties\r\n create mode 100644 src\/main\/resources\/application.yaml\n\r\n$ git push\r\nfatal: The current branch health has no upstream branch.\r\nTo push the current branch and set the remote as upstream, use\r\nCompressing objects: 100% (9\/9), done.\r\nWriting objects: 100% (11\/11), 1.00 KiB | 512.00 KiB\/s, done.\r\nTotal 11 (delta 5), reused 0 (delta 0), pack-reused 0\r\nremote: Resolving deltas: 100% (5\/5), completed with 5 local objects.\r\nremote:\r\nremote: Create a pull request for 'health' on GitHub by visiting:\r\nremote: https:\/\/github.com\/bullyrooks\/message-generator\/pull\/new\/health\r\nremote:\r\nTo github.com-bullyrook:bullyrooks\/message-generator.git\r\n * [new branch] health -> health\r\nBranch 'health' set up to track remote branch 'health' from 'origin'.\r\n<\/code><\/pre>\n\n\n\nWhen the build completes succesfully, merge to main<\/p>\n\n\n\n
$ git checkout main\r\nSwitched to branch 'main' \r\nYour branch is up to date with 'origin\/main'.\n\n$ git merge health \r\nUpdating 638c8dd..f69d658\r\nFast-forward\r\n helm\/message-generator\/templates\/deployment.yaml | 16 ++++++++--------\r\n pom.xml | 6 ++++++\r\n src\/main\/resources\/application.properties | 1 -\r\n src\/main\/resources\/application.yaml | 18 ++++++++++++++++++\r\n 4 files changed, 32 insertions(+), 9 deletions(-)\r\n delete mode 100644 src\/main\/resources\/application.properties\r\n create mode 100644 src\/main\/resources\/application.yaml\r\n\n$ git push\r\nTotal 0 (delta 0), reused 0 (delta 0), pack-reused 0\r\nTo github.com-bullyrook:bullyrooks\/message-generator.git\r\n 638c8dd..f69d658 main -> main\r\n<\/code><\/pre>\n\n\n\nAnd you should see it via postman<\/p>\n\n\n\n![\"\"](\"https:\/\/i0.wp.com\/bullyrooks.com\/wp-content\/uploads\/2022\/02\/image-5.png?resize=855%2C741&ssl=1\")
<\/figure>\n\n\n\nIt was about here that I discovered that my application wasn’t deploying the latest version because the publishing of the helm chart was taking longer than the time that github actions reached the helm repo update<\/code><\/p>I added this update to the helm github action in main.yaml to fix that<\/p><\/blockquote>\n\n\n\n
- name: Deploy
uses: WyriHaximus\/github-action-helm3@v2
with:
exec: |
helm repo add bullyrooks https:\/\/bullyrooks.github.io\/helm-charts\/
helm repo update
echo \"helm upgrade message-generator bullyrooks\/message-generator --install --version ${{ env.VERSION }}\"
sleep 45s
helm repo update
helm upgrade message-generator bullyrooks\/message-generator --install --version ${{ env.VERSION }}
kubeconfig: '${{ secrets.KUBECONFIG }}'<\/pre>\n\n\n\nHealth and Readiness for Cloud Application<\/h2>\n\n\n\n
Go ahead and do the exact same setup for cloud-application. Once you have that setup, we’re going to add some new health checks.<\/p>\n\n\n\n
When you deploy and run the health check you’re going to see that you have some additional health checks:<\/p>\n\n\n\n
\"discoveryComposite\": {\r\n \"status\": \"UP\",\r\n \"components\": {\r\n \"discoveryClient\": {\r\n \"status\": \"UP\",\r\n \"details\": {\r\n \"services\": [\r\n \"cloud-application\"\r\n ]\r\n }\r\n }\r\n }\r\n },\n...\n \"kubernetes\": {\r\n \"status\": \"UP\",\r\n \"details\": {\r\n \"nodeName\": \"gke-cloud-dev-b-1-b0b413d1-c9n4\",\r\n \"podIp\": \"10.8.17.59\",\r\n \"hostIp\": \"10.255.16.52\",\r\n \"namespace\": \"bullyrooks\",\r\n \"podName\": \"cloud-application-64dd9c6557-v2grm\",\r\n \"serviceAccount\": \"default\",\r\n \"inside\": true,\r\n \"labels\": {\r\n \"app.kubernetes.io\/instance\": \"cloud-application\",\r\n \"app.kubernetes.io\/name\": \"cloud-application\",\r\n \"pod-template-hash\": \"64dd9c6557\"\r\n }\r\n }\r\n },\n...\n \"mongo\": {\r\n \"status\": \"UP\",\r\n \"details\": {\r\n \"version\": \"4.4.12\"\r\n }\r\n },<\/code><\/pre>\n\n\n\nThe mongo one is built into the mongodb starter. It will check the connection to the database to confirm connectivity. The others come from the fabric 8 kubernetes framework. One tells you the other services that are available via discovery. The other tells you the status of the deployment so you can quickly debug.<\/p>\n\n\n\n
Now we’re going to create a new health check for the status of our dependent service (message-generator) and make the readiness status depend on the health of that service and the database.<\/p>\n\n\n\n
Create a Health Indicator<\/h2>\n\n\n\n
in com.bullyrooks.cloud_application.message_generator.client.dto<\/code> create a class called HealthCheckDTO <\/code>with this content<\/p>\n\n\n\npackage com.bullyrooks.cloud_application.message_generator.client.dto;\r\n\r\nimport lombok.Data;\r\n\r\n@Data\r\npublic class HealthCheckDTO {\r\n private String status;\r\n}\r\n<\/code><\/pre>\n\n\n\ncom.bullyrooks.cloud_application.message_generator.client<\/code> create a class called MessageGeneratorHealthClient <\/code>with this content<\/p>\n\n\n\npackage com.bullyrooks.cloud_application.message_generator.client;\r\n\r\nimport com.bullyrooks.cloud_application.message_generator.client.dto.HealthCheckDTO;\r\nimport org.springframework.cloud.openfeign.FeignClient;\r\nimport org.springframework.http.MediaType;\r\nimport org.springframework.http.ResponseEntity;\r\nimport org.springframework.web.bind.annotation.GetMapping;\r\n\r\n@FeignClient(contextId = \"healthClient\", name = \"message-generator\")\r\npublic interface MessageGeneratorHealthClient {\r\n\r\n @GetMapping(value = \"\/actuator\/health\",\r\n produces = MediaType.APPLICATION_JSON_VALUE,\r\n consumes = MediaType.APPLICATION_JSON_VALUE\r\n )\r\n ResponseEntity<HealthCheckDTO> getHealth();\r\n}\r\n<\/code><\/pre>\n\n\n\nThis will create a client that uses our message-generato<\/code>r service discovered through kubernetes to hit the health endpoint. We only care about status, so that’s the only field we map into the DTO.<\/p>\n\n\n\nNow add this content to application.yaml<\/code><\/p>\n\n\n\nfeign:
client:
config:
healthClient:
connectTimeout: 1000
readTimeout: 1000<\/pre>\n\n\n\nand in application.yaml<\/code> update this clause<\/p>\n\n\n\nmanagement:\r\n endpoints:\r\n web:\r\n exposure:\r\n include: \"*\"\r\n endpoint:\r\n health:\r\n probes:\r\n enabled: true\r\n show-details: always\r\n group:\r\n readiness:\r\n include: \"readinessState,mongo,messageGenerator\"\r\n<\/code><\/pre>\n\n\n\nHere we’re configuring our new feign client to timeout quickly if it can’t reach the health endpoint. Notice that the feign client contextId maps to the configuration. This allows us to use a separate timeout configuration for this client.<\/p>\n\n\n\n
Now create a new package: com.bullyrooks.cloud_application.config<\/code>. Create a class called MessageGeneratorHealth <\/code>and add this content<\/p>\n\n\n\npackage com.bullyrooks.cloud_application.config;\r\n\r\nimport com.bullyrooks.cloud_application.message_generator.client.MessageGeneratorHealthClient;\r\nimport com.bullyrooks.cloud_application.message_generator.client.dto.HealthCheckDTO;\r\nimport lombok.extern.slf4j.Slf4j;\r\nimport org.apache.commons.lang.StringUtils;\r\nimport org.springframework.beans.factory.annotation.Autowired;\r\nimport org.springframework.boot.actuate.availability.AvailabilityStateHealthIndicator;\r\nimport org.springframework.boot.actuate.health.Health;\r\nimport org.springframework.boot.actuate.health.HealthIndicator;\r\nimport org.springframework.boot.actuate.health.Status;\r\nimport org.springframework.boot.availability.ApplicationAvailability;\r\nimport org.springframework.boot.availability.AvailabilityState;\r\nimport org.springframework.boot.availability.ReadinessState;\r\nimport org.springframework.http.ResponseEntity;\r\nimport org.springframework.stereotype.Component;\r\n\r\n@Component\r\n@Slf4j\r\npublic class MessageGeneratorHealthIndicator implements HealthIndicator {\r\n\r\n @Autowired\r\n MessageGeneratorHealthClient messageClient;\r\n\r\n public Health health() {\r\n Health.Builder status = Health.up();\r\n try {\r\n ResponseEntity<HealthCheckDTO> healthCheckDTO = messageClient.getHealth();\r\n log.info(\"health check response: {}\\n{}\", healthCheckDTO.getStatusCode(), healthCheckDTO.getBody());\r\n if (!StringUtils.equals(\"UP\", healthCheckDTO.getBody().getStatus())) {\r\n status = Health.outOfService();\r\n }\r\n }catch (Exception e){\r\n log.error(\"error trying to get message generator health: {}\", e.getMessage(),e);\r\n status = Health.outOfService();\r\n }\r\n return status.build();\r\n }\r\n}\r\n\r\n<\/code><\/pre>\n\n\n\nHealthIndicator <\/code>is a spring actuator interface that will automatically detect this class and add its health()<\/code> check to the health endpoint results. In this class we use our new client to make sure that the m<\/code>essage-generator health endpoint returns UP<\/code>. If it doesn’t, we return DOWN <\/code>for this check. Additionally, the naming convention used for the application.yaml will be the name minus HealthIndicator<\/code>.<\/p>\n\n\n\nTesting with Message Generator Up<\/h2>\n\n\n\n
Message generator should already be running. Hit the endpoint for your cloud-application health endpoint (actuator\/health<\/code>). You should see something like this:<\/p>\n\n\n\n \"status\": \"UP\",\n...\n \"messageGenerator\": {\n \"status\": \"UP\"\n },\n...\n \"readinessState\": {\n \"status\": \"UP\"\n },\n<\/code><\/pre>\n\n\n\nand the readiness endpoint (actuator\/health\/readines<\/code>s) should look like this<\/p>\n\n\n\n{\r\n \"status\": \"UP\",\r\n \"components\": {\r\n \"messageGenerator\": {\r\n \"status\": \"UP\"\r\n },\r\n \"mongo\": {\r\n \"status\": \"UP\",\r\n \"details\": {\r\n \"version\": \"4.4.12\"\r\n }\r\n },\r\n \"readinessState\": {\r\n \"status\": \"UP\"\r\n }\r\n }\r\n}<\/code><\/pre>\n\n\n\nTesting with Message Generator Down<\/h2>\n\n\n\n
Log into okteto and destroy the message-generator deployment.<\/p>\n\n\n\n
Hit your readiness endpoint (\/actuator\/health\/readiness<\/code>)<\/strong>, you should see something like this:<\/p>\n\n\n\n {\r\n \"status\": \"OUT_OF_SERVICE\",\r\n \"components\": {\r\n \"messageGenerator\": {\r\n \"status\": \"OUT_OF_SERVICE\"\r\n },\r\n \"mongo\": {\r\n \"status\": \"UP\",\r\n \"details\": {\r\n \"version\": \"4.4.12\"\r\n }\r\n },\r\n \"readinessState\": {\r\n \"status\": \"UP\"\r\n }\r\n }\r\n}<\/code><\/pre>\n\n\n\nyou can also see the changes in the \/actuator\/health endpoint<\/p>\n\n\n\n
\"status\": \"OUT_OF_SERVICE\",\n...\n \"messageGenerator\": {\r\n \"status\": \"OUT_OF_SERVICE\"\n...\n \"readinessState\": {\r\n \"status\": \"UP\"\r\n },<\/code><\/pre>\n\n\n\nI’m not sure why readinessState here looks UP<\/strong>. However, you must be very careful with the readiness and liveness probes and kubernetes can take your service out of availability if they are failing. The docs are very specific about this<\/a>:<\/p>\n\n\n\nAs for the \u201creadiness\u201d probe, the choice of checking external systems must be made carefully by the application developers. For this reason, Spring Boot does not include any additional health checks in the readiness probe. If the\u00a0readiness state of an application instance<\/a>\u00a0is unready, Kubernetes does not route traffic to that instance. Some external systems might not be shared by application instances, in which case they could be included in a readiness probe. Other external systems might not be essential to the application (the application could have circuit breakers and fallbacks), in which case they definitely should not be included. Unfortunately, an external system that is shared by all application instances is common, and you have to make a judgement call: Include it in the readiness probe and expect that the application is taken out of service when the external service is down or leave it out and deal with failures higher up the stack, perhaps by using a circuit breaker in the caller.<\/p><\/blockquote>\n\n\n\nI do not recommend implementing it this way for our use case. Cloud application is available when message generator is offline (you can provide a message and it should still work). I’m only using this as an example of how to hook up readiness probes for dependent systems.<\/p>\n","protected":false},"excerpt":{"rendered":"