Using Prometheus to monitor apps on Oracle Application Container Cloud

Monitoring is hard.. more so in production, and even more so, if the production environment happens to be a PaaS component in the Cloud

You (generally) cannot access the underlying infrastructure/containers/VMs
There are a plethora of languages (polyglot) and each one of them has umpteen number of frameworks

There are multiple solutions out there of which Prometheus is a (relatively) new but promising option. With the help of a simple example, this blog demonstrates how to use Prometheus to monitor a Spring Boot app on Oracle Application Container Cloud

Before we begin…

.. it’s worth pointing out that, Oracle Application Container Cloud supports basic constructs such as

Ability to profile your (Java) applications using JFR and downloading them for further analysis
Application specific (per instance) memory metrics
Access to application logs

Let’s dive in

the source is available on Github

abhirockzz/accs-prometheus-spring-boot-example
accs-prometheus-spring-boot-example - Using Prometheus to monitor Spring Boot apps on Oracle Application Container…github.com

The high level flow (and the sample app) is dead simple

Prometheus scrapes the Spring Boot app (HTTP GET) for metrics and stores them (in this case its the local storage)
you can then use the Prometheus UI to introspect the metrics in the form of graphs

Let’s explore some of the important components

Prometheus

It is an open source monitoring service which was originally developed at SoundCloud

part of the CNCF
supports time series data model along with a query language
integration hooks with external systems (e.g. Grafana etc.)

Overview | Prometheus
An open-source monitoring system with a dimensional data model, flexible query language, efficient time series database…prometheus.io

Spring Boot (+ Spring Actuator)

Our Spring Boot app is simple and it exposes a REST endpoint using @RestController

Spring Actuator auto-magically enables the app monitoring part using HTTP endpoints

There are many endpoints which Spring Actuator (spring-boot-starter-actuator) exposes (e.g. info, health, dump, env, trace etc.), but the sample app only activates the metrics endpoint which exposes information such as JVM memory, threads, system uptime and more etc.

These metrics details are exposed as JSON — this is not the format which Prometheus understands. So, in addition to this, the sample app uses a adapter/transformer layer provided as an add-on module of the Prometheus Java client.

It helps convert the Spring Actuator metrics info (originally JSON) into the plain text format which Prometheus can consume (by default, it is exposed at the /prometheus endpoint)

prometheus/client_java
client_java - Prometheus instrumentation library for JVM applicationsgithub.com

The following Maven dependency is all that’s required

Maven dependency for Prometheus Spring Boot client

All you need to do is to use the annotations (highlighted below) to activate the plugin

@SpringBootApplication
@RestController
@EnablePrometheusEndpoint
@EnableSpringBootMetricsCollector

public class Bootstrap {

public static void main(String[] args) {
 SpringApplication.run(Bootstrap.class, args);
 }
.....
}

That’s about it for an overview… Now it’s time to prepare the application and send it on it’s way to the cloud

Build & deployment

With Oracle Application Container Cloud, you have multiple options in terms of deploying your applications. This blog will leverage PSM CLI which is a powerful command line interface for managing Oracle Cloud services

other deployment options include REST API, Oracle Developer Cloud and of course the console/UI

To begin with, download and setup PSM CLI on your machine (using psm setup) — details here

Start by cloning the git repo — git clone https://github.com/abhirockzz/accs-prometheus-spring-boot-example

Build the Spring Boot app

cd accs-springboot-app
mvn clean install

The build process will create accs-spring-boot-hello-world-dist.zip in the target directory

Push it to cloud

psm accs push -n SpringBootOnACCS -r java -s hourly -m manifest.json -d deployment.json -p target/accs-spring-boot-hello-world-dist.zip

Once executed, an asynchronous process is kicked off and the CLI returns its Job ID for you to track the application creation

Wait for the application to get created and note its URL

Simple Spring Boot app deployed to Oracle Application Container Cloud

Build the Prometheus server

cd prometheus-on-accs
Download Prometheus from here and place it into the directory

For latest release versions, please refer to the page below

Download | Prometheus
An open-source monitoring system with a dimensional data model, flexible query language, efficient time series database…prometheus.io

At the end, your directory structure should look like this

Now just zip/tar up the contents — zip accs-prometheus-server.zip prometheus-2.0.0.linux-amd64.tar.gz prometheus_config.yml start.sh

One last thing before pushing Prometheus to the cloud…..

Update Prometheus configuration

Edit the prometheus_config.yml file to enter the URL of your Spring Boot app on Oracle Application Container Cloud (the one you deployed in the previous step)

Replace this with the URL specific to your environment

Deploy the Prometheus server

You are just a command away from deploying Prometheus to Oracle Application Container Cloud

psm accs push -n PrometheusServer -r golang -e dockerhub -s hourly -m manifest.json -d deployment.json -p accs-prometheus-server.zip

Wait for the Prometheus server to get provisioned and bootstrapped

Prometheus on Oracle Application Container Cloud

Click on the application name to navigate to the details screen

Test drive

To begin with, let’s start off with a couple of sanity checks

Access Spring Actuator metrics endpoint

curl -X GET https://springbootonaccs-ocloud109.apaas.us2.oraclecloud.com/metrics

It should give back a JSON with the metrics information and attributes such as heap, threads etc.

Access the prometheus endpoint

As mentioned before, the Spring Actuator metrics information is transformed to Prometheus (plain text) format and exposed (by default) on the prometheus endpoint

curl -X GET https://springbootonaccs-ocloud109.apaas.us2.oraclecloud.com/prometheus

Check the Prometheus configuration

Access the Prometheus server on Oracle Application Container Cloud to open its console and select Configuration from the Status menu

The value for targets is the Spring Boot app on which you just deployed and value of the metrics_path is the endpoint at which the prometheus formatted metrics data is available

How is my target doing ?

Check Target under the Status menu. You will see that the Spring Boot app is up and is being scraped to extract monitoring metrics

Prometheus target (Spring Boot app) status

Explore the metrics

On the main page, the drop down displays all the available metrics which are generated by the Spring Boot app e.g. classes, heap etc.

There are a lot of metrics, but let’s quickly look at a few of them — used memory, free memory and used heap

Selecting mem (total memory being used by the JVM) from the metrics drop down shows the graph. Here we see that for 15 minute window, the memory utilization was increasing

Analyzing, mem_free (the memory which is free for the JVM) shows that it is on a decreasing trend — now that’s obvious since we just saw that the memory utilization was on the rise

The heap usage (evident using heap_used metric) is a part of the the used JVM memory (mem). From the graph, it’s clear that the heap utilization (for the selected 15 min time window) was on an upward trend

Additional considerations

Things which are interesting but not covered (maybe some other blog ?)

Custom metrics
Non Spring apps
Grafana integration
Remote storage

Quick recap of what we did

Deployed Spring Boot app on Oracle Application Container Cloud
Deployed the Prometheus server (golang runtime) on Oracle Application Container Cloud
Monitored the Spring Boot app using metrics which were sent to Prometheus (thanks to Spring Boot Actuator)

That’s it for this blog post!

Don’t forget to…

check out the tutorials for Oracle Application Container Cloud — there is something for every runtime!

Oracle Application Container Cloud Service — Create Your First Applications
Tutorials for Oracle Application Container Cloud Service. Learn to create your first applications.docs.oracle.com

other blogs on Application Container Cloud

Latest stories and news about App Container Cloud — Medium
Read the latest writing about App Container Cloud. Every day, thousands of voices read, write, and share important…medium.com

Cheers!

The views expressed in this post are my own and do not necessarily reflect the views of Oracle.