The Spring Cloud Sleuth project provided functionality for distributed tracing; the core of this project has now been moved to Micrometer. The distributed tracing provided by Micrometer is fully featured and has some overlap with the functionality provided by the spt-development-cid-* projects. In this post I will demonstrate how to combine spt-development-audit-spring and spt-development-logging-spring with Micrometer to get production quality logging and auditing together with the distributed tracing features of Micrometer.

Micrometer Tracing

All of the code from this post can be found in the spt-development-micrometer-tracing-demo project, which was forked from the spt-development-demo project. The spt-development-demo project already included the spring-boot-starter-actuator dependency, so the only other dependency required to get tracing working was to add the tracing bridge dependency.

To see the trace ID (and span ID) included in the logs, the loggging pattern needed updating.

Specifically [%X{traceId:-},%X{spanId:-}] was included in the pattern.

Further detail around configuring tracing and shipping logs to a distributed tracing system such as Zipkin, can be found in the documentation. Note, by default Spring Boot samples only 10% of requests to prevent overwhelming the trace backend, so it may be necessary to change the management.tracing.sampling.probability property; this is not necessary for this demo where we are only logging to the console.

Gotcha!

One thing that I struggled to find mentioned in the documentation was that tracing is switched off by default in tests. This is intended and therefore when I ran the tests initially, the trace ID and span ID were not set. To override this default behaviour it was necessary to add the AutoConfigureObservability annotation to the integration tests, which can be used to enable/disable metrics and tracing independently.

Auditing

Adding the spt-development-audit-spring-boot-starter dependency to the project pulls in the required dependencies and automatically configures the auditor. Since we are going to be using the Micrometer trace ID in place of the correlation ID, we can exclude the spt-development-cid dependency. This is not a mandatory step and the demo would work as expected without this exclusion; in the future we may change the audit libraries so that spt-development-cid is no longer brought in as a transitive dependency automatically.

To use the trace ID in place of the correlation ID implementation we need a custom CorrelationIdProvider bean that returns the current trace ID.

With this in place and methods annotated as described in this post, your application will be fully audited and audit events can be correlated using the trace ID.

Actuator audit events

If you are capturing actuator events as described in the previous auditing post you need to inject the Tracer bean into your audit event listener and again use the current trace ID in place of the correlation ID.

JMS Propagation

The demo project makes use of JmsAuditEventWriter simply by configuring the spt.audit.jms.destination property. The Micrometer trace context is not currently propagated in JMS messages by default; there is currently an outstanding issue to add this feature. For now, it is necessary to add custom code to propagate the trace context and for the trace ID to be included in the logs when the JMS messages are processed asynchronously. For this demo, a simple aspect has been added which extracts the trace ID from the JMS correlation ID header - this header is populated with the correlation ID returned by the CorrelationIdProvider.

Logging

As previously written about in more detail here simply add the spt-development-logging-spring-boot-starter as a dependency to get production quality logging added to your application without any further code changes. With the logging format changes above, you will then see logging for all of your public methods with the trace ID and span ID referenced.

As with the audit starter, we can again exclude the spt-development-cid dependency if we want to reduce the number of libraries brought into the project.

All of the changes required to migrate the original demo project to use Micrometer tracing, can be seen in this commit.