telemetry_logging

SLF4J

• Simple Logging Facade for Java
• It is a facade library which provides abstraction for logging libraries like:
  • log4j
  • logback
  • java.util.logger

Spring Boot Logging:

• Default log pattern:
  • Date and Time: Millisecond precision and easily sortable.
  • Log Level: ERROR, WARN, INFO, DEBUG, or TRACE.
  • Process ID.
  • A --- separator to distinguish the start of actual log messages.
  • Thread name: Enclosed in square brackets (may be truncated for console output).
  • Logger name: This is usually the source class name (often abbreviated).
  • The log message.

2014-03-05 10:57:51.253  INFO 45469 --- [ost-startStop-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
 
2014-03-05 10:57:51.253  INFO 45469 --- [ost-startStop-1] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 1358 ms

• Spring Boot Actuator:
  • Metrics library
  • Exposes metrics via http endpoint
• Spring Boot Admin:
  • Web Application to manage and monitor
  • Uses actuator and shows up in centralized way
• Spring Cloud Sleuth:
  • Adds TraceID to track single request, job or action
  • logs of different services will share the same trace ID for a single request

Log Levels

• Change log levels via application.properties:
  • package level: logging.level.com.my.package=TRACE
    • seems like if set at package level then all the classes/packages will inherit it
  • application level: logging.level.root=TRACE
• Change logging level at runtime without restarting application:
  • with spring boot actuator endpoint: /actuator/loggers
  • using logback autoscan functionality
  • with spring cloud config library
  • with spring boot admin

Logging library

• spring-boot-starter-web > spring-boot-starter-logging
• Default for starter library is logback
• co.elastic.logging:logback-ecs-encoder
  • ECS: Elastic Common Schema
  • Transforms Java application logs into JSON format following the ECS

Elastic APM

• https://www.elastic.co/apm
• https://github.com/elastic/apm
• https://www.elastic.co/blog/elasticon-global-observability
• https://docs.staging.external.byp.ai/features/monitoring/#overview-over-the-elastic-stack
• allows you to monitor applications in real-time:
  • incoming and outgoing HTTP requests
  • database queries
  • calls to caches etc.
• collects unhandled errors/exceptions and pickup host level metrics like JVM metrics in Java agent
• Stratosphere support both Elastic APM agent or OpenTelemetry client
• Stratosphere collects both metrics and traces via OpenTelemetry protocol
• For logs it collects, stdout and stderr, we format them in ECS (Elastic Common Scheme) for easier consumption
• Elastic search indices (can be found in Management > Kibana Dataviews):
  • Logs: filebeat-*
  • Metrics: metrics-prometheus.collector-default,metricbeat-*
  • Traces: apm-*
• Elastic agent seeks to improve over beats
  • https://www.elastic.co/guide/en/fleet/current/beats-agent-comparison.html