Critical Context
Currently, the log configuration present for each microservice is not suitable for a production environment.
- There are too many log messages present in the various files, this prevents us to connect the stack to Datadog.
- The log messages are written to files, this could cause disk pressure and forces us to mount a specific volume for logs, which might be huge, cf the point above
Request
We want to have a log configuration for our production environments and for each microservice.
- Such configuration should have the root logger level set to WARN by default.
- Loggers which need to have a level set to INFO should be explicitly set, such as the startup message. Debug/Info logs should be filtered out as much as possible as they are too many of them. For reference, the whole stack should log at most 100K per day.
- The log appender should only write to the console and not to files
- The log output should be in JSON
- As we're running the solution in Kubernetes, we would like to be able to reference a configmap for this configuration
- We would like to have a way to make this configuration hot reloadable (for example on log4j2: https://logging.apache.org/log4j/2.x/manual/configuration.html#Automatic_Reconfiguration), as it we can change the configmap to change log levels
- The proposed configuration would contain commented logger to uncomment in case of production incidents in order to help to diagnostic the issue
