Uploaded image for project: 'Nuxeo Platform'
  1. Nuxeo Platform
  2. NXP-28639

Improve Datadog metrics using tagging

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 11.1
    • Fix Version/s: 11.1
    • Component/s: Monitoring
    • Upgrade notes:
      Hide

      Many metrics names have been renamed in 11.1 in order to support tagging which is necessary for Datadog and Prometheus support. Visit the documentation for more information.

      Show
      Many metrics names have been renamed in 11.1 in order to support tagging which is necessary for Datadog and Prometheus support. Visit the documentation for more information.
    • Team:
      PLATFORM
    • Sprint:
      nxplatform 11.1.29, nxplatform 11.1.30, nxplatform 11.1.31, nxplatform 11.1.32
    • Story Points:
      0

      Description

      Today, Nuxeo publishes its Dropwizzard'smetrics to Datadog using the following addon:
      https://github.com/nuxeo/nuxeo/tree/master/addons/nuxeo-datadog-reporter

      To build a useful dashboard, metrics need to be manipulated using wildcards, for instance:

      # to aggregate the number of errors
      sumSeries(servers.*.nuxeo.org.apache.logging.log4j.core.Appender.{error,fatal}.count)"
                       
      # or to render the scheduled works in each workmanager pool:
      aliasByNode(servers.*.nuxeo.nuxeo.works.total.*.scheduled, 1, 6)

       

      But wildcards are not supported by Datadog the dimension for a metric should be expressed using tagging not using the metric name,
      we should rewrite the metric to:

      metric: nuxeo.log4j
      tagging: 
        server: <server-name>
        loglevel:<pool-name> 
      formula: sum:nuxeo.log4j{$var} by {loglevel}
      
      metric: nuxeo.works.scheduled
      tagging: 
        server: <server-name>
        workpool:<pool-name>
      formula: sum:nuxeo.works.scheduled{$var} by {workpool}
      

       

      Some investigation must be done on the addon to see if the latest [coursera lib|https://github.com/coursera/metrics-datadog] provides a way (maybe using DynamicTagsCallback) to create dynamically tagging based on the metrics name.

      Using tagging will reduce drastically the number of metrics in Datadog which can reduce the cost of monitoring.

       

       

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 2 days
                  2d

                    PagerDuty

                    Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.