Configure breakers metrics in Vets API for external services

Last Updated: 17 Jul 2024

Breakers is a Ruby gem that utilizes the circuit breaker pattern and integrates with Faraday middleware to manage service interactions, allowing metrics collected by Breakers to be displayed in Datadog via StatsD. This page provides steps to collect the metrics for your external service and surface them in a Datadog dashboard.

Configure external services in Vets-API

These steps use VAProfile::AddressValidation Service as an example.

1. Determine which configuration (configuration VAProfile::AddressValidation::Configuration, for example) your service is using and add it to config/initializers/breakers.rb.

2. Add the appropriate STATSD_KEY_PREFIX variable above your service’s configuration definition. See example:

   1. RUBY

      STATSD_KEY_PREFIX = "#{VAProfile::Service::STATSD_KEY_PREFIX}.address_validation".freeze
      configuration VAProfile::AddressValidation::Configuration

   2. Note: In the case of our example, the STATSD_KEY_PREFIX value utilizes VAProfile’s service.rb, which in turn calls VAProfile’s stats.rb—the latter handles the incrementation of metrics. Your service’s setup may be different, so please adjust accordingly. For more examples showing how to increment metrics with StatsD, please check out this page. 

3. Update your service class. Address Validation example.

   1. Include Common::Client::Concerns::Monitoring in your service in order to make use of its with_monitoring method.

   2. Wrap your central method or methods in a with_monitoring do block.

   3. Example:

      RUBY

      class Service < VAProfile::Service
        include Common::Client::Concerns::Monitoring
      
        STATSD_KEY_PREFIX = "#{VAProfile::Service::STATSD_KEY_PREFIX}.address_validation".freeze
        configuration VAProfile::AddressValidation::Configuration
        ...
        def address_suggestions(address)
          with_monitoring do
            candidate_res = candidate(address)
      
            AddressSuggestionsResponse.new(candidate_res)
          end
        end

4. Once your PR is reviewed, merged, and deployed, metrics will eventually populate in Datadog, though depending on traffic, this may take some time.

Surfacing breakers metrics in Datadog dashboards

Once your service has been configured using the steps above, it can take hours or more than a day for all of its metrics to surface in Datadog. A metric won’t populate until it starts receiving traffic. For example, if your newly-configured service has not yet experienced a failure, its “failed” metric will remain unsearchable until at least one failure is logged.

Is your metric showing up yet?

1. Using the left-side menu, click on Metrics > Summary.

2. Use the Search box at the top of the page to search for your specific service, e.g. VAProfile_AddressValidation

3. The search results show the types of metrics currently available for your service, which may include “success”, “failed”, and “median”, among others. 

4. If the desired metric shows up, it’s available to add to a dashboard. 

Add metrics to your dashboard

1. Assuming your dashboard is already created, navigate to it by clicking Dashboards from the left-side menu and entering its name in the Search box.

2. Once on your dashboard, add a new widget to it by clicking the plus sign in the large lavender box entitled, “Add Widgets or Powerpacks”.

3. Click the Timeseries Widget from the right-side menu that appears.

4. In the popup modal, under “Graph Your Data”, enter your service, e.g. VAProfile_AddressValidation

5. The dropdown from the search box shows available metrics associated with the service. Click the appropriate one. 

6. Give your graph a title.

7. Click “Save”.

Help and feedback

• Suggest content changes to this page.

• Submit new Platform Website content.

• Get help from the Platform Support Team in Slack.

• Submit a feature idea to the Platform.