Breakers Outages for EKS Environments
Introduction
This document details how breakers outages work in production in relation to the external services utilized by Vets API.
Resources
Breakers Dashboard (Move to Datadog currently in work. Link will be updated at a later time.)
Prepare to Run Rake Command
EKS Clusters (SOCKS PROXY required)
In the ArgoCD UI, go the desired environment cluster listed above.
Click into any vets-api-web pod, e.g. vets-api-web-696bf8799-pgpfn
Pointing to ArgoCD pods
Go to the terminal tab
Image of terminal inside pod
The following commands can be successfully ran inside the terminal.
List the Available Services
bundle exec rake breakers:list_services
e.g.
nonroot@vets-api-web-696bf8799-pgpfn:~$ RAILS_ENV=production bundle exec rake breakers:list_services
I, [2018-11-13T22:46:26.912071 #4133] INFO -- sentry: ** [Raven] Raven 2.1.2 ready to catch errors
/srv/vets-api/src/config/initializers/redis.rb:4: warning: already initialized constant REDIS_CONFIG
/srv/vets-api/src/rakelib/connectivity.rake:11: warning: previous definition of REDIS_CONFIG was here
Available Services:
["AppealsStatus", "Rx", "BB", "EmisMilitaryInformation", "EmisPayment", "EmisVeteranStatus", "EVSS/Claims", "EVSS/Common", "EVSS/Documents", "EVSS/Letters", "EVSS/PCIUAddress", "EVSS/GiBillStatus", "VHA_Access_PWT", "VHA_Access_SHEP", "VIC2", "GI", "HCA", "MHVAcctCreation", "MVI", "Preneeds", "SM", "Vet360/ContactInformation", "Search/Results", "OKTA", "CentralMail"]
Force an Outage of an External Service
⚠️ Forced outages do not recover automatically and can only be stopped by calling end_forced_outage ⚠️
bundle exec rake breakers:begin_forced_outage service=<SERVICE_NAME>
End a Forced Outage of an External Service
bundle exec rake breakers:end_forced_outage service=<SERVICE_NAME>
e.g.
nonroot@vets-api-web-696bf8799-pgpfn:~$ RAILS_ENV=production bundle exec rake breakers:end_forced_outage service=EVSS/ClaimsHelp and feedback
Get help from the Platform Support Team in Slack.
Submit a feature idea to the Platform.