Breakers Outages for EKS Environments
Last Updated: December 3, 2024
This document details how breakers outages work in production in relation to the external services utilized by Vets API.
Resources
Steps to force an outage
Pull up the terminal in ArgoCD
EKS Clusters (Only accessible behind the VA network [AVD/CAG/GFE]):
In the ArgoCD UI, go the desired environment cluster listed above.
Click into any vets-api-web pod, e.g. vets-api-web-696bf8799-pgpfn
Pointing to ArgoCD pods
Go to the terminal tab
Image of terminal inside pod
List the available services
bundle exec rake breakers:list_services
e.g.
nonroot@vets-api-web-696bf8799-pgpfn:~$ RAILS_ENV=production bundle exec rake breakers:list_services
I, [2018-11-13T22:46:26.912071 #4133] INFO -- sentry: ** [Raven] Raven 2.1.2 ready to catch errors
/srv/vets-api/src/config/initializers/redis.rb:4: warning: already initialized constant REDIS_CONFIG
/srv/vets-api/src/rakelib/connectivity.rake:11: warning: previous definition of REDIS_CONFIG was here
Available Services:
["AppealsStatus", "Rx", "BB", "EmisMilitaryInformation", "EmisPayment", "EmisVeteranStatus", "EVSS/Claims", "EVSS/Common", "EVSS/Documents", "EVSS/Letters", "EVSS/PCIUAddress", "EVSS/GiBillStatus", "VHA_Access_PWT", "VHA_Access_SHEP", "VIC2", "GI", "HCA", "MHVAcctCreation", "MVI", "Preneeds", "SM", "Vet360/ContactInformation", "Search/Results", "OKTA", "CentralMail"]
Force an outage of an external service
Forced outages do not recover automatically and can only be stopped by calling end_forced_outage
bundle exec rake breakers:begin_forced_outage service=<SERVICE_NAME>
End a forced outage of an external service
bundle exec rake breakers:end_forced_outage service=<SERVICE_NAME>
e.g.
nonroot@vets-api-web-696bf8799-pgpfn:~$ RAILS_ENV=production bundle exec rake breakers:end_forced_outage service=EVSS/ClaimsHelp and feedback
Get help from the Platform Support Team in Slack.
Submit a feature idea to the Platform.