MVI

Master Veteran Index

Overview and Terminology

The Master Veteran Index contains the golden record for a VA-affiliated individual (not always a veteran) and links their identity across systems.

• In operations terms, the service is frequently referred to as PSM or PSIM (Person Services Identity Management)

• PSM is in turn a child application of IAM.

• Finally, while PSM refers to the identity database, http://va.gov accesses this information via VAAFI (VA Authentication Federation Infrastructure) which provides an authentication layer and SOAP API on top of PSM.

• If a veteran is not found in the Master Veteran Index, MVI makes a call to the DMDC (Defense Manpower Data Center) fallback DEERS (Defense Enrollment Eligibility Reporting System). DMDC is part of the DOD (Department of Defense)

Monitoring

MVI Grafana dashboard Authentication Dashboard

Banners

Any time Pagerduty has a status other than "active" for MVI a downtime banner is in place on the sign in modal. Vets-api gets the status from PagerDuty once a minute.

Impact

When MVI is unavailable, MHV premium users are unable to log in to Va.gov. ID.me and DS Logon users will not be able to access health tools or MHV MyHealtheVet Single Sign on, because we rely on MVI to tie the logged in user to their ICN (internal control number, the unique identifier used by MHV)

Integration Endpoints

• Integration Type: SOAP

• Endpoints:

• Error Indicator:

  • StatsD: api.external_http_request.EVSS/Documents

  • Prometheus:

  api_external_http_request{service:MVI}
  api_external_http_request_success_total{service:MVI}
  api_external_http_request_failure_total{service:MVI}
  

  CODE

  • Rails Log: ^MVI.*

Trust Chains

int.services.eauth.va.gov

   0 s:/C=US/ST=Texas/L=Austin/O=U.S. Department of Veterans Affairs/OU=AcS/CN=dev.services.eauth.va.gov/emailAddress=EauthAdmins@va.gov
     i:/DC=gov/DC=va/CN=VA Internal Subordinate CA 1
   1 s:/DC=gov/DC=va/CN=VA Internal Subordinate CA 1
     i:/DC=gov/DC=va/CN=VA Internal Root CA
   2 s:/DC=gov/DC=va/CN=VA Internal Root CA
     i:/DC=gov/DC=va/CN=VA Internal Root CA

sqa.services.eauth.va.gov

   0 s:/C=US/ST=Texas/L=Austin/O=U.S. Department of Veterans Affairs/OU=AcS/CN=dev.services.eauth.va.gov/emailAddress=EauthAdmins@va.gov
     i:/DC=gov/DC=va/CN=VA Internal Subordinate CA 1
   1 s:/DC=gov/DC=va/CN=VA Internal Subordinate CA 1
     i:/DC=gov/DC=va/CN=VA Internal Root CA
   2 s:/DC=gov/DC=va/CN=VA Internal Root CA
     i:/DC=gov/DC=va/CN=VA Internal Root CA

services.eauth.va.gov

   0 s:/C=US/O=U.S. Government/OU=ECA/OU=IdenTrust/OU=DEPARTMENT OF VETERANS AFFAIRS/CN=services.eauth.va.gov
     i:/C=US/O=U.S. Government/OU=ECA/OU=Certification Authorities/CN=IdenTrust ECA 4
   1 s:/C=US/O=U.S. Government/OU=ECA/OU=Certification Authorities/CN=IdenTrust ECA 4
     i:/C=US/O=U.S. Government/OU=ECA/CN=ECA Root CA 2
   2 s:/C=US/O=U.S. Government/OU=ECA/CN=ECA Root CA 2
     i:/C=US/O=U.S. Government/OU=ECA/CN=ECA Root CA 2

Client Certificate Verification

• Per communication with Dinesh Punyala on 9/2017, PSIM only requires that client certificates are VA-issued.

• However, per communication with Aaron Levy (and experimentation in all environments), VAAFI does check the Subject of client certificates; each Subject must be authorized on a per-operation basis. VAAFI appears to include an IBM DataPower gateway that enforces client authorization.

End-to-end Test

It is possible to perform an end-to-end MVI request via cURL. This validates overall connectivity as well as authorization of the client certificate. An SSL handshake or even a WSDL request is not sufficient to verify client authorization.

To perform a request via cURL, you'll need the private key and certificate for the environment, a sample MVI request payload, and to either be on the VA network/VPN or on the appropriate forward proxy for the environment in question. The request payload in question is a query only and safe to invoke; in staging it should return a valid result; in production it will return a valid but empty result as the test user in question does not exist in production.

Note the URL path varies in different environments.

Staging:

curl --header "Content-Type: text/xml;charset=UTF-8" --data @mvi_request.xml --cert /tmp/vetsgov-mvi-qa-cert.pem --key /tmp/mvi.qa.key https://sqa.services.eauth.va.gov:9303/psim_webservice/stage1a/IdMWebService -v

Production:

curl --header "Content-Type: text/xml;chars" --data @mvi_request.xml --cert /tmp/vetsgov-mvi-prod-cert.pem --key /tmp/mvi.prod.key https://services.eauth.va.gov:9303/psim_webservice/IdMWebService -v

Required Configuration

The following environmental parameters must be set to establish connectivity.

• MVI_URL

• MVI_OPEN_TIMEOUT

• MVI_TIMEOUT

• MVI_CLIENT_CERT_PATH

• MVI_CLIENT_KEY_PATH

• MVI_PROCESSING_CODE

Credstash stores the proper keys for connecting to MVI. Keys are copied to their proper locations on each instance, which must be one of:

• mvi.int.key

• mvi.pint.key

• mvi.preprod.key

• mvi.prod.key

• mvi.qa.key

Scheduled Down Times

• Development: On Demand

• Stage 1A: Mondays starting at 3pm EST

• Pre Production: Wednesdays starting at 3pm EST

• Production: The 3rd Saturday of the months starting at 3pm EST

Emails are sent to the MPI Stakeholders mailing group. Shawn Arnwine is the USDS PoC for that list. Outages are also broadcast via ANR's.

[TODO: Add Devops mailing list for ANR broadcast]

Service Level Agreement

Escalation Procedure

1. Check the MVI Grafana dashboard

• If a significant number of requests are succeding, but we still have a high error rate consider the posibility that only the DMDC fallback is unavailable. \

• If the DMDC fallback is the problem we should see more DS Logon errors in the Failed saml_callback by context graph

1. Check that the forward proxy has at least one connection to the MVI servers run the following query on prometheus and validate there is more then 0 backends up

• sum(haproxy_backend_status{proxy="mvi_back"})

1. Check that the health check is returning a valid result

• curl https://internal-dsva-vagov-prod-fwdproxy-2075821597.us-gov-west-1.elb.amazonaws.com:4434/psim_webservice/stage1a/IdMWebService?WSDL

1. Check that the MVI services are available from within the VA network with the domain name:

• nc -z services.eauth.va.gov 9303

1. Check that the application responds, this cert should be pulled from AWS Parameter Store under the key 

   /dsva-vagov/fwdproxy/<env>/mvi_key

• curl -v --cert-type pem --cert <mvi client cert> https://services.eauth.va.gov:9303/psim_webservice/stage1a/IdMWebService?WSDL

1. Contact the NSD (1-855-673-4357) to file a ticket for "PSIM":
   Configuration Item: Person Services Identity Management (Alt CI ID: PSIM)
   Group: PSM Application Support Team

2. If the outage is severe, contact the VAAFI/eauth production support team:
   Internal Customer Contact Information (Non SSOe): ITSC@va.gov or 855-673-4357 (Option 3)

Contacts

On the Va.gov side, the MVI integration is owned by the VSP Engineers team.

IAM Team:

Production Support:

• VAAFI/eauth: ITSC@va.gov or 855-673-4357 (Option 3).

• MVI proper: AITCSoSWeblogicSupport@va.gov

Other helpful contacts for navigating production issues:

• Johnson, Ric (AITC) Ric.Johnson@va.gov Sustainment Manager
  Health Division
  Enterprise Product Support
  Transition, Release and Support(TRS)
  Enterprise Program Management Office (EPMO)
  Office: (512) 326-7895 ; Mobile: (512) 364-2988\

• Madipadga, Reddy (SMS) Reddy.Madipadga@va.gov
  Systems Analyst for System of Systems
  Enterprise Product Support, Health
  Transition, Release and Support (TRS), Enterprise Program Management Office (EPMO)
  VA Office of Information Technology
  1615 Woodward Street, Austin, TX 78722
  Phone: 512-827-1926

• Patricia L. Britten, PMP Patricia.Britten@va.gov
  Systems Analyst, Enterprise Product Support, Health (Contractor)
  Transition, Release and Support (TRS), Enterprise Program Management Office (EPMO)
  VA Office of Information Technology
  1615 Woodward St., Austin, TX 78772
  Office: 512-326-6598
  GFE Mobile: 512-585-9286
  Cell: 512-619-1122

• Thomas Sapp Thomas.Sapp@va.gov
  Sr. Systems Engineer
  Phone (727) 269-1294

Contact History

Client certificate expiry 9/2017

• PSM: Savita Garg, Reddy Madipadga -> Dinesh Punyala Dinesh was able to confirm cert verification for PSM.

• VAAFI: Eduardo LaGuerre -> Thomas Sapp -> Aaron Levy, Vijaya Chenna Thomas, Aaron, Vijaya were all directly involved with provisioning new client certificate in VAAFI layer.

3/9/19 MVI was unavailable for 3 hours

• Called 855-673-4357 and asked for a ticket to be created. Later, it was discovered that the ticket was assigned to http://va.gov devops, which is unhelpful.

• emailed AITCSoSWeblogicSupport@va.gov, no response

• emailed Patty.Britten@va.gov, no response

• emailed "Johnson, Ric" Ric.Johnson@va.gov, Reddy.Madipadga@va.gov recieved a response 2 days later from Ric.Johnson@va.gov, stating they were unaware of any issues

3/29/19 MVI's connection to DOD's DMDC (Defense Manpower Data Center) fallback was failing

• We emailed Ric.Johnson@va.gov who responeded and forwarded to Patty.Britten@va.gov who suggeted we contact cory.chin@va.gov and Pamela.oreilly@va.gov who forwareded to Thomas.Sapp@va.gov

• Thomas.Sapp@va.gov

   responded that the issue was fixed

Help and feedback

• Get help from the Platform

• Create an issue ticket to suggest changes to this page

• Submit a feature request