Polling the Benefits Intake API for Form Submission Status
Lighthouse offers a suite of APIs to access VA data. One of these APIs is the Benefits Intake API. The Benefits Intake API allows us to upload benefits claims documents to Central Mail. As we will explore in this guide, this API also allows us to get the status of previously-uploaded documents. Because validations occur asynchronously, we cannot always trust the 200 OK responses we get immediately from the Benefits Intake API; we need to periodically poll the API to learn what happened to our form submissions.
This document will give some background on the models and the sidekiq job that comprise the solution to the Benefits Intake API polling problem. Readers should feel empowered to implement the solution in their own code, registering their form submissions such that they get picked up by the sidekiq polling job.
The BenefitsIntakeStatusJob
Because the Benefits Intake API does not have a way to inform its clients of status updates for form submissions, and the fact that submissions can take hours to days to complete their journey through the various statuses (see Document Submission Statuses here), there is a daily job that records any status changes that have occurred since the last check.
Keen readers may wonder why we are polling instead of using a different architecture, such as webhooks. The short answer is that the Benefits Intake API does not support webhooks. However, there is evidence that webhooks had been considered and extensively worked-on.
The FormSubmission and FormSubmissionAttempt Models
The aforementioned daily job operates on FormSubmissionAttempts (defined here). A FormSubmissionAttempt is a state machine which moves between three possible states: pending, success, and failure. A FormSubmission (defined here) has many FormSubmissionAttempts.
Putting It All Together
If you’d like to have your form submissions tracked by the polling job, save a FormSubmissionAttempt with every submission. An example can be found here. If the FormSubmissionAttempt successfully saves, then the following day it will be queried along with all other FormSubmissionAttempts that are in the pending state.
Resubmitting Failed Submissions
The FormSubmission model has an encrypted form_data field, which can be used to re-process and re-submit the form, should the polling job report a failure.
The form_data will likely contain Personally Identifying Information (PII). Deleting successful instances of FormSubmission is recommended to reduce risk of leaking PII.
Using DataDog to Monitor and Alert
The BenefitsIntakeStatusJob uses StatsD Metrics to keep track of form submission status transitions. There is a DataDog Dashboard and a Monitor which goes to the #veteran-facing-forms-notifications channel on the OCTO Slack.
Help and feedback
Get help from the Platform Support Team in Slack.
Submit a feature idea to the Platform.