Rate Limiting Guide in Vets API

A complete guide: when to add rate limiting and how to implement it.

Part 1: When Should Your Team Add Rate Limiting?

Rate limiting in vets-api is opt-in. Not every endpoint needs it. Use the guidance below to decide whether to add a Rack::Attack rule for your endpoint.

First: Understand the Two Types of 429s

Before adding rate limiting, distinguish between two different sources of 429 errors in vets-api:

Should You Add a Rack::Attack Rule?

Ask yourself the following questions:

1. Is your endpoint unauthenticated or lightly authenticated?

Unauthenticated endpoints are the highest priority for rate limiting. Without authentication, there’s no barrier to abuse. See representation_management/next_steps_email as an example — without throttling it functioned as an open email relay.

2. Does your endpoint trigger expensive downstream calls?

If a single request fans out to multiple upstream services (e.g. Lighthouse FHIR APIs), a high request rate can cascade into upstream rate limit exhaustion. Consider rate limiting to protect both vets-api and your upstream dependencies.

3. Does your endpoint accept file uploads or send external communications?

File upload endpoints and anything that sends emails, notifications, or triggers external actions should be rate limited to prevent abuse and resource exhaustion.

4. Has your endpoint experienced a traffic spike or near-DoS incident?

Most existing Rack::Attack rules were added reactively after incidents. Don’t wait for an incident — if your endpoint is publicly accessible and handles sensitive operations, add a rule proactively.

5. Is your endpoint part of a form submission flow?

Form submission endpoints (POST) are good candidates for rate limiting. A legitimate user submitting a form rarely needs more than 15–30 submissions per minute.

You Probably Don’t Need Rack::Attack If…

• Your endpoint is fully authenticated and only accessible to credentialed users

• Your endpoint is read-only with low computational cost and no upstream fan-out

• Traffic to your endpoint is low and stable with no history of abuse

• You’re seeing 429s that trace back to upstream services rather than inbound request volume

Quick Decision Reference

When in doubt, reach out to the Platform SRE team in #vfs-platform-support on Slack or open a support request. They can help review your endpoint’s traffic patterns in Datadog and recommend an appropriate limit.

Part 2: How to Implement Rate Limiting

Rate limiting is configured in config/initializers/rack_attack.rb using the Rack::Attack gem. There is no global rate limiting — it is added per-endpoint as needed.

When to Add Rate Limiting (Checklist)

Rate limiting should be considered when:

• Your endpoint is publicly accessible

• The endpoint calls expensive upstream services

• The endpoint could be abused to cause denial of service

• A Staging Review or Security Review requires it

Reference: The Security Review checklist includes “Rate limits defined” as a required item.

How to Add Rate Limiting

Add a throttle block to config/initializers/rack_attack.rb:

throttle('your_endpoint_name/ip', limit: 10, period: 1.minute) do |req|

req.remote_ip if req.path.starts_with?('/your/endpoint/path')

end

Configuration Options

What Happens When Rate Limited

• Returns HTTP 429 Too Many Requests

• Includes headers: X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset

• Response body: “throttled”

Part 3: Determining the Right Rate Limit

For New Endpoints (No Existing Traffic Data)

When launching a new endpoint, you won’t have production traffic data to analyze. Here’s how to approach rate limiting without historical data:

Step 1: Model the User Journey

Map out realistic usage scenarios. Your rate limit should accommodate the power user scenario with headroom.

Step 2: Find a Similar Endpoint for Reference

Step 3: Check Upstream Service Constraints

If your endpoint calls external services, their limits set your ceiling:

• Your rate limit ≤ Upstream service limit / Expected concurrent users

• Contact the upstream service team to understand their constraints.

Step 4: Start High and Plan to Adjust

Recommended approach for new endpoints:

# Phase 1: Launch with permissive limit (2-3x expected peak usage)

throttle('new_endpoint/ip', limit: 60, period: 1.minute) do |req|
req.remote_ip if req.path.starts_with?('/v0/new_endpoint')
end

Then follow this timeline:

Step 5: Add Monitoring From Day One

Deploy with DataDog monitoring so you can adjust quickly:

# In your controller or service

StatsD.increment('api.new_endpoint.request', tags: ["ip:#{request.remote_ip}"])

Step 6: Document Your Assumptions

In your PR, document:

• Expected user behavior and request patterns

• Similar endpoints used as reference

• Upstream service constraints (if any)

• Plan for adjusting limits post-launch

Example PR description:

For Existing Endpoints (With Traffic Data)

If your endpoint already exists and has traffic, you can use DataDog to make data-driven decisions.

Step 1: Analyze Expected User Behavior

Think through the user journey: How many times would a legitimate user hit this endpoint in a session? Is it called once per page load? Multiple times during form submission? Are there any frontend polling patterns?

Step 2: Check Existing Traffic in DataDog

Before adding rate limiting, query DataDog for current traffic patterns:

# Requests per IP per minute

sum:vets_api.requests{path:/your/endpoint/*} by {client_ip}.rollup(count, 60)

Look for:

• P95/P99 requests per IP per minute — what do normal heavy users look like?

• Max requests per IP — what do potential abusers look like?

• Distribution — is there a clear gap between normal and abnormal traffic?

Step 3: Start Permissive, Then Tighten

Step 4: Consider Upstream Service Limits

If your endpoint calls an external service (PPMS, Lighthouse, etc.):

• What are their rate limits?

• Your limit should be lower than theirs to protect the upstream service

Step 5: Environment-Specific Limits

You can exclude non-production environments from rate limiting:

throttle('your_endpoint/ip', limit: 10, period: 1.minute) do |req|
req.remote_ip if req.path.starts_with?('/your/endpoint') &&
!Settings.vsp_environment.match?(/local|development|staging/)
end

Part 4: Reference

Safe Starting Points

Existing Rate Limits in rack_attack.rb

Monitoring After Deployment

Set up a DataDog dashboard to track:

1. 429 responses — How often is the limit being hit?

2. Unique IPs hitting limits — Is it one bad actor or many users?

3. Requests just below limit — Are legitimate users getting close?

DataDog query examples:

# Count of 429 responses

sum:vets_api.response{status:429,path:/your/endpoint/*}.as_count()

# Unique IPs hitting rate limits

count_distinct:vets_api.requests{status:429,path:/your/endpoint/*} by {client_ip}

Safe Rollout Strategy

1. Start with a limit of 2–3x your expected heavy user (e.g., if you expect 10 requests max, set 30)

2. Deploy to production with monitoring enabled

3. Watch DataDog for 1–2 weeks to observe actual traffic patterns

4. Tighten the limit based on observed data

5. Document your rationale in the PR for future reference

Additional Resources

• Rack::Attack Documentation

• Security Review Template

• Product Development Checklist

• Sidekiq Enterprise Rate Limiting (for worker-level rate limiting)

Questions?

Reach out in #vfs-platform-support on Slack.

Questions? Reach out in #vfs-platform-support on Slack.