Status: Resolved — 9:40 AM ET on April 30, 2026
An issue that affected multiple customers' ability to log in and launch applications was resolved at approximately 9:40 AM ET on April 30, 2026. Users who already had active sessions were able to continue working throughout.
Duration: 8:54 AM ET to 9:40 AM ET on April 30, 2026
Updates
- 9:40 AM ET — Resolved: Login and application launches were restored. Users were able to log in again.
- 8:54 AM ET — Active: Issue confirmed. Our team is investigating.
Root Cause Analysis Posted May 7, 2026
What Happened
On the morning of April 30, 2026, a defect in Venn's user provisioning logic caused our web servers to crash, which prevented users from logging in or launching applications. Users who already had an active session at the time remained logged in and could continue working. Service was restored at approximately 9:40 AM ET, and a fix addressing the underlying cause was fully deployed by 1:48 PM ET the same day.
Timeline
- 8:54 AM ET — Venn's monitoring detected the issue and our team began investigating immediately.
- 9:10 AM ET — Our team took manual control of traffic routing.
- 9:40 AM ET — Service was restored, and users were able to log in and launch applications again.
- 10:04 AM ET — The root cause was confirmed.
- 11:15 AM ET — We began deploying a targeted fix.
- 1:48 PM ET — The fix finished deploying across all servers.
Root Cause
The disruption was caused by a defect in Venn's user provisioning logic. Under specific conditions, this defect produced an error that crashed the servers responsible for processing login and application launch requests. Because our primary and backup systems shared the same underlying code, both were affected. We identified the cause, developed and tested a fix, and fully deployed it by 1:48 PM ET on the same day.
No data was accessed, exposed, or at risk at any point during this incident.
Next Steps
- Fix deployed: The defect has been repaired across all servers as of 1:48 PM ET on April 30, 2026.
- Expanded code review: We are expanding our code review process to automatically detect and flag high-risk patterns before they reach production.
- Proactive code scan: Our engineering team is reviewing our codebase to find and fix similar patterns before they can cause issues.
- Backup system improvements: We are reviewing how traffic is distributed across our systems and how backup systems take over, so they can reliably absorb load in future scenarios.
Comments
0 comments
Please sign in to leave a comment.