Errors with In-Person Card Payments with Stripe Readers
Incident Report for Dripos
Postmortem

Summary

On Friday we had an incident that affected the usage of our in-person card readers for about half an hour. Customers trying to complete a transaction with a physical card reader would be presented with an error screen. This incident was caused by a problem with our payment provider Stripe, and while our servers and product were not offline, we were unfortunately at the mercy of our payment provider at that time. Luckily, we’ve spent a lot of time working on our offline payment system and will continue to do so that incidents like this will be less impactful for our customers.

Timeline

This incident spanned from 1:47 pm to 2:21 pm on Friday March 31st. All times are in eastern standard time.

1:47 pm - The first customer reports to Dripos that errors were occurring when trying in-person transactions with our Stripe-issued card readers

1:52 pm - Dripos confirms the incident is affecting multiple locations and starts reaching out to customers to make them aware

1:53 pm - Dripos reaches out to payment provider Stripe, who confirms that an issue regarding their card readers is occurring. Stripe tells us they are investigating the case and will update us with more findings.

1:57 pm -  Dripos informs customers via email that we are aware of the issue and are investigating it alongside Stripe

2:07 pm - Stripe reports that mitigation on their end is working and that they are beginning to see resolution with card readers

2:21 pm - After testing, Dripos confirms the incident has been resolved and lets customers know

Analysis

‌After being told by customers that payments were not processing with our in-person card readers, we immediately contacted our payment provider, Stripe, to see if a known incident was occurring. We got an instant reply that they were aware of an issue and were deploying fixes as we spoke. We started to reach out to customers telling them to switch to offline or manual card entry payments while the incident was ongoing.

We had minor issues with specific customers having trouble with manual card payments, but our offline payment functionality saved many customers from losing sales during this time. Dripos servers and apps were running as expected, and customers could still use every part of the product besides in-person payments.

The fixes on Stripe’s end were resolved earlier than we mentioned to customers. However, we wanted to ensure no customers were still being impacted before reaching out to customers. We have backup systems in place, like an offline order system that lets customers still be able to take payments, and we wanted to make sure the fix by Stripe was 100% in place before telling them to use in-person card readers again.

Prevention

We’re always upset when a problem occurs in any part of our product, but even more when it’s payments related. Payment provider incidents are always the hardest since Dripos and our product were working and communicating as expected, so the origin of the issue was out of our control. The best thing we can do in cases like this is to have backup forms for taking payments for situations like this.

We’re continuing to work with Stripe to help eliminate these issues and are thankful that the incident was resolved promptly. We are confident in Stripe’s commitment to fixing any problem with their product as they are one of the biggest payment providers in the world (powering companies like Amazon and Google).

We know that it’s sometimes not the easiest to be aware of an incident and know how to switch to a backup form of payment, so when incidents like this occur, quick communication with our customers is our priority. Moving forward, we’re working on two new features to help clarify how to operate when an incident like this occurs. The first is creating an in-app alert system that will pop up on the point of sale explaining that an incident is happening. The second is a system allowing customers to go into an offline payments state, which will store payments offline but keep point-of-sale functions working that are not affected.

Our sincere apologies for any inconvenience caused by this incident. Please don’t hesitate to contact our team with any questions or concerns about this incident!

Posted Apr 02, 2023 - 12:05 EDT

Resolved
We have confirmed with customers and our payment provider that in-person card payments with Stripe card readers are working again. Let us know if anything funky is still happening on your side.
Posted Mar 31, 2023 - 14:21 EDT
Investigating
We have been made aware that card readers from Stripe are causing errors when trying to process a payment. Please use manual card entry in the meantime.
Posted Mar 31, 2023 - 13:57 EDT
This incident affected: Point of Sale.