Incident Report - Outages and Performance Issues 24/09/2020

On Thursday 24/09/2020 at approximately 10:54am Invoxy began experiencing database performance issues. 

This issue is causing errors while trying to perform many actions throughout Invoxy, including creating/editing time, invoices, pays and approvals. 

The Invoxy team are investigating as a matter of urgency and further information will be updated here when available. 

Update 24/09/2020: Following a restart of the app at 12:21pm NZT today, Invoxy performance has stabilised. We're continuing to monitor and investigate the cause of the incident.

Update 30/09/2020: Invoxy's performance has been back to normal since this issue was resolved on the day. We've identified that unusually high load on our mirrored database caused data replication to be throttled throughout the system, resulting in performance issues.

Steps have been take to reduce traffic to the mirrored database to prevent this occurring in future, and increased compute resources have been allocated so that it is more resilient to periods of high load.