Over the last few days, some premium email users notified us of problems with their email accounts. Here is a statement from our service provider to explain what happened.
Previous updates have been focused on resolution status as we worked through issues from the evening of Wednesday, July 12, 2017 through today, Saturday, July 15. We have compiled a preliminary view into the causes behind the system issues this week, and will have a final report available in the coming weeks. We recognize that these issues were significant for many of our customers, and we apologize for the difficulties you have experienced.
Here is a summary of what happened:
– We identified an issue with system performance during the week of July 3. While minor, we were concerned that it had the potential to grow worse over time. After discussing the issue with our vendors and consultants, we elected to perform proactive maintenance designed to double our total capacity. The additional capacity would provide plenty of headroom and prevent the issue.
– On the evening of Wednesday, July 12, the capacity was ready, and engineers initiated the process to rebalance users across the new hardware.
– An unknown bug associated with moving users caused high system load across affected systems, and impacted only those users who had been moved.
– During the time of impact, when an affected user tried to access their mailbox, they would have seen access errors, incomplete message listings, or other errors. Mail delivery was also slowed for a time, and this would have affected a broader set of users.
– Email messages were not lost or destroyed, but would have been inaccessible by some users during the time of impact.
– Our efforts through the end of the week of July 10 and into this weekend have been focused on reducing and shifting load, and then preparing to automatically clean up side effects where possible.
The high system load directly caused by the bug is now resolved, and performance and access have improved. We are in the process of preparing our systems to do a set of automated fixes for issues caused as side effects—the most serious of these are messages or folders that appear to be missing, but are actually present.
Please contact us if you have specific questions or concerns about your users.
We know that the email system is important to you and will continue to provide updates when we have more to share.