What happened and what we’re doing about it
We sincerely apologize for our recent outage: you trust us with your chats, your important documents, your cat gifs, your personal conversations, your system notifications, your internet memes – and we let you down.
Our team takes pride in making an important part of your (work) life, and we’re sorry.
Short version: the recent Mac client release which had the much anticipated “multiple account” feature also had a subtle reconnection bug that only manifested under very high load .
When a large network provider in the SF Bay Area had an issue Monday morning, it caused all of those clients to start reconnecting at once. This saturated our systems and prevented normal usage.
On Monday, we released an update to our backend systems, and Tuesday morning we released a new Mac app (v 3.3.1), both of which increased protection against this type of issue in the future.
We also fixed various other bugs related to reconnection in the Mac app that will prevent another connection overload like this one. And we continue to have teams building new, amazing technology that improves our system isolation, enhances our ability to do sophisticated load testing, supports even higher scale, and increases our server-side capability management (to disable misbehaving client functionality more directly, for example).
We never want you to be without HipChat. We fell far short of that Monday and are very sorry we let you and your teams down.
We just passed 6B messages delivered via HipChat, more than 2B of which have been delivered in 2015 – our platform is scaling and growing faster than ever thanks to teams like yours. We’re moving quickly to build a stronger HipChat as a result of the experience – thanks for your patience while we do.