zuhaib

Details on the recent outages

By zuhaib | 3 days ago | 0 Comments

HipChat has been growing like crazy. A year ago we blogged about how we passed 1 Billion messages after 4 years. Today, we are over 3 Billion messages sent–and growing faster every month! However, over the past few weeks we have had some growing pains – we’ve suffered a few different types of service outage with HipChat, and felt like a blog post would be helpful to explain the issues and how are moving forward.

We understand that HipChat is critical for your team – we live in HipChat all day as well – so we know how crippling it feels to lose that connection to your team. We take this responsibility very seriously, and we’re ensuring that even as we continue to scale, your HipChat experience continues to be amazing.

Web App Troubles

Starting Oct 1st, we started to see higher than normal load on our web tier, which caused some load issues at that tier, which in turn triggered all web clients to suddenly reconnect and cause a spike across our whole system.  We added more capacity throughout Oct 2nd in the web tier to handle the extra load and made several code & configuration changes to optimize how we used one of our databases, Redis, to support the higher load, and declared the issue resolved on Oct 2nd.

The optimization addressed the load issue (yey).  However, along the way we inadvertently introduced two additional issues into our system. On Oct 7th, we discovered and resolved an issue that resulted in our Android clients making extra requests and DDoSing ourselves. On Oct 9th, we discovered and resolved a second issue, which resulted in bad cache data that locked out a small percentage of users from logging into HipChat.

SSLv3 “POODLE” bites HipChat

Many of you likely have seen the report going around of a security vulnerability with SSLv3.  As soon as we had confirmation of the vulnerability we rolled out a patched version of our server code to the whole system, including new front-end XMPP servers.  In the process of adding new front-end XMPP servers, our automated tool which manages our DNS records failed to update our domain with the new servers’ addresses.  When we removed the old servers from rotation this resulted in users being unable to login for 15 minutes while we manually updated our DNS records.

Going Forward

We take our commitment to you, our end users as seriously at 3B+ messages as we did at the first message. It’s unacceptable for us to have this much downtime, and while the system cause of each issue is different, the net result to you our users is often the same. We continue to invest heavily in platform, scale, performance and reliability projects–we have several big ones in the works and we hope to blog about them soon. We have an entire team dedicated to scaling HipChat further, and we are looking to scale our team as well – come join us!

Sincerely

Your HipChat Team

 

 

Ian Monroe

HipChat 2.2.1221 for Linux and Windows released

By Ian Monroe | 3 days ago | 2 Comments

A new week, a new build from Bamboo to release. This patch release for both Linux and Windows has all the fixes from last week’s release for Linux, plus the ability to click on @-mentions to open up a new chat.

Feature – click on @-mentions to start a chat with that person
Feature – XHTML-IM messages now supported in 1-1 chats
Fixed – Various issues with large log file creation
Fixed – show only offline presence icons for others when offline

Jeff Park

Meet the new iPhone and iPad apps: faster, better, iOS 8ier

By Jeff Park | 4 days ago | 0 Comments

Hot on the heels of our newly redesigned HipChat for Mac, the brand new HipChat for iPhone and iPad is out today. The new HipChat is more responsive, more beautiful and more connected, so you can get work done faster, wherever you are.

The first thing you’ll notice is the new design, which is lighter, cleaner, and easier to navigate. It’s also faster; responding more quickly to get you directly to the right conversation. And HipChat uses some of the great new features of iOS 8 to connect with the other apps on your phone.

So what’s new?

Beautiful and efficient new design

We’ve redesigned HipChat to make it easier to read information and faster to navigate between conversations. The main chat window has been simplified and the chat windows have been improved to fit more context on your mobile screen.

We’ve also brought HipChat more in tune with iOS aesthetics, all while fitting beautifully within Atlassian’s Design Guidelines.

A fresh new chat view makes it easy to read your messages.A new home screen makes it easy to find your conversations.

Share to (and from) HipChat

The new HipChat makes it even easier to share with your team. 

You can now select any file from any other app and post them straight into HipChat, and vice versa. Let’s say there’s a photo you want to share with your team on HipChat. Simply click the share icon button, select HipChat, and choose which chat you want to share the photo with. Likewise, if you have a file in HipChat you want to share, just click the share icon inside HipChat and select where you want it to go.

Share on HipChat easily.

Interactive notifications

Now you can be even more efficient in HipChat thanks to our new interactive notifications. When you get a notification from another teammate or chatroom, you can pull down to read the whole message.

You can also respond directly from the notification. Click the pencil icon to type a reply. Or ‘quick reply’ with a thumbs up, thumbs down, OK or question mark. Interactive notifications let you carry on multiple conversations without constantly switching tabs. Get your point across without leaving your current chat.

Stay on top of your messages with rebuilt notifications.Quick reply to messages with our interactive notifications.

There’s a lot of other cool stuff in this update, so grab the new version on your iPhone or iPad. If you don’t have the app yet, you can download it for free at the App Store.

As always, we want to hear your feedback. Let us know what you think!

Ian Monroe

HipChat 2.2.1215 for Linux released

By Ian Monroe | 1 week ago | 0 Comments

1215 was a momentous year: the Mongols torched Beijing and the first Magna Carta was written. And now 1215 is also the Bamboo build number and the patch version of a rather small Linux-only update for HipChat!

Fixed – Various issues with large log file creation
Fixed – dependency problems for RPM

Josh Devenny

HipBots are here to work for you!

By Josh Devenny | 2 weeks ago | 0 Comments

At our recent annual conference, Atlassian Summit 2014, we talked a LOT about robots…

We’re adding them to many of the Atlassian products, and HipChat has had them for a long time. We thought you might like to see some of the Bots we’ve been playing around with in our Atlassian HipChat rooms!

Introducing HipChat Bot Lab

Bot Lab is a place where we will add HipChat Bots that are ready to be put to work, but aren’t ready for primetime. We’re going to be taking lots of diagnostics on the Bots, and making sure that they’re performing their duties correctly. They’ll be put through some rigorous tests, and those that prevail will be promoted to primetime – a listing on the HipChat Integrations page on HipChat.com!

We’d love for you to meet the very first class of Bots – Sassy, Karma, Standup, Mailroom, Hearsay, Untappd and Alias! If you have a Bot Building Degree (some Javascript knowledge), you can even build your own! While the Bots are in the Bot Lab, they are not officially supported. If you find that they need any tweaks, maintenance or new parts, get in touch with us via the Bitbucket repositories (links on the page), or email us.

Let’s see how many of them can pass the tests and graduate!

Take me to the bots!