Elena Gorman

Guest Post: Server Density’s Ops War Room

By Elena Gorman | 3 months ago | 2 Comments

This is a guest post by David Mytton, founder of server and website monitoring product, Server Density. He has been programming for over 10 years and has grown Server Density to now processing over 30TB of incoming data per month. You an email him on david@serverdensity.com or follow @davidmytton

Whether your company works entirely from an office, are all remote or a combination of the two, using HipChat to help run infrastructure operations is a common way to organize and coordinate teams.

At Server Density - a server and website monitoring service, where the majority of the team work remotely from the UK and Europe, we use HipChat extensively.

For us, HipChat acts as a news feed with events being piped in from all the services we use - GitHub, ZenDesk, JIRA, deploys, new signups and ops alerts. It’s this final one that I want to focus on in this post.

The ops war room

There is a lot of activity going on during each day and so our main room gets quite noisy with commits, builds, customer upgrades, etc.

This works well for staying up to date with what is going on but when there’s an infrastructure incident e.g. an outage, the response team immediately switch to sterile cockpit rules -  only essential communication is allowed.

To achieve this, we have a dedicated HipChat room which is used only to discuss ongoing incidents. Only critical information from our alerting system gets piped into this room, which is combination of Server Density’s own HipChat integration and alerts handled by PagerDuty. This has a number of advantages:

  1.  We have an easy way to see chronological timeline of exactly what has been happening for the first responder to triage and additional responders to review to get up to speed.
  2.  We have a single place to communicate for the responders.
  3. We have a permanent record of what happened for the followup post-mortem.

This seems to be a similar pattern amongst other companies. The privacy oriented search engine DuckDuckGo, users of both Server Density and HipChat, also have alerts piped into a single Ops room which is used to help increase sysops transparency across the whole team.

Communicating during outages

Every incident starts with a first responder doing some initial investigation to diagnose the issue. This involves them joining the Ops War Room, triaging the alerts and then generating a new incident tracking ticket in JIRA.
The most important thing is to keep a close-to-real-time record of what is being done and who is doing it. This helps any additional people who might join later, it helps to review what has already been done and really helps with the post-mortem analysis so you can review how to improve responses in the future.

We record events in several ways:

  1. Quick communication and initial investigation is done through HipChat text chat in the Ops War Room.
  2.  Actions performed e.g. commands run, failover scripts executed, etc are logged in JIRA with the command line and output. It’s important to know what commands were run so there’s no duplication, they can be considered for automation next time, and we have a history for review.
  3.  If there is a long running incident or there is some complexity, text based chat can become time consuming. Instead, we often switch to video conferencing so we can talk through what’s happening and coordinate individual responders. Even when there’s not much to say and it’s mostly silence when people are working, video is a good way to work with people remotely. We have been using Google Hangouts for this but are now testing the HipChat Video features.

A gathering place

The linking factor between many users of HipChat is how it acts as a gathering place for all teams. At idio, they are also using HipChat to help fight fires with the use of Airbrake to make them aware of code level exceptions, but it also brings together the dev teams with build events from Janky and Hubot.

The combination of developer and operations teams, with the ability to pipe events in real time and access crucial tools like video chat in a single location helps to improve ops response times, which all leads to better uptime – something which customers really notice!

Pete Curley

New HipChat app icons, take two!

By Pete Curley | 3 months ago | 23 Comments

After months of hard work, our team was really excited to announce 1-1 video chat and screen sharing for all of the apps. However, that news was completely overshadowed by another change: a new logo and app icon.

This is our new logo and the controversial OS X app icon we released a few days ago:

New logo Controversial OS X icon

The feedback was as plentiful as it was negative. 

There were hundreds of tweets about how ugly the icon was. Hundreds.

The weird thing is, we agree. So how did this happen?

The original goal of the new logo was to align it with the rest of the Atlassian tools. I think we achieved that and came up with a solid logo. The problem came with the implementation of the app icons themselves.

You could look at the feedback and say “people hate change” which is often true. But the problem here was that a large part of our team feels the exact same way as our users. There were two schools of thought when it came to the icon:

School #1 – “Consistency means the icon is the same on every device.”

School #2 – “Consistency means it pays homage to the style/design while respecting the look and feel of each OS.”

We fought. We argued. We gave each other the silent treatment, then argued some more. In the end, we chose option #1. But hey, if it sucks, we can fix it. We have zero problem admitting we made a mistake.

We made a mistake. The app icon sucks. Today we’re fixing it.

Lesson learned: respect the design guidelines and recommendations put forth by each operating system, even if you have to sacrifice your own internal guidelines. No one wants to see an OS X icon that looks like it belongs on iPhone, and people certainly don’t want to see an Android app with an iOS style icon. The part that made me both happy and sad was that people on Twitter proposed an alternative design, and it’s almost the exact app icon that we decided not to use in the first place (and I have the PSD’s to prove it!).

Here’s the the new app icon we’re releasing today for the desktop apps:

The new NEW OS X icon

It may seem that having hundreds of tweets shitting on your icon is a bad thing. Sure – it bummed us out, but it means our customers are passionate and they fucking care. It also means our team members that argued for the ugly icon have to buy the rest of the team a big box of apology donuts.

So let us know what you think of the new icons. I can’t promise we won’t make mistakes in the future, but I can guarantee we’ll own up to them and make things right.

Jeff Park

HipChat video and screen sharing are here

By Jeff Park | 3 months ago | 68 Comments

HipChat 1-to-1 video chat and screen sharing are now available on all of our apps. Start a video chat and share your screen with anyone on your team.

HipChat Video

Here’s what we’ve got waiting for you in our latest release:

  • Start and receive 1-to-1 video or audio calls
  • Share your screen with co-workers
  • A sweet new logo

Please note: If you’re currently using HipChat under our Free for 5 plan, you’ll be able to try video and screen sharing free for 30 days. After that, you’ll need to upgrade to a paid account to keep using the new features.

If you’re already on a paid account, HipChat video and screen sharing will be available at no extra cost.

Voice or video – it’s your call

Sometimes video is worth a thousand words. To start a video chat, open up a 1-to-1 chat with a co-worker and click the video icon. Want to start an audio call instead? Just click the phone icon. Easy.

Sync up with screen sharing

Get on the same page (literally) by sharing your screen so you can work together on any project. Just click the screen share button in any video chat and build genuine consensus to get your work done quicker.
HipChat Screen Sharing

Upgrade your app

HipChat video and screen sharing run natively on all our apps. Make sure you update your HipChat to the latest version. Check here to find which app versions support video.


Lastly, thanks for being dedicated HipChat users, and enjoy! We could not do this without you guys. awthanks

If you run into any issues or have feedback, don’t hesitate to let us know.

Garret Heaton

Hey, we’re changing our terms of service

By Garret Heaton | 4 months ago | 0 Comments


We can see from the feedback we’ve gotten that we need to clarify exactly what is changing with the new Terms. There is confusion around the last item, 1-to-1 chat history.

First, let us clarify that there is no way for an admin to access 1-to-1 chat history through HipChat itself.

In the past, our terms prevented us from releasing any 1-to-1 chat transcripts to our customers, regardless of the reason, without a subpoena. This caused a great deal of trouble for companies who use HipChat for business communication and would ordinarily have the rights to view that communication, just as they would with a service like Gmail. Many companies need access to all the conversations that occur through their systems for regulatory or legal reasons.

The new terms allow customers to request chat history from us, and allow us to provide it, if the customer has the right to view the communications of their employees. Those rights are very often granted through a company’s employee handbook or policies. So, for chats occurring after the terms become effective (May 27th at the earliest), if an account owner contacts us and requests their 1-to-1 chat history these new terms allow us to provide it to them.

We hope this helps clarify, and if you have more questions, please let us know at customer-agreement@hipchat.com.

It’s been two years since HipChat joined the Atlassian family, and we’re bigger and better because of it. Of course, when a product grows this quickly there are always lots of big changes going on. We need to rewrite certain bits of code to make them faster, change the way our servers run to make them more reliable, improve the way our team is organized so we can keep it productive, and implement new ways to support our users.

Today we’re announcing a change that many of you have asked us for; bringing HipChat under the same terms as other Atlassian products (currently the Atlassian Customer Agreement and the Atlassian Privacy Policy – together referred to as the “Atlassian terms”). No, seriously, we get asked about this a lot. Unfortunately we didn’t think through the original HipChat terms as much as we should have, and they have caused considerable pain for many users over the years. (Note to other startups: get someone to look over your terms because they’re tough to change!) Anyway, we know this stuff can be boring, but it’s important, so we hope you’ll take the time to read it.

Here’s an overview of the major differences between the HipChat terms and privacy policy and the Atlassian terms that will replace them:

  • The Atlassian terms are better for your company. The Atlassian terms have the structure and provisions that many businesses expect, which larger companies may appreciate. These include more specific termination provisions (rather than the right to terminate in our sole discretion), as well more favorable liability limitations for paying customers. Atlassian’s liability for paid accounts is now capped at amounts we’ve received from you in the last 12 months, rather than $100. In addition, the Atlassian terms include liability caps and disclaimers for customers, rather than being only in Atlassian’s favor. To accommodate our increased liability to paying customers, our liability for free products is being reduced to $20.
  • The Atlassian terms require binding arbitration for disputes. As HipChat grows, we need to have a cost-effective alternative to going to court. Our terms now mandate that we resolve disputes via arbitration. We expect binding arbitration to help contain legal costs and offer a faster path to resolution for both parties.
  • The Atlassian terms include a more comprehensive privacy policy. The Atlassian Privacy Policy covers the information we collect and how we use that information in much more detail. For example, it includes an explicit discussion of how we collect analytics information to improve our products and services. This analytics information could include elements of user content related to the function being performed.
  • The Atlassian terms allow companies to access 1-to-1 chat history (for future chats, not retroactively). The Atlassian Privacy Policy also removes a HipChat restriction that has caused a lot of confusion for business customers. Under HipChat’s support documentation (which is referenced in the HipChat privacy policy), HipChat administrators cannot view other users’ 1-to-1 chat history or the files that were shared. In many cases, this is inconsistent with an employer’s policy about employee communications occurring in the workplace, which employers typically have the right to access. Under the Atlassian Privacy Policy, HipChat administrators will have the right to access all information in the HipChat account they manage, including 1-to-1 chat history and files shared in those 1-to-1 chats. The HipChat-Specific Terms require customers (e.g. the account holder) to secure all required consents from users to allow for this level of access. Note that this change does not apply retroactively; 1-to-1 chats occurring before the Atlassian terms become effective are still covered by the prior HipChat policies.

The Atlassian terms will be effective starting May 27, 2014 for free accounts and new purchases. For existing subscriptions, the new agreement will take effect on your first renewal after this date. If your subscription is set to auto-renew, and you are not willing to accept the terms of the new agreement, please make sure to cancel prior to your next renewal.

Adopting the Atlassian terms will bring HipChat in line with the other Atlassian products you might be using and will help us stay focused on building a great product for you. If you have any questions you can direct them to customer-agreement@hipchat.com.


Jonathan Nolen

HipChat is Hiring!

By Jonathan Nolen | 4 months ago | 1 Comment

As you might have heard, we just opened a brand new office in Austin, Texas! And we’re looking for tons of new people to help us fill it up!

We’re growing every part of the HipChat team. We’re looking for developers of every stripe: iOS, Mac, Windows, Android, web, backend and ops devs. But that’s not all! We’re also looking for marketers, QA, product managers and community managers.

Most of these roles are slated for Austin, but part of our team is in San Francisco as well. So let us know if you’re in SF. If you live elsewhere, no worries! Naturally, we’d help out with relocation expenses (though you must be eligible to work in the US).

We all share the belief that HipChat brings teams together to do amazing work. HipChat is already helping thousands of the brightest teams around. Come help us bring HipChat to every team.

You can find all the details http://hipchat.com/jobs/.