Reflections on Our Latest Blackout
This one goes out to our Austin-Area colleagues. Around the office (and over the phone) we have been discussing the events of the past week or two – finally power is restored across town after a classic February cold-snap. It’s true that the ice storm we saw recently was unique, but it wasn’t totally unheard of. The storm we faced two years ago was much worse, overall in terms of placing strain on the state’s energy grid.
What made this storm different was not that the grid failed, but that the ice accumulating on trees caused limbs to crash to the ground, taking power lines with them. The many small fires (literally, in some cases) made it more difficult for energy providers to keep up with outages. Rather than having a large problem to focus on, they had to deal with each outage individually, manually repairing power lines and clearing downed trees. Often, only hours after restoring service to a line, another limb would come crashing down, taking out the newly restored line. So people who’d just had their power restored would once again lose it, and end up at the back of the queue for service.
This is not something that Austin Energy could simply prevent, and no one should really hold them accountable for the fact that these small outages occurred, but, as some of us sat in the dark on Tuesday, Wednesday, and Thursday, we realized there are some takeaways from the way they responded that we can apply to our own lives – and businesses.
1. Have Redundancy Systems in Place
If something is critical, you need to make sure that you have a backup. This applies to your hard drive as well as processes. In Austin Energy’s case, some could argue that with a real “grid,” one interrupted line should not mean a house loses power down the line. We are not electricians, so can’t say for certain how efficient redundant electrical lines would really be, but it certainly got us thinking about redundant systems in our own life.
On a personal level, some of us own generators or solar panels, home redundancy systems to ensure that if one line of power fails, our houses can still stay warm. We back up our computers to a hard drive and to the cloud, so that if the hard drive is destroyed or the cloud is hacked, it still exists somewhere. We recognize that it is important to not let one failure derail our entire system.
Redundancy in Software Development
This applies to many realms of life and industries – including software development. When building a development team, you don’t want to base it around one star player, you want a strong team that will survive even if a given individual suddenly quits or is unable to communicate without power. We see this time and again working with our customers. They say how critical it is to have their own people “own” some piece of their product, so they keep that piece of code to themselves. But then they put it in the hands of one or two people. This introduces enormous risk when one or both move, or change jobs, or (in one notable case) are bitten by a rattlesnake (he’s fine now, but was out for several scary days). Great teams are resilient and continue to produce through adversity.
Redundancy is critical in other areas as well including data and connectivity. You cannot develop software effectively without these. As mentioned, it’s important to keep your data in multiple safe places as well. No one hard drive, or system, should be responsible for carrying the weight of your business or your code. You may use a RAID array – but what if the building burns down at night? There are several great tools and storage options available through cloud offerings for software development that makes this easy to address. Connectivity is much more challenging. By the same token, having teams in multiple areas reduces your risk as it is unlikely that every area would face connectivity issues at the same time. This is one reason that Calavista provides teams from all over the world.
2. Set Proper Expectations
As I said, it’s not Austin Energy’s fault per se that this storm happened and destroyed so many power lines, – we have to give them that. However, one area where they did fail was properly communicating to the public how long power would be out.
It is understandable that many small outages caused by physical damage would make it difficult to give a broad, accurate estimate of when power would come back on for everyone, but if you do not have enough information to make a judgement, it is important to be upfront about that, rather than throwing a dart at a wall. If people knew they would be out of power for days, they would behave differently than if they expected it to come on in just a few hours.
Setting Expectations as a Business
Likewise, we recognize how important it is to set proper expectations for development timelines and budgets. At Calavista, we have a thorough estimation process that we go through with every potential client. This requires more labor than most people put in upfront, but we believe it is the only way to give an accurate estimate and set realistic expectations. Sometimes, we’ll be told that our time estimates are higher than a competitor’s. That implies they either think that there’s less to be done than we do, or they agree on the amount of work to be done, but just think it can be done more quickly. When we ask for clarification, more often than not the customer tells us the competitor didn’t give them a breakdown of the work as we did – just a total time. And when those projects go to the competitor, they generally fail.
The point is, without a mature and realistic estimation process, you might as well throw a dart at the wall, in which case you will more than likely end up overpromising. Overpromising is how you set up your clients to be disappointed and to not trust you, undermining your relationship early on. It is important to not only gain clients, but to make them happy and keep them happy while you work together. Taking time to understand the situation, make detailed estimates, and clearly setting expectations in the beginning can have a massive payoff over the life of a project – and over the hopefully longer life of the client relationship.
As the work progresses, we continually provide visibility into the status and set expectations properly with the customer. No one likes surprises in business – you want to know what is coming and when so that you can plan accordingly. There is no value in delivering early if proper expectations are not set.
3. Communicate and Own Failures
This point is closely related to setting proper expectations but deserves its own segment for how difficult it can be sometimes. It will also be an important step if you fail to set proper expectations. For example, during the power outages this past week, many Austinites were told that their power would be turned back on soon, then nothing. Then again, a message that it would be on soon, with nothing. This leaves customers frustrated and distrusting.
How to Successfully Fail as a Business
If you fail, customers are going to be frustrated, period. If you get out in front of it, own it, apologize for it, you may be able to avoid losing their trust. If you notice you are running behind, burning through the budget faster than expected, or are on track to disappoint in any other regard, it is best to get ahead of it and notify your customer. Most customers, if you actually ask, will say something like, “I can live with a delay. I can’t live with not knowing about the delay until it’s here.” Try to work together to understand or explain how this happened and hopefully this will allow you to point to ways to avoid it happening again. It’s amazing the impact that owning a failure, and then inviting the customer into the review and lessons learned process as part of the team, can have on a relationship.
Conclusion
At the end of the day, we are not electrical or civil engineers, and don’t understand quite what goes into maintaining a complex energy network for one of the biggest cities in the nation. However, we do understand how to be resilient to unforeseen challenges. Having a resilient business model means many things. It means being able to scale and tackle challenges on the business side of things-declining business, recession, inflation, what have you- but also in the real world. We are not untouchable by disasters like epidemics, storms, power outages, yet Calavista’s operational model means we suffer relatively little when touched.
We have taken great care to build a model that is robust, redundant, and resilient. One, where, even if we cannot make it into the office, cannot safely meet face to face, our business is not affected. When COVID burst upon us, an email went out advising people to not come to the office for awhile. That’s all it took – the infrastructure to support remote work had been in place for years and was regularly exercised. Likewise, when some team members were taken off-line when their country was invaded, others in different areas seamlessly picked up the slack. When a large hosting provider suffered an outage, we shifted easily to another. Work can stay on track and keep being done through many challenges, even disasters. For work to grind to a halt at Calavista, means something incredibly serious, and totally unexpected, would have to occur. Maybe a meteor. You can’t plan around everything.
