A middle-of-the-night phone call is never a good thing – especially when the director of technology operations is on the other end. It was 2:00 a.m. in the summer of 2003 when I was abruptly awakened from my phone’s vibration.
My nightmare started as the director of technology operations reported that the system was down with no resolution in sight.
A company system outage is comparable to cutting off blood flow to the brain. When the system is down, there’s no cash and the business starts to die. No matter the size or stature of a company, technology leaders constantly carry the fear that even the smallest system outage could seriously damage their work. While this fear is hidden deep inside the psyche, it’s a reality that all tech leaders learn to live with.
My system outage was no different from eBay’s in the summer of 1999. The eBay auction site suffered from a series of system outages – the longest outage lasting 22 hours.
That outage cost eBay $5 million of transaction revenue. This $5 million may sound like a lot, but, in reality, it was nothing compared to the $4 billion drop in the company’s market value as the result of the outage.
WHAT ABOUT NOW?
Almost two decades later, and technology experts are still experiencing major complications in their systems.
In July 2015 alone, recent outages and system failures have affected the New York Stock Exchange, United Airlines, Department of State’s Visa system, Apple’s iStore, and – most notably – the Royal Bank of Scotland’s IT systems, which reported that a half-million financial transactions vanished from the system due to an unknown error.
They say “time heals all wounds,” but system outages may be the exception to this rule, as effects can be severe.
“For the Fortune 1000, the average total cost of unplanned application downtime per year is $1.25 billion to $2.5 billion,” says Stephen Elliot, IDC Analyst. “The average cost of a critical application failure per hour is $500,000 to $1 million.”
We are still not immune to these outages and we must take great care in avoiding these issues, or risk losing time, money and business.
WHAT IS HAPPENING?
Today’s systems are growing exponentially more complicated. Rising demand, volumes of aging data, patchwork of software, and network infrastructure each impact a system’s complexity and deployments. IDC estimates that the average amount of monthly deployments will double in two years.
To combat the case of system failures, technology leaders must adjust to an era of instant consumption – the “have to have it now” era. The world is no longer satisfied with singular massive updates to their systems every 12 or 6 months. Rather, we need to “deploy on demand,” where software can be updated several times per day with 100% resilience.
In other words, we need DevOps.
THE WAVE OF CHANGE IS HERE
Simply described, DevOps is the collaboration and communication between software developers and technology professionals in the IT value chain to deploy software to customers. Gene Kim, author of The Phoenix Project refers to DevOps as “the outcome of applying Lean Principles to the IT value stream.”
To achieve greatness, DevOps demands leadership vision and involvement. This requires sponsorship, so operational and cultural norms can change. It’s likely that your company will need to incorporate all of these changes to ensure long-term success.
DevOps is successful because it dramatically reduces a company’s operational risk by creating conditions that advance company culture, interactions, and tools.
Imagine a world where product, development, QA, infosec, and operations are orchestrating together to deliver business value at the fastest pace possible in an “IT value stream.” And fast execution isn’t the only benefit here – the process also has high predictability and low risk. This symphony of establishing a reliable flow across the organization – along with cultivating the right culture – is the foundation on which change can be made.
In May 2011, LinkedIn’s valuation doubled to $9 billion on its second day after IPO. With the stock soaring and a flood of new users flocking to the professional social networking site, LinkedIn was Wall Street’s golden child. Kevin Scott, LinkedIn’s top engineer didn’t feel as confident. Scott knew that the system and its engineers were being crushed by it’s own technology infrastructure inhibiting growth.
In a bold move, Scott launched Project InVersion, an initiative where all new feature development for LinkedIn stopped so that every engineer focused on rebuilding its core technology infrastructure. “You go public, have all the world looking at you, and then we tell management we’re not going to deliver anything new while all of engineering works on this project for the next two months,” Scott says. “It was a scary thing.” This work centered on LinkedIn’s ability to build out DevOps so that it could scale and accelerate while eliminating technical risks.
This resulted in extending the company’s deployment capabilities so that it can deploy changes at a moments notice at any time of day. Further, it helped support the growth of LinkedIn’s user base to over 364 million members and a market cap of $28 billion.
THE THREE LEADERSHIP MUST-HAVES
Gene Kim describes DevOps as a “philosophical movement.” And he’s right. As DevOps garners more attention, experts are deliberating its “best practices” and developing tools to support those practices.
To enable success, I have found there are 3 “musts” that leadership should have when launching a DevOps movement. These “musts” are based on the premise that DevOps requires disruptive leadership.
1. Executive Involvement
Leaders, including the CTO and the CEO, must work together to make DevOps a strategic priority. Just as soldiers, airplanes, satellites, and technology are strategic assets for the military, technology leaders need to utilize DevOps assets to achieve their goals. Leaders should engage with business counterparts when harnessing the strategic value of DevOps.
Successful DevOps transformations require executive participation and understanding. With the DevOps’ unification of the technology value stream, it becomes a unique strategic capability that enables faster innovation and faster time to market.
2. Organizational Design Focused on Agile Value Delivery
DevOps transformations are not simple. They are difficult and require creativity which leads to a journey that not all people in your company are prepared to take.
Value Driven Organizations
The best way to confront this challenge is to develop a healthy organization design. Separate organizational silos split by domains may be traditional; however, they are no longer effective. Many organizations, particularly those using Agile, are experiencing success by building cross-functional teams. Each team creates work in segments of time, or “sprints.” Each sprint results in the team delivering potentially shippable increments of work product. Moreover, place more emphasis on grouping team to swarm on delivering shared objectives. This structure will have a powerful effect on your company’s ability to collaborate and build business value.
This approach places more emphasis on teamwork. The teams design, build, and test as a team. And, throughout the development process, these teams actively coordinate with Technology Operations, InfoSec, and others to ensure that their work can be deployed.
Craftsmanship & Automation
Great DevOps companies require thoughtful and deliberate decisions to encourage great engineering craftsmanship. This craftsmanship ensures software is built with practices that encourage high quality product. The practices we follow should focus on receiving fast feedback on whether or not the code really works.
Today, practices like Test Driven Development (TDD) are used to create lines of tests before the code is written. By writing the code to after the tests are created, developers create a collection of code that, by definition, is already tested before it’s finished, thus reducing errors in increasing quality.
Automation is another key element to the product development flow. Once automated, a developer can automatically test the code with a simple click. The system can test the changes across thousands of developers’ new code in a fraction of time compared to manual tests.
3. Synchronized Product Planning and DevOps Planning
Several successful DevOps groups are also accelerating their delivery capabilities with support teams. Technology operations, infosec, architecture, and risk/compliance teams are often involved in product planning.
This results in a higher degree of coordination in the product development cycle. Aspects of security, scalability, reliability, are baked into the solution from the earliest stages of planning. Moreover, by tying together areas of release management practices at the beginning, the organization’s ability to coordinate product delivery matures faster.
DevOps may seem like a lot of work, but technology leaders should consider it a smart business investment. Companies unwilling or unable to adapt will be left behind and trapped under the weight of their own antiquated practices. Those slow to react will not be able to compete due to limitations of deployment speed and resiliency. However, it’s the companies employing DevOps that will outmaneuver and outpace their competition, leaving others in the dust.
Stacey Louie is the CEO of Bratton & Company, a leading Agile Transformation consultancy based in the Silicon Valley. As an Enterprise Agile Coach, he was instrumental in PayPal’s 400 team global agile transformation as well as supporting other Fortune 500 companies including Cisco, Hewlett Packard, and eBay. He also held the position of division CTO/CIO of public companies including Verisk Analytics and Stewart Information Systems.