Key Principles That Lead to High-Performing Engineering Teams.
How to build a culture that consistently delivers a quality product, shortens lead times, and improves continuously.
Delivering software products and services remains a challenge for many organizations. Failed deliveries can cause serious damage, not just financially but also reputationally. Some more recent examples are the launch of HealthCare.gov and the faulty transactions by Royal Bank of Scotland.
Why is it so hard to deliver software products? Fred Brooks states in his book Mythical Man-Month:
“Many of the classical problems of developing software products derive from this essential complexity and its nonlinear increases with size. From the complexity comes the difficulty of communication among team members, which leads to product flaws, cost overruns, schedule delays. From the complexity comes the difficulty of enumerating, much less understanding, all the possible states of the program, and from that comes the unreliability.”
Software in many respects is more art than science. We talk about software design, build abstractions, and architecture. As long as we follow the right syntax, the machine doesn’t care. There are no inherent boundaries or controls that prevent us from building bad software. The complexity arises with the human factor.
That’s why we use methodologies and principles to infuse software engineering with discipline and structure. However, it is very hard to quantify its success. Every project is unique. Teams change, the technology stack evolves, dependencies vary, and methodologies are customized to fit the context of the project.
So how do we measure success? Most often, it is about the business outcome: Making sure we successfully launch, on time, with a stable and reliable system and a scope everybody is happy with. We can also look at the team’s velocity over time or the amount of bugs reported. However it doesn’t give us much indication of what practices worked and to what degree. This makes it pretty hard to learn and improve our craft.
Measuring the impact of changes in process and practice is not straightforward; however, we can always measure our lead and cycle time. We want to continuously reduce the time from idea to production while still delivering a high-quality product. In my opinion, a team who can deliver on that promise makes a high-performing team.
Building high-performing engineering teams is about culture. It’s about a value system, habits, and behaviors. How do we build a culture that consistently delivers and creates great software products? Let’s have a look at seven principles I strongly believe will lead to high-performing teams that won’t just deliver a quality product but are able to shorten the lead time and improve continuously.
Hire the best talent.
Talent defines and drives the culture. It is core to a successful delivery. Smart, dedicated engineers want to work with like-minded peers. This principle is essential for growing a team. While the concept may seem obvious, it’s imperative to structure the interview process in a way that supports it.
Finding the right talent takes a lot of time and effort. A well-defined and structured interview process helps avoid wasting time on the part of both the team and the candidates. Having a process in place also allows for more repeatability and consistency, and leaves a much better impression with the candidate.
The goal of the interview process is to predict of how well a candidate will perform on the job. Most likely the candidate won’t have to solve a complex algorithmic problem under high stress in a very short amount of time in his or her normal day. To create a more realistic environment, take-home coding challenges are great to assess a candidate's ability to code and allow for consistent evaluation. That take-home test can then be extended during the day the interviewer comes in. This ensures the code was written by the candidate and provides a more realistic challenge, as the candidate has the context of a problem already.
Interviewing is everyone’s job. Engineers will most likely spend a significant amount of time interviewing others. It generally is a rewarding experience as it is training; it also confers ownership and allows team members to assess the potential cultural contributions of the candidate. While it can be intimidating for the candidate, having two interviewers in the room will allow everyone to learn from each other and improve the process by providing feedback.
Make sure everybody is comfortable with a candidate. If there is doubts or someone disagrees, it is better to pass. Written feedback is good for traceability but at the end of an interview, the interview team should come together and discuss a candidate. It is important to make the call as a team.
Once you have the team in place, make sure you foster an environment of collaboration, trust and mutual respect. You can have the best talent on the planet, but if they don’t work well with each other, they won't help you.
Reducing cycle time requires quick decisions. To do that, we need to reduce the dependencies to other teams and systems. Werner Vogels, CTO of Amazon, spoke about the challenges they faced as they grew and the changes they made:
“The many things that you would like to see happening in a good software environment couldn’t be done anymore; there were many complex pieces of software combined into a single system. It couldn’t evolve anymore. The parts that needed to scale independently were tied into sharing resources with other unknown code paths. There was no isolation and, as a result, no clear ownership… The services model has been a key enabler in creating teams that can innovate quickly with a strong customer focus. Each service has a team associated with it, and that team is completely responsible for the service—from scoping out the functionality, to architecting it, to building it, and operating it.”
Giving teams more autonomy and independence demands proper structure. Teams should be small, cross-functional, and provide business value. Aligning them against a service or product slice makes the most sense. A service model allows for independent decisions, fosters innovation, enables more effective communication, and reduces operational overhead.
A short feedback loop allows teams to validate ideas quickly and reduces risk of failure as releases are much smaller. The ability to release often requires mature processes based in test and deployment automation. It also changes the way teams approach their work. A release demands features to be complete and therefore shows real progress instead of fake velocity.
Continuous delivery has established itself over the years as a series of practices that ensure deployable software passing a rigorous set of automated tests in a production-like environment. Continuous deployment builds on that; any code change that passes the automated tests is deployed to production automatically. Both approaches require incredible rigor but allow teams to shorten the feedback loop and the risk of failure no matter the size or complexity of the business.
There is a cost to maintaining automation, and reaching the right level of automation takes time. What’s important is the continuous focus on it. Over time, team performance will improve significantly and the team can focus on feature work instead of worrying about releases.
Evolve the architecture with the product.
Product development is an evolutionary process, and as the product evolves over time, so does the software architecture. Decisions shouldn’t be made based on technology trends but rather on requirements. Marek Kirejczyk wrote a nice essay about Hype Driven Development and how teams bring doom on themselves by making decisions based on technologies that are trending on Twitter.
Microservices and Serverless, two architectural approaches that have become popular in the past few years are such examples. Both fit well with the idea of creating more autonomous teams and are the new darlings in the tech community. The core concept is simple: Separate your concerns and break the product down into small, independently deployable services.
While both look incredibly appealing on paper, teams may find the implementation challenging, especially at scale. Before committing to an architectural change, it is important to understand the pros and cons and balance the additional complexity with some of the advantages. In either case, starting small, establishing learnings, and making incremental changes is often a better choice than going all the way in.
Software lives in a constantly evolving environment. New ideas and problems will lead to new requirements which will demand change to our software systems. It is important that we embrace change and make sure we plan for it.
Build quality in.
Never take shortcuts when it comes to quality. We should have confidence that our systems are functional at any given time.
That means that we focus on working software and have the right suite of automated tests in place to get fast feedback and push changes to production or a production-like system comfortably. It also means that any quality attributes such as performance and security are met.
Maintaining a healthy codebase is equally important. Developers spend much more time reading the code than writing it. Clean code is easier to understand, maintain, and extend on. It will be easier to find and fix bugs as well.
Collective code ownership, pair programming, and code reviews are all practices which will lead to better code quality. Collective code ownership reduces potential bottlenecks created by one person owning the code, and helps introduce new ideas. Pair programming and code reviews are great tools to get feedback and learn from others.
Quality is a team effort. The more close the collaboration between QA and development, the better the result. We still see too many cases where QA is an afterthought and not part of the iterative development process. Without QA’s input early on, risk of potential rework increases significantly. QA should also be the gatekeeper that will close out a story. Avoid logging bugs. Often, a simple conversation during the huddle, before closing out a story, will keep processes lean and teams much more efficient.
Today’s organizations completely depend on their systems and data. A simple failure can cost millions of dollars. Quality should have utmost importance for any team.
Team efficiency is about reducing waste. The team might be wasting time in meetings, with regularly missed requirements, or by manually deploying to a production environment. No matter the waste, it increases the cycle time, so a team's goal should be to reduce it as much as possible.
Automation is essential for reducing technical waste. It increases speed and introduces repeatability. In most cases it is a sign of maturity, technically but also operationally. Especially in larger organizations, the operational challenges often surpass the technical ones due to extensive approval processes, locked down environments, and so on. The ability to follow a continuous deployment process requires us to get the right people in the room and convincing them that breaking out of old habits is feasible and can be beneficial for the organization. The DevOps mindset has helped break the barriers between operations, development, and testing, and provides collaboration and clear operational efficiencies.
Retrospectives are a great tool to spot inefficiencies, and facilitate reflecting on the process and moving toward more efficiencies.
“‘Kaizen’ is a Japanese term that captures the concept of continuously making many small improvements. It was considered to be one of the main reasons for the dramatic gains in productivity and quality in Japanese manufacturing and was widely copied throughout the world. Kaizen applies to individuals, too. Every day, work to refine the skills you have and to add new tools to your repertoire.” — Andrew Hunt and David Thomas, The Pragmatic Programmer
Most software engineers are generally curious and motivated to learn. However it is important to provide the right structure for continuous learning.
We learn by doing. The most experience we gain is on the job. Especially for more junior engineers, it is important to have the right support and someone to go to. When it comes to coding, pair programming is extremely effective and helps get engineers up to speed very quickly.
Workshops, tech breakfasts, and lunch and learns are all great ways to encourage regular sharing of knowledge and learnings. Online services such as Safari Books and Pluralsight provide an on-demand solution.
Finally, teams should learn continuously and improve on the overall processes. Postmortems help us learn from mistakes and make sure we improve.