If you are a software company, it's common to think about the User Experience (UX), how they experience your product when they use it, how intuitive it is to interact with, how quickly someone can accomplish an activity within your product, and how they feel about that process when it is completed. The Developer Experience (DX) is similar, but instead of how your customer experiences using your software, it’s how your Developers experience the process of adding features, fixing bugs, doing code reviews, meeting testing objectives, coordinating on QC/QA requirements, and fulfilling their responsibilities in the release process. It’s important to think macro when considering the developer experience: DX wraps up all the positives and negatives of the working environment in which developers do their work, often encompassing the entire Software Development Lifecycle (SDLC) within your organization, including local tools, SaaS platforms, Cloud Platforms, automation (both presence and/or absence), testing standards, security standards, and company or departmental policies!. If you speak to developers and engineers in different organizations, you’d be hard pressed to get a consistent answer of what defines the Developer experience, and often it comes up in the context of what is NOT good about the Developer experience in a given organization.
Let’s look at the big picture: the software development landscape has changed significantly in the last few years as technology has evolved at a rapid pace and DevOps best practices have placed more responsibility in the hands of the Engineering team, often being responsible for their code all the way to production. Becoming cloud native solves a lot of deployment, management,and scaling problems. However, being cloud native comes with its own set of new complexities and abstractions. Where a Makefile used to be enough to build and unit test your code, Kubernetes-orchestrated applications are often made up multiple microservices, and have their own Makefile, Dockerfile, Helm Charts, and other CI/CD-related metadata files. This added complexity definitely influences the Developer Experience, and gives companies that invest in optimizing this experience an advantage in keeping their Developers happy...and happy Developers are easier to retain than unhappy ones!
DevOps and the Developer Experience
In his New York Times Bestselling Book “The Unicorn Project”, Gene Kim spelled out Five Ideals which help organizations better understand how to improve their business and engineering processes through DevOps best practices. In Gene’s own words, his “main objective is to confirm the importance of the DevOps movement as a better way of working, and delivering better value, sooner, safer, and happier.” Of those Five Ideals, three of them directly deal with the Developer Experience, specifically:
- Locality and Simplicity: “Locality relates to the degree of which a development team can make local code changes in a location as opposed to many places, without impacting various teams and other locations.”
- Focus, Flow, and Joy: “When developers are able to focus on developing their code with minimum dependencies, delays and impediments, it creates flow of value, therefore joy. When they are absorbed in their work, they’re really having fun, losing sense of time, and even their sense of self.”
- Improvement of Daily Work: “The elite organizations...are successful because they all made the conscious decision to pay down their technical debt. They all did whatever it took to make sure that developers’ daily work could be done and could flow with as little interruption and impediments as possible.”
The Complexity of Developing Modern Code
This is our first look at the developer experience: cloud solutions that create further complex problems: just what level of complexity is it going to involve to solve a problem; make a change to the code; and see whether the change accomplished the end result?
Traditionally, devs would make solutions work locally on their laptops then say goodbye to the code as they pass work on to “Operations” or “Systems” teams to deploy and manage it in Production. The DevOps methodology changed that kind of thinking. Now, Developers have to consider not only about writing code, but how that code should be deployed, managed, and supported as well. In addition, with the rise of cloud native architecture and containers, there's an extra hurdle or two to clear to show that code is running the right way in a container.
Another problem is within organizations, different developers are each approaching such hurdles differently. There’s sometimes no uniformity to bug fixing or hurdle jumping or code styling. The main issue is that no one addresses the pain points of this for developers or engineers. In many companies, dealing with DX is typically secondary to trying to make the end user experience as positive as possible. This approach is unfortunate—developers are users too!
Another Dev Tool Just Won't Cut It
Fundamentally for developers, running cloud native makes certain aspects of the work a little bit harder. Which is why tools are starting to be created that improve the developer experience when developing for containers on Kubernetes. Dev tools like Skaffold or Tilt, for example. But improving the developer experience is about more than just finding another good tool to improve efficiency in the write/build/run process that happens iteratively in the pipeline of adding a feature or fixing a bug.
Organizations want to move forward and modernise; to become cloud native. Yet, the teams and developers in the trenches sometimes don't really know what they're in for with such a major change in approach. Such transformational pivots need more than simply top down leadership to work. It’s important to get everyone in the development team onboard with the strategic direction that you're going in—culturally as well as technically.
Otherwise, you can make major investments in tech and cloud solutions, but your devs will still be struggling with their day to day work because tasks have become more difficult in the pursuit of these organizational aims. It’s very similar to the transitional difficulties that companies can face in the initial period of DevOps adoption or Agile adoption. The cure is harder to handle initially than the ongoing illness.
This is one snapshot of the developer experience, let’s examine the second.
Microservices and the Integration Testing Problem
Microservices architecture and cloud native applications go hand in hand. Most organizations leverage a microservice architecture to decouple and achieve greater scale, as without it you have too many people changing the same code, causing velocity to slow as friction increases. Where in monolithic architecture, teams would be bumping into each other to merge, release, and deploy their changes to the monolith, in a microservices architecture, each team can clearly define the interfaces between their components, limiting the size and complexity of the code base they are managing to that of a smaller, more agile team.
Each team can move more quickly since they can focus on the components they own. Their level of friction and velocity can be that of just the group working on that component, not that of the larger development organization. This means fewer merge conflicts which are less complicated to resolve. In addition, fewer pending code reviews on the same software at the same time.
But this creates its own problems as well, a key being the complexity of needing to ensure the cohesive whole also gets tested and functions together as a complete software product. Such consequences have made things like integration testing very important. Depending on the product, it may also produce the need for a proper shared “Dev environment” and/or “Test environment” so developers can validate their most recent changes against the most recent changes completed by other Dev teams prior to them being promoted together to upper-level environments (and eventually Production).
A key tool in making this time-efficient was the adoption of automated build/release pipelines. It’s important in a development pipeline to maintain velocity and move forward to get features and production code released.
However, with multiple developers across multiple teams releasing and integrating code synchronously, it’s crucial to ensure seamless integration through testing. The developer experience is seriously impacted by the pressure to maintain velocity from organizational and market demands, yet integration and testing takes toil, time and energy to make local code work at scale.
However, if you rely only on your automated testing in your pipelines, this too can have an impact on the developer experience. We’ve talked with many Engineering managers who complain about developer velocity. When submitted to scrutiny, such velocity turned out to be tied to bloated pipelines that took too long to run, but which were fully relied upon to know whether the code could be approved and promoted.
Through the perspective of a Developer who has just finished a bug fix; the idea of merging your code and then having to wait two or three hours just to find out that one automated test failed and you how need to make a one line change and wait another two or three hours to see if it worked can feel very much like a personal seventh ring of hell! This is also an aspect of the Developer experience that comes into play when having to integrate your changes with those of other teams around you.
There are tools that do help in this situation too. For example, Telepresence is a great tool designed to facilitate rapid, more interactive integration testing; giving a Developer the ability to run a piece of code on their local machine, while testing its interaction with other engineers’ code running in a shared Dev Kubernetes environment alongside live versions of dependent components. No waiting for a deploy pipeline to see how the component interacts with its dependencies. No need to configure and deploy other team’s components in your local dev environment.
While of course, this isn’t suitable for every scenario, Telepresence is an example of how thinking about the sources of Developer pain and friction at the point of integration can give rise to options to make them a little more palatable.
There are also approaches to pipeline tooling which can help. Such as moving away from monolithic tools like a shared Jenkins server and looking at build/release and CI/CD tools which support concepts like distributed pipeline workers/runners. Industry leaders like GitHub (within GitHub Actions) and GitLab both implement this approach, as do many cutting-edge CI/CD tools and projects. This way you can have shared pipeline definitions so those can be jointly maintained and re-used, while letting each team or environment be able to execute those pipelines in a way that supports both scale and team independence.
Organizations want the complication barrier around velocity, integration and testing to be as low as possible but with organizational growth comes next level complications, all of which can influence the developer experience.
Summary
Many companies have begun to recognise the pain of these impacts in the wake of their own digital transformation and move to becoming cloud native. More and more organizations are facing the reality that it is important to reflect on the toll that increased technological solutions can have on this concept of the developer experience. The next step is being the company that seeks to improve the developer experience rather than maintain a route that only feeds the endless exacting loop that it can become.
Creating a working environment where your team’s satisfaction and happiness are important will directly and positively impact the success of your project. Contented developers are more inclined to create and innovate exceptional software long term.
Caylent provides a critical DevOps-as-a-Service function to high growth companies looking for expert support with Kubernetes, cloud security, cloud infrastructure, and CI/CD pipelines. Our managed and consulting services are a more cost-effective option than hiring in-house, and we scale as your team and company grow. Check out some of the use cases, learn how we work with clients, and read more about our DevOps-as-a-Service offering.
Caylent Team
