Adopting DevOps is no easy undertaking, but the promise of cutting delivery times for projects from months down to days makes the feat worthwhile. For those keen to implement a DevOps transformation, understanding its roots will assist when applying the principles. Among its influences from Lean Manufacturing comes the infamously visible feature known as the Andon Cord.
Let’s go back in time first to check out Lean’s foundations for quality.
Deming & DevOps
Dr. William Edwards Deming, an American statistician credited for Japan’s rise as an industrial nation after WWII, developed core concepts for building quality into the manufacturing pipeline. His name, as well as being woven into the fabric of Japanese history during this era, is synonymous today with the infusion of quality in production through precision, consistency, and efficiency. As well as the 14 points of Total Quality Management which Deming is renowned for, he is also responsible for the concept known as Plan, Do, Change, Act (PDCA). One of the core fundamentals of his 14 points—point 3 in fact—was to “Cease dependence on inspection to achieve quality.” Deming argued instead that we should build quality into the product at every step.
In development, this translates to tests on code quality at each stage in the pipeline, rather than waiting until the end.
Manual to Automated Testing
Traditional quality assurance is usually realized through expensive and unreliable manual testing. In addition to these factors—and perhaps most importantly—manual testing does not improve quality, it merely highlights that the output lacks quality.
Consider, for example, that within your pipeline you produce a circular component, .i.e, a circle. Manual testing of our circle helps us determine that yes, it is a circle, it’s made of the right material, and looks like it fits our needs. The problem straight away with manual testing is that we introduce risk due to the human element of manual testing. Sadly, no developer—however experienced—is infallible.
Automated testing will always go above and beyond manual testing. Not only will it deduce all the above in our analogy, but automated testing will determine just how perfect or imperfect the component really is by testing both more accurately and in depth. And so, we reduce the element of human error and improve our quality of work.
If you’re looking for ways to integrate DevOps into your organization, give Caylent a try here.
Quality Assurance Over Quality Control
Automated testing is both vital to achieving DevOps and quality. A further component to automated testing as part of DevOps is the idea of shifting all testing left in order. The aim here is to improve the flow of feedback within the software development process downstream and find quality faults earlier rather than later. Automated testing is not only crucial to the building of applications but also as part of the fabrication of the software’s infrastructure used to build them. Integrate automated testing throughout the process from the beginning, but it should also continue into production. Deming’s belief in quality transfers well into software development. If teams weave quality into the code from the start rather than test it into the code post-production, they will achieve quality assurance over quality control.
Just as the PDCA system works in manufacturing, in DevOps it’s advised to establish fast flowing feedback loops. Do this by using telemetry and monitoring. This rotating cycle of steps aims to encourage teams to continuously improve the process as well as their output.
Toyota’s Andon Cord
Another Lean manufacturing principle and tool used to address problems in the production pipeline is the Andon Cord. This tool comes directly from the Toyota Production System (TPS) fathered by Taiichi Ohno. Pioneered by Toyota, the Andon Cord is a Jidoka quality method designed to signal to colleagues and management that there is a problem in the assembly line. If the problem persists after an allotted time the whole production comes to a standstill until the team can find a solution.
Pulling the Andon Cord is not just about solving a problem though. As part of the continuous improvement process, it offers a further opportunity to build quality into the production system—or development pipeline. Production stops, improvement starts.
While the concept in software development is more challenging, the principles should remain the same. In Toyota, if something is wrong, it is not needlessly escalated up the chain-of-command. Any worker on the assembly line can, and should, pull the Andon Cord. This signals to the team leader that there is a problem, if the worker and the team leader cannot find a resolution in a pre-defined amount of time, then the assembly line is stopped. See the image below from The DevOps Handbook:
Toyota Andon Cord – The DevOps Handbook (1st ed. 2016) Kim et al.
The Andon Cord in Software Development
During development, bugs are talked about openly, but the project moves forward regardless, and the bugs continue down the process. Automated testing can be the Andon Cord equivalent within the development pipeline, by flagging up issues to the team.
If a test fails, the team leader and developer should attempt to address the issue. Preferably, within an agreed timeframe (e.g., 5-10 mins). If they cannot resolve the issue, everything stops. The team can swarm the problem until they address the code and clear the pipeline.
For this process to work, a team’s automated testing must be comprehensive. Tests should be comprehensive and cover all aspects of the software development from code quality and performance, to compliance and security, and everything in between. This way when a test fails and production stops until resolved; there is reassurance that at least the event won’t repeat itself.
DevOps Culture
Fundamental to all of these ideas is the culture of DevOps which encourages individuals to highlight errors as part of a blameless working environment. By addressing mistakes and inefficiencies in this manner, you also create opportunities for ongoing learning; another key DevOps principle. Two birds with one stone, as they say.
Caylent is a cloud-native services company that helps organizations bring the best out of their people and technology using AWS. We are living in a software-defined world where technology is at the core of every business. To thrive in this paradigm, organizations need to empower their people and processes through technology. Caylent is uniquely positioned to fuel that engine of innovation by bringing ambitious ideas to life for our customers.
Caylent works with customers to build, scale and optimize sophisticated cloud solutions using deep subject matter expertise to deliver world-class outcomes through an agile co-delivery model.
