Infrastructure & DevOps Modernization
Quickly establish an AWS presence that meets technical security framework guidance by establishing automated guardrails that ensure your environments remain compliant.
As code repositories have become more complex to reflect the nature of the intricate microservices architecture that they support—and teams have started using multiple repos to manage different projects—challenges have started to appear in their management.
For starters, it is not always possible to maintain coding best practices in code repos when multiple projects are structured differently. There are also bottlenecks in packaging and deploying codes from multiple repositories.
Organizations are starting to transition from polyrepos to monorepos as a way to bring complex structure back to a manageable level. Having multiple codebases for web and mobile applications alone can be quite complex. Is shifting from polyrepos to monorepos really the answer? How can such transition be done smoothly?
The simple reason behind keeping services, modules, and libraries in separate repositories is actually very easy to understand. You are dealing with different components, and many of them – while may be interchangeable or used by multiple frontend components – are developed by different teams. Keeping them in separate repositories keeps things tidy.
However, problems start to occur when you try to integrate components from multiple repositories as the development cycle comes to its conclusion. A change in UI element, for example, can have a significant effect in how the entire app functions, and keeping track of that specific change is not easy when there are multiple codebases to track.
Other issues are starting to become more common too. The overhead needed to maintain multiple repositories can be huge when the project is not managed for efficiency. A set of services may require the same modules and dependencies, and each repo needs to configure those dependencies separately for the development to work.
Slowdowns become quite common. Simple tasks such as fixing bugs require thorough assessment, and the process itself becomes incredibly slow at times. Even with microservices in multiple repos linked correctly, it is still not always possible to assess the effects of a major change to the rest of the app.
The biggest concern with polyrepos is the complexity involved in tracking revisions and package updates. You cannot pinpoint which particular revision solves a specific bug, which means rollbacks and annotations are unnecessary complex too. These small difficulties add up, and they result in polyrepos becoming unsuitable for some organizations.
To make matters worse, teams with their own repositories don’t always use standardized tools and frameworks. Each team could have their own preferred tools, and they often introduce unnecessary differences in cofigurations; when I say differences, I mean variations that still require adjustments by the time you have to integrate microservices and push updates.
Centralizing the entire development project using a single repository, the monorepo approach, makes sense when the complexities mentioned earlier starts to affect your development cycle. You cannot just combine everything and be done with it though. Some adjustments are still needed before the monorepo can be effective for the whole team.
Simplifying component sharing is another thing that needs to be done when transitioning to a monorepo. Fortunately, you have tools like Bit helping you with the process. Bit in particular automates this step by immediately isolating components and their dependencies. When changes are made to one component, Bit makes sure that its dependents are updated too.
The three tools can be used in tandem. Bit + Lerna + Yarn Workspaces are notoriously reliable. Lerna can manage larger packages, while Bit takes care of the small components. Overhead is kept at a minimum when components and resources are shared between multiple packages. Yarn Workspaces, on the other hand, optimizes workflows and ties everything together.
Naturally, transitioning to monorepo is not without its challenges, the biggest of which is keeping testing streamlined and efficient. You can test specific packages when using multiple repositories. You can easily run tests on individual microservices since microservices are already separate and have their own dependencies configured. With monorepo, further steps are needed.
How you test specific packages and components in monorepo depends on the packages that you want to test and the tools you use. PHPUnit, which handles repos for PHP-based development, has functions that allow for package testing individually. The same can be said for NX, another popular monorepo tool; however, NX does not support publishing of individual packages.
Bit uses a slightly different approach. It associates each package with a compiler, and then link that compiler to a tester that handles everything on a component level. Every component needs to be associated with a tester, but once you configure everything, you can run granular testing across the repository without hassle.
Deployment in another story. Most of the tools are not designed to deploy each incremental change. However, you don’t have to bring the entire system down just to update a single component. As long as you have the right complier and deployment pipeline configured, you can still push selective updates based on which parts of the repository are changed.
Transitioning from polyrepo to monorepo does not always require refactoring, but actually conducting a thorough review and refactoring your packages are steps worth taking if you really want to gain the most advantage from the transition. Sure, you can move multiple repositories into one and continue development, but it will not be efficient.
This is very similar to when refactoring was needed during cloud migration. Monolithic apps can run just fine in any cloud environment, but you gain so much more flexibility and substantially increase efficiency in resource usage by making the transition to smaller components, microservices, and cloud-native packages.
There are some appealing advantages to gain indeed.
So, is it time to make the shift from polyrepos to monorepos? The answer still depends on the nature of your application and development project. In most cases, the answer is YES. Besides, you always have the option to make the transition only for certain packages.
Caylent provides a critical DevOps-as-a-Service function to high growth companies looking for expert support with Kubernetes, cloud security, cloud infrastructure, and CI/CD pipelines. Our managed and consulting services are a more cost-effective option than hiring in-house, and we scale as your team and company grow. Check out some of the use cases, learn how we work with clients, and read more about our DevOps-as-a-Service offering.
Data streaming eliminates the need to write custom applications for transferring data. Caylent’s Kennery Serain provides a reference architecture and code examples to showcase how to ingest data on OpenSearch using Kinesis Data Streams in near real-time.
Learn how to effectively mock AWS services for better testing and development with Moto to make code testing more effective & efficient.