Jorge Goldman
Caylent Services
Cloud Native App Dev
Deliver high-quality, scalable, cloud native, and user-friendly applications that allow you to focus on your business needs and deliver value to your end users faster.
Learn some simple changes you can make to better monitor, manage, and maintain more efficient costs with AWS cloud services.
We’re often amazed by how “The Cloud” scales efficiently to keep up with our demands and we try to tell ourselves that we only pay for what we use, which is partially true, but we’ve all seen the occasional rogue spend slip through the cracks only to be noticed when the bill arrives. Small companies have run into big problems by not understanding and estimating the costs of the solutions they build. On the other hand, large enterprises can find themselves paying thousands of dollars per month for resources they don’t use and don’t need. Neither of the two types of companies mentioned above had taken into consideration that having a healthy cost structure could make their business not only more profitable but also easier to manage and plan for future growth.
This article will provide the reader with a basic understanding of their options for becoming a healthier organization with respect to their Cloud Cost Management and which strategies are the ones to start implementing towards this path. Establishing a FinOps practice in any company is a long process that will take time and training. Do not attempt to fly before learning to walk or crawl!
Design our AWS Organization with a Master Payer Account
The first step towards a healthy cloud cost structure is to create an AWS Organization that contains all the accounts under a Master Payer Account that has no resources in it. This will help us have a better understanding and segregation of costs. Beyond that, it will help us manage reservations and savings plans in an even and organized way.
Design our AWS Organization with Enough Accounts to Cover our Business Strategy
Don’t be afraid to create as many accounts as needed in order to segregate our resources and manage permissions efficiently. There’s little to no incremental cost for AWS accounts.
Very often we tend to use only one or two accounts, like Production and the rest of the environments in the other one, which leads to very common mistakes that can impact other tasks like Business Support, QA, and development.
Even having a playground account for holding our company’s hackathons and non-business activities is a great idea, as it leads to healthier resource management. Here are some best practices for designing a healthy account management plan.
Design an Effective Tagging Policy
An effective tagging policy will provide our organization with increased granularity visibility and management over our allocated resources but it will require standardization and grouping into categories.
The following tag categories can help us achieve these goals:
- Environment – Distinguishes between development, test, and production infrastructure.
- Application – Identifies resources that are related to a specific application for easy tracking of spend change and turn-off at the end of projects.
- Cost Center/Business Unit – Identifies the cost center or business unit associated with a resource, typically for cost allocation and tracking.
- Owner – Used to identify who is responsible for the resource.
- Project – Identifies resources that are related to specific projects or new developments.
Enforcement of tagging during resource creation, will help our organization maintain a solid cost profile. There are 2 types of tagging policies:
- Soft: Users who create a resource without proper tagging are notified through email, nevertheless the creation is allowed
- Hard: Users are unable to create resources without proper tagging (Best practice)
Another recommended strategy regarding tagging is to have a list of fixed values each tag can take in order to avoid typos or multiple values for the same type of resource (for example: DataLake, data-lake, dataLake).
Enable Cost Allocation Tags
Once the account setup is done, from the Master Payer Account, we must enable the Cost Allocation Tags. This will help us identify and group all the resources created in the accounts linked to the one mentioned before. This is done under the Billing Dashboard option, and each tag needs to be selected individually. If more tags are enabled at a later stage, this will only be reflected in the costs from the moment they were enabled, not prior to the time they were enabled.
Enable Cost and Usage Report (CUR)
Once Cost Allocation Tags is enabled, the first step in this journey is to enable the Cost and Usage Report in AWS.
CUR can be enabled by any administrator, it will output parquet files with our resources usage in the destination folder specified. Our suggestion is to use the hourly granularity to be able to make a deeper analysis.
Amazon Athena will provide us with a tool to perform ad hoc queries over our cost structure.
CUR vs AWS Cost Explorer
At this point we are probably wondering why using CUR if we have AWS Cost Explorer, right? Well, that’s a great question!
Let’s understand the benefits of both as they complement each other:
- While having AWS Cost Explorer with customizable dashboards is really nice to have, it lacks a deeper level of analysis that can be achieved through CUR.
- Let’s assume we want to build a more fancy cost dashboard and have the chance to drill down our cost structure, the only way to achieve this, in a fancy way with a dashboarding tool, is through the CUR data.
- Also having the CUR data will let us create more complex queries through Amazon Athena, that would be impossible to achieve through AWS Cost Explorer.
Budgets and Alerts
AWS provides us with tools for automating the monitoring of our costs and usage for our resources. Organizations can easily set up daily, weekly and monthly reports that are automatically delivered to the corresponding users, besides being able to set thresholds for sending alerts.
Cloud Intelligent Dashboard
AWS provides us with an outstanding tool out of the box that can be easily configured with an AWS CloudFormation template that will create several dashboards that can be used for cost monitoring and forecasting.
These Amazon QuickSight dashboards are known as CUDOS mostly. follow the link below for setup and usage tips – Cloud Intelligence Dashboard with Amazon QuickSight.
Conclusion
It is never too late nor too early to start with our cloud cost management strategies, whether we are a small business just trying to get started or a big company that needs to have a healthier cost allocation strategy for cloud resources. The decision of not following these strategies can greatly affect our monthly bill and the discovery of any issues related to it. This is not only a technical challenge that has to be implemented but also a cultural change where people training is critical for an effective strategy.
Yes, it will take time to have it done, but it is better to start as soon as possible!
If you’d like to leverage our expertise to accelerate your AWS initiatives, while ensuring that your environment is optimized for costs, scale and efficiency, get in touch with our team.
Cost Optimization
Jorge Goldman
Jorge Goldman is a Sr. Big Data Architect with over 12 years of experience in diverse areas from SRE to Data Science. Jorge is passionate about Big Data problems in the real world. He graduated with a Bachelors degree in Software Engineering and a Masters degree in Petroleum Engineering and Data Science. He is always looking for opportunities to improve existing architectures with new technologies. His mission is to deliver sophisticated technical solutions without compromising quality nor security. He enjoys contributing to the community through open-source projects, articles, and lectures, and loves to guide Caylent's customers through challenging problems.
View Jorge's articles
Related Blog Posts
Driving Down Costs for Native AWS Apps
Save costs while simultaneously improving your application performance and developer velocity by modernizing applications on AWS
Application Modernization
Cost Optimization
SaaS App Scaling & Unit Cost Insights
Leveraging the AWS Cloud and a Serverless infrastructure for your SaaS apps, allows you to remove granular infrastructure considerations such as compute and storage. Learn how this simplifies your unit costs per user and helps you build SaaS products more efficiently.
Cloud Native App Dev
Cost Optimization
Video
Enabling Cost Controls in AWS Control Tower
Learn how enabling cost controls in AWS Control Tower can help you bring down unintended infrastructure expenditure & allocate funds to innovate.
Cost Optimization
Video