Learn how to effectively mock AWS services for better testing and development with Moto to make code testing more effective & efficient.
When it comes to mocking AWS services, there are several options and strategies to consider. For example, overwriting an AWS SDK client or method to intercept the AWS request and fake the response would work, but that is not always the best approach. It can be challenging to cover all possible behaviors and errors that can occur, and keeping the code up to date with SDK updates can be a hassle. That's where tools like Moto come in.
Moto is a popular option for mocking AWS services. It's a Python library that can be used to emulate AWS services and can be run in server mode to work with other languages through SDKs. But what sets Moto apart is that it can also be run as Python code, making it an ideal solution for unit testing in Python. This means you can avoid the need for server configuration or an extra Docker image in your CI/CD pipeline and get straight to testing your code.
Another description for mock testing in software development is the idea of overwriting some piece of code with another piece of code during runtime so that it runs a mock method rather than making an external call. It is like saying, "When the code reaches this part, do this other thing instead". Can be used to input/output desired data, or to call the desired method.
Let’s say we are testing an online bookstore. An end-to-end test would consist of:
Once a user submits an order for buying a book, a check_credit_card method must be called to know if the card is valid. Considering that this method needs to make a paid API call to verify if the credit card is valid, mocking it will speed up your test and avoid costs from being charged on every test run, especially when it becomes frequent, such as during the development phase. The benefit is that we are not negatively affecting the core business by mocking that API call, and we are also guaranteeing the system will run as it should if we receive that mocked expected response.
While mocking is a good approach to unit tests, there are many ways to do it. We can mock an entire library, a method like check_credit_card, or an endpoint like requests.post(url='https://wanted_api.com/post/'). Because the mocked code will run in place of the actual code, the more granular the mock the better. The closer you can get to mocking a request or single line of code, the more you know that the rest of your logic is being tested on its own and running exactly as it should.
This is extremely useful when working with cloud providers, like AWS.
Now, let's analyze some code. Keep in mind that the code shown is very simplistic and focused on demonstrating how to mock AWS services and test your code. No exceptions are considered here, which means that changing the input is a great way to discover what could happen in advance. The code used along this article can be found on this git repository.
Considering the online bookstore again, let's imagine an author wants to register a new book on the platform. On the backend, we’ll have the following flow:
Below is the Lambda’s handler:
This handler gets the parameters to create a book from the Lambda input, instantiate the manager class, create our book, and if nothing breaks, returns a successful message.
The manager class creates all the boto3 clients to prepare for the book creation and also wraps all the environment variables used by this Lambda. It helps both the production and alpha environments to use the same code by changing only these variables.
The way we presented the client and environment variables declarations is not a best practice, because it will run on every Lambda instance. But, it helps for demonstrating purposes and sets this class ready to be used as a singleton, running only on Lambda’s cold start. To keep it simple, let’s continue as it is:
The create_new_book method will then orchestrate the AWS services calls, like our diagram, to build the S3 key based on the author and title, save the book’s information on the database, upload the book file, and broadcast the message of its creation for all the subscribers.
This is a simple bit of code, but it will work to demonstrate the use of Moto on the tests. The test was developed using the well-known library pytest. To develop this same logic with unittest.TestCase you can check Moto’s documentation here.
This test will begin running the main_fixture, responsible for activating Moto’s mock, and configuring the local AWS environment created. Then, it continues building an input, the same as the Lambda would receive, and calling the handler method we want to test, our run method. To guarantee the book was created correctly, we make three asserts:
Let’s now analyze what is happening under the hood, inside our main_fixture:
The pytest decorator will ensure this code will run before the actual test if we add the method name as a parameter on the test - no need to import.
To ensure the credentials were configured, and that the system could not make any real call, dummy values were added before activating the mock. That way we avoid the case of a non-mocked service hitting the real AWS environment.
Generating this local AWS environment consists of redirecting the calls of actual systems (mock_ methods) and creating resources as the real environment would have. To keep it all encapsulated in this fixture we make use of context managers, yielding the already configured MainFixture instance to use inside the test.
The MainFixture instantiation begins storing the path to the test resources. This will be especially useful to upload a dummy file to the mocked bucket for testing the S3 logic. The env var will be copied to be applied as environment variables allowing us to simulate Lambda’s run closely. Next, we have the clients of the three AWS services called by our handler, which will help us with the asserts. For the last piece, the sns_backend contains all the messages sent to any SNS topic.
Insert formatted text here
With the instance now ready, we can use its attributes to set up the empty local AWS environment to be created. The exit_stack will help us to work with all the context managers we need.
As we have an empty infrastructure, to be able to save book information to a DynamoDB table, the table needs to be created first. The same logic applies to the other services, so we create the S3 bucket and the SNS topic. While creating the topic, we save its ARN, to be used in the message assert.
The environment variables copied will be added to our context - avoiding using the @patch decorator on every test. Any other patch can be added like that if needed.
The code above demonstrates how those services are created, and the cached_property decorator will make sure this topic is created just once and only the topic ARN is returned on the following calls.
And finally, let’s analyze the asserts made:
The first two are basically the opposite of what was presented inside the NewBookManager, making sure the right S3 object is in the bucket, and that the right item was added to the DynamoDB table. The third assert needs to call Moto’s sns_backend, which is the engine behind Moto’s SNS mock. It persists all messages published to topics, so with the topic’s ARN the notifications sent can be retrieved and compared with the expected message.
Mocking AWS services can be complicated without the right tools, but solutions like Moto make it more trustworthy. With the ability to emulate the behavior of AWS services, and to do it offline, developers can test their code more effectively, efficiently, and with no real calls.
The hands-on example presented in this post provides not only an explanation of how to build tests that behave as AWS real resources but also provides a template that can be reused across other examples. When developing tests, some resources tend to be the same across several tests, which means that with only a few changes the template provided can be adapted to other services and other tests. As you begin implementing this in your own code you will solidify your application, gain increased confidence in deployments, and improve the overall experience for your users.
The full code can be found in this repository, along with another example using SES: mock_aws_Moto
Michael Camilo is a Senior Software Engineer in the Cloud Native Applications practice at Caylent. After developing applications using C, C++ and Lua early in his career, Michael now likes to spend his time coding with Python in the AWS ecosystem. When he’s not busy building technical solutions you can find Michael hiking, traveling, dancing, or casting fireballs in D&D tables.
View Michael's articles
Learn how we helped an advertising customer setup automated, cost-effective pipelines to ensure compliance for sensitive data in their existing processes.
Data streaming eliminates the need to write custom applications for transferring data. Caylent’s Kennery Serain provides a reference architecture and code examples to showcase how to ingest data on OpenSearch using Kinesis Data Streams in near real-time.
Learn how to automate testing and safeguard your JavaScript apps using Jest with AWS CodeBuild and CodePipeline.
Michael Camilo