In the previous post, I wrote an introductory post about getting started with mocking tests in Python. You can find it here: Python testing - Introduction to mocking.

In this post, I'll discuss how you can mock HTTP requests made using urllib or requests package.

A simple example¶

This is a very simple example:

# main.py

import requests

def url_exists(url):
    r = requests.get(url)
    if r.status_code == 200:
        return True

    elif r.status_code == 404:
        return False

The url_exists function takes a url and makes an HTTP request to that url. If the response status code is 200, it returns True, if status code is 404, it returns False.

Now, we'll test if it works as expected. But instead of making actual requests to the url, we'll mock the requests and set the desired status codes for our mock object manually.

# tests.py

from unittest import TestCase
from mock import patch # for Python >= 3.3 use unittest.mock

from main import url_exists


class FetchTests(TestCase):
    def test_returns_true_if_url_found(self):
        with patch('requests.get') as mock_request:
            url = 'http://google.com'

            # set a `status_code` attribute on the mock object
            # with value 200
            mock_request.return_value.status_code = 200

            self.assertTrue(url_exists(url))

    def test_returns_false_if_url_not_found(self):
        with patch('requests.get') as mock_request:
                url = 'http://google.com/nonexistingurl'

                # set a `status_code` attribute on the mock object
                # with value 404
                mock_request.return_value.status_code = 404

                self.assertFalse(url_exists(url))

We can control the returned value of a mocked object using return_value attribute. Above, we're setting status codes on mock_request.return_value, which means that mock_request.return_value returns a "blank" object which we can modify in any way we want, just like we've done above by setting a status_code attribute on it.

The reason we've done it is because requests.get returns a response object, if we mock it, mock_request should also return a response object. But, as mentioned above, it returns a "blank" object, not an actual response object. So, we set a status_code attribute on it to mimic a response.

If all this is confusing, you can create a separate class for a fake response. See below.

Let's run the tests:

$ python -m unittest tests

# Output

..
----------------------------------------------------------------------
Ran 2 test in 0.003s

OK

So it worked!

Testing if the mocked object actually gets called¶

In our tests above, we only test the return value of url_exists function. But there's a problem with our tests. We don't know for sure if the url_exists function calls requests.get method because we've mocked it in our tests. We have to know for sure that url_exists function really calls requests.get method.

Well, mock makes it easy for us to test that. To test if url_exits tries to call requests.get once with the given url, we can use assert_called_once_with attribute on the mocked object.

NOTE: From our small example, it is very obvious that url_exists does call requests.get method. But in a large piece of code, with a lot of variables and conditional statements, things don't remain so obvious any more - you will have to write tests to make sure things work as expected.

Let's modify our test:

class FetchTests(TestCase):
    def test_returns_true_if_url_found(self):
        with patch('requests.get') as mock_request:
            url = 'http://google.com'

            # set a `status_code` attribute on the mock object
            # with value 200
            mock_request.return_value.status_code = 200

            self.assertTrue(url_exists(url))

            # test if requests.get was called 
            # with the given url or not
            mock_request.assert_called_once_with(url)


    def test_returns_false_if_url_not_found(self):
        with patch('requests.get') as mock_request:
                url = 'http://google.com/nonexistingurl'

                # set a `status_code` attribute on the mock object
                # with value 404
                mock_request.return_value.status_code = 404

                self.assertFalse(url_exists(url))

                # test if requests.get was called 
                # with the given url or not
                mock_request.assert_called_once_with(url)

Run it again and it should still work.

Faking the response¶

At certain times, just mocking requests isn't enough because you'd need to do some operations on the response. For those purposes, you can make the mocked request object return a fake response for the purpose of testing.

Let's write another function in the main.py file called process_response.

# main.py

import requests

def process_response(url):
    r = requests.get(url)
    return r.content

The process_response function just requests a url and returns the content of the response. It's a very simple example.

Since mock allows you to set attributes on the mocked object on-the-fly while testing, setting a fake response is pretty straight forward. We can even create attributes of attributes. So, to mock the content, we'll need to set a content attribute on the the return_value of the mock_request, something like - mock_request.return_value.content = "Fake content".

It's very convenient that we can create attributes of attributes like this. In a general Python class, you'll first need to create an attribute before you can create it's sub-attribute.

Let's test our code:

# tests.py

from unittest import TestCase
from mock import patch

from main import process_response

class ProcessResponseTests(TestCase):
    def test_response_content_is_not_empty(self):
        with patch('requests.get') as mock_request:
            url = 'http://google.com'

            # set fake content
            mock_request.return_value.content = "Fake content"

            response = process_response(url)

            self.assertNotNone(response.content)

Run the tests again and they should all pass.

Creating a dedicated class for fake response¶

In this post, I've used mock_request.return_value to set status code, content, etc on the fake response. This can get repetitive and there's a better way to deal with fake responses.

You can create a separate class for the fake response where you'd set default values for the response. That should reduce some of the repetitive code. It would work like this:

class FakeResponse(object):
    # default response attributes
    status_code = 200
    content = "Some content"


# then use it like this:
fake_response = FakeResponse()

mock_request.return_value = fake_response

# change the default values if you want
fake_response.status_code = 404
fake_response.content = "Hello, world"

Conclusion¶

In this post we saw how we can mock HTTP requests by mocking requests.get or (urllib.urlopen). We also saw that we can mock requests.get directly or we can mock the funciton/code that calls requests.get. Both the ways work, so it doesn't matter which way we choose.