How to Continue Test if Assert Fails

There is much debate in the testing community about how many assertions a single UI automated test should have. Some standby the one test, one assert method, meaning that each test should only check or validate a single item. Others are quite happy for the test to check multiple items throughout.

Whatever the approach, I think it's pretty safe to say that tests should remain clear, concise, readable and of course maintainable. I personally have no issue with multiple assertions in a test, so long as I'm only focusing on one area of functionality.

Take this Registration form, for instance:

As part of a Register User test, I may want to check a couple of things. Perhaps check that the congratulations message popup is displayed and maybe that the user is routed to the login page following successful registration.

I wouldn't then want to test that the user can login successfully, I can do that part in another test.

The Problem

The trouble with Python's Inbuilt assert method is that it stops the test on first failure. This makes perfect sense in some scenarios. There is no point in continuing a test if part of the application is not working. However, there are cases where it makes more sense to make all your checks regardless of if one fails, especially if all your assertions relate to a single page at the end of your test.

This is an approach we often use in web service testing. When validating a response body, we probably want to check each value returned and then report on which ones were not as expected rather than stopping the test when the first assert fails.

The Solution: Pytest-check Plugin

Pytest-check (created by Brian Okken) is a Pytest plugin that enables you to wrap up all test assertions into a single pass/fail result. For example, if you have 3 assertions in a test and the first one fails, Pytest-check will continue to run the remaining 2 assertions. It will then report the test as failed if one or more checks fail. Nice, huh? ✨

❗ TestProject's Python OpenSDK also supports Pytest – So if you have existing pytest-based Selenium tests, it is very easy to convert these to TestProject-powered tests. You can read more about it in my latest article over at HowQA Blog, and check out a step-by-step getting started tutorial here.

Let's see how the Pytest-check plugin works 👇

Selenium Test

First, we need a test. I'll be using the registration page of this test application: https://docket-test.herokuapp.com/register

import selenium.webdriver as webdriver from selenium.webdriver.common.by import By   def test_register_user():     # Arrange     url = "https://docket-test.herokuapp.com/register"     # set the driver instance     driver = webdriver.Chrome()     # browse to the endpoint     driver.get(url)     # maximise the window     driver.maximize_window()      # Act     # Complete registration form      # enter username value     driver.find_element(By.ID, "username").send_keys("Ryan")     # enter email value     driver.find_element(By.ID, "email").send_keys("[email protected]")     # enter password value     driver.find_element(By.ID, "password").send_keys("12345")     # enter repeat password value     driver.find_element(By.ID, "password2").send_keys("12345")     # click register button     driver.find_element(By.ID, "submit").click()        

We have a test, but we need to write our checks. First let's do that using assert method:

# Assert # confirm registration has been successful  # check if congratulations message contains the correct text message = driver.find_element(By.XPATH, "/html[1]/body[1]/div[1]/div[1]/div[1]/div[1]/form[1]/div[1]").text assert message == "Congratulations, you are now registered"  # check user is routed to login page current_url = driver.current_url assert current_url == "https://docket-test.herokuapp.com/login"

If we run it now, everything should pass:

Great so far, but what happens if both our assertions fail? Let's change our assertions so we can see:

# Assert # confirm registration has been successful  # check if congratulations message contains the correct text message = driver.find_element(By.XPATH, "/html[1]/body[1]/div[1]/div[1]/div[1]/div[1]/form[1]/div[1]").text assert message == "Well done, You've Registered"  # check user is routed to login page current_url = driver.current_url assert current_url == "https://docket-test.herokuapp.com/register"  driver.quit()

So, we've updated our expected message and changed the expected URL so if we run this test again it will now fail:

As expected, the test fails with an assertion error.
Great! Our test has done its job, we've found a bug, well, sort of 🤔

What you can see from the error message though is that the test has failed due to the wording in the message. It hasn't even checked the URL. The test stops as soon as it finds the first failure, so we have no way of telling from the result if that was correct, hmmm… In this scenario, we would have to fix our expected message text and then rerun the test. So let's do that now.

Now we've changed the expected message back to 'Congratulations, you are now registered' now we can run it again:

Ah ha! Another failure, this time due to the expected URL.

Yeah, I know what you're thinking wouldn't it be great if we could have just captured both of these failures in one test run, well you're in luck. Enter Pytest-check.

Pytest-Check

Installing

We can install pytest-check using Pip install pytest-check Once installed pytest-check we can import it into our test with.

import pytest_check as check

Ok, now that's done, we can make some changes to our assertions. From now on we're not going to be using the assert statement, instead, we'll use the pytest-check syntax as follows.

To check the message we'll use the check.equal function and provide the Expected and the actual text, like so:

check.equal(message, "Congratulations, you are now registered1")

We can do the same thing for our URL check but for the purposes of showcasing a different method, we can use check.is_in instead.

check.is_in("login", current_url)

Our full test now looks like this:

import selenium.webdriver as webdriver from selenium.webdriver.common.by import By import pytest_check as check   def test_register_user():     # Arrange     url = "https://docket-test.herokuapp.com/register"     # set the driver instance     driver = webdriver.Chrome()     # browse to the endpoint     driver.get(url)     # maximise the window     driver.maximize_window()      # Act     # Complete registration form      # enter username value     driver.find_element(By.ID, "username").send_keys("Ryan8")     # enter email value     driver.find_element(By.ID, "email").send_keys("[email protected]")     # enter password value     driver.find_element(By.ID, "password").send_keys("12345")     # enter repeat password value     driver.find_element(By.ID, "password2").send_keys("12345")     # click register button     driver.find_element(By.ID, "submit").click()      # Assert     # confirm registration has been successful      # check if congratulations message contains the correct text     message = driver.find_element(By.XPATH, "/html[1]/body[1]/div[1]/div[1]/div[1]/div[1]/form[1]/div[1]").text     check.equal(message, "Congratulations, you are now registered")      # check user is routed to login page     current_url = driver.current_url     check.is_in("login", current_url)      driver.quit()

At this stage We can also revert the expected values back to their original state so the test will pass if we run it. So let's do that first.

Great! Now let's see what happens if both our checks fail. First, we'll need to update the code to make the checks fail once again:

# check if congratulations message contains the correct text message = driver.find_element(By.XPATH, "/html[1]/body[1]/div[1]/div[1]/div[1]/div[1]/form[1]/div[1]").text check.equal(message, "Congratulations, you are now registered!")  # check user is routed to login page current_url = driver.current_url check.is_in("1", current_url)

So let's run it.

Like last time our test has failed as expected, but this time it's recorded that 2 checks failed: first it reports that the message is not as expected, followed by the URL check. As far as pytest is concerned this is treated as a single test failure but we now know that multiple checks have failed.

Only if all the checks in the test pass, will the test be marked as a pass ✅

Awesome, well that's Pytest-check. Find out more by reading the documentation here.

Happy Testing Everyone! 🐍😉

mccoycluell.blogspot.com

Source: https://blog.testproject.io/2020/08/11/non-blocking-assertion-failures-with-pytest-check/

Share this post