We need to talk about quaternions…

I know, I know, this doesn’t seem like anything that has to do with testing. However, I found this concept to be very challenging and I think some people might be interested in knowing about it, especially if they are considering automating objects on VR.

The problem:

You want to be able to rotate the main camera with Google VR. Google design has reached the conclusion that moving the camera in virtual reality should be something that the user does, not the developer (I think it was possible in earlier versions of Unity and Google VR sdk).

So I decided to create an empty object and assign the camera as a child object of it. To move the camera, I move this object. To undo the camera movement done by the user, I move the object the opposite way. Should be easy, shouldn’t it?

Well, sometimes… but other times it does not work. I spent a very long time searching online for other people that had this issue and it was either working for them or not, nobody would explain why this was the case. I believe the reason to be the effect of gimbal lock.

I know, that sounds like a made up word, but this is actually an important concept that you should be aware of, let me explain:

Gimbal lock is an effect in which the object loses one degree of rotation, which is common when you use three dimensional vectors. At some point, two of the rotation coordinates get lined up in a way that every time you move one, the other one moves as well. If you are a visual person, this video from GuerrillaCG (a Youtube channel with information about 3D modelling and animation) explains it quite clearly.

How did I understand what this mean and how did I figure this was the issue? I decided to encapsulate the camera object in another object, and then that object in another one. Then I assigned them some colored spheres and pointers and I run some tests. After a while, it became clear that movements were not coordinated and that the rotation variables were not working as expected.

Attempts to understand 3D issues

Introducing quaternions:

The difference between Euler vectors (the usual 3 components vector that indicates a point in the 3D space) and quaternions (a 4 components vector) could help us avoid the Gimbal lock effect.

Quoting wikipedia: ” In mathematics, the quaternions are a number system that extends the complex numbers. […] Quaternions find uses in both pure and applied mathematics, in particular for calculations involving three-dimensional rotations such as in three-dimensional computer graphics, computer vision,[…] “

I know, it still sounds a bit gibberish… Basically (very basically), quaternions are a mathematical way of representing 3D objects by using 4 coordinates (representing them in 4D space instead), that an Irish mathematician (called William Rowan Hamilton) came up with and decided to literally set in stone, on a bridge.

https://upload.wikimedia.org/wikipedia/commons/a/a2/William_Rowan_Hamilton_Plaque_-_geograph.org.uk_-_347941.jpg
Commemorative Hamilton plaque on Broome Bridge. From wikipedia

They are difficult to understand and to visualise (as we don’t intuitively know how a 4D space should be visualised), but the maths applied to them are easier than to the 3D vectors. For example: if you want to rotate an object on an x plane, on the y plane and on the z plane, the object will not be in the same state if you perform the entire rotation on x and then on y and then on z, that if you do it in a different order (because each coordinate rotates based on the others too). However, with quaternions, you will always get the same result, as you can rotate two planes at once (which is particularly convenient when you are trying to undo a movement or automate a 3D object)

I am linking to a lot of articles below for you to deep dive, but two main points to remember are:

  1. You should keep one of the values of the quaternions fixed (or set to 1) as this is the way of telling the quaternions that there has been a rotation.
  2. Rotation angles in quaternions are half the expected in 3D rotation (so be careful when calculating the rotations in 4D). This is known as double-cover and it’s actually quite useful once you get the grasp on it.

If you are interested in knowing more about quaternions, you can check this and this videos from 3Blue1Brown (a Youtube channel full of interesting mathematical concepts with a very easy explanation, I really recommend it) Also, I enjoyed the explanations on this article. If you are considering working with 3D objects movement at some point, you should definitely watch these videos and play around with the simulators to understand everything better.

Conclusion

It is not my intention to go against the design of a system such as Google VR, and you should listen to the platform owners as, in general, it is wise not to tinker with the camera. However, sometimes I find it to be useful to undo a user’s movement, for example, for automation purposes or if the gyroscope is drifting on its own (more about phone’s sensors in this interesting article).
In these cases, the use of quaternions is so fundamental, that it is definitely worth it to spend sometime learning about them.
The next step after the automation of the camera and other 3D objects would be to automate the hands movements, but that’s…well…another story

Automating the automation: Automating Vstest

As part of my career, I have, on multiple occasions, found it useful to automate visual studio tests (VS tests) execution. The goal of this article is to show how to do so and why. I hope you can find it useful no matter your experience; for the experienced readers, you might get some ideas on how to use the capabilities of vstest, batch files and logs generated, setting up the basics to revisit some of the parts in the future.

If you have never created or run tests in Visual Studio, take this as a starter guide on executing automation. If you have ever run Visual Studio Unit Tests, you should be familiar with the Test Explorer tool:

At times, we might want to include these tests as part of a different system rather than executing them from Visual Studio. It is possible to use some existing functionality in the command line to help you doing so: VSTest.Console.exe is a command-line tool to run Visual Studio tests.

Steps 1 and 2 are for setting up the test data of this post, feel free of skipping them if you consider them too simple. Otherwise, I hope they can help you get started with Unit test in visual studio and general automation with it.

Disclaimer: I’m explaining this from a Windows Operating System’s perspective, but it could be done similarly from anywhere you could run vstest console tool.

Step 1: Create an unit test project

Note: needless is to say that you need Visual Studio installed to follow these steps.

We are going to start creating a test project in Visual Studio. To do so, we can click File -> New -> Project and select a C# template ‘Test‘ as in the image below.

Select the destiny folder and insert a name for it. Refer to the Visual Studio documentation if you have any issues.

Step 2: Create tests

The project should come with a unit test class by default. If for some strange reason that’s not the case, or you want to create another class, you can right click the Project and select Add->Unit Test (please note if you do this but you already had a class, then you might have a duplicate TestMethod1 in your list later on)

Now we are going to add a command inside the test that comes by default (because we are not adding test logic for now, we just add the ‘Fail’ command as shown below). We are also adding a Priority above this test. Then we can copy and paste the lines from [TestMethod] (line 9 in the image below) up to the last ‘}’ (line 14) three times to get a few tests in our test explorer. Change the names of the methods so each have a different name (for the example we have TestMethod1, TestMethod2, TestMethod3, TestMethod4.

For the example, all odd tests (TestMethod1 and TestMethod3) have priority 2 and even tests (TestMethod2 and TestMethod4) priority 1.

To be able to see the test explorer we should go to the menu Test->Windows -> Test Explorer.

And to be able to see the tests we have created we should go to Build -> Rebuild Solution.

Now, you can try and run the tests on the left hand side. They should fail because we are not adding any logic to them. If you are interested in how to do this, leave a comment below and I’ll write another article with more details on this. You can also look up the Visual Studio documentation to learn more.

In the next step we are going to learn how to execute the test cases outside Visual Studio.

Step 3: VSTest

Now, if we want to execute these tests outside Visual Studio’s environment we can use the console that is typically install under the following path:

C:\Program Files (x86)\insert VS version here\Common7\IDE\CommonExtensions\Microsoft\TestWindow

Where “insert VS version here” would be something like ” Microsoft Visual Studio 14.0 ” or “Microsoft Visual Studio\2017\Enterprise”… basically, you can navigate to your Program Files (x86) folder and look for a folder that has Microsoft Visual Studio within its name. Then continue with the rest of the path as above.

Inside the aforementioned folder you can find “vstest.console.exe“. Alternatively, you can download the nuget package and search it in the path where it’s installed.

Typically we would access this file from the command line (be it admin cmd or visual studio’s native tool command prompt)

We can open cmd (as administrator, by right clicking on it) and type “cd path” where path is the path to the above file with the correct visual studio version for your installation. It is convenient that this path is surrounded by quotes (“”), just in case there are spaces the program can recognise them as part of the path and not as a different command.

Now, you can select what type of tests to run by adding some parameters to the call, but you can test it by calling “vtest.console.exe” followed by space an the path to the dll of the test project. For example: vstest.console.exe c:\....\UnitTestProject1\UnitTestProject1\bin\Debug\UnitTestProject1.dll

You should see something like this:

Since you’ve set up your tests to fail. Later on, once your tests are finished and they are all passing you would see something like this:

If you are new to testing, you are probably wondering why would we want to run them in this complicated way instead of using the beautiful UI from Visual Studio. If so, keep on reading, this is where things start to get interesting.

Now, imagine we want to run just the first and third test method, for this we should make the following call: vstest.console.exe pathtodll /Tests:TestMethod1,TestMethod3

As you can see, now only TestMethod1 and TestMethod3 were executed (you can use the names of your test methods). Note that it should be failing for you, I’m just adding the passing image because is cleaner.

Remember we setup priorities before? So how to run the tests with higher priority? vstest.console.exe pathtodll /TestCaseFilter:”Priority=1″

In Microsoft vstest.console documentation, there are much more ways we can use this tool, including parallel runs. Have you started to see why using this tool could be very powerful? What else could we do with this?

Step 4: Creating batch files

The coolest part is that we can create a file with these calls and then, we can use that file virtually anywhere (continuous integration, night builds, as part of an agent to be executed on a remote computer…) These sort of files are called “batch files” because they run a set or batch of operations.

The first line of the batch file would be to CD into the vstest console folder. Then we can add calls to the tool for running the tests we want. Finally, we add a pause to verify that this is working fine.

To do this, just use the window’s notepad and type the instructions below. When saving it, save it with .bat extension instead of .doc or .txt.

cd C:\Program Files (x86)\insert VS version here\Common7\IDE\CommonExtensions\Microsoft\TestWindow
vstest.console.exe pathtodll /Tests:TestMethod1,TestMethod3 /Logger:Console
pause

Remember to change pathtodll to your actual project and add the right VS version. Now, if you execute this newly created file (as administrator), you should see the same results as before. Pressing any letter closes the console that opens up.

If you don’t want to see the results in a console (as if would be happening if you integrate this file with other projects), just remove the last command (pause). The logs of the results are saved in the current folder (the one for the vstest program) and we will analyse them on the last section.

Explaining continuous integration or multi-project execution would be a bit more complicated and out of the scope of this post (but do leave a comment or reach out in Twitter if you want me to explain it). But I can explain how to set up your computer to run this file every night with Windows!

For this, you need to open the task Scheduler (you can search for it in the lower left search box on Windows). Then click on “create a new basic task” on the right hand side and go through the assistant. The most important things are to specify the frequency you want the file to run and browse to select the saved .bat file. Now this file will be running with the indicated frequency automatically (if your computer is turned on at that time).

The next thing you want to do is to check the logs that this file has generated every time it was executed.

Step 5: Saving logs

Running automatic tasks and tests is awesome, but not really useful unless we know the results so we can do something about them.

First of all we should check where the logs are saved. Usually they are saved into a folder called “Test results” within the folder where you’ve run the file. Because we were using cmd (admin) and navigating to the vtests.console folder, it would be created there. In fact, that’s the reason we need administrator permission to run the file. There is a parameter with to run vtest.console to change this location, although my vstest.console was not recognising it, so I stick to the vstest.console folder for the purposes of this article.

I think trx logs are useful and should be always installed by default. To get them we can add a parameter to the vstest “ /Logger:trx“.  The generated file can be opened with the notepad and it will give you information about the run tests. However, we would focus on /Logger:Console as it is simpler.

Another way of retrieving the logs is by using the capabilities associated with Windows batch system. We just need to add ” > pathToFile\file.txt” where path to file would be the path all the way to a file with a txt extension (this file does not need to exist, you can create it with this command). This way a file will be saved with the contents of the console.

cd C:\Program Files (x86)\insert VS version here\Common7\IDE\CommonExtensions\Microsoft\TestWindow
vstest.console.exe pathtodll /Tests:TestMethod1,TestMethod3 /Logger:Console > pathToFile\file.txt
pause

You might want to save different files (by adding date and time) or replace the latest one (by keeping the same name as above), depending on the frequency that it is generated (if it takes long enough, we don’t mind replacing it).

Using the parameter “ /Logger:TimelineLogger” can give you a bit more information of the times of execution, but it will make it harder to parse later on.

Step 6: Playing with the logs

Now we have a text file with the logs…but reading all the files all the time, might be a bit boring..what to do with it? You get it, automate it!

Let’s output just the number of test case that have failed. We can do this with any programming language, but let’s keep going with batch. Why? Because I feel people underestimate it, so here it is:

@echo off
set /a FAILED=0
for /f %%i in ('findstr /i /c:"Failed" file.txt') do (
set /a FAILED=FAILED + 1
)
set /a FAILED = FAILED- 1
if %FAILED% gtr 0 (
echo Failed: %FAILED%
)
pause

The first line of the file allows the output to come up on the screen. Then we create a variable to save the number of times that we find the string “Failed”, we perform a loop with the search over the file called “file.txt”, take one out (because at the end there is a summary with the word “Failed” on it, which we don’t want to count) and only if the result is greater than 0 it is printed.

When executed for the file with priority 1 test cases failing we can see this result on a console:

If everything passes, nothing is printed for this example.

Please keep in mind that with this method, any test case that has the word “Failed” within its name will also increment the search, so this is just for demonstration purposes.

Maybe we would prefer to indicate as well the names of the failed test cases, or to print the last sentence on the file, which has already a summary.

We can also create some code that would send us an email if there are any failed test cases, or push the results into a graph, or send some sort of alert, or even a slack message… There are many possibilities, but they are…well…another story.

Automating the automation: Dealing with dynamic objects IDs

As testers, we sometimes find that developers don’t take testing into account while writing the system and design things that are very difficult for us to test. I’ve some experiences about it that I would like to share in case they help or inspire you.

Localised ID’s

A developer on my team was once tired of redeploying many times to rename objects because of business decisions. He decided to put into place a system to store the objects so people from localisation and internationalisation could change the text themselves without need of redeployment from his side.

It sounds like a great idea, but the issue was that the id’s were also on that table, and the people that were on charge of translating the text, would also translate those ID’s without understanding what they were for.

What was before me, was a set of localised pages that “most of the times” would work.

In this case, the solution I did was to check the developer’s code to understand how he was retrieving those objects and do the same from my code.

Automatic object creation

Another case of dynamic objects Id’s that I’ve seen is when the developers would create an automatic number of items dynamically on the page. In this case, the objects were created always with id’s that didn’t really identify what object they were, but would follow an structure.

For example, if they populate a list of users based on an input, each of them could have an id on the sort of “id_1, id_2, id_3…” (at least they had id’s).

Before me, they were doing manual tests because “it was not something we could automate”.

For this case, what I did was searching for id’s in the page. Something like this:

boolean caseExit = false;

int i=0;

while (!caseExit)

{

if (getObject("ID_" + i++).exits)

// do something

else

caseExit = true;

}

The “do something” part could be many things. For example, we could check that all the users had a certain property.

If your “exists” throws an exception you might need a try-catch instead of if statement in here.

An important note: do this if you are not able to retrieve the objects somehow else. For example, if what you have is a table you can easily access the different rows as long as the table has an id (you don’t need ID per row). It could also be similar if this is an html list.

The problem is when you have a serie of new unrelated objects. Moreover, if this is the case for all the objects in the page (they auto populate them all somehow), maybe auto-creating the page would help better for the case (explained on my previous post).

Angular/react dynamism

With introduction of JIT (a great article explaining here: buff.ly/2qFL28g) the browsers could start handling more dynamism in their websites. AngularJS (insert increasing version number here), ReactJS or VueJS are examples of frameworks that allow for this to happen.

But, as these frameworks started to get popular, some other tools were also created that would allow us to deal with this new dynamism. For example, AngularJS team created Karma, which is a NodeJS application that allows you to input your tests in the command line and aligns well with Jasmine and others tools for testing.

For end to end, you could check tools like protractor, nightwatchjs and testcafe.

There are many frameworks, extensions and customisations in the open source community. They are starting to move almost as fast as the front end tools (The frustration on front end is very nicely explained in this post: https://hackernoon.com/how-it-feels-to-learn-javascript-in-2016-d3a717dd577f )

Each of them would be tailored for a particular case scenario, so if you are building a framework you need to do a good research first, and ask many questions to your team.

POM for dynamic objects

Many people start forgetting about page object design pattern when they start automating on dynamic objects. However I would recommend you still incorporate it as much as you can because it really helps maintaining a clean code and reducing the time to write it.

Even if you have a lot of tests, you usually don’t have the time resources to execute every single one of them. For this, you need to decide what test to run, and this could also be a difficult, repetitive and automate-able process. But that’s…well…another story.

Automating the automation

Have you ever find yourself writing the exact same basic tests for certain objects over and over again? With the only difference on the id or path or even the element retrieval?

Wouldn’t it be nice to have some way of doing this bit faster? Automation is about making your computer do repetitive tasks for you, and this can be applied to everything, including, yes, writing code.

The process to automation is as follows:

  1. Identify repetitive tasks
  2. Write code that would do these tasks for you
  3. Identify when the code needs to be executed
  4. Execute the code or automate the execution

In order to give an example or something being repetitive during automation, I would like to introduce the page object model. If you already know about this, maybe you should skip the next section.

Page object model

The page object, in short, it’s a model for organising functional testing classes and objects so the code is cleaner and easier to modify. These sort of models are also known as design patterns (note to any developer reading this: yes, there are design patterns for testing and you should be using them!).

There is plenty of information about POM (page object model), starting on selenium website (http://www.seleniumhq.org/docs/06_test_design_considerations.jsp#page-object-design-pattern), so I am just going to summarise it here.

In POM, we have (at least, depending on the implementation) two different types of classes:

  1. A class (page class) that would gather all the information of the objects that we need from a website’s page (for example, a button or a text box) and micro-methods to preform common tasks (such as click, type, select, verify existence…). If you are very newbie, just open your browser’s developer mode (usually f12 does it) and using the selector hover over the website to see the objects that it has and their associated values.
  2. Another class (model class) would implement the more sophisticated methods (test cases) using the objects and methods exposed by our page class.

In this way, if the website changes layouts it would be easier to retrieve the objects again without the need of finding every instance of that object throughout the code.

1. Identify the repetitive tasks

So, you need to extract some of the elements on a website page, and add small known methods… We have found our repetitive tasks!

However, our job is not done here. There are still many things to take into account for automation:

  1. The tools: What tools are the developers using? What programming language? And how are we going to design the system in respect to this? We should, within possible, try to align with the development tools, but sometimes this is not the best choice. For example, imagine that developers use both javascript and ruby to build a website, but that you have a team full of knowledgeable test engineers experienced in Java, shall we train them and align with the developers or take advantage of their current skills? This would have to be taken case by case.
  2. Dynamic programming: Would we need to extend our framework to support elements that would be refreshed in the screen without fully re-loading the website? (Nowadays, you most likely would!)
  3. Number of elements/iframes: If we have a lot of iframes, or elements in the website (such as many nested divs), but we only need to access certain elements, we might prefer to write an automated solution that allows us to input the elements that we want. However, it could be possible that we want to define everything on a page automatically, because that takes away one manual step while keeping a reasonable load.

2. Write code that does the tasks for you

To give an example and in order to simplify things, let’s say that we have decided to create a solution in Java, that there is no dynamic loading and that we are going to say which elements and properties we want rather than getting them all.

The idea of the code is as follows:

Allow input of a list of elements, the type, the selector type and the selector for each of them. We could use a csv file for this, for example:

name= loginbox, type=input, selector=id, id=loginbox;
name= password, type=input, selector=xpath, id=…;
name= loginButton, type=button, selector=name, id=loginButton;
name= textSuccess, type=textBox, selector=text, id=success;

  1. For each of the lines on the input file, create an object with the name and selector given.
  2. For each of the lines on the input file, create a method for the object Type:
    1. For input: type method
    2. For button/href: click method
    3. For textBox: method to verify test
    4. For all: exist method

It would look something like this:

// retrieval of the fileLines left out for exemplification
for (Object o : fileLines) {

switch(o.type) {

case "input":

toPrintInPageClass += "WebElement " + o.name + " = driver.findElement(By." + o.selector+ "(\"" + o.id + "\"));\n";

toPrintInPageClass += "void " + o.name + "_type(string toType) { \n\t" + o.name + ".sendKeys(toType); \n}";

break;

case "button":

...... (you should have gotten the idea now)

This should take less than an hour to build, but after that, every page would be built in no time and the code would be less prone to errors as opposed to copy-pasting the same code over and over again. (You write one class as opposed to one per page)

An additional benefit is that if you have a manual tester in the team, now you can assign him or her the definition of the inputs and execution of the code. And you just made that person able to “program” with this “very high level programming language”.

Bonus: After this is built, we can create another class that would extract the items from the website into the input file for this class. That way we can retrieve all the elements in one go without human intervention.

3. Identify when the code needs to be executed

What happens if a developer changes the website and adds an element? Shall we execute the whole thing again or just add manually the missing element?

Even if it is tempting to add the missing element manually, I would suggest to add it to the input file, otherwise if someone decides to execute this code, your element would be missed. It is likely to still be faster to execute the code than to add an element manually.

But what if it is not? What if we have so many elements at the moment than executing the code can take longer than adding just one more?

I would still run the code rather than add elements manually, because it could as well be that some of those elements do not exist anymore. But if it is only one quick change, please remember to change the input file too.

As an addition, we could add into the code functionality to modify the class already created rather than create a new one, but I’d say this could be a lot of overhead for the benefit that you can get out of it.

4. Execute the code or automate the execution

Lastly, especially if you have the object retrieval automated too, you might want to automate the execution. For example, you can say to the computer (using a cron file could be a way): run this code every evening at 8pm. Or once a week… once a month… That way you could “completely” forget about the page definition side of the POM and just focus on the functionality.

Alternatively, if you have a way of retrieving the objects from the website, you could check that the original input is the same one as the newly generated and only execute the page class creation if they are different. That should be faster to run and it would allow you to change the actual code only when required.

However, be careful with this approaches, because you might miss when you need to add new functionality.

This is quite common in automation: you need to keep an eye on it. After it is built, it usually needs maintenance and refactoring. The better built, the less maintenance, but that does not mean 0 maintenance.

Conclusions:

Many sources would recommend to make short and simple tests, with the idea of identifying the failing parts easily and clearly. Nonetheless, sometimes you can spend longer in creating these simple tests than in actually testing the parts that are likely to fail, so the right balance is important.

What we’ve looked at today would help creating many simple tests, but you still should be careful not to over-test things as sometimes it could be too expensive or not really needed.

On the other hand, this could be a difficult task to do if we find with dynamically created objects or/and dynamically assigned id’s/properties. There are ways of dealing with these as well, but that’s…well…another story.