Automating the automation: Automating Vstest

Posted on by noemielis

As part of my career, I have, on multiple occasions, found it useful to automate visual studio tests (VS tests) execution. The goal of this article is to show how to do so and why. I hope you can find it useful no matter your experience; for the experienced readers, you might get some ideas on how to use the capabilities of vstest, batch files and logs generated, setting up the basics to revisit some of the parts in the future.

If you have never created or run tests in Visual Studio, take this as a starter guide on executing automation. If you have ever run Visual Studio Unit Tests, you should be familiar with the Test Explorer tool:

At times, we might want to include these tests as part of a different system rather than executing them from Visual Studio. It is possible to use some existing functionality in the command line to help you doing so: VSTest.Console.exe is a command-line tool to run Visual Studio tests.

Steps 1 and 2 are for setting up the test data of this post, feel free of skipping them if you consider them too simple. Otherwise, I hope they can help you get started with Unit test in visual studio and general automation with it.

Disclaimer: I’m explaining this from a Windows Operating System’s perspective, but it could be done similarly from anywhere you could run vstest console tool.

Step 1: Create an unit test project

Note: needless is to say that you need Visual Studio installed to follow these steps.

We are going to start creating a test project in Visual Studio. To do so, we can click File -> New -> Project and select a C# template ‘Test‘ as in the image below.

Select the destiny folder and insert a name for it. Refer to the Visual Studio documentation if you have any issues.

Step 2: Create tests

The project should come with a unit test class by default. If for some strange reason that’s not the case, or you want to create another class, you can right click the Project and select Add->Unit Test (please note if you do this but you already had a class, then you might have a duplicate TestMethod1 in your list later on)

Now we are going to add a command inside the test that comes by default (because we are not adding test logic for now, we just add the ‘Fail’ command as shown below). We are also adding a Priority above this test. Then we can copy and paste the lines from [TestMethod] (line 9 in the image below) up to the last ‘}’ (line 14) three times to get a few tests in our test explorer. Change the names of the methods so each have a different name (for the example we have TestMethod1, TestMethod2, TestMethod3, TestMethod4.

For the example, all odd tests (TestMethod1 and TestMethod3) have priority 2 and even tests (TestMethod2 and TestMethod4) priority 1.

To be able to see the test explorer we should go to the menu Test->Windows -> Test Explorer.

And to be able to see the tests we have created we should go to Build -> Rebuild Solution.

Now, you can try and run the tests on the left hand side. They should fail because we are not adding any logic to them. If you are interested in how to do this, leave a comment below and I’ll write another article with more details on this. You can also look up the Visual Studio documentation to learn more.

In the next step we are going to learn how to execute the test cases outside Visual Studio.

Step 3: VSTest

Now, if we want to execute these tests outside Visual Studio’s environment we can use the console that is typically install under the following path:

C:\Program Files (x86)\insert VS version here\Common7\IDE\CommonExtensions\Microsoft\TestWindow

Where “insert VS version here” would be something like ” Microsoft Visual Studio 14.0 ” or “Microsoft Visual Studio\2017\Enterprise”… basically, you can navigate to your Program Files (x86) folder and look for a folder that has Microsoft Visual Studio within its name. Then continue with the rest of the path as above.

Inside the aforementioned folder you can find “vstest.console.exe“. Alternatively, you can download the nuget package and search it in the path where it’s installed.

Typically we would access this file from the command line (be it admin cmd or visual studio’s native tool command prompt)

We can open cmd (as administrator, by right clicking on it) and type “cd path” where path is the path to the above file with the correct visual studio version for your installation. It is convenient that this path is surrounded by quotes (“”), just in case there are spaces the program can recognise them as part of the path and not as a different command.

Now, you can select what type of tests to run by adding some parameters to the call, but you can test it by calling “vtest.console.exe” followed by space an the path to the dll of the test project. For example: vstest.console.exe c:\....\UnitTestProject1\UnitTestProject1\bin\Debug\UnitTestProject1.dll

You should see something like this:

Since you’ve set up your tests to fail. Later on, once your tests are finished and they are all passing you would see something like this:

If you are new to testing, you are probably wondering why would we want to run them in this complicated way instead of using the beautiful UI from Visual Studio. If so, keep on reading, this is where things start to get interesting.

Now, imagine we want to run just the first and third test method, for this we should make the following call: vstest.console.exe pathtodll /Tests:TestMethod1,TestMethod3

As you can see, now only TestMethod1 and TestMethod3 were executed (you can use the names of your test methods). Note that it should be failing for you, I’m just adding the passing image because is cleaner.

Remember we setup priorities before? So how to run the tests with higher priority? vstest.console.exe pathtodll /TestCaseFilter:”Priority=1″

In Microsoft vstest.console documentation, there are much more ways we can use this tool, including parallel runs. Have you started to see why using this tool could be very powerful? What else could we do with this?

Step 4: Creating batch files

The coolest part is that we can create a file with these calls and then, we can use that file virtually anywhere (continuous integration, night builds, as part of an agent to be executed on a remote computer…) These sort of files are called “batch files” because they run a set or batch of operations.

The first line of the batch file would be to CD into the vstest console folder. Then we can add calls to the tool for running the tests we want. Finally, we add a pause to verify that this is working fine.

To do this, just use the window’s notepad and type the instructions below. When saving it, save it with .bat extension instead of .doc or .txt.

cd C:\Program Files (x86)\insert VS version here\Common7\IDE\CommonExtensions\Microsoft\TestWindow
vstest.console.exe pathtodll /Tests:TestMethod1,TestMethod3 /Logger:Console
pause

Remember to change pathtodll to your actual project and add the right VS version. Now, if you execute this newly created file (as administrator), you should see the same results as before. Pressing any letter closes the console that opens up.

If you don’t want to see the results in a console (as if would be happening if you integrate this file with other projects), just remove the last command (pause). The logs of the results are saved in the current folder (the one for the vstest program) and we will analyse them on the last section.

Explaining continuous integration or multi-project execution would be a bit more complicated and out of the scope of this post (but do leave a comment or reach out in Twitter if you want me to explain it). But I can explain how to set up your computer to run this file every night with Windows!

For this, you need to open the task Scheduler (you can search for it in the lower left search box on Windows). Then click on “create a new basic task” on the right hand side and go through the assistant. The most important things are to specify the frequency you want the file to run and browse to select the saved .bat file. Now this file will be running with the indicated frequency automatically (if your computer is turned on at that time).

The next thing you want to do is to check the logs that this file has generated every time it was executed.

Step 5: Saving logs

Running automatic tasks and tests is awesome, but not really useful unless we know the results so we can do something about them.

First of all we should check where the logs are saved. Usually they are saved into a folder called “Test results” within the folder where you’ve run the file. Because we were using cmd (admin) and navigating to the vtests.console folder, it would be created there. In fact, that’s the reason we need administrator permission to run the file. There is a parameter with to run vtest.console to change this location, although my vstest.console was not recognising it, so I stick to the vstest.console folder for the purposes of this article.

I think trx logs are useful and should be always installed by default. To get them we can add a parameter to the vstest “ /Logger:trx“.  The generated file can be opened with the notepad and it will give you information about the run tests. However, we would focus on /Logger:Console as it is simpler.

Another way of retrieving the logs is by using the capabilities associated with Windows batch system. We just need to add ” > pathToFile\file.txt” where path to file would be the path all the way to a file with a txt extension (this file does not need to exist, you can create it with this command). This way a file will be saved with the contents of the console.

cd C:\Program Files (x86)\insert VS version here\Common7\IDE\CommonExtensions\Microsoft\TestWindow
vstest.console.exe pathtodll /Tests:TestMethod1,TestMethod3 /Logger:Console > pathToFile\file.txt
pause

You might want to save different files (by adding date and time) or replace the latest one (by keeping the same name as above), depending on the frequency that it is generated (if it takes long enough, we don’t mind replacing it).

Using the parameter “ /Logger:TimelineLogger” can give you a bit more information of the times of execution, but it will make it harder to parse later on.

Step 6: Playing with the logs

Now we have a text file with the logs…but reading all the files all the time, might be a bit boring..what to do with it? You get it, automate it!

Let’s output just the number of test case that have failed. We can do this with any programming language, but let’s keep going with batch. Why? Because I feel people underestimate it, so here it is:

@echo off
set /a FAILED=0
for /f %%i in ('findstr /i /c:"Failed" file.txt') do (
set /a FAILED=FAILED + 1
)
set /a FAILED = FAILED- 1
if %FAILED% gtr 0 (
echo Failed: %FAILED%
)
pause

The first line of the file allows the output to come up on the screen. Then we create a variable to save the number of times that we find the string “Failed”, we perform a loop with the search over the file called “file.txt”, take one out (because at the end there is a summary with the word “Failed” on it, which we don’t want to count) and only if the result is greater than 0 it is printed.

When executed for the file with priority 1 test cases failing we can see this result on a console:

If everything passes, nothing is printed for this example.

Please keep in mind that with this method, any test case that has the word “Failed” within its name will also increment the search, so this is just for demonstration purposes.

Maybe we would prefer to indicate as well the names of the failed test cases, or to print the last sentence on the file, which has already a summary.

We can also create some code that would send us an email if there are any failed test cases, or push the results into a graph, or send some sort of alert, or even a slack message… There are many possibilities, but they are…well…another story.

Testing on VR world

Posted on by noemielis

Previously, I’ve written a couple of posts about how to get yourself started on VR in which I promised some stories about testing on this world.

Why do I call it world instead of application? Because the goal of virtual reality is to create realistic synthetic worlds that we can inspect and interact with. There are different ways of interacting in these worlds depending on the device type.

Testing an application in VR would be similar to testing any other application, while we would have to take into account some particularities about the environment. Later on we will see different kinds of testing and think about the additional steps for them in VR, but first let’s see the characteristics of the different types of devices.

Types of devices:

Phone devices (Google Cardboard or DayDream) – allows you to connect your phone (or tablet) on the device to be able to play a VR app in there.

This is possible because most of smartphones nowadays come with gyroscopes: a sensor which uses the Earth’s gravity to determine the orientation.

Some Cardboards (or other plastic versions) can have buttons or a separate trigger for actions on the screen (as it is the case for DayDream), but the click is usually not performed in the object. Instead, it is done anywhere in the screen while the sight is fixated on the object. If the device does not have a button or clicker, the developer have to rely on other information for interaction, such as entering and exiting objects or analyzing the length of time the user was on the object.

Cardboard VR device – Picture credit mentatdgt

Computer connected devices (HTC, Oculus, Samsung VR…) that generally comes with (at least) a headset and handset, have an oled device with high resolution and supporting low persistence embedded in the headset, so you don’t need to connect it to a screen. They detect further movement, as it is not just about the movement of the head but also the movement on the room itself and the hand gestures. This is done differently depending on the device itself.

We have moved from being able to detect user head movement (with reasonable size devices), to use sounds, to use hand gestures… so now, testing VR applications is getting more complicated as it now require the test of multiple inputs. The handset devices usually have menu options as well.

Before going on, I’d like to mention AR. AR is about adding some virtual elements in the real world, but with AR we do not create the world. However, AR has a lot in common with VR, starting with the developing systems. Therefore, the testing of the two platforms would be very similar

We have talked about the hardware devices in which the VR applications would run, but we should also talk about the software in which the applications are written.

Samsung gear + one handset

Developing platforms:

Right now there are two main platforms for developing in VR: Unity and Unreal, and you can also find some VR Web apps. Most of things that are done with Unity use C# to control the program. Unreal feels a bit more drag and drop than unity.

Besides this, if you are considering to work in a VR application, you should also take into account the creation of the 3D objects, which is usually done with tools such as blender or you can find some already created onlin.e

But, what’s different in a VR application for testers?

Tests in VR applications:

VR applications have some specifics that we should be aware of when testing. A good general way of approaching testing on VR would be to think about what could make people uncomfortable or difficult.

For example, sounds could be very important, as they can create very realistic experiences when done appropriately, that make you look where the action is happening or help you find hidden objects.

Let’s explore each of the VR testing types and list the ways we can ensure quality in a virtual world. I am assuming you know what these testing are about and I’m not defining them deeply, but I will be giving examples and talking about the barriers in VR.

Usability testing:

It ensures that the customer can use the system appropriately. There are additional measurements when testing in VR such as verifying that the user can see and reach the objects comfortably and these are aligned appropriately.

We are not all built in the same way, so maybe we should have some configuration before the application for the users to be able to interact properly with the objects. For example, the objects around us could be not seen or reached easily by all our users as our arms are not the same length.

You should also check that colors, lighting and scale are realistic and according with the specifications. This could not only affect quality, but change the experience completely. For example, maybe we want a scale to be bigger than the user to give the feeling of shrinking.

It is important to verify that the movement does not cause motion sickness. This is another particularly important concept for VR applications that occurs when what you see does not line up with what you feel, then you start feeling uncomfortable or dizzy. Everyone have a different threshold for this, so it is important to make sure the apps are not going to cause it if used for long time. For example, ensure the motions are slow or placing the users in a cabin area where things around them are static, or maintaining a high frame-rate and avoiding blurry objects.

Sitting experience – Picture credit rawpixel.com

If there is someone on your team that is particularly sensitive to motion sickness, this person would be the best one to take the tester role for the occasion. In my case, I asked for the help of my mother, who was not used at all to any similar experiences and was very confused about the entire functioning.

Accessibility testing

Is a subset of usability testing that ensures that the application being tested can be appropriately used by people with disabilities like hearing, color blindness, old age and other disadvantaged groups.

Accessibility is especially important in VR as there are more considerations to make than in other applications such: mobility, hearing, cognition, vision and even olfactory.

For mobility, think about height of the users, hand gestures, range of motion, ability to walk, duck, kneel, balance, speed, orientation…

To ensure the integration of users with hearing issues, inserting subtitles of the dialogs would be a must, and ensuring those are easily readable. The position of the dialogs should be able to tell the user where the sound is coming from. In terms of speech, when a VR experience require this, it would be nice if the user could also provide other terms of visual communication.

There are different degrees of blindness, so this should be something we want to take into account. It is important that the objects have a good contrast and that the user can zoom into them in case they are too far away. Sounds are also a very important part of the experience and it would be ideal that we can move around and interact with the objects based on sound.

I realized on how different the experience could be depending on the user just by asking my mother to help me test one of my apps. She usually wears glasses to read, so from the very beginning she could not see the text as clearly as I did.

I mentioned before that in VR it is possible to interact with object by focusing the camera on them for a period of time. This is a simple alternative to click without the need of hand gestures for people with difficulty using them.

There are many sources online about how to make fully accessible VR experiences, and I am sure you can come up with your own tests.

Integration testing

Its purpose is to ensure that the entire application functions as is should in the real world and meets all requirements and specifications.

To test a VR application, you need to know the appropriated hardware, user targeting and other design details that we would go through with the other type of testing.

Also, in VR everything is 360 degrees in 3 coordinates, so camera movement is crucial in order to automate tests.

Besides, there might be multiple object interaction around us that we would also need to verify, such collisions, visibility, sounds, bouncing…

There are currently some testing tools, some within Unity that could help us automate things in VR but most are thought from a developer’s perspective. That’s one more reason for us to ensure that the developers are writing good unit tests to the functions associated with the functionality and, when possible, with the objects and prefabs. In special, unit test should focus in three main aspects for testing: the code, the object interaction and the scenes. If the application is using some database, or some API to control things that then change in VR, we should still test them as usual. These tests would alleviate the integration tests phase.

Unity test runner

Many things are rapidly changing on this area, as many people have understood the need for automation over VR. When I started with unity, I did not know the existence of the testing tools, and tested most of it manually, but there are some automated recording and playback tools around.

Performance testing

Is the process of determining the speed or effectiveness of a system.

In VR the scale, a high number of objects, different materials and textures and number and of lights and shadows can affect the system performance. The performance would vary between devices, so the best thing to do would be to check with the supported ones. This is a bit expensive to do, that’s why some apps would only focus in one platform.

Many of my first apps were running perfectly well in the computer but they would not even start on my phone.

It is important to have a good balance to have an attractive and responsive application. New technologies also make it important to have a good performance, so the experience is realistic and immersive. But sometimes in order to improve performance we have to give up on other things, such as quality of material or lights, which would also make the experience less realistic.

In the case of unity, the profiler tool would give you some idea of the performance, but there are many other tools you can also use. In VR, we need to be careful with the following data: CPU usage, GPU usage, rendering performance, memory usage, audio and physics. For more information on this, you can read this article.

Unity profiler

Also, you can check for memory leaks, battery utilization, crash reports, network impact…. and use any other performance tools available on the different devices. Some of these get a screenshot of the performance by time and send it to a database to analyze or set up alerts if anything goes higher for you to get logs and investigate the issue, while others are directly installed on the device and run on demand.

Last but not least, VR applications can be multiplayer (that’s the case of the VRChat) and so we should verify how many users can connect at the same time and still share a pleasant experience.

Security testing

Ensures that the system cannot be penetrated by any hacking way.

This sort of testing is also important in VR and as the platforms evolve, new attacks could come to live. Potential future threats might be virtual property robbery, especially with the evolution of cryptocurrency and with monetization of applications.

Other testing

Localization testing: as with any other application, we should make sure that proper translations are available for the different markets and we are using appropriate wordings for them.

Safety testing: There are two main safety concerns with VR (although there might be others you could think of)

1. Can you easily know what’s happening around you?

Immersive applications are the goal of VR, but we are still living in a physical world. Objects around us could be harmful, and not being aware of alarms such as a fire or carbon monoxide could have catastrophic results. Being able to disconnect easily when an emergency occurs is vital on VR applications. We should make sure the user is aware of the immersion by ensuring we provide some reminder to remove objects nearby.

Smoke could be unnoticed – Picture credit CARLOS ESPINOZA

Every time I ask someone to give a test to some app with the mobile device, they start walking and I have to stop them otherwise they might hit something. And this is the mobile device, not the entire immersive experience.

Even I, being quite aware of my surrounding, give a test to some devices that include sound, and I hit my hand with several objects around me. Also, I could not hear what the person that handed over the device was telling me.

2. Virtual assaults in VR:

When you test an application in which users can interact with each-others in VR, the distance allowed between users could make them feel uncomfortable. We should think about this too when testing VR.

Luckily, I haven’t experienced any of these myself but I have read a lot of other people talking about this issue. Even some play through online on VR chat, you can see how people break through the comfort zone of the players.

Testing WITH VR: There are some tools being developed in VR to allow for many different purposes and technologies as emulation of real scenarios. Testing could be one of them, we could, for example, have a VR immersive tool to teach people how to test by examples. I have created a museum explaining the different types of testing, maybe next level could be to have a VR application with issues that the users have to find.

What about the future?

Picture credit Harsch Shivam

We have started to seen the first wireless headsets and some environment experiences such moving platforms and smelling sensations.

We shall expect the devices to get more and more accurate and complete and therefore there would be more things to test and that we should have into account. We are also expecting the devices to get more affordable with the time, which would increase the market.

Maybe someday we would find it hard to differentiate between what’s real and what’s a simulation… maybe… we are already in a simulation.

How do I get into…? guide

Posted on by noemielis

One of the most common question I get when people reach out to me about virtual reality (VR) is: how could I get started? Even thought I already have written an article about this, but maybe I should talk about my own experience instead so you can get more specific examples about it. If you are not into VR, please bear with me, as what I’m about to tell you can help you getting into any topics you wish to get into.

My Story

My first experience with a VR device was during a hackathon in Microsoft, when one of the interns brought his Oculus Rift. Back then it was a very expensive device, so it was very interesting to be able to play around with one. But I found that it would still have some issues to solve, starting for adding hands gestures.

As life goes, sometimes you get stuck in what you are doing at work and don’t get the time to investigate interesting new things. In my case, I bought a house and there was a lot of stress related to this. It was not until years later that I actually got the chance to try again another device, this time it was over mobile on a meetup called “tech for good” in Dublin. In this meetup, they were using VR mobile devices to provide social impact. It was my first experience with phone VR and I thought: Ok, now this is something that anybody can use and get, therefore it is something that is going to need testing.

After that, another hackathon (this time an open Nasa hackathon) got my interest in VR and AR back. I highly recommend this hackathon as I made really good friends there and we had so much fun building a AR/VR experience to navigate a satellite. My team (who won the local People’s choice award in Dublin) created an application that simulate a satellite around the orbit (on AR) and translate to see the view from that satellite (VR). If you are interested, here is our project

When I found myself having more time, I started looking for information about VR. I found a Udacity course on VR and decided to take it on. Back when I started it, the course covered many topics, although they made the decision of separating the courses in different specialties, which makes much more sense. If you are interested in seeing some of the projects I made during this course, check my Github account.

After that, I got interested in open source projects on AR and wanted to start doing testing there… However, life got in the way again when I moved to China. It’s still on my to-do list.

I was lucky enough to start working for Netease Games in China right after, so I had then enough flexibility and hardware access to do some more research in VR including some automated testing with Google Cardboard, which it should be now integrated in Airtest project (I know, not many people are using Google Cardboard anymore but, hey, you need to start somewhere.. the other research is still ongoing)

I also was lucky to have the opportunity to attend to the second Sonar in Hong Kong, which is a music and technology festival, and it showcased some cool new technologies and devices in VR (including aroma experiences and snow-surfing)

Besides that, I started to think of plans and ways of testing VR applications too (as Netease was working in some projects like nostos, which I had the opportunity to try myself and really enjoyed it).

Around that time, I gave a talk in Selenium conference in India gathering all this gained knowledge (which I talked about on this post). In order to prepare for this talk I played around and created my own ‘conference simulator’ just to get prepared for it.

Another thing I do frequently to gain knowledge in VR is to watch playthroughs and online reviews, as you can learn a lot from watching others play and it could be very good to understand your potential users if you are working on a game. I also have read some books on the matter (shout out to packtpub which gives away free IT books everyday!)

Have you found a pattern?

I know you have, if you are a reader of this blog you surely are a clever Lynx by now, but just in case, I have highlighted it in bold letters: Attending to (and after a while starting) hackathons, meetups, talks and festivals, watching or reading online related content and books, and playing around in courses, open source projects, at work and on your own projects will get you into anything you are interested to get into.

It sure sounds like a lot of things to do, but the tools are already around you, and I’m talking about years worth of experience here. Just take one thing at a time and you will too become an expert of that thing you are into. The biggest difficulty is to pick which one to take at any given time and to be honest about yourself and how much time you can spend on that. (I regret a ton not having put more efforts on the AR open source project when I had the chance)

Of course, if you are not really into it, then it would sound like a lot of work, in which case it’s probably better to save yourself time and pick something else. I like to think of it as a test of passion, or on the words of ‘Randy Pausch’ from his talk ‘Achieving Your Childhood Dreams’: “brick walls”. (By the way, this is one of the best motivational talks I’ve ever watch, and I actually re-watch it at least once a year to keep me motivated. Also, it mentions VR too 🙂 )

As you would imagine, this is not the only subject I spent time with or I gave my attention to, another big one for me is artificial intelligence, but that’s…well…another story.



Hacking social media

Posted on by noemielis

I know, I still owe you some stories, but I am now inspired to talk about something else. Besides, today’s issue is easier to put it into words. I don’t need to sit down and think carefully on a way of explaining some technical concept such as artificial intelligence while not sounding boring. But, just so you know, I am still working on the other stories.

I would like to show you how dangerous social media could become and on one hand highlight the need of asking the right questions when a new technology comes along in order to set proper tests and barriers (lynx are curious animals, aren’t we?).

On the other hand, highlight the importance to take breaks from it and think about yourself and things that would make YOU happy instead of thinking of things that ‘would make other people think that you are happy’ and therefore approve of you. I hope you enjoy reading this article.

Let’s think about it: the most viewed YouTube channels from independent creators are from people below their 30’s (or just on them), many started them 5-9 years ago. They have been getting a lot of pressure from fans and companies that would like a piece of their influence. Celebrities with less direct exposure to their fans have done crazy things in the past because of social pressure. Yet these influencers are not invited (that I know of) to Davos or famous lists of most influential people, besides demonstrating incredible marketing strategies, knowledge of new technologies, having charisma and being very intelligent (more than they let to be seen in some cases)

Social media affects society, not only these influencers, but many people are actually feeling depress or harm themselves because of social media. It is also a potential source for propaganda of all types and an a source for advertisement of all sorts by using the platforms ‘algorithms‘ in their favor.

There is a very important point to consider, which is its the potential for hacking (if you are interested on this, there is more information here and here). So, imagine that someone could actually go there and decide what you are going to see… how could this affect you?

Techniques and prevention:

Let’s imagine a platform that accepts comments and likes (let’s forget about dislikes). How could someone socially hack it?

1) Removing the likes: We would need to intercede the information that this platform is showing to the user and eliminate the likes the user will see. Maybe we should only eliminate them partially, so the user is not suspicious of not seeing any likes at all. For this, we could have a pondered random variable that would eliminate or not eliminate each like. How would you feel if all of the sudden nothing you write gets any likes? Prevention: From test side, make sure the like system works properly and cannot be done by anonymous sources. Make sure accounts are real. Make sure the user sees only real data. From user perspective, when you see something that you like, mention it in person, start a conversation about it instead of just clicking a button.

2) Liking specific posts: This is a bit fancier. Based on above, we could have some sort of AI algorithm that could classify the posts. Then we can decide which comments are going to show as liked for the user. How would you feel if only some type of your posts would get liked? Would that change your way of writing? Prevention: From test side, make sure all information is shown to the user. From user side, find your audience and focus on them. Also, consider talking with this people directly too (or in conferences). Try to keep honest to your goals for writing and who you want to reach.

3) Filtering comments: This would require some form of classification as with the previous point. Instead of targeting the likes, we would target the comments but the idea would be the same. Eliminate from view those that are not ‘interesting’. What would you think if you only receive a certain types of comments? Prevention: From test side, make sure all information is shown to the user. Maybe have a conversation about the feature itself and allow users to hide all comments. From user side, as above.

4) Creating comments: We could create new comments with AI. You might think the user would realize about this, but if done carefully they might not even notice this or confront the person making that comment. Besides, the social media platform might allow for not-logged in comments. This adds to the feeling of the previous one. Prevention: Have a conversation about blocking anonymous comments or disabling them. From user side, if you see a strange comment from someone that does not add up, clarify this with the person. It can also help with misunderstandings. Option 2, disable or stop reading comments.

5) Changing the advertisement around the website to a convenient one for propaganda or for harm (only for some types of social media). Prevention: Most of sites have a way of deciding what advertisements you are more interested of. Also, try cleaning cookies regularly or use private browsers and VPN services.

6) Extracting automatically interesting information for malicious purposes. Prevention: Be careful with what you post, don’t use information that is available for anybody as your passwords or security questions or pictures with personal data (such passports or train tickets). If you really want to share pictures of a trip, try uploading them after the trip and enjoy while in there!

7) Connecting certain types of people. I am not 100% sure how this could be used for malicious purposes but surely someone would find a way. Making sure you can block people is also very important.

8) Taking things out of context. Prevention: It’s very hard to delete something from the internet once it is out there, but some platforms allow it. Have you ever read your old posts? It is a good idea to do some clean-up every so often. Also, if this happens to you, keep a track of the entire context. Maybe have a system in which you can remove what you have written before it goes online, take some hours before posting to make sure you want to post that.

Why am I talking about social hacking social media in a test blog? Well, because, if you happen to be working on developing a social media project, you should make sure that these attacks are not possible and think about how the user could feel for other features to come.

(Please take some time to go throughout the articles I liked above to know more)

Thoughts about being addicted to being connected:

When was the last time you did something good for someone but did not tell anybody about it? When you do something good for someone and post about it, how do you know you are doing it because of the other person and not to have a better image of yourself in front of others?

When was the last time that you went for a trip and didn’t share the pictures with anybody? What about not taking any pictures at all? There is something truly special about having a memory that is just yours to have.

Conclusion

Social media has evolved quickly in very short time, and we need to consider a lot of new things, more so if we are in the development team of one of these platforms. We should really stop and agree about what is ethical and not with the particular platforms and maybe even list the set of contraindications as if we do with addictive substances. For example, would it be ethical for some platforms (maybe for young people?) to change what the users see in order to protect them from the bad critic? Consider this could, in theory, save some lives, but, in contrast, it would take away some potential good feedback disguised as bad comments. Maybe this is a feature you want people to turn on and off? If not, maybe you should list in your contraindications that it could be an issue. And I don’t mean terms and conditions, it’s not about saving your behind if anything happens, it’s about actually alerting the user about what could be experienced. Terms and conditions are…well.. another story.

Sources:

Some sources if you are interested in this area that helped me control my internet usage:

Book: “How to break up with your phone” by Catherine Price.

Watch: Crash courses on navigating digital information

Talking in selenium conference

Posted on by noemielis

Recently I’ve had the amazing opportunity to present in none other that Selenium conference in India. Even though I have done some other talks in the past, this was a big challenge.

I would like to share my experience, for those lynx curious about talking in or attending to conferences.

1) Submitting proposals:

The power of a deadline can sometimes be very impressive…at least for me.

I was keeping a couple of proposals for things I wanted to share for quite a while, always postponing to actually sit down and properly write them.

However, when I received a message stating that the last call for proposals for the selenium conference was in a couple of days, I told to myself: “hey, nothing to lose, just write them down and send them on before is too late. Let’s just get out of our comfort zone!”

2) Getting accepted and preparing for the conference:

After submitting the ideas, I’ve gotten contacted for one of my proposal asking for more details. There was some exchange of communications and after that…silence… nothing. I’ve gotten no idea if they were happy with my replies at all.

A few days later, I received an email. Bad news: my proposal was not accepted. I was shocked, after so many messages, I thought they would have been interested on it. Luckily they provided a rejection reason: “We’ve already accepted another proposals from the same speaker”.

Really?? I suddenly remembered: I did send two proposals. And right after that, I realised I was accepted to speak in THE selenium conference… and I started to freak out. What now?

seconf

I took a deep breath.

I reviewed the published list of speakers and then I realised that I had the pleasure of having met one of them before: Maaret Pyhajarvi. I brace myself, took all my courage and decided that if I was going to speak in the conference, I should be able to reach out for help.

I was very lucky that she was very nice and helpful. We arranged some online meetings (which was not easy because of the time difference) and she gave me plenty of valuable advice. I’m so happy that I asked Maaret for help and so grateful for her advice. I highly recommend that you watch her presentation and lightning talks, they were absolutely brilliant and inspiring.

OK, what else can I do to prepare for the day? (Besides planning and creating the slides and content, which is easier said than done)

As I was talking about virtual reality, I thought it was only fair to create an app for practising. So I asked the organisers for some pictures of the room I would present on and I created it on VR, even adding some original murmuring sounds. If you are interested, it’s uploaded here.

3) Travelling to India. Getting around. Impressions:

One of the reasons for me to apply for the selenium conference in India was that I was, at the time, located in China, so it was very close. I didn’t need many days off, the trip is shorter and it also was fully covered by the conference. Also, I’ve never been in India, great excuse to visit it!

I was told to first book the trip and then request reimbursement. They actually helped me to pick a better flight that was also a bit cheaper, and they dealt with the currency exchange. I am thankful for the organisers to help so much with this.

They also booked me in the hotel where the conference was taking place and arrange a car to collect me from and to the airport. I didn’t know how useful this would be until I arrived there. I would have had no idea about how to get to the hotel once there, so once again: so grateful for this.

I didn’t have time to travel a lot, just meet with the other speakers for food. That said, it seemed that Bangalore didn’t have much to see within the city but if you go a bit outside you can visit places. Unfortunately I didn’t have time for this , so my opinion about India can’t really count much from this trip. I’m going to have to go back for a better impression 😊

4) Impressions on the selenium conference:

I feel it was well organised, food was delicious (although I did get sick coming back home, but apparently this could happen during your first days in India for what I’ve heard).

I didn’t have much time to see all the talks, but I’ve watched some of the videos afterwards. I feel there was a good balance of speakers and that people should take into account the levels explained on the descriptions.

It was a pity that I was not able to attend to all the talks, but it is OK because of the recordings. I also felt pressured because I was presenting, so I’m happy I presented early enough and I could relax afterwards.

I really enjoyed it overall, it was a great experience and amazing opportunity for networking. All the speakers were very approachable, which I think it’s one of the biggest values of attending a conference.

5) The presentation. Self-feedback. Lighting talks.

What you are about to read now is a series of self-improvement tips, because, as a good Lynx, I am always learning. I am writing them more for myself than for you, but maybe this could help you too, if you ever consider presenting anything.

Looking back, I wish I reviewed much more and I trusted my own experience of needing more content than what it seems enough when practising. I felt it ended up being a bit short. Learning: prepare extra content and practice a lot.

For some reason (probably the fact that it was THE SELENIUM conference) I was as nervous as if it was my first conference ever. Learning: breathe slow, try relaxation techniques before the talk.

I wish I would have kept it more natural, after the matter I came up with more ideas to break the tension with the audience, but before that I didn’t know how to do it. Learning: get information about audience and think of how to break tension with them and make the content more interesting.

I wasn’t so sure about the content: it was a beginner talk but because it was very generic I could not go deep in any of the testing types… I wish now I would have spent longer in some of them or done some demo. However, I was relying in the questions to go a bit deeper…shockingly, there were not many questions. I think it could be a cultural thing because other presenters told me that the same happened to them as well. Learning: keep only 5 minutes for questions and show more technical bits. Talk about what you think it is important instead of waiting for them to ask.

I did blank out and forgot many many things I was meant to say. I took my safety notes with me but yet I lost my way throughout. This was what I was most scared of. Learning: slow down, move the notes as I go along. Have notes on the laptop, not only paper, it looks more natural to look in there.

Performance: muy accent didn’t come up clean. I should pursue to speak slower. I also looked and sounded tired, I was. I should try to get better sleep before the presentation day. Learning: just before the presentation, get as much sleep as possible. Travel earlier to avoid jet-lag. Record yourself to check if the pronunciation is correct or you need to change some words.

The lights were not the best – neither my shirt nor my slides matched with the background and projector. Learning: bring spare clothes and presentation with two background colours and try them before the meeting.

Video: it started late, my introduction was not recorded. I didn’t know I wasn’t supposed to move and I disappear every so often. Learning: Ask video person to record a short video and let you see it. Ask if you can move around or better stay in one place.

Luckily I could make up a bit during the lighting talks, in which I seem more like myself. But, unfortunately, not many people attended and it’s not part of the video of the talk.

6) Conclusion

Even though I feel I could have done much better, I enjoyed the entire experience a lot. I learned plenty from it and I will be much more prepared for the next one.

If you attended or have watched the video, you might be interested on knowing more about a project that I mentioned I am working on…but that’s, well, another story.

Hands on: automating scratch with AirtestIDE

Posted on by noemielis

On one of my firsts posts in this blog, I went through a hands on about how to automate scratch. The goal was to be able to teach kids how to test, hand by hand with how to develop programs.

Back at the time, the only program that I could find that would allow me to do this automation was not open source. The experience was very nice but since the goal was to use it on coderdojo , the conclusion was that it would not work out to use a paid solution.

Currently, I’m part of the development team on an open source project called ‘Airtest‘. Please note the intention on this post is not to advertise this product, but to provide a different solution that could actually be used in coderdojo. However, I want to disclaim it clearly: I work on this project, also, I am very fond of it.

Part 1: scratch development

Scratch is a simple tool for getting started with developing just it by dragging and dropping easy sentences. I’ve already explained how to create an app with scratch in my previous post, so I am going to assume you know how to do this.

Let’s just use the project that I created as example: http://scratch.mit.edu/projects/48422496/

This program moves a cat (called sprite) 10 pixels every time the user clicks on it. If it reaches a corner it supposes to bounce. The goal is to verify that the cat bounces when reached the corner.

Part 2: Using Airtest project to automate scratch

    1. Go to the official website: http://airtest.netease.com/ and download Airtest IDE. (Note: it is possible that you see some Chinese around as we are based in China, but don’t get too scared, everything should also be written in English)
    2. Click on “File” -> “New” -> “.air Airtest Project”,  to create a new airtest test file1
    3. Make sure you have “Selenium window” options active by clicking “Window”->”Selenium Window” (you can close Airtest assistance, Poco Assistance and devices).                2
    4. You also want to make sure you have your Chrome path configured in “Options”->”Settings”. If you have Chrome installed and are using Windows, this is generally under C:/Program Files (x86)/Google/Chrome/Application/chrome.exe 3
    5. Make sure your cursor is at the bottom of the script editor.  Then, click on the globe icon at the top left of the “Selenium Window” to open a browser for test.4
    6. Click “Yes” on the yellow message that appears on the top. 5Now you could continue with the recording method (right hand side top button on the Selenium Window). However, since scratch is done in flash, we won’t be able to retrieve a selenium element. we would proceed with Visual testing.
    7. Make sure your cursor is at the bottom of the script editor. Click the “start_web” button and input the url in the quoted yellow text. 6 It would look like this: 
      driver.get("https://scratch.mit.edu/projects/48422496/")
    8. Make sure your cursor is at the bottom of the script editor. Click the “airtest_touch” button and select the flag so it is within the highlighted rectangle. Double click inside it to confirm the selection  7
    9. Insert a this piece of code (this is a loop to repeat the next sentence 30 times, which would be the cat click): 
       for x in range(0, 30):
    10. Make sure your cursor is at the bottom of the script editor. Click tab once. Click the “airtest_touch” button and select the cat. Double click to confirm the selection. Your code should look like this (I have added a sleep command to wait for the elements but it should not be needed): 8
    11. Click play to see how the program works (make sure to activate flash on the browser for the program to load the first time). Feel free of getting some snack while your computer does your job for you 😉
    12. Bonus step: verify that the cats bounces. You can do this by inserting an “assert_template” and selecting the cat (same as with airtest_touch). However, the program would pass if it finds a cat, in any position, so even if it is upside down. Alternatively, you can create a snapshot to check manually this part.

Conclusion

Since Airtest project’s back-end (poco) is based on Python (a programming language) you can insert loops and other instructions easily.

Because of its powerful IDE which allows for record/playback or the use of simple buttons, it is easy for beginners.

Since visual testing is possible, even within a Selenium test, it works with programs or websites in which object hierarchy (DOM) is not easy/possible to retrieve (such as flash applications).

Because it is open source and the IDE is free, it can be used in teaching.

There is much, much more that this one tool can do that what is shown here. However, that is… well, another story.

Automating test case decision (using AI in testing part I)

Posted on by noemielis

1. The problem (and possible actions):

While testing, we need to decide carefully what test cases we will create, maintain, remove and execute per deployment.

Imagine that you join a company and get handled over a long list of test cases. You know absolutely nothing about them and you need to decide which ones to use for production (you have a time restriction of 10 minutes to execute them). What would you do?

  1. Try to understand which of the existing tests are needed and decide manually which ones to run:
    1. Check the priority of these test cases. Unfortunately, not many people review the priority of the test cases, so you can have obsolete test cases that are still marked as high priority but might be covered by other tests or the original functionality no longer be in place.
    2. Check the creation date. However, sometimes, an old test case might still make sense or be important.
    3. Ask the existing testers. Although, sometimes they have moved out of the company by the time you join and if not, things change so quickly that they might not be able to help anymore.
  2. Scrap it all and start over. I think this is a drastic solution, it might work out, but you might be wasting time re-doing something that might already be working fine.
    1. You could decide to just test the latest feature and not do any regression (trusting that the system was well enough tested before)
  3. Spend days learning about the features, executing all the test cases and figuring out what tests what and which tests you need to re-do. It’s a very analytic approach, but you are not likely to have the time for this, even if you have a lot of resources to execute them in parallel (which you should try to do). Also, maybe you need to refactor some of them, so you still need to do a selection.
    1. You could decide to leave comprehensive test for after deployment and only focus on a small set of features before that.
    2. You could do the deployments at hours where the load is small and do them more often (although this is generally painful for the team)
  4. Use new technologies to figure out which test cases to run (for example AI).
  5. Mix and match: Implementing point 4 on its own could be tricky. The best would be to mix it out with the others, analyzing and reviewing test cases, selecting higher current priorities, executing them in parallel to verify the percentage of success, eliminating test cases that don’t make sense anymore or that constantly fail…

As lynx, we are curious animals and we tend to ask many questions to understand the system. For example, some of the questions you could ask are:

  • How often are the iterations of the projects? If there are fast iterations, chances are that old test cases are not needed anymore.
  • How long do we have to verify a build?
  • Are the technologies from development changing? If they are, it would be a good moment to change on testing too, and point 4 could be a good solution here. I think it’s always good to have similar technologies between development and testing so both teams feel aligned and can help each other in a better way.
  • Do you have available testers in the company to whom to ask about the recent features and tests? If so, you can start with 3 adding up to 1 and 2 (so you don’t bother people with silly questions).
  • Is priority aligned within the company? Is priority per build or per feature? Is there a clear list of features per build? Is there a clear way of tracking which old features might be affected by the new ones?

It’s important is to balance well the test cases to get as many defects as possible and as early as possible, and also to ensure there is no overhead on the process.

Some tests can create false failures or be not reliable. Also, I’d like to highlight that sometimes writing tests takes too long or needs too many resources and some testers would write those test for the sake of ticking the “automated” box. That is not a good practice, be careful with these.

2. Understanding the process (how do we test)

Every time we want to automate anything (in this case we want to automate human decisions), we need to think about the manual way of doing it: When, as human, we decide which test cases to execute, what are we basing our decisions on? We want to check priority (of test cases and feature) and creation date. We might also take into account the severity of the test and feature (how costly would it be to fix a defect related with those). Another thing could be to look at previous runs and check how many times has this test case been failing or how many defects have raised already.

Note that the measures themselves are also estimated – it is important to have a good process in terms of the estimation. The first thing is to clean up the test cases and the system (process) itself. Having good documentation around when something is considered high priority or high severity could help out when aligning the system across the team or the company.

The second thing we need to do for automating tests decision is to decide which variables we are going to take into account for our system. Some of the above mentioned could actually measure the same thing. Having a short and clear number of variables is essential in order to build a correct system, since the more variables the more complicated the system would be and the longer it would take for it to make decisions.

An example or two variables that could be measuring the same thing could be the priority of the test case and the priority of the feature, if the system is well assigned.

There are tools and algorithms thought to identify automatically which variables are actually more important for the data or what sort of relationship there are among them, as this is sometimes not obvious for a human. Just have this in mind when creating your system (as this is usually topic 1 in any machine learning related book).

3. What’s AI

In order to automate these decisions, we could make use of one of the technologies that is being trending recently because of the new systems being able to compute it faster and the creation of better algorithms: Artificial intelligence.

According to Arthur Samuel in 1959, Artificial intelligence gives “computers the ability to learn without being explicitly programmed.”

Artificial intelligence is a big area, and there are many ways we could use it to help with testing.

Note also, that this is not a simple topic and there are many people who have dedicated their entire careers to artificial intelligence. However, I am simplifying it as much as possible since I’m taking this as an introduction and overview.

For this story, I am going to focus in using artificial intelligence to decide among test cases. I found two interesting ways of doing this. The first one is called “rule based system”.

4. Rule based system:

A rule based system is a way to store and manipulate knowledge to interpret information in a useful way. For us, we would like to use fixed rules in order to get an automatic decision of if we want to execute our test case or not. Imagine this as if you wanted to teach it to a newbie who needed your logic to be written down in notes.

For example: If risk is low and priority is low and test case has run at least once before, then do not run the test case. This rule would not act on its own, but mixed with a long list of rules written in this style (which is related to logic programming, in case you want to learn more about it). The group of rules is called “knowledge base”.

In this system, there is no automatic inference of the rules (which means that they are given by a human and the machine does not guess them). But there are some cycles that the machine goes through in order to make a final decision:

  1. Match: The first phase would try to match all the possible rules with each test case creating a conflict set with all satisfied rules.
  2. Conflict-Resolution: One of the possible rules is chosen for execution for that test case. If no rules are satisfied, the interpreter halts.
  3. Act: We mark the test cases as execute or not execute. We then execute and can return to 1 as the action have changed the property of the tests (last executed, passed or failed…)

 

5. Fuzzy logic – hands on:

If you ask experts to define things like ‘high priority’, ‘new test case’ or ‘medium risk’, they probably will not agree among them. They can agree that a test case is important, but when exactly are they marking it with priority 3 or 2 or 1 (depending on your project’s scale) would be a bit more difficult to explain.

In a fuzzy system, such as our, we define things with percentages and probabilities.  If we gather the information of the particular definitions for a variable, we will find it follows a specific function, such a trapezoid, triangle or Gaussian.

Imagine that we asked a lot of experts and come up with the example below:

Let’s define ‘low’ as a trapezoidal function starting on the edge (minimum value) and travelling to 20 and 40.

‘Medium’ would be the same function on the points 20, 40, 60, 80 (note that they overlap)

‘High’ shall be 60, 80 and maximum value.

The graph would represent our system as such:

fuzzylowmedhi

If we decide on the variables (for example ‘priority’) and definitions (also called labels, for example, ‘low’), the functions that compose those labels (as the graph above) and the rules among the variables, we should be able to implement a system that would decide for us if we should run a test or if it is safe to go without it. Let’s do so!

After a bit digging for a good C# library to implement this sort of things (maybe using F# would have been easier), I came across: http://accord-framework.net which seems to be a good library for many AI related implementations. We can install its NuGet Package with visual studio.

The first thing we need to do is define a fuzzy database to keep all these definitions:

Database fdb = new Database();

Then we need to create linguistic variables representing the variables we want to use in our system. In our case, we want to look at priority, risk, novelty of test case and pass-failure rate. Finally, we will like to define a linguistic variable to store the result, that we are calling ‘mark execute’.

 LinguisticVariable priority = new LinguisticVariable("Priority", 0, 100);
 LinguisticVariable risk = new LinguisticVariable("Risk", 0, 100);
 LinguisticVariable isNew = new LinguisticVariable("IsNew", 0, 100);
 LinguisticVariable isPassing = new LinguisticVariable("IsPassing", 0, 100);
 LinguisticVariable shouldExecute = new LinguisticVariable("MarkExecute", 0, 100);
// note on the last one that the name of the variable does not have to match the name for the rule,
//      which is the string literal that we are assigning it

After that, we define the linguistic labels (fuzzy sets) that compose above variables. For that, we need to define their functions.

For demonstrative purposes, let’s say that we have the same definitions for low, medium and high for priority and risk. For novelty, pass rate and mark execute, we are going to define a yes/no trapezoidal function. Note that we cannot use ‘no’ as it is a ‘reserved word’ for the rule specifications (more below), so we would call it ‘DoNot’. The yes/no function graph that we are using looks like this:

fuzzyyn


// defining low - medium - high functions
TrapezoidalFunction function1 = new TrapezoidalFunction(20, 40, TrapezoidalFunction.EdgeType.Right);
FuzzySet low = new FuzzySet("Low", function1);
TrapezoidalFunction function2 = new TrapezoidalFunction(20, 40, 60, 80);
FuzzySet medium = new FuzzySet("Medium", function2);
TrapezoidalFunction function3 = new TrapezoidalFunction(60, 80, TrapezoidalFunction.EdgeType.Left);
FuzzySet high = new FuzzySet("High", function3);

// adding the labels to the variables priority and risk
priority.AddLabel(low);
priority.AddLabel(medium);
priority.AddLabel(high);
risk.AddLabel(low);
risk.AddLabel(medium);
risk.AddLabel(high);

// defining yes and no functions
TrapezoidalFunction function4 = new TrapezoidalFunction(10, 50, TrapezoidalFunction.EdgeType.Right);
FuzzySet no = new FuzzySet("DoNot", function4);
TrapezoidalFunction function5 = new TrapezoidalFunction(50, 90, TrapezoidalFunction.EdgeType.Left);
FuzzySet yes = new FuzzySet("Yes", function5);

// adding the labels to novelty (isNew), pass rate (isPassing) and markExecute (shouldExecute)

isNew.AddLabel(yes);
isNew.AddLabel(no);

isPassing.AddLabel(yes);
isPassing.AddLabel(no);

shouldExecute.AddLabel(yes);
shouldExecute.AddLabel(no);

// Lastly we add the variables with the labels already assigned to the fuzzy database defined above

fdb.AddVariable(priority);
fdb.AddVariable(risk);
fdb.AddVariable(isNew);
fdb.AddVariable(isPassing);
fdb.AddVariable(shouldExecute);

That was a bit long, still with me? We are almost done.

We have defined the system, but we still need to create the rules. Next step is creating the inference system and assigning some rules.

Note that for this implementation the rules are not weighted. We can make it a bit more specific (and complicated) assigning weight to the rules to denote their importance.

Also, note that these rules are defined in plain English, making it easier for the experts and other players on the project to contribute to them.

InferenceSystem IS = new InferenceSystem(fdb, new CentroidDefuzzifier(1000));

// We are defining 6 rules as example, but we should take them from experts on the particular system. The rules don't necessarily need to work out for every system.
IS.NewRule("Rule 1", "IF Risk IS Low THEN MarkExecute IS DoNot");
IS.NewRule("Rule 2", "IF Priority IS High OR Risk IS High THEN MarkExecute IS Yes");
IS.NewRule("Rule 3", "IF Priority IS Medium AND IsPassing IS Yes then MarkExecute IS Yes");
IS.NewRule("Rule 4", "IF Risk IS Medium AND IsPassing IS DoNot THEN MarkExecute IS Yes");
IS.NewRule("Rule 5", "IF Priority IS Low AND IsPassing IS Yes THEN MarkExecute IS DoNot");
IS.NewRule("Rule 6", "IF IsNew IS Yes THEN MarkExecute IS Yes");

Finally, we need to set the actual inputs or values from the tests. The ideal scenario would be that we retrieve them from a file. We could automate the extraction of the variables of our tests into this file from our test case database.

For this example we are typing the values directly. Let’s think of a test case with low priority (20% low), low risk, quite new (is 90% new) and with low passing rate (since it is new, that makes sense). This would be defined as this:

IS.SetInput("Priority", 20);
IS.SetInput("Risk", 20);
IS.SetInput("IsNew", 90);
IS.SetInput("IsPassing", 10);

If we want to define a test case with high priority and risk, old and with high passing rate, the variables would look something like this:

 IS.SetInput("Priority", 90);
 IS.SetInput("Risk", 90);
 IS.SetInput("IsNew", 10);
 IS.SetInput("IsPassing", 90);

For now, let’s get the outputs directly on the console. It would look like this:

try
{
float newTC = IS.Evaluate("MarkExecute");
Console.WriteLine(newTC);
Console.ReadKey();
}
catch (Exception e)
{
Console.WriteLine("Exception found: " + e.Message);
Console.ReadKey();
}

The result of passing the first test case to this system is that we should execute it with 49.9% of security and for the second we get 82.8%.

After playing around for a while with this particular set of rules, I’d say that the system is a bit pessimistic and plays a bit too safe. It’s hard to get values under 50% (which we could assume it’s safe not to execute those test cases).

6. Rule based system – conclusions:

  • An expert / experts are needed to specify all the rules (we might influence the system. In the example above, I’m making the system too safe)
  • These rules won’t automatically change and adapt; we need to add new rules if the situation changes
  • The rules are hard to define: shall we always run all the cases when risk is high and feature is old?
  • Fuzzy definitions and fuzzy results make the system a bit complicated to understand and, again, to define
  • There could be relationships between the variables that are not obvious to us
  • We need to parse the test case variables in order for them to make sense in the system (a bit more of automation)

The problem about a human deciding the rules and the variables is that some of these variables could be measuring the same things or relate to each other without it being obvious to us.

An example could be: when a feature is new and the risk is high there might be a low probability of the test case to fail, so we might not need to execute it. This could happen because, knowing that the risk is high, developers might put more efforts on the code. (Note: This is hypothetical, not necessarily the case)

That is why, while it is important to analyse as many variables as possible, we still need to get a compromise and try not to fall on these cases, for which we need the experts… or a system to discover automatically the importance of the variables. But this is…well…another story.

 

 

Virtual Reality start pack (and VR Udacity nanodegree experience)

Posted on by noemielis

If you are a regular reader of my blog, you are likely to expect a testing story. This is not exactly such, but have some patience, as I will link it with testing in upcoming posts.

Virtual reality is a field I was always curious about, from when I was a little lynx, before it was even possible to bring it to the users (as the devices were not quite portable back then).

On the other hand, I was looking to do a nanodegree course from Udacity to learn something new and keep myself updated. When you are working as a developer for a while (especially developers in test) you need to keep up to date. I once heard a feedback about a candidate that was interviewed by some friends, that he did not really have 10 years of experience, but he had a repeated 1 year of experience in a loop of 10, and this can easily happen to anybody. To avoid this, what I do and I recommend doing is: keep learning new stuff.

And so, I decided to course the VR nanodegree course in Udacity.

Advises if you are considering VR nanodegree course:

The first thing you need to know about virtual reality is that you will need a device for it, otherwise you won’t be able to test anything you do.

The second thing you need to know about VR: if you want to work in multiple platforms, you also need multiple devices. This might seem obvious for you, but most of the current technologies (think about mobile, for example) have emulators to be able to deploy and test in different devices. However, for VR this is not there yet (as for the time I’m writing this article and my knowledge).

So, if you are planning in getting into the nanodegree course and into actual VR development, get ready to purchase an HTC vibe or an Oculus Rift, unless you are lucky enough to be able to borrow one or unless you prefer to take the speciality about cinematics and 360 recording. I ended up picking this speciality. Not that I did not want to spend the money on a VR cool device that would allow me to also play cool games, but I have recently moved countries (and continents, in fact) and I did not want to carry much stuff with me around.

One more thing to take into account: VR devices might come with minimum computer’s specifications, so you might also need to update your computer in order for the device to work properly.

Lucky for us, in VR we can also develop for mobile, which only requires a cheap device in which to connect your phone (and you could even build your own one). You can’t do as many things as if you also have hands controllers and body movement detectors, but you can still do some cool things. For the nanodegree first modules, this is all you need and the course provides a cardboard to the students (which is great because it has a clicking option and some other devices don’t have this one)

However, there is another thing that you could also get stuck with, although I think there are some workarounds but it would at least slow you down: you cannot directly develop for IOS phone from a windows device, you should do it from a MAC.

In terms of content, I would advise you to be interested in multimedia and gaming if you decide to go through the course.

Feelings about the course itself

I actually really enjoyed the course (except for the speciality that I was forced to course because of lack of devices). I think the content is quite good and the projects are challenging and open for creativity.

It’s also great to network with other people with interest in VR.

In terms of testing on VR, there is currently not a module about this, but they do explain many things about performance, VR and what a good VR application should look like, so I believe this content is covered across the course.

Where should I start if I want to learn more about VR?

First of all, I think you can do this course for free if you don’t mind not having the degree (you cannot access the evaluations). That could be a good starting point and you could always join for the assessments after, which might save you a bit of time and money.

However, I’d say that the best way to get started is to actually try a device and try some apps. In Android you can download the ‘cardboard’ app and expeditions. Also, you can look for VR apps or games in the phone store (whichever your phone’s OS). Another way could be checking in steam (with a more expensive device), youtube or even in github to see someone’s code. For example, you can check out mine.

Last but not least, you can also install unity as it has an emulator that might give you an idea on how the world would look like and try to start playing around. There is plenty of documentation about it. Another good tool to check out is unreal, you don’t need as much development skills with this one.

What next?

So, you have checkout some VR apps and devices. You might even have created some small apps. The next step would be to be able to tell if your apps (or someone else’s) are of good quality (this is a test focused blog, after all).  For this, we should have in mind some new considerations for every type of testing, but that’s…well…another story.

Examples of AI applications and how possibly test them

Posted on by noemielis

Recently I attended an online crowdchat hosted by ministry of testing about testing AI applications.

The questions were very interesting, but it was hard to think of a right answer for all AI applications, as this is a very broad field. Explaining it over twitter would be confusing, so I thought I may as well create a post giving some examples.

Kudos to someone on Twitter that mentioned supervised and unsupervised learning at the end of the chat. I was very sleepy at the time (the chat started at 4am my time) so I was not able to find his tweet in the morning to vote for it. I think that we could understand better the types of AI applications that we could have if we divide them in supervised vs unsupervised. More information here.

Supervised learning examples

The idea behind it is easy to understand: These applications have a phase of learning in which we keep feeding them with data and rewarding them if they produce a correct result, while punishing them when they don’t until the produced results match the expected results within a threshold. (In other words, until we are happy with the results).

Let’s ignore for now the exact ways in which we can punish or reward a machine and just focus on the general idea.

After this learning phase, we generally just use the application and no more learning takes place. We “turn off” the learning. This is called inference phase. Not all the applications have inference phase; sometimes we want them to keep learning from the users, but this can turn out to be problematic, as we will see further on.

I think these are the easiest AI applications to test functionally speaking, as we just need to pass new data and check the results obtained against the expected. Apart from this, they behave just like any other application and we can also go through the other types of testing without many changes (performance, security, system..)

NPR / OCR:

Imagine for example a number plate recognition system – once the system learns how to recognize the numbers in the license plate, you don’t have to keep training it. The application can use the learned patterns to verify the new number plates.

There are many tests we could think of here, without caring for how the application gets the results: try with characters that have strange typography (if allowed in the country), tilt the number plate, check the boundary in distance from the vehicle…

An OCR (optical character recognition) application could also be done with this technique. In fact, the number plate recognition system could be considered as a specific type of OCR.

Digital personal assistance (Cortana, Siri, Alexa…):

Quite common nowadays, they help you find out information using voice commands. They could also use supervised learning (although, I believe the right classification for them would be “semi-supervised learning”, but let’s think of them as just supervised for the sake of the example). However, in this case the application keeps learning from the users. It stays in the learning phase.

The reason they can ‘safely’ do this it is because they collect data from the users but not their direct input in whether the result was to be penalized or rewarded. An example of application getting direct input from the user to keep learning would be a chatbot that guesses something and asks if that guess was correct. This could be easily tricked by dishonest users.

Applications that keep learning are much trickier to test, even functionally, as if we pass wrong inputs to test, they will learn wrong. If I had to test one of these, I would use a copy of the state of each iteration we would like to test in an isolated environment, so we don’t break the acquired good learning. For performance testing it would be best to use valid data, to ensure the learning process continues well.

If anybody is concerned about AI gaining consciousness, this type of applications would be the problematic one, as they could be learning things we are not aware of depending on the power that the programmer and the user gave them and the data they are able to collect. This brings up the question: Should testers be responsible to test consciousness?

Unsupervised learning examples

The key of these applications is to discover relationships on the data without direct penalization or reward. They are very useful when we are not sure of what the output should be, and to discover things that we would not naturally think of being related.

There are two types: Clustering (when the system discovers groupings in data) and association (for discovering rules describing data). I won’t go deep on them in this post, as it is a lot of information as it is.

Tailored content-advertising (Amazon, Netflix, Google…)

These apps want to be able to predict what the customers that bought something would be interested on next. In fact, digital personal assistance tools could also use this data to help you find what you want (that’s why I mentioned before they should be classified as ‘semi-supervised’ learning). I cannot think of any ways of testing this except checking on the impact on the sales after the application is in place, but this could potentially be subjective to chance or other factors not related with the application itself.

Apart from that, the test of the application should be the same as we already do with non-AI applications (not just the results, but how the user inputs the data and how the application responds and shows back the data…) Imagine this as a feature of a bigger product, all the other features would also need to be tested as well.

The moral impact of these applications, in my opinion, is that at some point they might be telling you (as a user) what you want, even before you know you wanted it.

What could possibly go wrong?

What should we be careful about in AI that might not need so much attention in other  apps?

Things could go very wrong if we leave apps learning constantly and we leave the users to provide the penalization or rewards. You probably have heard of applications such as image recognition systems and chatbots becoming racists and sexists. Sometimes this is because the test data given to the application is biased, but it could also be because of trolls playing around with the application in unexpected ways and giving rewards when the application is wrong.

Also leaving apps learning on their own is not the best idea, as we do not control what they are actually learning, as mentioned before.

If you are interested, I found an article with some more examples of issues with AI applications here.

What else have you got?

Below is a list of readings that I found very interesting while researching for this post (a couple of the links are about video games and AI):

How “hello neighbor” game’s AI works

AI predicting coding mistakes before developers make them

Examples of AI

Game examples of AI

How would you test these applications?

What do you think about the moral connotations?

If used well, AI could be harmless and powerful. In fact, it could also be a good tool that we could use for automating our testing, but that’s…well…another story.

Interviewing your interviewer (17 tips)

Posted on by noemielis

This topic is probably not only for testers, but, since I have been dealing with it recently, I thought some people might find it interesting. Also, I know, I owe you a couple of other posts (I’m working on them, but I am also moving houses and I have limited time to write right now)

I have been told that I ask good questions at the end of the interviews, so I’d like to share my system in case someone else finds it useful. Tip: the important thing is not to ask questions to impress your interviewer, but to use this time frame to find out things about the job and the company.

I usually get very nervous when I do interviews, even though I have interviewed people myself as well. I find, that a good trick for me not to get so nervous, is to think of the process as a two-ways interview, in which you are also interviewing the company to verify if you really want to work in there.

I know this might be hard to take, especially for beginners: “How would I be interviewing a company that I want to work for? I just want the job, if I didn’t like the company I would not have applied for it.” However, it is important to know as much as possible about the job you are about to be doing for a good while. That is why it is crucial that you ask as many questions as you can in order to understand how they work in that particular team and what is expected of you.

Another point, if you can, is to double check the answers with the interviewer’s reaction to the question. I mean, they are supposed to say good things about the company… imagine how bad they would look if the candidate says something like “well, I need to drop my application because the interviewer told me this is an awful place to work for”. But, if you are paying enough attention, you can see some reactions such as long pauses or struggles, that could lead you to believe that they did not feel comfortable with the question or they are trying to sugar coat their answers.

By now, you are probably thinking that this could be a good advise, but you would like to see actual examples of these questions. I owe you a couple of code samples by now, so I won’t let you down on this one. Be careful, some of them might be in the job description, and asking them might show little research and be so annoying for the interviewer as when they ask you to walk them through your CV (which usually means they have not bothered to read it fully). Below some examples of things you might need to know before joining a company (note, when I do an interview these come naturally to me, it depends on the specific job, these are general examples that I can think of right now):

  1. What process are they following? (Agile, waterfall…)
  2. Would you be joining the team of the interviewer or a different one?
  3. How would you relate to the interviewer in the company?
  4. What technology are they using? (The description usually would mention one or two, but you might ask what would be the biggest used, or for a technology that is not in the description, for example, what they use for source control)
  5. Do they do code reviews?
  6. What’s the relationship between the developers and testers? Do they sit together? Share code? Do common code reviews?
  7. How often do the interviewer…? (meet clients, have team meetings, create new features, spend in paperwork such as performance review…)
  8. How long are the springs? (if using agile)
  9. How many times did the interviewer use *insert benefit* this month?
  10. How do they do the performance reviews? How do they measure performance?
  11. What are they expecting of the candidate?
  12. Is there a possibility of *insert benefit*? (getting bonus, stock, gym membership, learning expenses… this depends on what you are looking for in the company)

Some extra tips:

  1. Try to ask the right questions for the right interview: technical interviewers might not know the answer to an hr questions (for example benefits) and hr people might not have an answer for technical questions (for example technologies they use). You might be wasting their time and not getting your answer anyways, so it is better to save the questions for the right round. (Be sure to learn about the rounds to know when to ask what)
  2. If you can, try to say them in a way that sounds a bit more personal for the interviewer, they are more likely to give you honest answers if you are asking for their opinions than for the company’s protocols. For example, the style of number 9 is more personal than number 12 for knowing about benefits. While number 9 gives you information about the actual behaviour of your co-workers and the non-spoken politics in the company, number 12 can give you room to negotiate a particular benefit that is not usually given (you can use this style with hr).
  3. I have said this already, but: don’t ask them for the sake of asking questions. Think about what it is not clear to you ahead of time and take a pen and paper next to you when you ask them to write down the answers. This can help you not to repeat yourself and to remember everything at the end of the process. It might be something you can negotiate or it might be something for you to discard the company (or the specific team). This does not only give you valuable information but puts you in a more powerful and confident position when doing an interview rather than feeling under test.
  4. Don’t get angry or depressed if you don’t get the job: sometimes it’s just a matter of being lucky and getting an interviewer that connects better with you. Sometimes a company might offer different positions or to work for teams that use different technologies that might be more aligned with your experience. And above it all: if your interviewer is pushing you back for not having the exact same expertise as him/her, you are probably better off not working with that person anyways. I think the trick for a company to work well is to have people with different set of expertise: they might not know something with the same depth as you do, but they might know a lot about something else and you both can learn from each other.
  5. Keep trying and practising: doing interviews might be extenuating but, with enough practice and asking the right questions, sooner or later, you’ll get just the job you want.

Let me know in the comments if you can think of other good questions to ask and let me know if you like these sort of posts about the interview process. I could tell more about it, but that’s… well… another story.