The HONEST (anti)guide to WFH

With the lock down that many countries are having these days and the need to work remotely, I’ve been noticing an increasing number of posts about how to work from home. I think many of the advises are quite obvious, but some of us tend to gravitate towards a “different” experience when we start doing it. So, here is my satiric guide to how to NOT work from home:

Free-Photos / 9091 images – Example of a perfectly fine home ‘desk’
  1. Turn off your alarm. No commuting means extra time, and the office has flexible starting time, what could possibly go wrong?
  2. Wake up, go to get some breakfast, put your phone to charge and realize you had a meeting 10 minutes ago. Which genius decided to set up a meeting at 9am when everyone usually starts arriving at 9.30m?
  3. Put on a shirt. Don’t worry about the bottom part (it’s an online call and you’ll be sitting down) If you are female improvise some make up without washing your face (no time for that).
  4. Take your laptop and sit in the living room. Everyone is still asleep, surely they wont disturb you.
  5. Try to pretend you’ve been in the call all this time when it’s your time to speak.
  6. Your favorite relative appears in the background and turn the TV on. Turn microphone and camera off while taking the laptop to a more private room.
  7. Come back to the call and apologize for having “connectivity issues”. Turn the camera off and on a few times to prove your point. Finish your talk.
  8. Your relative walks behind you and starts speaking to you without noticing you are in a call. Shut them up.
  9. Realize your microphone was on. Turn off.
  10. The call continues. Realize that the camera is pointing down and you were still wearing your pajamas. Turn it up and hope nobody noticed it. They have.
  11. Finally the meeting is off. Go to speak with your relative who is now very angry and offended because you shut them up and try to explain the situation. They don’t quite understand it.
  12. Go back to your improvised desk. Start checking your slack and email.
  13. Realize is time for lunch. Go to have lunch.
  14. Go back to work. Notice everyone started pinging you while you were having lunch.
  15. Reply to your coworkers, even the ones that are sending you blogs about how to work from home and other jokes.
  16. Go get a snack.
  17. Go back to work. Start reading blogs about how to work from home.
  18. Go to get a drink, the snack made you thirsty.
  19. Go back to work. Remember you were working and close the blogs. Do a bit of work.
  20. Go to the toilet (badly needed now because of the drink you had before).
  21. Go back to work. Realize the day is almost done and you have not done anything useful. Start doing some work.
  22. Your relative comes over. You chat for a bit because you feel guilty about before.
  23. Go back to work. Read the latest news about the global pandemic.
  24. Your neck hurts and back hurt because your desk and chair are not ergonomically set and you have the habit of sitting down in weird positions. Go lie down.
  25. Go back to work but immediately pick up a personal phone call from someone asking how everyone is getting on.
  26. Go back to work, it’s time to turn off the laptop but since you’ve done nothing and you are at home you can make up staying a bit longer…
  27. It’s 9pm. Realize you’ve done nothing really useful today, but tomorrow… tomorrow you’ll be super productive! All you need is to do is to set up some daily milestones and read one more blog about how to work from home.

I hope some of these made you laugh a little and was worth your quarantine’s time. These days I am working on a side project, but that’s…well..another story.

Test architecture in a nutshell

Picture from ulleo / 3990 images – a plant in a nutshell, literally

In my career I’ve gotten a lot of questions about test architecture: How to start testing when nothing has been done for years? How many test projects should there be? How many different type of testing do we need? Who should be in charge of testing in percentages? How to escalate testing? How to deal with big applications? How to deal with existing tests that are incomprehensible? What tools should we use? How many browsers should we test (are there more than two!?)?

First things first: Each application is different, each company is structured in a different way and with different resources and budget and each one has a different maturity state (which does not mean they should aspire to go to the next state, maybe they are right were they should be). There is not a ‘one size fits all’ solution and that’s why I recommend you check it with an expert. But here are some of my previous experiences, in case you find them useful.

On getting started

The first things I ask when I start a conversation about starting testing for a company are:

  1. What process are they following? (Agile, waterfall…)
  2. What technology are you using? (For development, any existing tests, code sharing, code control, task management…)
  3. What’s the relationship between the developers and testers? Do they sit together? Share code? Do common code reviews?
  4. What size is the company?
  5. Do developers write unit tests?
  6. How many people are dedicated (or plan to be) exclusively to tests?

Do they ring a bell? Here are some of the questions I suggested you could ask your interviewer if you are interviewing for a job. Of course, depending on the answers to these questions, I would suggest a different set of tools and practices (or if I am looking for a job, I could tell if I would be able to grow professionally and help the company given the circumstances and what to expect of the position)

Some more that are truly valuable are:

  1. What does your app or company do?
  2. How many clients do you have/plan to have?
  3. What’s the current vs potential growth?
Image from StartupStockPhotos / 121 images – all architecture need a study of the company first

In my experience, when someone asks for help getting started with testing, there are three status the company could be at, and therefore three things might be needed:

  1. Convincing upper management of the need to test: Sometimes you need to be able to explain to high management of the need for testing so you can get started with it. Their main fears are usually time and budget. Addressing those fears are not always easy and depend on each case, but the most important question is “Why do we test?” I leave this one for you for now, but if you are able to explain it, this point would be very easy to deal with.
  2. Convincing other team parties: Other times, they know testing is needed but they don’t know where to start (hint: unit test and TDD) When you don’t have a test team in place yet, I usually suggest the developers start getting into test. Then it would be nice to have someone with test experience to orchestrate these tests, to educate the team about testing and to start creating a test system for end to end and other sort of testing, or even doing manual checks. However, from when you get someone to when you have it all up and running, it might take a while, that’s why it is good that the developers start spending time in unit testing, as ideally they would keep doing this afterwards anyways.
  3. Getting a full framework up and running: In this case, the company have some unit testing and/or manual testing but they need help creating an automation framework and identifying other testing needs. Maybe they are also feeling insecure about the current tests and want to make sure they are not missing anything or want to improve their system (see the next two points).

On keeping it going

Sometimes people are struggling with their current test systems. They might be wondering if they are doing their best as it is, or if the tests they have provide actual value. The questions you need to ask yourself are:

  1. Is there any repetition that could be reduced? (how many current tests and what type of tests do you have? If they run over 15 minutes, this should likely be improved)
  2. Is what we are testing really important? Sometimes we test things that don’t really provide any value but it’s taking time and budget to implement and keep.
  3. Are there any tests that are not up to us? For example, downloading something from a browser, unless your app IS a browser, it’s probably something you should not test.
  4. Are we executing out of date tests? Keeping the system clean is highly recommended, if a feature comes back, you should be able to retrieve those tests too.
  5. Could we use any tools for tracking the issues to understand where we need or don’t need tests?
  6. Do we have the right tests for the right moment? Integrating tests with CI/CD in different branches means you need to decide what to run and check and when.

On escalating

Horizontal scaling means that you add more machines to your testing. For example using the cloud to test in different systems (more web browsers, mobile vs machine, different OS…)

Vertical scaling means that you scale by adding more type of tests (security, accessibility, performance…)

Identifying the type of tests you need and what systems should be covered would be part of an architecture design.

From Free-Photos / 9091 images – Don’t forget to have a conversation with the team before and after designing the test architecture. Everyone should be on board

On moving on

You’ve set up one or more test frameworks. Your team (developers/SDETS…) are managing the new tests which are carefully planned and being executed as part of the CI/CD structure. The analytics are set up and alerting of any problems or improvements… Could you ever be done with testing?

Most people in quality would straight away jump to the easy answer: “you are never done with it”. However, I feel like bringing up the argument: I believe sometimes you are. The same way as you might be done with development too. You will never be 100% happy about it, but sometimes you’ve done your best and maintenance it’s all there is left, at least for that project.

What do you do when you are developing and you are done with that project? You move on to the next. Of course, someone should stay to fix potential issues and provide support, but you wont need such a big of a team or the same sort of skills for this. I argue here, that the same could be said for testing. Development and testing come hand to hand, when there are less changes in development there would be less changes in testing.

Does that mean nobody should look into it anymore? No, that means you need someone with other set of skills to do that. Not worse or better, just different. Some people enjoy being at the beginning of the projects (I love doing research and creating new stuff) but other people prefer maintaining them. And this does not need to be after all it’s escalated, maybe your company is in the place it should be, and investing in more testing might not be necessary, even if you have the budget for it. The problem is, of course, discerning when that is the case, for which I highly recommend talking to a professional.

From JACLOU-DL / 5460 images – look at this cat: this cat is moving on, why can’t you?

On hiring the right test manager/architect

As I mentioned before, having a test expert helping figuring out the maturity of the company and what is needed to improve the quality of the product is very important. What do we call this expert? Test manager and test architect should not be the same position, although, sometimes companies use the descriptions a bit freely.

Test manager: A test manager makes sure the tests are done correctly (being them from test or dev team) and good practices are implemented. The position requires deep knowledge about testing and people skills.

Test architect: The architect designs the frameworks and tools to be used by testing. The position requires deep technical knowledge and experience planning and coding things from the start (also deep knowledge about testing).

Not all companies need both of the positions, and sometimes both jobs would be done by the same person. But being called whatever it’s called, someone has to make sure the system to do the testing is in place and someone has to make sure there are good practices and the right people understand how and what to look for in the quality area.

Therefore, in order to look for the right skills, you need to understand what you need. I have been asked many times for help defining test roles (I talk about the roles I’ve been in here), interviews and even salaries. In fact, there was a time I had to tell my interviewers how to interview me (and, funny story, somehow I ended up not getting the job!) But that’s, well, another story…

Making friends with the impostors inside

Disclaimer: These are only my experiences and this should not at any case be taken as a substitute for any sort of medical advice.

Photo by fotografierende from Pexels

First things first: It is extremely difficult for me to talk about feeling like an impostor. I’ve read about the power of words and how when you say “don’t think about a red hammer”, what do you imagine? A red hammer. So, if anybody is to say: “I feel like an impostor”, whoever is listening is going to imagine you as one. I’ve read about this syndrome before and I couldn’t help myself from thinking: “hmm… if you are feeling like one… maybe you are?” And maybe I am, let me explain…

When people talk about the “impostor’s syndrome” they usually talk about feeling like an impostor. I think it’s more a feeling of not being good enough or feeling different than the others, or both, and that we have multiple of these “impostors” inside.

So, yes, I’ve felt “not good enough” and I’ve felt “not fitting the mold”, and I was probably right every time: I wasn’t good enough for my standards and to myself and I was different from the rest, but everyone is unique in some form.

An impostor on Test

When you are an SDET (Software developer engineer in Test) you definitely are not fitting the mold: Developers tend to shy away from these positions and testers need development skills for them. I remember trying to convince people applying for Microsoft to join the SDET teams instead of SDE teams, because they felt they would be paid less or have less importance.

Photo by mentatdgt from Pexels (not necessarily an impostor)

Microsoft decided to have a separate team for making tools and tasks related to testing (as explained in this book, at least the version I read years ago). This should work out the same way as there are specialized developers for front end or back end. Many other companies followed this approach. But there are so many different companies that work in different structures and call different positions different names, that sometimes concepts get mixed up. And when you are a SDET you get offered all sort of positions, and usually, yes, less salary than a SDE. After many years, Microsoft decided to move everyone to Software Engineers and split the tasks evenly between the teams (which I discussed on previous articles talking about test roles and the future of testing).

On a different company, I’ve worked with SDE’s that have only worked with manual QA’s before, so they could not understand that I was a developer too. These are some of the things I had to hear:

  • “You are lucky, SDETs now are being paid almost as much as SDEs” – I’m soooo relieved that I won’t have to decide between selling my computer science degrees or moving to a position that takes away one letter from my title
  • “I am not doubting you have used it but… you can’t know the same about X technology that someone that uses it daily” – Do you REALLY use it daily? I heard tons of front end engineers talking about back end stuff and nobody is doubting their knowledge based on their lack of daily use
  • “Testing is not part of a developer’s job” – Not sure if you saying this is a bad or a good thing…
  • “We would not mind you to be in our code reviews, but… it would slow us down, better to restrict them to only two people (dev architect and dev lead) reviewing the entire team” – Ignoring the bottle neck you are creating, finding issues during code review would make the process FASTER… and if it is slowing it down you should definitely worry about the quality of the code being written
  • “It’s great that you checked the developers unit tests in your previous company.. But, were they so bad that they needed a tester checking that?” – What they were was actually good team players that split the work evenly and accepted advise
  • “It’s nice of you to be interested on being added to the design meeting, but it is really technical, you will get bored” – You know what? Just take notes afterwards, so I don’t bother you with my “lack of knowledge” (and I save myself from another meeting in which you are discussing technicalities about implementation to assess dominance)
  • “So..your university…was it technical? I mean, did you do technical stuff or was it theoretical but was giving out technical degrees?” – Well, to be honest, I was once walking down the street and I found a degree title with my name on it…(rolling eyes, the average of time for people to finish my degree in my university was twice the expected)
  • “Well, but there are different things in development, for example, design patterns…” – OOh design patterns, really? it sounds so fancy!! Please keep talking… I learnt this stuff on the second year of university: they are just repetitive ways of doing something that people with experience realized about and shared so other people with less experience can brag about… sorry use them too.
From knowyourmeme.com (my face on the last comment)

Of course, at the time, I didn’t think quickly enough these responses. I mostly stared blankly at the person without quite believing what I was hearing. It takes a while to build on responses, but what good would it have made anyways? Sometimes, people just talk because they like hearing their own voices. In Plato’s words “Wise men speak because they have something to say; Fools because they have to say something”.

Sometimes, it does not really matter what you can tell them, they will be stuck on their own beliefs and versions. Sometimes people need to feel, to see, to experience… in order to change their minds. And I believe I did my job with that, since at some point they started coming to me for advise and trusted me much more. But it took a while.

A quick note on degrees: I think having a degree does not necessarily give you better skills. I think degrees are good for having the bases and guidelines. But, no, you don’t need to have a computer science degree to be an SDET, but neither you do for an SDE position (I’ve known plenty of SDE without any sort of certification). However, you have to be really applied and be able to demonstrate skills to do the job; having a degree, experience or some projects you can show are some ways for proving this.

How do I deal with my impostors

Well, as the title of this post states it: I made friends with my impostor’s feelings. I have not overcome them. They are always here, part of me, waiting to get my attention, but I now acknowledge them and learn from them.

Picture by DigiPD from Pixabay (an impostor kitty)

I may not be taking the right steps but here is what I have told my impostor in the past :

  1. Use it on your favor. People that underestimate you are the ones that get most surprised when you get to show your skills, if you feel this is the case, maybe it is a good thing.
  2. Let people be wrong. Haters gonna hate, you don’t need to proof yourself to anybody but you. This is really difficult sometimes, but I learnt to care a bit less about it.
  3. Be kinder to yourself. Most of the times, this is all in your head.
  4. Use it to improve, learn more, grow… I think feeling you are not good enough is a great incentive to get better. Think of the words of Mark Twain: “The secret of getting ahead is getting started.”
  5. Be kinder to others. When someone makes you feel bad you realize you might be making others feel bad too, it can make you more empathic. Use it to reflect about your own acts and also try to understand why the other person is acting like that towards you…maybe they are the ones with insecurities?
  6. Realize that nobody was born knowing everything. Even when they do know about one thing that does not make them automatically know everything else there is to know. A lot of people love flexing. A lot of people have actually not an idea of what they are doing. In the end, everyone is constantly learning, that’s what life is about.
From https://knowyourmeme.com/memes/i-have-no-idea-what-im-doing. I like to imagine this dog with the face of the people that seems incredibly talented or never wrong.

At times, the impostor can get louder, for example when a deadline is coming closer or when public speaking. It does not matter how many times I do this I still can get nervous and anxious about a talk.

A especially bad time for feeling as an impostor is when you get laid off, it does not matter the reason for it, you always feel that if you really were an “A player” it should not happen to you. Also interviewing… this is especially troublesome, as your value might be calculated based in a company’s needs and not your actually qualities: When you need a pencil, a pen will never be as good.

Finally, when you get to join a company the impostor might be calling out the expectations people could have of you.

Those times that the impostors are louder, are when a special effort is required, by listening, understanding and talking to them, but every time it gets a little bit easier. I personally have many other examples when I felt like other impostors, but those are…well…another story.

API Testing

What’s an API?

An API (Application programming interface) is a set of calls that an application does to communicate parts of the application. For example, the user’s view (browser or UI) with some software component (in a remote server or within the user’s computer) that makes the necessary operations for an application to function.

If you are curious about how this looks for a web application, you just need to check the ‘network’ tab on the developer’s tool on your browser. You can see there are many calls happening in the background when trying to reach a website.

Network tab on Chrome for google.com URL

If you click in one of the calls of the left hand side, you will get some information as on the right hand side. The request URL is the address that the call was trying to hit.

For these web calls, there are two main methods: GET (to request some information from the server) and POST (to send some information to the server). You can learn more about other methods here.

The next field is also very important. This is the message that we get from the server when the call is executed. In this case it is 307 (a redirection). If you are curious about what other statuses number mean, you can check this web, if you are a cat person, or if you are more of a dog person this one.

There are two widely used protocols for sending information between devices: SOAP(stands for Simple Object Access Protocol) that sends information with XML format and REST(Representational State Transfer) that sends information with several formats such as json, html, xml, and plain text (see this article for further explanation in the formats).

Tools to test API’s?

Please keep in mind that the tools mentioned below are not the only ones in that you can use for API testing. I’m talking specifically about these ones because they are the ones I’ve used in the past.

In the section after this one, I’ll show an example about how to do an API test.

Swagger:

According to their website, Swagger is an open source and professional tool-set that “Simplifies API development for users, teams, and enterprises”

https://swagger.io/tools/swagger-ui/

I have used swagger UI as a way to easily check API URLs and understand the calls to then add them into my test code, but, I have not tried all the Swagger’s tooling set yet. I think this is a easy way to communicate changes on the API across the team and document it.

Alternatively to this, the developers should document their API calls in some other form, generally some list, as Twitter does here. The problem I have had with this option is that sometimes the documentation might be out of date and then you need to figure out in the development’s code what’s the exact API call. With Swagger, the list of calls should come directly from the code (clarification UPDATE: depending on the API implementation and if the changes are back tracked) , which makes it easier to handle and up to date (clarification UPDATE: you shouldn’t assume anything in testing).

Swagger is supported by SmartBear, same as SoapUI, so for API testing with it, please check below.

Soapui and Postman,

SoapUI is ” a Complete API Test Automation Framework for SOAP, REST and more”. There is an open source version and a professional one featuring more functionality. The API testing part looks like this:

https://www.soapui.org/docs/endpoint-explorer.html

I took the image from their website and it is quite self-explanatory. Besides, there is a lot of documentation to get you started in there.

Postman is a “collaboration platform for API development”. In the development sense, it provides automatic documentation, so there is no problem with developers making change on functionality and forgetting to upload it.

For API testing, it’s very easy to get started. You can create REST, SOAP, and GraphQL queries. It supports multiple authentication protocols (I talk about this later) and certificate management. Please refer to their website for further information.

Wireshark and Fiddler:

These two programs are very useful to analyse network packets. They are powerful programs and a must known for security, network and performance testing and checking the packets at micro level. You can actually see the exact data sent over the network. However, if what you are looking for are tools for API testing, I would probably not go for them but the ones above, because they are more high level and specific for that.

That said, I have used them before in order to test API’s that required specific secure certificates and for debugging issues (specially performance ones). If you are interested in knowing more about Fiddler, I recommend this article. For Wireshark, this one.

https://blog.wireshark.org/

How to do this programmatically?

If you want to add this testing into your automation code, you have some help on with the tools mentioned before. However, there are many ways of doing these type of calls with different programming languages. For example, here is how to do a rest call with python:

import requests
# make a get call
response = requests.get("URL")

# do something with the result
response.status_code # this gives you the status code as mentioned above
response.json() # this gives you a json with the response on it

# make a post call
response = request.post("URL", {json data})

It gets a bit more difficult when you need to add parameters, authentication or parsing some types of data, but there is plenty of documentation about it all. Let’s see an specific example using the API provided by numbersapi.com.

import requests

response = requests.get("http://numbersapi.com/42?json")

print(response.status_code)
print(response.json())

The result when you execute the code above is:

200
{'text': '42 is the result given by the web search engines Google, Wolfram Alpha and Bing when the query "the 
answer to life the universe and everything" is entered as a search.', 'number': 42, 'found': True, 'type': 'tr
ivia'}

With Python, you could play with the json data to easily retrieve and validate the text, or the number, that there is some result…

For more information about what exactly test when testing API, I think this post is wonderfully well explained (they use postman as example).

Why should I care? UI VS API testing

UI (User interface) testing is the best way to simulate the actual behaviour of the users. However, we tend to re-test things in the UI that could be covered already by testing the API (and in some companies this could be done by a different group or team).

Let’s say a developer changes a call for an API. Let this call be the list of movies that someone liked. Now imagine this API call is not modified in some part of the application, the result being the user cannot find its liked movies. What’s happening in the UI test?

We will then get the UI Test couldn’t find an object. This could be due to the API call being wrong or a bug in the automation, an update on the way of the object needs to be retrieved, a button not working properly, the object being hidden somehow…

However, if you have an API test for it, you should be able to understand that the call is not retrieving anything. If you need to verify things such as results of a search, it’s probably best to use API to check the entire list (which could be done with a quick comparison) and let the UI verify that a result appears where it should rather than the result itself. Also, you should be verifying that the API call is correct (and update the test call if it is not).

Level up:

API calls are less likeable to change than UI objects and they generally come in different versions when they do, not to disturb previous releases of the application. This means you might want to add functionality to verify which version is being tested.

It is also interesting to use this to speed up our UI testing. The most common example being the Login method. This method is usually a bottle neck for the rest of the tests, and if it fails, you don’t know what else might be failing or passing and you are blocked until the fix. Whilst it’s super important to have a Login test to make sure that your users can log into the application, performing UI login each time it’s needed for another test, slows down your execution.

Google login screen

What’s the solution? You can use API testing to skip the login bit. Be careful when doing this, it wouldn’t be secure to have an API to do this in production environment. An example could be to set up some unique tokens (see an example about doing this with soapUI here) that are of quick expiration to perform this skip and send it alongside the URL, or have an API call that sets some cookies or session to be logged in.

If you have other repetitive dependent tests, you should consider running API calls for them before continuing with the test that depends on it. This would considerably speed up your test execution and your results would give you more trustful information about the problem.

That said, UI testing is the best way of ensuring everything works as per user behaviour and E2E and integration tests should not be substituted by API tests, use it only as a help and if it is not increasing the complexity and failures of your tests.

Yet another level up: stats

Another interesting thing that you can do thanks to API calls is to find out information about your application and about how users are using it. To analyse and visualise calls in a bigger scale you can use tools such as elastic search and kibana, and even use artificial intelligence to get conclusions from such calls, but that’s… well… another story.

2019 Review and 2020 plan

While everyone has been preparing this year’s resolutions, I wanted to share the ones I had for last year. I was getting the feeling that I did not do very well with them, but after writing them down, it turns out I accomplished a lot and it makes me want to achieve more next year.

Photo by freestocks.org from Pexels

Doing more

One of the things I am often asked by people is: “how do I manage to get to do so many things on my free time?” When I hear this question, my first thought is: “well, you obviously haven’t visited my blog often…”

To me if I can help one single person throughout the contents of my blog or inspire someone with my talks, I would achieve my goal: I would have made the world a bit better. For this reason, I tend to wait for inspiration to write; I might be doing a research for the next post, or working on learning a skill.

Therefore, once you are aware of your end goal, I think the first step to get more things done is to find inspiration. You can get this from reading books, articles, attending to talks, having active friends, joining courses or projects… But, be careful, sometimes this could be counterproductive. Maybe seeing how everyone around you manage their time so well, discourages you for not being able to do as good. Don’t let this bring you down, everyone has their own tempo and you can start at any point in your life.

A trick I use sometimes when watching or listening to some content, it what I call the ‘2x factor’: set it up 2 times the normal speed. I don’t recommend you do this with everything, for me it generally depends on the speaker’s speaking pace. When the speaker is talking slow or about something I already know or don’t find very interesting, I increase the speed of the video. I find generally 1.50x is a good speed for me for most things, 2x might be too fast and whilst I can understand everything, it makes me nervous.

https://www.pexels.com/@neo8iam

If getting inspiration is the first step, the second one, to me, is to get to action. Sounds easy? Get started on anything, even for 5 minutes a day, and you will be achieving more than you are right now. I particularly enjoy the ‘pomodoro technique’ (with rewards), in which you try to work 25 minutes straight followed by a 5 minutes break (in which I grab some drink or snack). I find it very hard to finish the 25 minutes the first two or three times but then I start hating the alarm for taking breaks because it interrupts my flow.

Books

One of my new year resolutions was to track my readings. I’ve noticed I’ve been reading here and there, never really stopping to think about my progress.

First thing I did to firstly, check my progress and secondly inspire myself to read more is to join the goodreads reading challenge. Since I was not sure how much I was currently reading, I set it to be 24 books.

Photo by Pixabay from Pexels

I was shocked to realise that I read 6 books the first month. Ok, let’s be fair, half of those were audiobooks, I love walking around and having them help me get to more information. Because of traveling or other commitments, I didn’t get to 6 books every month, but I also didn’t track them all.

I discovered that sometimes Goodreads didn’t have the book I was reading and sometimes it just removes books from your shelves without noting. I contacted their support (as suggested on that link) and all I was told was to re-add them and make up the dates I read them… This proved to me that it is not a good place to track books so I’ll be happy to hear of other options if you have any.

At the end of the year, taking all of this into account, I achieved almost 60 books in 2019, so I set this as my 2020 goal. However, I want to keep my focus on quality over quantity and I believe this number may also depend on the length and difficulty of the chosen books.

Commitments

In order to motivate myself, I like having some commitments to push myself to do them, for example, participating in meetings and conferences such as the automation guild (it’s an online conference, you might be still on time to join this year’s). The videos for it took me longer than expected, but I learnt a lot because of that. I presented in a total of 5 conferences around the world and spoke in a couple of meetups. If you want to know more of my past and future appearances, you can do so here.

Another way I have of motivating myself is by joining courses. Last year I wanted to finish the VR course I’ve been pushing aside for a while. I’ve been doing some research in VR and that has taken priority over the course, but paying a course monthly and not having time to work on it is really painful. On the other hand, not paying for a course makes me less committed to it. This year, I want to work more on my AI knowledge.

Activities

Another thing you can do to get inspired is to write a list of life achievements or values you want. If you have it nearby and read it often, it could help you focus on what’s more important for you. Also, try activities that get you excited, find other passions.

Some months I was focused on improving Chinese, practising piano and meditating. Besides, on summer I went back to Spain from China, so was recovering from jet-lag for a couple of weeks and then decided to enjoy some time off too.

Another new year resolution was to practice yoga at least 3 times a week and going to the gym at least 3 times too. I’ve been quite regular with this, even when travelling, except the times I’ve been sick.

Photo by Cedric Lim from Pexels

Next year, I would like to plan better the activities and periods of break, the same as I plan for work or conferences. Otherwise, commitments can get in the way and push everything out and compensation later on does not seem to be the best solution.

Traveling

One of my 2019’s resolutions was to travel more, and I think I surpassed my expectations: I travel through China (Beijing, Xian, Zhangjiajie, Tongren, Guilin..), US (San Francisco), Japan (Tokyo), Spain (Valencia), UK (London), The Netherlands (Amsterdam), Germany (Berlin, Dusseldorf..), Switzerland (Zurich) and Russia (Moscow).

I moved to a new city but, because I was meeting many new people throughout the trips and did a lot of sight seeing of the visited places, I did not feel like engaging in many activities there. I did go to a Muse concert (which was, as usually, quite amazing) and walked around a lot. I like doing this when I go to a new city as I feel getting lost is actually the best way of getting to know a place.

Photo by Porapak Apichodilok from Pexels

Whats next

Something else that works for me is to list the things I’ve done in the day at the end of it. Same for the year, as you’ve read. This is a way of being appreciative, which makes me want to achieve more.

Being grateful is also really important. Another daily writing or morning meditation, if you wish. I struggled with this because some days I might not interact with anybody as to be grateful to them. However, you can be grateful to yourself and your achievements and for the opportunities and the good that’s on its way and the good that happens to others.

I’ve counted 12 post that I have half written, so keep checking because there is a lot to come. A couple of them are on VR (I might merge them into one), thoughts to share, AI, API testing… If you are especially interested in one of those, let me know so I give it priority.

I like most of my last year’s resolutions, I want to keep them, and I haven’t added up much to this year: do things to make the world better, block times of rest, have more patience, look at the brighter side of everything, get an AI specialisation, write more on my blog, find a new app for my reading… Would I manage to add all of that? That’s…well…another story.

How to test a time machine

I have recently watched a video from PBS space time that got me thinking: if we were to have a time machine, how would we test it? I’ve seen a lot of “how would you test X” type of questions in interviews, but I don’t think I’ve ever seen this one before (I am not trying to give you ideas for interview questions!)

It’s not rocket science… it’s rocking testing science!

I couldn’t help but compare it firstly to a spaceflight, so I started wondering: how do they test spacecrafts? And what better one than Apollo 11 to start with? If only we had a time machine for getting its source code… Yes, I have been looking at Assembly code trying to make sense of potential testing routines, like this one… and… guess what I found there?

This section of the code is checking the Gimbal lock of the accelerometers! Do you remember the concept from my last post? Maybe I just have a case of Baader-Meinhof, but I do feel Gimbal lock is an important concept to learn, so check it out.

Testing in Assembly was not as ‘easy’ as nowadays (for example, macros does not seem to be a thing that I could find in the Apollo11 programming syntax). Do not expect a page object model or a library with tests or testing functions. Nor common methods for before and after tests. Actually, don’t expect any sort of OOP, to start with.

In my search I could find some files with tests on them, but they are mostly for stressing the hardware by sending signals to the different devices and recovering from bad statuses. Also, spacecrafts might need to check correctness of bits to make sure there are no catastrophic arithmetic errors.

https://upload.wikimedia.org/wikipedia/commons/thumb/3/3c/Ariane_5ES_with_ATV_4_on_its_way_to_ELA-3.jpg/1200px-Ariane_5ES_with_ATV_4_on_its_way_to_ELA-3.jpg
from @wikipedia

Time traveling tests

Imagine we have covered the unit and integration tests for both hardware and most of the software for our machine that could potentially match any currently existing ones. What are the specific cases we should cover for our time machine? The logical cases to think of:

1) The machine should not kill the traveler: We should insert some devices to measure that the cabin of the machine is livable. Also, keep in mind for the rest of the tests, that usually spaceships are first tested with robots inside, rather than humans. (Security test)

2) There must be some ways of safe interrupting the process (risk assessment/testing): In case anything wrong happens, what’s the risk involved for the passenger? (Security test)

3) As the machine goes forward or backwards, the traveler state stays while the surrounding world changes: We could measure this by introducing some other object before that we know it decays and checking how long time it has passed in the machine. (Usability test)

4) There is a way to return: I assume we will want this, so we should test it. (Security test?)

5) Performance tests: how many times it can go backwards and forward in time? (Performance but also, regression test?)

6) What is the minimum time that can travel? (test it) If the time machine requires a rapid negative speed (as the video above suggested as one of the possibilities), I imagine traveling in time as playing billiard or golf. Let me explain: When you try to position a ball on a particular hole, you need to give it the right original impulse (not too much, not too little) but also the right angle depending on the starting point. We are likely to travel space as well as time, so it might be particularly difficult to stop on a particular moment or place. (Which may explain why nobody made it to the time travelers party) (Usability test)

Photo by Tomaz Barcellos from Pexels

7) Test boundaries scenarios: go back to start of the earth (or universe) and forwards to its end. What happens if just before or just after? Does time always exist? Can we travel to a point there is no time? Technically these should be tests we would like to do, but I think they are probably not doable, realistically speaking (can I use this expression in this context?)

8) Try to change a small thing in the past…does it change the future? If a change makes a parallel universe, then we should not ever recover the machine. If we test this, we should look for an event we can easily undo (like maybe turn a light on/off?) (Exploratory test)

9) Time traveler meeting same time traveler. Test paradoxes. (Exploratory test)

11) Test placing a box where the machine is positioning before a trip. Then take the box away, position the machine on that same spot and test traveling to before we positioned the box. Does the box o machine break? (Integration test)

12) Test traveling to when the box is still in place; do the machine or the box break? (Integration test)

13) Try to travel twice to the same exact time and place. (Integration test)

14) Could the machine travel between different universes? (If we have a way of doing so, which might be more likely if we are using wormholes for the trips than negative speed) (Integration test)

From futurism.com (and an interesting article)

15) Is there a maximum/minimum size of the machine? If so, test them. (Boundary test)

16) Can everybody use the machine or only qualified people? (Accessibility test)

Conclusion:

Maybe we have already invented a time machine but kept it secret because it is dangerous. Or maybe we have gone back in time to stop ourselves from inventing it, creating a parallel universe and as a result, our universe is the one in which such machine was never invented (time version of the Fermi paradox)

Whichever the answers to these questions, I hope you’ve found this exercise fun and enjoyed reading this post as much as I enjoyed writing it. Let me know if you can come up with more tests we could do to the time machine (if we were to have one). I will resume my serious posts soon, they will be… well… another story.

We need to talk about quaternions…

I know, I know, this doesn’t seem like anything that has to do with testing. However, I found this concept to be very challenging and I think some people might be interested in knowing about it, especially if they are considering automating objects on VR.

The problem:

You want to be able to rotate the main camera with Google VR. Google design has reached the conclusion that moving the camera in virtual reality should be something that the user does, not the developer (I think it was possible in earlier versions of Unity and Google VR sdk).

So I decided to create an empty object and assign the camera as a child object of it. To move the camera, I move this object. To undo the camera movement done by the user, I move the object the opposite way. Should be easy, shouldn’t it?

Well, sometimes… but other times it does not work. I spent a very long time searching online for other people that had this issue and it was either working for them or not, nobody would explain why this was the case. I believe the reason to be the effect of gimbal lock.

I know, that sounds like a made up word, but this is actually an important concept that you should be aware of, let me explain:

Gimbal lock is an effect in which the object loses one degree of rotation, which is common when you use three dimensional vectors. At some point, two of the rotation coordinates get lined up in a way that every time you move one, the other one moves as well. If you are a visual person, this video from GuerrillaCG (a Youtube channel with information about 3D modelling and animation) explains it quite clearly.

How did I understand what this mean and how did I figure this was the issue? I decided to encapsulate the camera object in another object, and then that object in another one. Then I assigned them some colored spheres and pointers and I run some tests. After a while, it became clear that movements were not coordinated and that the rotation variables were not working as expected.

Attempts to understand 3D issues

Introducing quaternions:

The difference between Euler vectors (the usual 3 components vector that indicates a point in the 3D space) and quaternions (a 4 components vector) could help us avoid the Gimbal lock effect.

Quoting wikipedia: ” In mathematics, the quaternions are a number system that extends the complex numbers. […] Quaternions find uses in both pure and applied mathematics, in particular for calculations involving three-dimensional rotations such as in three-dimensional computer graphics, computer vision,[…] “

I know, it still sounds a bit gibberish… Basically (very basically), quaternions are a mathematical way of representing 3D objects by using 4 coordinates (representing them in 4D space instead), that an Irish mathematician (called William Rowan Hamilton) came up with and decided to literally set in stone, on a bridge.

https://upload.wikimedia.org/wikipedia/commons/a/a2/William_Rowan_Hamilton_Plaque_-_geograph.org.uk_-_347941.jpg
Commemorative Hamilton plaque on Broome Bridge. From wikipedia

They are difficult to understand and to visualise (as we don’t intuitively know how a 4D space should be visualised), but the maths applied to them are easier than to the 3D vectors. For example: if you want to rotate an object on an x plane, on the y plane and on the z plane, the object will not be in the same state if you perform the entire rotation on x and then on y and then on z, that if you do it in a different order (because each coordinate rotates based on the others too). However, with quaternions, you will always get the same result, as you can rotate two planes at once (which is particularly convenient when you are trying to undo a movement or automate a 3D object)

I am linking to a lot of articles below for you to deep dive, but two main points to remember are:

  1. You should keep one of the values of the quaternions fixed (or set to 1) as this is the way of telling the quaternions that there has been a rotation.
  2. Rotation angles in quaternions are half the expected in 3D rotation (so be careful when calculating the rotations in 4D). This is known as double-cover and it’s actually quite useful once you get the grasp on it.

If you are interested in knowing more about quaternions, you can check this and this videos from 3Blue1Brown (a Youtube channel full of interesting mathematical concepts with a very easy explanation, I really recommend it) Also, I enjoyed the explanations on this article. If you are considering working with 3D objects movement at some point, you should definitely watch these videos and play around with the simulators to understand everything better.

Conclusion

It is not my intention to go against the design of a system such as Google VR, and you should listen to the platform owners as, in general, it is wise not to tinker with the camera. However, sometimes I find it to be useful to undo a user’s movement, for example, for automation purposes or if the gyroscope is drifting on its own (more about phone’s sensors in this interesting article).
In these cases, the use of quaternions is so fundamental, that it is definitely worth it to spend sometime learning about them.
The next step after the automation of the camera and other 3D objects would be to automate the hands movements, but that’s…well…another story

Automating the automation: Automating Vstest

As part of my career, I have, on multiple occasions, found it useful to automate visual studio tests (VS tests) execution. The goal of this article is to show how to do so and why. I hope you can find it useful no matter your experience; for the experienced readers, you might get some ideas on how to use the capabilities of vstest, batch files and logs generated, setting up the basics to revisit some of the parts in the future.

If you have never created or run tests in Visual Studio, take this as a starter guide on executing automation. If you have ever run Visual Studio Unit Tests, you should be familiar with the Test Explorer tool:

At times, we might want to include these tests as part of a different system rather than executing them from Visual Studio. It is possible to use some existing functionality in the command line to help you doing so: VSTest.Console.exe is a command-line tool to run Visual Studio tests.

Steps 1 and 2 are for setting up the test data of this post, feel free of skipping them if you consider them too simple. Otherwise, I hope they can help you get started with Unit test in visual studio and general automation with it.

Disclaimer: I’m explaining this from a Windows Operating System’s perspective, but it could be done similarly from anywhere you could run vstest console tool.

Step 1: Create an unit test project

Note: needless is to say that you need Visual Studio installed to follow these steps.

We are going to start creating a test project in Visual Studio. To do so, we can click File -> New -> Project and select a C# template ‘Test‘ as in the image below.

Select the destiny folder and insert a name for it. Refer to the Visual Studio documentation if you have any issues.

Step 2: Create tests

The project should come with a unit test class by default. If for some strange reason that’s not the case, or you want to create another class, you can right click the Project and select Add->Unit Test (please note if you do this but you already had a class, then you might have a duplicate TestMethod1 in your list later on)

Now we are going to add a command inside the test that comes by default (because we are not adding test logic for now, we just add the ‘Fail’ command as shown below). We are also adding a Priority above this test. Then we can copy and paste the lines from [TestMethod] (line 9 in the image below) up to the last ‘}’ (line 14) three times to get a few tests in our test explorer. Change the names of the methods so each have a different name (for the example we have TestMethod1, TestMethod2, TestMethod3, TestMethod4.

For the example, all odd tests (TestMethod1 and TestMethod3) have priority 2 and even tests (TestMethod2 and TestMethod4) priority 1.

To be able to see the test explorer we should go to the menu Test->Windows -> Test Explorer.

And to be able to see the tests we have created we should go to Build -> Rebuild Solution.

Now, you can try and run the tests on the left hand side. They should fail because we are not adding any logic to them. If you are interested in how to do this, leave a comment below and I’ll write another article with more details on this. You can also look up the Visual Studio documentation to learn more.

In the next step we are going to learn how to execute the test cases outside Visual Studio.

Step 3: VSTest

Now, if we want to execute these tests outside Visual Studio’s environment we can use the console that is typically install under the following path:

C:\Program Files (x86)\insert VS version here\Common7\IDE\CommonExtensions\Microsoft\TestWindow

Where “insert VS version here” would be something like ” Microsoft Visual Studio 14.0 ” or “Microsoft Visual Studio\2017\Enterprise”… basically, you can navigate to your Program Files (x86) folder and look for a folder that has Microsoft Visual Studio within its name. Then continue with the rest of the path as above.

Inside the aforementioned folder you can find “vstest.console.exe“. Alternatively, you can download the nuget package and search it in the path where it’s installed.

Typically we would access this file from the command line (be it admin cmd or visual studio’s native tool command prompt)

We can open cmd (as administrator, by right clicking on it) and type “cd path” where path is the path to the above file with the correct visual studio version for your installation. It is convenient that this path is surrounded by quotes (“”), just in case there are spaces the program can recognise them as part of the path and not as a different command.

Now, you can select what type of tests to run by adding some parameters to the call, but you can test it by calling “vtest.console.exe” followed by space an the path to the dll of the test project. For example: vstest.console.exe c:\....\UnitTestProject1\UnitTestProject1\bin\Debug\UnitTestProject1.dll

You should see something like this:

Since you’ve set up your tests to fail. Later on, once your tests are finished and they are all passing you would see something like this:

If you are new to testing, you are probably wondering why would we want to run them in this complicated way instead of using the beautiful UI from Visual Studio. If so, keep on reading, this is where things start to get interesting.

Now, imagine we want to run just the first and third test method, for this we should make the following call: vstest.console.exe pathtodll /Tests:TestMethod1,TestMethod3

As you can see, now only TestMethod1 and TestMethod3 were executed (you can use the names of your test methods). Note that it should be failing for you, I’m just adding the passing image because is cleaner.

Remember we setup priorities before? So how to run the tests with higher priority? vstest.console.exe pathtodll /TestCaseFilter:”Priority=1″

In Microsoft vstest.console documentation, there are much more ways we can use this tool, including parallel runs. Have you started to see why using this tool could be very powerful? What else could we do with this?

Step 4: Creating batch files

The coolest part is that we can create a file with these calls and then, we can use that file virtually anywhere (continuous integration, night builds, as part of an agent to be executed on a remote computer…) These sort of files are called “batch files” because they run a set or batch of operations.

The first line of the batch file would be to CD into the vstest console folder. Then we can add calls to the tool for running the tests we want. Finally, we add a pause to verify that this is working fine.

To do this, just use the window’s notepad and type the instructions below. When saving it, save it with .bat extension instead of .doc or .txt.

cd C:\Program Files (x86)\insert VS version here\Common7\IDE\CommonExtensions\Microsoft\TestWindow
vstest.console.exe pathtodll /Tests:TestMethod1,TestMethod3 /Logger:Console
pause

Remember to change pathtodll to your actual project and add the right VS version. Now, if you execute this newly created file (as administrator), you should see the same results as before. Pressing any letter closes the console that opens up.

If you don’t want to see the results in a console (as if would be happening if you integrate this file with other projects), just remove the last command (pause). The logs of the results are saved in the current folder (the one for the vstest program) and we will analyse them on the last section.

Explaining continuous integration or multi-project execution would be a bit more complicated and out of the scope of this post (but do leave a comment or reach out in Twitter if you want me to explain it). But I can explain how to set up your computer to run this file every night with Windows!

For this, you need to open the task Scheduler (you can search for it in the lower left search box on Windows). Then click on “create a new basic task” on the right hand side and go through the assistant. The most important things are to specify the frequency you want the file to run and browse to select the saved .bat file. Now this file will be running with the indicated frequency automatically (if your computer is turned on at that time).

The next thing you want to do is to check the logs that this file has generated every time it was executed.

Step 5: Saving logs

Running automatic tasks and tests is awesome, but not really useful unless we know the results so we can do something about them.

First of all we should check where the logs are saved. Usually they are saved into a folder called “Test results” within the folder where you’ve run the file. Because we were using cmd (admin) and navigating to the vtests.console folder, it would be created there. In fact, that’s the reason we need administrator permission to run the file. There is a parameter with to run vtest.console to change this location, although my vstest.console was not recognising it, so I stick to the vstest.console folder for the purposes of this article.

I think trx logs are useful and should be always installed by default. To get them we can add a parameter to the vstest “ /Logger:trx“.  The generated file can be opened with the notepad and it will give you information about the run tests. However, we would focus on /Logger:Console as it is simpler.

Another way of retrieving the logs is by using the capabilities associated with Windows batch system. We just need to add ” > pathToFile\file.txt” where path to file would be the path all the way to a file with a txt extension (this file does not need to exist, you can create it with this command). This way a file will be saved with the contents of the console.

cd C:\Program Files (x86)\insert VS version here\Common7\IDE\CommonExtensions\Microsoft\TestWindow
vstest.console.exe pathtodll /Tests:TestMethod1,TestMethod3 /Logger:Console > pathToFile\file.txt
pause

You might want to save different files (by adding date and time) or replace the latest one (by keeping the same name as above), depending on the frequency that it is generated (if it takes long enough, we don’t mind replacing it).

Using the parameter “ /Logger:TimelineLogger” can give you a bit more information of the times of execution, but it will make it harder to parse later on.

Step 6: Playing with the logs

Now we have a text file with the logs…but reading all the files all the time, might be a bit boring..what to do with it? You get it, automate it!

Let’s output just the number of test case that have failed. We can do this with any programming language, but let’s keep going with batch. Why? Because I feel people underestimate it, so here it is:

@echo off
set /a FAILED=0
for /f %%i in ('findstr /i /c:"Failed" file.txt') do (
set /a FAILED=FAILED + 1
)
set /a FAILED = FAILED- 1
if %FAILED% gtr 0 (
echo Failed: %FAILED%
)
pause

The first line of the file allows the output to come up on the screen. Then we create a variable to save the number of times that we find the string “Failed”, we perform a loop with the search over the file called “file.txt”, take one out (because at the end there is a summary with the word “Failed” on it, which we don’t want to count) and only if the result is greater than 0 it is printed.

When executed for the file with priority 1 test cases failing we can see this result on a console:

If everything passes, nothing is printed for this example.

Please keep in mind that with this method, any test case that has the word “Failed” within its name will also increment the search, so this is just for demonstration purposes.

Maybe we would prefer to indicate as well the names of the failed test cases, or to print the last sentence on the file, which has already a summary.

We can also create some code that would send us an email if there are any failed test cases, or push the results into a graph, or send some sort of alert, or even a slack message… There are many possibilities, but they are…well…another story.

Testing on VR world

Previously, I’ve written a couple of posts about how to get yourself started on VR in which I promised some stories about testing on this world.

Why do I call it world instead of application? Because the goal of virtual reality is to create realistic synthetic worlds that we can inspect and interact with. There are different ways of interacting in these worlds depending on the device type.

Testing an application in VR would be similar to testing any other application, while we would have to take into account some particularities about the environment. Later on we will see different kinds of testing and think about the additional steps for them in VR, but first let’s see the characteristics of the different types of devices.

Types of devices:

Phone devices (Google Cardboard or DayDream) – allows you to connect your phone (or tablet) on the device to be able to play a VR app in there.

This is possible because most of smartphones nowadays come with gyroscopes: a sensor which uses the Earth’s gravity to determine the orientation.

Some Cardboards (or other plastic versions) can have buttons or a separate trigger for actions on the screen (as it is the case for DayDream), but the click is usually not performed in the object. Instead, it is done anywhere in the screen while the sight is fixated on the object. If the device does not have a button or clicker, the developer have to rely on other information for interaction, such as entering and exiting objects or analyzing the length of time the user was on the object.

Cardboard VR device – Picture credit mentatdgt

Computer connected devices (HTC, Oculus, Samsung VR…) that generally comes with (at least) a headset and handset, have an oled device with high resolution and supporting low persistence embedded in the headset, so you don’t need to connect it to a screen. They detect further movement, as it is not just about the movement of the head but also the movement on the room itself and the hand gestures. This is done differently depending on the device itself.

We have moved from being able to detect user head movement (with reasonable size devices), to use sounds, to use hand gestures… so now, testing VR applications is getting more complicated as it now require the test of multiple inputs. The handset devices usually have menu options as well.

Before going on, I’d like to mention AR. AR is about adding some virtual elements in the real world, but with AR we do not create the world. However, AR has a lot in common with VR, starting with the developing systems. Therefore, the testing of the two platforms would be very similar

We have talked about the hardware devices in which the VR applications would run, but we should also talk about the software in which the applications are written.

Samsung gear + one handset

Developing platforms:

Right now there are two main platforms for developing in VR: Unity and Unreal, and you can also find some VR Web apps. Most of things that are done with Unity use C# to control the program. Unreal feels a bit more drag and drop than unity.

Besides this, if you are considering to work in a VR application, you should also take into account the creation of the 3D objects, which is usually done with tools such as blender or you can find some already created onlin.e

But, what’s different in a VR application for testers?

Tests in VR applications:

VR applications have some specifics that we should be aware of when testing. A good general way of approaching testing on VR would be to think about what could make people uncomfortable or difficult.

For example, sounds could be very important, as they can create very realistic experiences when done appropriately, that make you look where the action is happening or help you find hidden objects.

Let’s explore each of the VR testing types and list the ways we can ensure quality in a virtual world. I am assuming you know what these testing are about and I’m not defining them deeply, but I will be giving examples and talking about the barriers in VR.

Usability testing:

It ensures that the customer can use the system appropriately. There are additional measurements when testing in VR such as verifying that the user can see and reach the objects comfortably and these are aligned appropriately.

We are not all built in the same way, so maybe we should have some configuration before the application for the users to be able to interact properly with the objects. For example, the objects around us could be not seen or reached easily by all our users as our arms are not the same length.

You should also check that colors, lighting and scale are realistic and according with the specifications. This could not only affect quality, but change the experience completely. For example, maybe we want a scale to be bigger than the user to give the feeling of shrinking.

It is important to verify that the movement does not cause motion sickness. This is another particularly important concept for VR applications that occurs when what you see does not line up with what you feel, then you start feeling uncomfortable or dizzy. Everyone have a different threshold for this, so it is important to make sure the apps are not going to cause it if used for long time. For example, ensure the motions are slow or placing the users in a cabin area where things around them are static, or maintaining a high frame-rate and avoiding blurry objects.

Sitting experience – Picture credit rawpixel.com

If there is someone on your team that is particularly sensitive to motion sickness, this person would be the best one to take the tester role for the occasion. In my case, I asked for the help of my mother, who was not used at all to any similar experiences and was very confused about the entire functioning.

Accessibility testing

Is a subset of usability testing that ensures that the application being tested can be appropriately used by people with disabilities like hearing, color blindness, old age and other disadvantaged groups.

Accessibility is especially important in VR as there are more considerations to make than in other applications such: mobility, hearing, cognition, vision and even olfactory.

For mobility, think about height of the users, hand gestures, range of motion, ability to walk, duck, kneel, balance, speed, orientation…

To ensure the integration of users with hearing issues, inserting subtitles of the dialogs would be a must, and ensuring those are easily readable. The position of the dialogs should be able to tell the user where the sound is coming from. In terms of speech, when a VR experience require this, it would be nice if the user could also provide other terms of visual communication.

There are different degrees of blindness, so this should be something we want to take into account. It is important that the objects have a good contrast and that the user can zoom into them in case they are too far away. Sounds are also a very important part of the experience and it would be ideal that we can move around and interact with the objects based on sound.

I realized on how different the experience could be depending on the user just by asking my mother to help me test one of my apps. She usually wears glasses to read, so from the very beginning she could not see the text as clearly as I did.

I mentioned before that in VR it is possible to interact with object by focusing the camera on them for a period of time. This is a simple alternative to click without the need of hand gestures for people with difficulty using them.

There are many sources online about how to make fully accessible VR experiences, and I am sure you can come up with your own tests.

Integration testing

Its purpose is to ensure that the entire application functions as is should in the real world and meets all requirements and specifications.

To test a VR application, you need to know the appropriated hardware, user targeting and other design details that we would go through with the other type of testing.

Also, in VR everything is 360 degrees in 3 coordinates, so camera movement is crucial in order to automate tests.

Besides, there might be multiple object interaction around us that we would also need to verify, such collisions, visibility, sounds, bouncing…

There are currently some testing tools, some within Unity that could help us automate things in VR but most are thought from a developer’s perspective. That’s one more reason for us to ensure that the developers are writing good unit tests to the functions associated with the functionality and, when possible, with the objects and prefabs. In special, unit test should focus in three main aspects for testing: the code, the object interaction and the scenes. If the application is using some database, or some API to control things that then change in VR, we should still test them as usual. These tests would alleviate the integration tests phase.

Unity test runner

Many things are rapidly changing on this area, as many people have understood the need for automation over VR. When I started with unity, I did not know the existence of the testing tools, and tested most of it manually, but there are some automated recording and playback tools around.

Performance testing

Is the process of determining the speed or effectiveness of a system.

In VR the scale, a high number of objects, different materials and textures and number and of lights and shadows can affect the system performance. The performance would vary between devices, so the best thing to do would be to check with the supported ones. This is a bit expensive to do, that’s why some apps would only focus in one platform.

Many of my first apps were running perfectly well in the computer but they would not even start on my phone.

It is important to have a good balance to have an attractive and responsive application. New technologies also make it important to have a good performance, so the experience is realistic and immersive. But sometimes in order to improve performance we have to give up on other things, such as quality of material or lights, which would also make the experience less realistic.

In the case of unity, the profiler tool would give you some idea of the performance, but there are many other tools you can also use. In VR, we need to be careful with the following data: CPU usage, GPU usage, rendering performance, memory usage, audio and physics. For more information on this, you can read this article.

Unity profiler

Also, you can check for memory leaks, battery utilization, crash reports, network impact…. and use any other performance tools available on the different devices. Some of these get a screenshot of the performance by time and send it to a database to analyze or set up alerts if anything goes higher for you to get logs and investigate the issue, while others are directly installed on the device and run on demand.

Last but not least, VR applications can be multiplayer (that’s the case of the VRChat) and so we should verify how many users can connect at the same time and still share a pleasant experience.

Security testing

Ensures that the system cannot be penetrated by any hacking way.

This sort of testing is also important in VR and as the platforms evolve, new attacks could come to live. Potential future threats might be virtual property robbery, especially with the evolution of cryptocurrency and with monetization of applications.

Other testing

Localization testing: as with any other application, we should make sure that proper translations are available for the different markets and we are using appropriate wordings for them.

Safety testing: There are two main safety concerns with VR (although there might be others you could think of)

1. Can you easily know what’s happening around you?

Immersive applications are the goal of VR, but we are still living in a physical world. Objects around us could be harmful, and not being aware of alarms such as a fire or carbon monoxide could have catastrophic results. Being able to disconnect easily when an emergency occurs is vital on VR applications. We should make sure the user is aware of the immersion by ensuring we provide some reminder to remove objects nearby.

Smoke could be unnoticed – Picture credit CARLOS ESPINOZA

Every time I ask someone to give a test to some app with the mobile device, they start walking and I have to stop them otherwise they might hit something. And this is the mobile device, not the entire immersive experience.

Even I, being quite aware of my surrounding, give a test to some devices that include sound, and I hit my hand with several objects around me. Also, I could not hear what the person that handed over the device was telling me.

2. Virtual assaults in VR:

When you test an application in which users can interact with each-others in VR, the distance allowed between users could make them feel uncomfortable. We should think about this too when testing VR.

Luckily, I haven’t experienced any of these myself but I have read a lot of other people talking about this issue. Even some play through online on VR chat, you can see how people break through the comfort zone of the players.

Testing WITH VR: There are some tools being developed in VR to allow for many different purposes and technologies as emulation of real scenarios. Testing could be one of them, we could, for example, have a VR immersive tool to teach people how to test by examples. I have created a museum explaining the different types of testing, maybe next level could be to have a VR application with issues that the users have to find.

What about the future?

Picture credit Harsch Shivam

We have started to seen the first wireless headsets and some environment experiences such moving platforms and smelling sensations.

We shall expect the devices to get more and more accurate and complete and therefore there would be more things to test and that we should have into account. We are also expecting the devices to get more affordable with the time, which would increase the market.

Maybe someday we would find it hard to differentiate between what’s real and what’s a simulation… maybe… we are already in a simulation.

How do I get into…? guide

One of the most common question I get when people reach out to me about virtual reality (VR) is: how could I get started? Even thought I already have written an article about this, but maybe I should talk about my own experience instead so you can get more specific examples about it. If you are not into VR, please bear with me, as what I’m about to tell you can help you getting into any topics you wish to get into.

My Story

My first experience with a VR device was during a hackathon in Microsoft, when one of the interns brought his Oculus Rift. Back then it was a very expensive device, so it was very interesting to be able to play around with one. But I found that it would still have some issues to solve, starting for adding hands gestures.

As life goes, sometimes you get stuck in what you are doing at work and don’t get the time to investigate interesting new things. In my case, I bought a house and there was a lot of stress related to this. It was not until years later that I actually got the chance to try again another device, this time it was over mobile on a meetup called “tech for good” in Dublin. In this meetup, they were using VR mobile devices to provide social impact. It was my first experience with phone VR and I thought: Ok, now this is something that anybody can use and get, therefore it is something that is going to need testing.

After that, another hackathon (this time an open Nasa hackathon) got my interest in VR and AR back. I highly recommend this hackathon as I made really good friends there and we had so much fun building a AR/VR experience to navigate a satellite. My team (who won the local People’s choice award in Dublin) created an application that simulate a satellite around the orbit (on AR) and translate to see the view from that satellite (VR). If you are interested, here is our project

When I found myself having more time, I started looking for information about VR. I found a Udacity course on VR and decided to take it on. Back when I started it, the course covered many topics, although they made the decision of separating the courses in different specialties, which makes much more sense. If you are interested in seeing some of the projects I made during this course, check my Github account.

After that, I got interested in open source projects on AR and wanted to start doing testing there… However, life got in the way again when I moved to China. It’s still on my to-do list.

I was lucky enough to start working for Netease Games in China right after, so I had then enough flexibility and hardware access to do some more research in VR including some automated testing with Google Cardboard, which it should be now integrated in Airtest project (I know, not many people are using Google Cardboard anymore but, hey, you need to start somewhere.. the other research is still ongoing)

I also was lucky to have the opportunity to attend to the second Sonar in Hong Kong, which is a music and technology festival, and it showcased some cool new technologies and devices in VR (including aroma experiences and snow-surfing)

Besides that, I started to think of plans and ways of testing VR applications too (as Netease was working in some projects like nostos, which I had the opportunity to try myself and really enjoyed it).

Around that time, I gave a talk in Selenium conference in India gathering all this gained knowledge (which I talked about on this post). In order to prepare for this talk I played around and created my own ‘conference simulator’ just to get prepared for it.

Another thing I do frequently to gain knowledge in VR is to watch playthroughs and online reviews, as you can learn a lot from watching others play and it could be very good to understand your potential users if you are working on a game. I also have read some books on the matter (shout out to packtpub which gives away free IT books everyday!)

Have you found a pattern?

I know you have, if you are a reader of this blog you surely are a clever Lynx by now, but just in case, I have highlighted it in bold letters: Attending to (and after a while starting) hackathons, meetups, talks and festivals, watching or reading online related content and books, and playing around in courses, open source projects, at work and on your own projects will get you into anything you are interested to get into.

It sure sounds like a lot of things to do, but the tools are already around you, and I’m talking about years worth of experience here. Just take one thing at a time and you will too become an expert of that thing you are into. The biggest difficulty is to pick which one to take at any given time and to be honest about yourself and how much time you can spend on that. (I regret a ton not having put more efforts on the AR open source project when I had the chance)

Of course, if you are not really into it, then it would sound like a lot of work, in which case it’s probably better to save yourself time and pick something else. I like to think of it as a test of passion, or on the words of ‘Randy Pausch’ from his talk ‘Achieving Your Childhood Dreams’: “brick walls”. (By the way, this is one of the best motivational talks I’ve ever watch, and I actually re-watch it at least once a year to keep me motivated. Also, it mentions VR too 🙂 )

As you would imagine, this is not the only subject I spent time with or I gave my attention to, another big one for me is artificial intelligence, but that’s…well…another story.