I have recently watched a video from PBS space time that got me thinking: if we were to have a time machine, how would we test it? I’ve seen a lot of “how would you test X” type of questions in interviews, but I don’t think I’ve ever seen this one before (I am not trying to give you ideas for interview questions!)
It’s not rocket science… it’s rocking testing science!
I couldn’t help but compare it firstly to a spaceflight, so I started wondering: how do they test spacecrafts? And what better one than Apollo 11 to start with? If only we had a time machine for getting its source code… Yes, I have been looking at Assembly code trying to make sense of potential testing routines, like this one… and… guess what I found there?
This section of the code is checking the Gimbal lock of the accelerometers! Do you remember the concept from my last post? Maybe I just have a case of Baader-Meinhof, but I do feel Gimbal lock is an important concept to learn, so check it out.
Testing in Assembly was not as ‘easy’ as nowadays (for example, macros does not seem to be a thing that I could find in the Apollo11 programming syntax). Do not expect a page object model or a library with tests or testing functions. Nor common methods for before and after tests. Actually, don’t expect any sort of OOP, to start with.
In my search I could find some files with tests on them, but they are mostly for stressing the hardware by sending signals to the different devices and recovering from bad statuses. Also, spacecrafts might need to check correctness of bits to make sure there are no catastrophic arithmetic errors.
Time traveling tests
Imagine we have covered the unit and integration tests for both hardware and most of the software for our machine that could potentially match any currently existing ones. What are the specific cases we should cover for our time machine? The logical cases to think of:
1) The machine should not kill the traveler: We should insert some devices to measure that the cabin of the machine is livable. Also, keep in mind for the rest of the tests, that usually spaceships are first tested with robots inside, rather than humans. (Security test)
2) There must be some ways of safe interrupting the process (risk assessment/testing): In case anything wrong happens, what’s the risk involved for the passenger? (Security test)
3) As the machine goes forward or backwards, the traveler state stays while the surrounding world changes: We could measure this by introducing some other object before that we know it decays and checking how long time it has passed in the machine. (Usability test)
4) There is a way to return: I assume we will want this, so we should test it. (Security test?)
5) Performance tests: how many times it can go backwards and forward in time? (Performance but also, regression test?)
6) What is the minimum time that can travel? (test it) If the time machine requires a rapid negative speed (as the video above suggested as one of the possibilities), I imagine traveling in time as playing billiard or golf. Let me explain: When you try to position a ball on a particular hole, you need to give it the right original impulse (not too much, not too little) but also the right angle depending on the starting point. We are likely to travel space as well as time, so it might be particularly difficult to stop on a particular moment or place. (Which may explain why nobody made it to the time travelers party) (Usability test)
7) Test boundaries scenarios: go back to start of the earth (or universe) and forwards to its end. What happens if just before or just after? Does time always exist? Can we travel to a point there is no time? Technically these should be tests we would like to do, but I think they are probably not doable, realistically speaking (can I use this expression in this context?)
8) Try to change a small thing in the past…does it change the future? If a change makes a parallel universe, then we should not ever recover the machine. If we test this, we should look for an event we can easily undo (like maybe turn a light on/off?) (Exploratory test)
9) Time traveler meeting same time traveler. Test paradoxes. (Exploratory test)
11) Test placing a box where the machine is positioning before a trip. Then take the box away, position the machine on that same spot and test traveling to before we positioned the box. Does the box o machine break? (Integration test)
12) Test traveling to when the box is still in place; do the machine or the box break? (Integration test)
13) Try to travel twice to the same exact time and place. (Integration test)
14) Could the machine travel between different universes? (If we have a way of doing so, which might be more likely if we are using wormholes for the trips than negative speed) (Integration test)
15) Is there a maximum/minimum size of the machine? If so, test them. (Boundary test)
16) Can everybody use the machine or only qualified people? (Accessibility test)
Maybe we have already invented a time machine but kept it secret because it is dangerous. Or maybe we have gone back in time to stop ourselves from inventing it, creating a parallel universe and as a result, our universe is the one in which such machine was never invented (time version of the Fermi paradox)
Whichever the answers to these questions, I hope you’ve found this exercise fun and enjoyed reading this post as much as I enjoyed writing it. Let me know if you can come up with more tests we could do to the time machine (if we were to have one). I will resume my serious posts soon, they will be… well… another story.
Previously, I’ve written a couple of posts about how to get yourself started on VR in which I promised some stories about testing on this world.
Why do I call it world instead of application? Because the goal of virtual reality is to create realistic synthetic worlds that we can inspect and interact with. There are different ways of interacting in these worlds depending on the device type.
Testing an application in VR would be similar to testing any other application, while we would have to take into account some particularities about the environment. Later on we will see different kinds of testing and think about the additional steps for them in VR, but first let’s see the characteristics of the different types of devices.
Types of devices:
Phone devices (Google Cardboard or DayDream) – allows you to connect your phone (or tablet) on the device to be able to play a VR app in there.
This is possible because most of smartphones nowadays come with gyroscopes: a sensor which uses the Earth’s gravity to determine the orientation.
Some Cardboards (or other plastic versions) can have buttons or a separate trigger for actions on the screen (as it is the case for DayDream), but the click is usually not performed in the object. Instead, it is done anywhere in the screen while the sight is fixated on the object. If the device does not have a button or clicker, the developer have to rely on other information for interaction, such as entering and exiting objects or analyzing the length of time the user was on the object.
Computer connected devices (HTC, Oculus, Samsung VR…) that generally comes with (at least) a headset and handset, have an oled device with high resolution and supporting low persistence embedded in the headset, so you don’t need to connect it to a screen. They detect further movement, as it is not just about the movement of the head but also the movement on the room itself and the hand gestures. This is done differently depending on the device itself.
We have moved from being able to detect user head movement (with reasonable size devices), to use sounds, to use hand gestures… so now, testing VR applications is getting more complicated as it now require the test of multiple inputs. The handset devices usually have menu options as well.
Before going on, I’d like to mention AR. AR is about adding some virtual
elements in the real world, but with AR we do not create the world.
However, AR has a lot in common with VR, starting with the developing
systems. Therefore, the testing of the two platforms would be very
We have talked about the hardware devices in which the VR applications would run, but we should also talk about the software in which the applications are written.
Right now there are two main platforms for developing in VR: Unity and Unreal, and you can also find some VR Web apps. Most of things that are done with Unity use C# to control the program. Unreal feels a bit more drag and drop than unity.
Besides this, if you are considering to work in a VR application, you should also take into account the creation of the 3D objects, which is usually done with tools such as blender or you can find some already created onlin.e
But, what’s different in a VR application for testers?
Tests in VR applications:
VR applications have some specifics that we should be aware of when testing. A good general way of approaching testing on VR would be to think about what could make people uncomfortable or difficult.
For example, sounds could be very important, as they can create very realistic experiences when done appropriately, that make you look where the action is happening or help you find hidden objects.
Let’s explore each of the VR testing types and list the ways we can ensure quality in a virtual world. I am assuming you know what these testing are about and I’m not defining them deeply, but I will be giving examples and talking about the barriers in VR.
It ensures that the customer can use the system appropriately. There are additional measurements when testing in VR such as verifying that the user can see and reach the objects comfortably and these are aligned appropriately.
We are not all built in the same way, so maybe we should have some configuration before the application for the users to be able to interact properly with the objects. For example, the objects around us could be not seen or reached easily by all our users as our arms are not the same length.
You should also check that colors, lighting and scale are realistic and according with the specifications. This could not only affect quality, but change the experience completely. For example, maybe we want a scale to be bigger than the user to give the feeling of shrinking.
It is important to verify that the movement does not cause motion sickness. This is another particularly important concept for VR applications that occurs when what you see does not line up with what you feel, then you start feeling uncomfortable or dizzy. Everyone have a different threshold for this, so it is important to make sure the apps are not going to cause it if used for long time. For example, ensure the motions are slow or placing the users in a cabin area where things around them are static, or maintaining a high frame-rate and avoiding blurry objects.
If there is someone on your team that is particularly sensitive to motion sickness, this person would be the best one to take the tester role for the occasion. In my case, I asked for the help of my mother, who was not used at all to any similar experiences and was very confused about the entire functioning.
Is a subset of usability testing that ensures that the application being tested can be appropriately used by people with disabilities like hearing, color blindness, old age and other disadvantaged groups.
Accessibility is especially important in VR as there are more considerations to make than in other applications such: mobility, hearing, cognition, vision and even olfactory.
For mobility, think about height of the users, hand gestures, range of motion, ability to walk, duck, kneel, balance, speed, orientation…
To ensure the integration of users with hearing issues, inserting subtitles of the dialogs would be a must, and ensuring those are easily readable. The position of the dialogs should be able to tell the user where the sound is coming from. In terms of speech, when a VR experience require this, it would be nice if the user could also provide other terms of visual communication.
There are different degrees of blindness, so this should be something we want to take into account. It is important that the objects have a good contrast and that the user can zoom into them in case they are too far away. Sounds are also a very important part of the experience and it would be ideal that we can move around and interact with the objects based on sound.
I realized on how different the experience could be depending on the user just by asking my mother to help me test one of my apps. She usually wears glasses to read, so from the very beginning she could not see the text as clearly as I did.
I mentioned before that in VR it is possible to interact with object by focusing the camera on them for a period of time. This is a simple alternative to click without the need of hand gestures for people with difficulty using them.
There are many sources online about how to make fully accessible VR experiences, and I am sure you can come up with your own tests.
Its purpose is to ensure that the entire application functions as is should in the real world and meets all requirements and specifications.
To test a VR application, you need to know the appropriated hardware, user targeting and other design details that we would go through with the other type of testing.
Also, in VR everything is 360 degrees in 3 coordinates, so camera movement is crucial in order to automate tests.
Besides, there might be multiple object interaction around us that we would also need to verify, such collisions, visibility, sounds, bouncing…
There are currently some testing tools, some within Unity that could help us automate things in VR but most are thought from a developer’s perspective. That’s one more reason for us to ensure that the developers are writing good unit tests to the functions associated with the functionality and, when possible, with the objects and prefabs. In special, unit test should focus in three main aspects for testing: the code, the object interaction and the scenes. If the application is using some database, or some API to control things that then change in VR, we should still test them as usual. These tests would alleviate the integration tests phase.
Many things are rapidly changing on this area, as many people have understood the need for automation over VR. When I started with unity, I did not know the existence of the testing tools, and tested most of it manually, but there are some automated recording and playback tools around.
Is the process of determining the speed or effectiveness of a system.
In VR the scale, a high number of objects, different materials and textures and number and of lights and shadows can affect the system performance. The performance would vary between devices, so the best thing to do would be to check with the supported ones. This is a bit expensive to do, that’s why some apps would only focus in one platform.
Many of my first apps were running perfectly well in the computer but they would not even start on my phone.
It is important to have a good balance to have an attractive and responsive application. New technologies also make it important to have a good performance, so the experience is realistic and immersive. But sometimes in order to improve performance we have to give up on other things, such as quality of material or lights, which would also make the experience less realistic.
In the case of unity, the profiler tool would give you some idea of the performance, but there are many other tools you can also use. In VR, we need to be careful with the following data: CPU usage, GPU usage, rendering performance, memory usage, audio and physics. For more information on this, you can read this article.
Also, you can check for memory leaks, battery utilization, crash reports, network impact…. and use any other performance tools available on the different devices. Some of these get a screenshot of the performance by time and send it to a database to analyze or set up alerts if anything goes higher for you to get logs and investigate the issue, while others are directly installed on the device and run on demand.
Last but not least, VR applications can be multiplayer (that’s the case of the VRChat) and so we should verify how many users can connect at the same time and still share a pleasant experience.
Ensures that the system cannot be penetrated by any hacking way.
This sort of testing is also important in VR and as the platforms evolve, new attacks could come to live. Potential future threats might be virtual property robbery, especially with the evolution of cryptocurrency and with monetization of applications.
Localization testing: as with any other application, we should make sure that proper translations are available for the different markets and we are using appropriate wordings for them.
Safety testing: There are two main safety concerns with VR (although there might be others you could think of)
1. Can you easily know what’s happening around you?
Immersive applications are the goal of VR, but we are still living in a physical world. Objects around us could be harmful, and not being aware of alarms such as a fire or carbon monoxide could have catastrophic results. Being able to disconnect easily when an emergency occurs is vital on VR applications. We should make sure the user is aware of the immersion by ensuring we provide some reminder to remove objects nearby.
Every time I ask someone to give a test to some app with the mobile device, they start walking and I have to stop them otherwise they might hit something. And this is the mobile device, not the entire immersive experience.
Even I, being quite aware of my surrounding, give a test to some devices that include sound, and I hit my hand with several objects around me. Also, I could not hear what the person that handed over the device was telling me.
2. Virtual assaults in VR:
When you test an application in which users can interact with each-others in VR, the distance allowed between users could make them feel uncomfortable. We should think about this too when testing VR.
Luckily, I haven’t experienced any of these myself but I have read a lot of other people talking about this issue. Even some play through online on VR chat, you can see how people break through the comfort zone of the players.
Testing WITH VR: There are some tools being developed in VR to allow for many different purposes and technologies as emulation of real scenarios. Testing could be one of them, we could, for example, have a VR immersive tool to teach people how to test by examples. I have created a museum explaining the different types of testing, maybe next level could be to have a VR application with issues that the users have to find.
What about the future?
We have started to seen the first wireless headsets and some environment experiences such moving platforms and smelling sensations.
We shall expect the devices to get more and more accurate and complete and therefore there would be more things to test and that we should have into account. We are also expecting the devices to get more affordable with the time, which would increase the market.
I know, I still owe you some stories, but I am now inspired to talk about something else. Besides, today’s issue is easier to put it into words. I don’t need to sit down and think carefully on a way of explaining some technical concept such as artificial intelligence while not sounding boring. But, just so you know, I am still working on the other stories.
I would like to show you how dangerous social media could become and on one hand highlight the need of asking the right questions when a new technology comes along in order to set proper tests and barriers (lynx are curious animals, aren’t we?).
On the other hand, highlight the importance to take breaks from it and think about yourself and things that would make YOU happy instead of thinking of things that ‘would make other people think that you are happy’ and therefore approve of you. I hope you enjoy reading this article.
Let’s think about it: the most viewed YouTube channels from independent creators are from people below their 30’s (or just on them), many started them 5-9 years ago. They have been getting a lot of pressure from fans and companies that would like a piece of their influence. Celebrities with less direct exposure to their fans have done crazy things in the past because of social pressure. Yet these influencers are not invited (that I know of) to Davos or famous lists of most influential people, besides demonstrating incredible marketing strategies, knowledge of new technologies, having charisma and being very intelligent (more than they let to be seen in some cases)
There is a very important point to consider, which is its the potential for hacking (if you are interested on this, there is more information here and here). So, imagine that someone could actually go there and decide what you are going to see… how could this affect you?
Techniques and prevention:
Let’s imagine a platform that accepts comments and likes (let’s forget about dislikes). How could someone socially hack it?
1) Removing the likes: We would need to intercede the information that this platform is showing to the user and eliminate the likes the user will see. Maybe we should only eliminate them partially, so the user is not suspicious of not seeing any likes at all. For this, we could have a pondered random variable that would eliminate or not eliminate each like. How would you feel if all of the sudden nothing you write gets any likes? Prevention: From test side, make sure the like system works properly and cannot be done by anonymous sources. Make sure accounts are real. Make sure the user sees only real data. From user perspective, when you see something that you like, mention it in person, start a conversation about it instead of just clicking a button.
2) Liking specific posts: This is a bit fancier. Based on above, we could have some sort of AI algorithm that could classify the posts. Then we can decide which comments are going to show as liked for the user. How would you feel if only some type of your posts would get liked? Would that change your way of writing? Prevention: From test side, make sure all information is shown to the user. From user side, find your audience and focus on them. Also, consider talking with this people directly too (or in conferences). Try to keep honest to your goals for writing and who you want to reach.
3) Filtering comments: This would require some form of classification as with the previous point. Instead of targeting the likes, we would target the comments but the idea would be the same. Eliminate from view those that are not ‘interesting’. What would you think if you only receive a certain types of comments? Prevention: From test side, make sure all information is shown to the user. Maybe have a conversation about the feature itself and allow users to hide all comments. From user side, as above.
4) Creating comments: We could create new comments with AI. You might think the user would realize about this, but if done carefully they might not even notice this or confront the person making that comment. Besides, the social media platform might allow for not-logged in comments. This adds to the feeling of the previous one. Prevention: Have a conversation about blocking anonymous comments or disabling them. From user side, if you see a strange comment from someone that does not add up, clarify this with the person. It can also help with misunderstandings. Option 2, disable or stop reading comments.
5) Changing the advertisement around the website to a convenient one for propaganda or for harm (only for some types of social media). Prevention: Most of sites have a way of deciding what advertisements you are more interested of. Also, try cleaning cookies regularly or use private browsers and VPN services.
6) Extracting automatically interesting information for malicious purposes. Prevention: Be careful with what you post, don’t use information that is available for anybody as your passwords or security questions or pictures with personal data (such passports or train tickets). If you really want to share pictures of a trip, try uploading them after the trip and enjoy while in there!
7) Connecting certain types of people. I am not 100% sure how this could be used for malicious purposes but surely someone would find a way. Making sure you can block people is also very important.
8) Taking things out of context. Prevention: It’s very hard to delete something from the internet once it is out there, but some platforms allow it. Have you ever read your old posts? It is a good idea to do some clean-up every so often. Also, if this happens to you, keep a track of the entire context. Maybe have a system in which you can remove what you have written before it goes online, take some hours before posting to make sure you want to post that.
Why am I talking about social hacking social media in a test blog? Well, because, if you happen to be working on developing a social media project, you should make sure that these attacks are not possible and think about how the user could feel for other features to come.
(Please take some time to go throughout the articles I liked above to know more)
Thoughts about being addicted to being connected:
When was the last time you did something good for someone but did not tell anybody about it? When you do something good for someone and post about it, how do you know you are doing it because of the other person and not to have a better image of yourself in front of others?
When was the last time that you went for a trip and didn’t share the pictures with anybody? What about not taking any pictures at all? There is something truly special about having a memory that is just yours to have.
Social media has evolved quickly in very short time, and we need to consider a lot of new things, more so if we are in the development team of one of these platforms. We should really stop and agree about what is ethical and not with the particular platforms and maybe even list the set of contraindications as if we do with addictive substances. For example, would it be ethical for some platforms (maybe for young people?) to change what the users see in order to protect them from the bad critic? Consider this could, in theory, save some lives, but, in contrast, it would take away some potential good feedback disguised as bad comments. Maybe this is a feature you want people to turn on and off? If not, maybe you should list in your contraindications that it could be an issue. And I don’t mean terms and conditions, it’s not about saving your behind if anything happens, it’s about actually alerting the user about what could be experienced. Terms and conditions are…well.. another story.
Some sources if you are interested in this area that helped me control my internet usage: