Mixed reality is the combination of the real world with digital content, allowing developers to bring games and interaction to the real world around you. This relatively new medium presents a number of unique challenges not normally faced by “normal” game AI. Because both the real world and the virtual world need to understand the other, AI and data parsing are huge areas for AI work. In a talk called Bringing Pixels To The World at Casual Connect Europe 2018 in London, Aleissia Laidacker, Interaction Director and Brian Schwab, Director of Interaction Lab at Magic Leap
discussed the ins and outs of mixed reality. This session explained the “what, why, and how” of mixed reality to develop truly immersive experiences. Instead of bringing you into the story [like screen games in MR] the digital objects are in our world, like Roger Rabbit. Tune in to the session below for more insights.
Brian Schwab is the Director of the Interaction Lab at Magic Leap. The Interaction Lab is a rapid prototype group in the Systems Engineering Organization where they do experience exploration and turn this into feedback and requirements for the hardware, software and perception groups.
After leaving his last job in the game industry, Brian wanted to grow as a creator and experiment with the future of interactive media. Directing the Interaction Lab presented the ideal opportunity.
Feedback Is King
Brian has spent twenty-five years in the game industry as a gameplay and artificial intelligence programmer, focusing in realtime feedback and input systems. He describes, “I have lived my life watching people play with the toys I make, take good notes and then tune them for more goodness. Feedback is king, both in using the system and in making the system great.”
While at Sony, Brian was involved in multiple product launches, including a few of the consoles themselves, which involved using unfinished hardware that needed iteration work.. He “squeezed fun out of a noisy electric circuit by bending it to my will.”
Conducting Across the Disciplines
Being an AI programmer prepared him to be an ambassador between disciplines. He explains that game AI is about holistically using all the tricks of a team, so Brian becomes good friends with the team members in order to conduct them just as an orchestra conductor works with the orchestra; Brian makes the AI sing with direct help from the designers, engineers, animators, and audio staff.. At Magic Leap, where he works with so many different disciplines, this is an especially important ability.
Brian particularly enjoys the feeling his work gives of helping to change the face of interaction between human and computer. He is finding world-bending new paradigms in user experience and knows that he is just scratching the surface of what is possible.
The Beginning of a Career in Games
This fascinating career began with Brian attending university to study electrical engineering and planning to get into robotics in order to “make the teddy bear from Supertoys Last All Summer Long”. Eventually he switched to computer science to get into making games after realizing that his free time alternated between playing video games and making video games. He realized this could actually be a job.
Brian was only nine years old when he wrote his first game. This was a game where the player could run around a maze and also transform into a car to ram walls. And in high school Brian learned on his own 6502 assembly programming and then ported Zelda to Apple II.
He first became interested in coding in about 1979. The computers at his school had BASIC built in, and Brian loved the fact that he could exit out of what he was supposed to be doing and make his own things. By 1986 he became interested in design. At this time Brian started to critically break down the games he was playing and try to replicate them or to improve on them.
To those who would like to have a similar career, Brian advised building a lot of things, but build them all the way to completion. Get lots of feedback. Understand the feedback is about the product, not you. Try not to react to feedback, except to ask clarifying questions.
Creating the Experience
Brian begins his creative process by doing extensive research to discover what has been done before. Then, he decided what he wants the experience to be like and what emotional triggers he is aiming for. He considers, “How does the human animal experience those emotions? Are there any common human biases or mental blind spots around the area that I should know about? I try to not only have a good idea, but to fully understand my audience as far as possible. What knowledge might they bring with them already? How easy will it be to learn something new? How can I make each part of the experience fun?”
Inspiration for designs comes every day, from minute to minute. Brian believes that if a person is open, they will constantly see places where people are having a sub par time but would prefer to have a more crafted experience of their work, play or whatever they are doing. So if you know the customer and their pain point, if you can solve it you have the basis for a viable design.
The most challenging part of game development, Brian claims, is 3D math. Even for those who are good at math, “it gets hairy fast when you want to do advanced things in a 3D world”.
The most rewarding aspect of game development Brian has found is having someone tell you they were having a tough time and your game made it bearable. Or that the reason they got into games was because of something you said.
If Brian could create any game, without limitations of time and resources, it would involve a series of little games that the computer would play with you. The system would try to get to know you better, by asking you direct questions, finding the types of challenges you enjoyed, what kind of pacing you like, your preferred rewards and what motivates you. Then, the game could tweak the experience and make things more enjoyable.
Dealing with Blocks
Creative blocks are an inevitable fact in game development. Brian suggests dealing with them by explaining your stuck situation and what you think is the problem to someone who has no skills similar to yours. When you have to explain the situation in a very simplistic way, it forces your brain to fully deal with each step instead of making the jumps that you would make because of common knowledge among peers or even in your own head. But Brian maintains that creative blocks are a constant you must develop very sharp tools to fight against. Being aware enough to notice the start of a block early is your best weapon.
Rewards and Challenges
Getting his first job in games was the proudest moment Brian remembers in his career. He had been homeless for almost a year while trying to get a job with a specific company. Eventually he started searching more broadly and landed that first job. “I was so proud that I hadn’t given up and taken a real job,” he recalls.
But there are challenges in the game industry that can be painful. There was a co-worker that Brian offended without realizing it, through using too direct feedback. This, of course, made working difficult. When Brian did understand the problem, he talked to the co-worker directly, worked things out and began rebuilding the relationship. However, it took a long time and was very frustrating while he was in the dark. Brian learned that if you feel disconnected it is because you have a blind spot.
The next big trend in the game industry, according to Brian, will undoubtedly be mixed reality. “It’s a vast improvement in how to use computers,” he insists, “and leads to a much more intimate and enabling connection to our technology.” Brian is preparing for it by being on the front lines of creating that future.
When Brian has free time he reads voraciously on a wide variety of subjects: fiction, technical works, research papers, cognitive science, sociology and many others. At any time he will be reading multiple books, a habit he began as a child. He also enjoys working out as a meditative discipline. As a former professional movie reviewer, Brian continues to watch a lot of movies, no doubt criticizing them in detail.
Catherine Quinton is a staff writer for www.gamesauce.org. Catherine loves her hobby farm, long walks in the country and reading great novels.