An NYU Research Project Is Trying to Make Virtual Reality Less Lonely
Source: Roy Graham

There’s been a lot of money dumped into virtual reality. To be more specific, over $765 million dollars have been invested in virtual reality and augmented reality companies as of last July, according to a collected list by UploadVR. That list of investors includes giants such as Google, Facebook, and Intel, all of which are fixed on defining what commercially available virtual reality will look like.



At the computer science department of New York University, a team of designers and programmers aren’t very interested in the immediate future of a commercial Virtual reality market. Their project, titled “Holojam,” is about what comes after everybody’s plugged in.

“Holojam is a set of practical experiments to prototype a future that does not yet exist,” said Ken Perlin, principal investigator of Holojam. “Suppose it’s ten or twenty years in the future, and the sort of technology that Vernor Vinge describes in his novel Rainbow’s End is a reality. People in the morning can pop in their contact lenses and see whatever they want. What happens between people? We’re interested, not about when it’s cutting edge and exciting, but when it’s boring.”



Two people in virtual reality. Image: David Lobser

The equipment used by Holojam, on first glance, doesn’t appear particularly high-tech. Optical markers, which allow the motion tracking cameras encircling the room to capture movement, are affixed onto Samsung Gears with electrical tape. The get-up which renders one’s own body in virtual space includes dollar-store gloves with the middle two fingers cut off. The home-made look of Holojam’s gear belies the sheer ambition of what the team is attempting to do, though: put two people in the same virtual reality space, untethered by cables of any kind.

“One of the phrases I’ve come up with to describe the feeling of Holojam is Harry Potter meets Harold and the Purple Crayon,” Perlin tells me. While a minor legend in the realm of computer science, he’s best known for developing the Perlin noise, a gradient noise used to create textures on CGI that appear natural. He won an Academy award developing the technique for the 1982 movie TRON—one of the most famous depictions of a computer-generated reality to date.



A computer processes the data captured by the camera rig. Image: David Lobser

“In Harry Potter, as in many fantasy works, you’re not thinking about how it’s done. You’re thinking about the relationships between people. They’re able to do magic things in service of those relationships, not as an end in itself.” The reference to Harold and the Purple Crayon, says Perlin, is more about playfulness than anything else, but it’s one his team chooses to take rather literally. The first time I enter into one of the virtual realities constructed by Holojam, it’s like stepping into a child’s drawing. The walls of the simulated house have a deliberately sketched feeling, and the scene is bathed in warm, bright colors.

My hands, when I look at them, are bright red animated discs. When I move them, the OptiTrack motion capture cameras around the room log it and send the data to a computer, which renders our figures in virtual reality accordingly. Across from me is Holojam’s senior artist in residence, David Lobser, who is rendered as a bright blue cartoon character. He’s doing jumping jacks.

“Seeing somebody in VR gives you a sense of them that you don’t get from video,” explains Lobser, who’s done all of the art for Holojam so far. “The way that they express themselves becomes more cartoony. Certain things are simplified, and certain things are amplified.”

In virtual reality, he demonstrated by switching from leg to leg, balancing on each. There’s not much subtlety possible in here, which lends itself to more extravagant gestures. He offers me a high five, which is his favorite way to blow the minds of the uninitiated; making physical contact with something you can see in virtual reality is, I can confirm, wild.

Inside Holojam, this is what another person looks like. Image: David Lobser

While the team works on Holojam in some form throughout the week, Thursdays are the most exciting time. From 5:00 to 8:00 on Thursday evenings, graduate students and faculty gather in one of NYU’s computer science buildings to talk, plan, code and occasionally just hang out in virtual reality. Running a project with this many volunteers can be messy—I haven’t been inside the development of Sony’s PlayStation VR, but I expect it’s different than the consensus-run meetings of Holojam. In addition to designing some groundbreaking code for the project, students demand an introspective mindset from the older members of the team. “Students come with fewer preconceptions, whereas people such as us have been doing this a long time,” says Lobser. “Having minds that are more of a blank slate forces us to question our own assumptions, too.”

On November 1 of last year, the Holojam team held their first event. Entitled “Dia de Los Holos,” participants could put on a virtual reality headset and find themselves atop a float in a parade of skeletons and ghosts.

“What we’ve made so far has been very participatory experiences,” said Lobser. “Right now, we’re making a something that will be designed for two people, in VR, to give a performance in.” Scheduled for the Scores+Traces NYC exhibition in March, Ken Perlin will share the virtual stage with a professional dancer.

Refinement, for Lobser and Perlin, takes a backseat to innovation—after one successful interactive event, they’re already moving on to live, improvisational performance in virtual reality. While much of the focus on virtual reality development has been on the well-funded corporate world, the inventive and curious experiments of the Holojam team might end up as a better road-map to the future of virtual reality. If the breadth of their projects isn’t impressive enough, their ambition certainly is.