Meet Google’s Next Generation Virtual Reality Platform

At Google’s I/O conference yesterday, Google showed off the future of virtual reality — and not everything is made of cardboard. It’s surprising because Google’s previous ventures into space have been, well, a bit indecisive.

Don’t get me wrong: Google Cardboard, the minimalist virtual shell for smartphones, is certainly neat, but its limitations are obvious. This is one of the famous 20% of Google’s projects — a labor of love from a few employees, and it shows. Cardboard has a narrow field of view, motion blur, no positional tracking, and lousy latency. This is a recipe for bad diving and VR disease.

Google Cardboard doesn’t have a headband, and that’s no omission. Without a head strap, users must hold it over their faces, forcing them to turn with their torso. This slows them down with a noticeable delay. And this is normal — we are talking about a headset made of cardboard. It’s still a valuable tool for bringing people into virtual reality, but it won’t be a game changer.

All this is starting to change. Google uses a significant share of the world’s intellectual resources, and at this year’s IO conference began to demonstrate some of these capabilities in the virtual reality space. Ads three times.

• First, there is a significant improvement to the Cardboard headset.
• Secondly, there is a VR camera built to create a solid 3D image.
• Finally, there is Tango VR, a new VR/AR platform built on Google’s enigmatic 3D touch pad.

Ready? Let’s dig into.

New cardboard set

The new design for Google Cardboard is a definite improvement — it adds support for larger phones (up to six inches), includes iOS support, and adds a small physical lever on the side of the headset for typing. It is also designed to be easier to build and control.

Other than that, it’s pretty much the same story: small, standard lenses, a cardboard shell, and a lot of motion sickness if you use it for too long. It’s a notable improvement, but we’re pretty close to the limit for cardboard shells and Google is on hold until they start building something more useful.

VR camera

The new camera, developed in collaboration with GoPro, is called Jump . It consists of 16 GoPro cameras (about $10,000 total) mounted on a round plastic mount. So far mundane: similar setups for VR video production already on the market. What’s getting really cool is the so-called «Jump Assembler», a piece of software made by Google to cut down on sharp VR artifacts.

To understand this software, you must understand the problem it is trying to solve. John Carmack of Oculus talks about problems with 3D panoramic capture.

«What they end up doing is you have cross-sections taken from multiple cameras, so straight ahead that’s proper stereo […]and then here it goes for that [угла]. But that means if you’re looking at what was right for the eyes here, but you’re looking out of the corner of your eye here, […] this is not a correct inequality for the eyes. And then, what’s worse, if you [закатываете голову]things get bad because it’s only tuned to the eyes. «

The jump is the way to fix it. Google uses machine vision technology to compare images and determine the scene’s 3D geometry. The software uses this reconstruction to create a clean, stitch-free perspective with perfect stereo from all angles.

Google plans to allow this sort of content to be viewed via YouTube using Google Cardboard — though there’s no word yet on support for more sophisticated VR headsets like the Oculus Rift.

This is already a huge victory. However, technology can be used to do more—much more. During the demo, Google shows a virtual camera moving around inside the ring, smoothly updating the perspective. Right now, Google is converting 3D data into stereo panoramas. The illusion will work from any turn, but will break if you turn your head or move.

However, in the future, Google may show you the restored 3D geometry directly, giving you a full VR video experience. Perhaps one that supports every axis of rotation as well as a significant degree of positional tracking. It’s not yet implemented, but it’s a tantalizing opportunity for a big step forward in VR video.

Tango V.R.

Google Cardboard has a lot of problems, but even the best example of mobile VR, the Gear VR Gear VR headset there is still one significant drawback — there is no positional tracking.

Tracking rotation is quite simple. You can use gyroscopes to determine how much your head is turning, and the Earth’s magnetic poles are a handy guide to avoid drift. This amounts to a couple of dollars for sensors that are already installed in most smartphones.

Positional tracking is a whole different Putin record. Different companies have dealt with this problem in different ways. Oculus uses an external camera paired with tracking dots on the headset, allowing it to be tracked using machine vision.

HTC and Valve are taking a different approach with their Vive headset. using a variety of Lighthouse base stations with spinning drums to sweep the entire room with laser light. The trackers on the headset and controllers are covered in light sensors that can detect (with extremely precise timing) when they are hit by this laser. By checking the exact time and performing some basic geometric calculations, any three sensors can together determine the exact position of the device in space relative to the base stations.

Both of these approaches have their advantages, and we can discuss their relative merits. However, both have a drawback that becomes critical in the mobile world: you need external hardware to track the headset.

Needless to say, this is a problem for a mobile headset. The point of mobile VR is to free yourself from the cables and infrastructure required for high-performance desktops. So what can we do about it? The answer, at least according to Google, lies in using «SLAM» (simultaneous location and display) technology in combination with a depth sensor to track points in the world. This creates a 3D environment mesh to determine where the device is in space relative to its starting position.

Google already shipped some early hardware with this functionality in the form of the Tango tablet. that we have covered here before. Google believes the sensor could revolutionize the way we interact with devices. As developer Johnny Lee writes in the video below,

“Instead of cramming our entire lives into this very small rectangle [экрана смартфона]Project Tango develops technology that helps everything and everyone knows exactly where they are. Everywhere. Not only where there is good GPS coverage, good WIFI coverage […] The devices we use will share our understanding of space.”

This technology has exciting applications in virtual reality, and recently Google started showing them off.

It’s impressive, but it’s also early days. Valve uses a pre-made holster for their VR experience along with NERF’s honest guns. The tablet itself is not designed for virtual reality — it’s heavy, with a lot of latency, and the ergonomics are just terrible.

However, this might be the most impressive mobile VR demo the world has ever seen — strictly because of the positional tracking it provides. The headset allows you to move freely in any room (provided that the camera is well lit). This opens up some exciting possibilities for mixed reality applications.

Many of the shortcomings will soon be corrected: Google also announced that it has partnered with processor maker Qualcomm to create a smartphone containing the full set of Tango processors for the project and the hardware to support it. These smartphones will be the reference design for a future Tango compatible smartphone, which should be much better suited for VR applications.