How Does Mixed Reality Work: From Sensors to Spatial Computing

The defining characteristic of Mixed Reality (MR) is not the quality of the graphics, but the device’s understanding of the physical world.

In simple Augmented Reality (AR), like a phone filter, graphics are often just overlaid on a 2D video feed. If you move the camera too fast, the digital object slips or floats awkwardly. In true Mixed Reality, digital objects abide by the laws of physics. They can sit on a real table, disappear behind a real sofa (occlusion), and remain in the exact same spot even if you leave the room and come back an hour later.

To achieve this, an MR headset must solve one of the most difficult problems in computer vision: it must figure out exactly where it is located in 3D space, in real-time, while simultaneously drawing a map of that space.

Here is the technical breakdown of the reality loop: how the hardware sees, maps, renders, and blends digital content with your physical environment.

1. Environmental Perception: Sensor Fusion

Before an MR device can project a hologram, it must digitize the physical world. It does this through Sensor Fusion combining data from multiple types of sensors to create a reliable truth.

Most modern MR headsets (like the Apple Vision Pro or Meta Quest 3) rely on three specific inputs working in tandem:

• Tracking Cameras (Grayscale/IR): These are often low-resolution black-and-white cameras pointed in different directions. They aren’t looking for pretty pictures; they are looking for high-contrast “feature points” corners of tables, patterns on a rug, or the edge of a doorframe. By tracking how these points move frame-by-frame, the device calculates your head’s velocity and rotation.
• Depth Sensors (LiDAR or Time-of-Flight): Cameras only see 2D images. To understand distance, devices use LiDAR (Light Detection and Ranging) or Time-of-Flight sensors. These emit pulses of invisible infrared light and measure how long it takes for the light to bounce back. This generates a Point Cloud a sparse collection of dots that tells the processor, “There is a solid surface 1.2 meters away.”
• Inertial Measurement Units (IMUs): These are ultra-fast accelerometers and gyroscopes (similar to those in your phone). They detect rotation and movement roughly 1,000 times per second. They fill in the micro-gaps between camera frames to ensure tracking feels instant.

2. The Brain: SLAM and Spatial Anchors

Once the sensors collect the data, the processor executes an algorithm called SLAM (Simultaneous Localization and Mapping).

This is the core engine of Mixed Reality. As you look around a room, the device builds a 3D map of the environment (Mapping) while simultaneously calculating its own coordinates within that map (Localization).

The Coordinate System (0,0,0)

When you turn on an MR headset, it establishes a “world origin” a coordinate of (0,0,0) usually right where you are standing. Every feature point in the room is then assigned a coordinate relative to that origin.

If you place a virtual chessboard on your physical desk, the device doesn’t know what a desk is. Instead, it creates a Spatial Anchor. It locks the chessboard’s coordinates to the feature points of the desk. If you walk to the other side of the room, the device calculates your new position relative to those anchors, keeping the chessboard mathematically pinned to the desk.

3. The Display Method: Passthrough vs. Optical

This is where the technology splits into two distinct approaches. How do you actually see the mix of real and digital?

Video Passthrough (The Modern Standard)

Used by: Apple Vision Pro, Meta Quest 3/Pro

In this method, you are not looking at the real world. You are looking at opaque screens inside the headset.

1. High-resolution cameras on the outside of the headset capture the world.
2. The processor takes that video feed and digitally inserts the virtual objects into the frames.
3. The combined image is displayed on the screens in front of your eyes.

The Trade-off: This allows for perfect opacity (virtual objects look solid) and complete control over lighting. However, it requires massive processing power to keep latency low. If the video feed lags behind your head movement, your inner ear will disagree with your eyes, causing motion sickness.

Optical See-Through (The Legacy/Enterprise Standard)

Used by: Microsoft HoloLens 2, Magic Leap 2

In this method, you look through transparent glass lenses (waveguides).

1. You see the real world naturally with your own eyes.
2. Micro-displays project light into the edges of the lenses.
3. The lenses bounce that light internally and project the hologram directly into your retina.

The Trade-off: This provides the most natural view of the real world and causes less eye strain. However, light is additive. You cannot project “black” (which is the absence of light). Therefore, holograms in optical devices always look slightly ghost-like or transparent, and they struggle in bright sunlight.

4. Occlusion and Physics: The Realism Layer

The difference between a gimmick and a useful tool is Occlusion. This is the ability of a real-world object to hide a digital object.

If you hold your real hand in front of a virtual floating menu, your hand should cover the menu. If the menu floats over your hand, the illusion breaks, and your brain struggles to judge depth.

How Occlusion Works

The device uses its real-time depth map (created by the LiDAR/Depth sensors) to create an invisible “mask.”

1. The system recognizes that your hand is 0.5 meters away.
2. It knows the virtual menu is placed 1.0 meter away.
3. The graphics engine renders the menu but “cuts out” the pixels where your hand is located.
4. In Video Passthrough devices, it composites your real hand over the digital layer.

Physics and Mesh Interaction

To make a virtual ball bounce off a real floor, the device wraps your physical room in an invisible Collider Mesh. This is a geometric wireframe that mirrors your furniture. When the digital ball hits the coordinates of the invisible mesh, the physics engine reverses its velocity, making it look like it bounced off your carpet.

5. The Latency Threshold (Motion-to-Photon)

The final piece of the puzzle is speed. Mixed Reality requires a Motion-to-Photon latency of under 20 milliseconds.

This means that from the moment you begin to turn your head, the sensors must detect the movement, the SLAM algorithm must update your position, the graphics engine must render the new perspective of the hologram, and the display must light up the pixels all in less than 20ms.

If this loop takes 30ms or 40ms, the digital world will feel like it is “swimming” or trailing behind your movements. This intense requirement for speed is why standalone MR headsets require such powerful mobile processors and generate significant heat.

---

Related Topics for Further Learning

• LiDAR vs. Time-of-Flight: A deeper comparison of depth sensing technologies.
• Waveguide Optics: How glass lenses channel light in optical see-through devices.
• Spatial Computing Use Cases: How industries are applying SLAM technology beyond gaming.

Conclusion

Mixed reality is a new form of immersive technology that combines elements of both virtual and augmented reality to create a seamless blend of the physical and virtual worlds. By using a combination of hardware and software, mixed reality allows users to interact with digital content as if it were part of the real world. The potential applications of mixed reality are vast, and as the technology continues to evolve, we can expect to see it being used across a wide range of industries.

Explore our comprehensive AI Key Concepts and Definitions article for detailed explanations and essential terms.

FAQs: How Mixed Reality Works