Latency

Latency is the delay between action and reaction. In augmented reality headset systems, it is the time between head movement and display update. Having low latency is crucial when using a Head-mounted Display for VR or AR. If there is a long delay, it results in an effect like the visuals are floating, rather than anchored in place. As you move your head, your HMD should display the resulting changes immediately.

Any significant delay in a VR system will cause your brain to reject the legitimacy of the virtual world and cause you to lose presence. High latency can also cause simulator sickness. In AR, high latency can cause virtual content to be misaligned with the real world, resulting in a swimmy effect.

Human brains can detect very small latency in visual and audio systems. Minimizing latency is one of VR and AR's biggest challenges.

Motion-to-photon latency / End-to-end latency

Motion-to-photon latency is the delay between the movement of the user's head and the change of the display of VR gadget reflecting the user's movement. As soon as the user's head moves, the VR scenery should match the movement. The more delay (latency) between these 2 actions, the more unrealistic the VR world seems. To make the VR world realistic, VR systems want low latency of <20ms and even really low latency of <7ms.

Software

Latency can be reduced in software by using Timewarp and preventing GPU buffering/Vsync.

Hardware

In hardware design, latency is dependent on various factors, including the display's Pixel switching time and Refresh rate.

In VR and AR, pixel switching time affects latency of the device. Reducing pixel switch time can reduce latency. Pixel switching time can be improved by using a different display technology. When VR gadgets such as Oculus Rift switched from LCD to OLED, pixel switch time was significantly reduced.