MR immersive drama makes Shanghai debut

Taixu Dreamscape, the world's first mixed reality immersive drama, made its debut in Shanghai in June, delivering a theatrical experience that dissolves the boundary between the physical stage and the digital world.

Staged at Shanghai's 1862 Theater, the production draws its soul from Dream of the Red Chamber — one of China's four great classical novels — and pioneers the full integration of MR technology with 4D Gaussian splatting, using Apple Vision Pro headsets as the interactive medium for each audience member.

"We kept asking ourselves, is there a technology that can give audiences the warmth of live performance while also opening up infinite possibilities in a blended reality?" said Zhong Zhuo, the play's general director and president of Shanghai SeasonONstage group. "Every piece of technology we deployed exists solely to serve the story and amplify the emotional truth of each moment."

The story is set within a 1940s theatrical troupe staging a performance of Dream of the Red Chamber. Audiences enter as "phantom" in the system, free to roam a film-grade set spanning more than 400 square meters while wearing Apple Vision Pro headsets that track their position and hand gestures.

The production team deployed 150 industrial cameras in a matrix formation within a professional studio, capturing performers from every angle simultaneously. Proprietary AI algorithms then reconstruct that data in real time into digital avatars that audiences can explore from any direction.

Volumetric video technology embeds those avatars into the physical theater space, where virtual characters walk alongside real actors, their past lives materializing in parallel dimensions that audience members can interact with.

"The greatest challenge was technology," said Zhong, noting that the project consumed 14 months from conception to curtain, with the majority of that time devoted to achieving a truly seamless fusion of art and engineering.

"Our biggest hurdle was allowing audiences to witness an actor's past and present lives, all within the same time and space, and to participate in that narrative without ever breaking immersion," said Jiang Xuefen, the production's supervising producer.

"Immersive performing arts have been iterating alongside XR technology for years," Jiang added. "This new paradigm of MR and 4D Gaussian splatting at commercial theater scale will empower a much wider range of performance content and cultural tourism spaces. We hope to keep creating works that let technology breathe new life into cultural heritage."