Jump to content

HEAT: Difference between revisions

From OpenVerse Wiki
VisualWikitext
Created page with "=== HEAT Project === {| class='wikitable' style='margin:auto' |- ! CORDIS Reference !! Start date !! End date !! Coordinator |- | https://cordis.europa.eu/project/id/101135637 || 01/06/2024 || 31/05/2027 || UNIVERSITA DEGLI STUDI DI CAGLIARI / Italy |} === Project description === Hybrid Extended reAliTy (HEAT) is born to pave the way for the next-generation distributed experiences by addressing major challenges to make those experiences that up to now could only be in..."
 
No edit summary
 
Line 8: Line 8:
=== Project description ===
=== Project description ===
Hybrid Extended reAliTy (HEAT) is born to pave the way for the next-generation distributed experiences by addressing major challenges to make those experiences that up to now could only be in our imagination: being realistically immersed (holo-ported) within real captured omnidirectional and navigable hyper-realistic 3D spaces, feeling their atmosphere, and sharing these experiences with others, regardless of their location. The aim of this proposal is to integrate immersive media technologies such as point cloud/holographic imaging, multi-sensorial media, Social VR in a multi-user, feedback-enabled communication system to provide the construction of compelling context-aware and embodied experiences for innovative hybrid XR applications, where remote users can experience a real captured environment through immersive VR, while in presence users can visualise and interact with the holograms of remote users integrated in the real environment through holographic rendering. The system will aim at facilitating the exploitation of agile (multi-sensory) 3D data acquisition techniques, enhancing performance while reducing technology costs. It will create a scalable communication pipeline embodying either encapsulation of different media from classical audio-video to multisensorial to holographic video or a combination of them, providing means for efficient and scalable encoding, processing, storage, (real-time) streaming and rendering. In such a way, the system will allow users to provide/exploit the features according to the acquisition/rendering system available. The project will provide well designed and fully tested scenarios in real-world environments for enhanced XR experiences: a blended learning, a modern theatre act, a music festival and an opera show. All pilot actions will ensure that GDPR and ethics are addressed for end-users (privacy and ethics by design methodology).
Hybrid Extended reAliTy (HEAT) is born to pave the way for the next-generation distributed experiences by addressing major challenges to make those experiences that up to now could only be in our imagination: being realistically immersed (holo-ported) within real captured omnidirectional and navigable hyper-realistic 3D spaces, feeling their atmosphere, and sharing these experiences with others, regardless of their location. The aim of this proposal is to integrate immersive media technologies such as point cloud/holographic imaging, multi-sensorial media, Social VR in a multi-user, feedback-enabled communication system to provide the construction of compelling context-aware and embodied experiences for innovative hybrid XR applications, where remote users can experience a real captured environment through immersive VR, while in presence users can visualise and interact with the holograms of remote users integrated in the real environment through holographic rendering. The system will aim at facilitating the exploitation of agile (multi-sensory) 3D data acquisition techniques, enhancing performance while reducing technology costs. It will create a scalable communication pipeline embodying either encapsulation of different media from classical audio-video to multisensorial to holographic video or a combination of them, providing means for efficient and scalable encoding, processing, storage, (real-time) streaming and rendering. In such a way, the system will allow users to provide/exploit the features according to the acquisition/rendering system available. The project will provide well designed and fully tested scenarios in real-world environments for enhanced XR experiences: a blended learning, a modern theatre act, a music festival and an opera show. All pilot actions will ensure that GDPR and ethics are addressed for end-users (privacy and ethics by design methodology).
=== Project outputs ===
==== Publications ====
{| class="wikitable sortable"
! Domain !! Type of output !! Title !! DOI URL
|-
| AI, Machine Learning & Data Science || Conference proceedings || Forecasting Ethereum Prices with Machine Learning, Deep Learning, and Explainable Artificial Intelligence Using Multi-source Market Articles and Hybrid Sentiment Analysis || https://doi.org/10.1007/978-3-032-04339-9_12
|-
| AI, Machine Learning & Data Science || Conference proceedings || Pixelating to the Edge: Generative AI Art on Edge Devices || https://doi.org/10.1109/BMSB65076.2025.11165718
|-
| Computer Vision, 3D Modeling & Rendering || Peer reviewed articles || Low-Complexity Patch-Based No-Reference Point Cloud Quality Metric Exploiting Weighted Structure and Texture Features || https://doi.org/10.48550/ARXIV.2503.15001
|-
| Computer Vision, 3D Modeling & Rendering || Peer reviewed articles || CNN-Based 360° Scene Recognition for Automatic Generation of Omnidirectional Scent Effects || https://doi.org/10.1109/TMM.2025.3618574
|-
| Ethics, Society, Arts & Culture || Conference proceedings || Proxemic Behavior in a Multi-User Virtual Reality Experience for Multilingual Cultural Heritage Education || https://doi.org/10.1109/ISPA66905.2025.11259464
|-
| Extended Reality (VR/AR/MR) & HCI || Conference proceedings || Immersive Live Concert: A Multi-Sensory Experience Based on Real-Time Lyrics Detection from Spatial Audio Data || https://doi.org/10.5753/IMXW.2025.1139
|-
| Extended Reality (VR/AR/MR) & HCI || Conference proceedings || Social eXtended Reality (XR) and Virtual Production: Toward New Engaging Immersive Experiences || https://doi.org/10.1145/3706370.3731642
|-
| Extended Reality (VR/AR/MR) & HCI || Conference proceedings || Immersive Application for Real-Time Interactive Music Performances Using Spatial Audio || https://doi.org/10.1109/BMSB65076.2025.11165553
|-
| Extended Reality (VR/AR/MR) & HCI || Conference proceedings || Live Feedback for Immersive Music Performances - A Case Study || https://doi.org/10.5753/IMXW.2025.8520
|-
| Extended Reality (VR/AR/MR) & HCI || Conference proceedings || aDapT-XR: Adaptive Data Allocation and Prioritization for Synchronizing Real and Virtual Worlds in XR Digital Twins || https://doi.org/10.1109/BMSB65076.2025.11165558
|-
| Extended Reality (VR/AR/MR) & HCI || Conference proceedings || EmoLoop: A Bi-Directional System for Emotion-Driven Interaction Between Remote Audiences and Live Performers || https://doi.org/10.1109/IS264627.2025.11284604
|-
| Extended Reality (VR/AR/MR) & HCI || Conference proceedings || A Fast Volumetric Capture and Reconstruction Pipeline for Dynamic Point Clouds and Gaussian Splats || https://doi.org/10.1145/3756863.3769713
|-
| Extended Reality (VR/AR/MR) & HCI || Conference proceedings || An Architecture for Real-Time Interactive Music Performances || https://doi.org/10.1109/MEDCOMNET65822.2025.11103534
|-
| Extended Reality (VR/AR/MR) & HCI || Conference proceedings || Enhancing XR Theatre with Remote Scent Delivery Using Audio Speech Recognition and Convolutional Neural Network-Based Scene Detection || https://doi.org/10.1109/IS264627.2025.11284558
|-
| Extended Reality (VR/AR/MR) & HCI || Conference proceedings || Social eXtended Reality (XR) and Virtual Production: Crafting New Frontiers in Immersive Storytelling || https://doi.org/10.5753/IMXW.2025.1505
|-
| Extended Reality (VR/AR/MR) & HCI || Conference proceedings || Toward Interactive and Distributed Hybrid Extended Reality Experiences || https://doi.org/10.1145/3672406.3672424
|-
| Extended Reality (VR/AR/MR) & HCI || Conference proceedings || Social VR with multi-user holo-portation: toward a new medium for rich interactive shared media consumption || https://doi.org/10.1109/BMSB65076.2025.11165698
|-
| Extended Reality (VR/AR/MR) & HCI || Peer reviewed articles || Social VR With Holographic Comms: Enablers for New Engaging Experiences Within the TV / Video Consumption Landscape || https://doi.org/10.1109/TBC.2025.3570869
|-
| Networks, Cloud & Telecommunications (5G/6G) || Conference proceedings || Collaborative 360-Degree Video Streaming: A Multi-User Synchronization Solution with Socket.IO || https://doi.org/10.1109/BMSB65076.2025.11165535
|-
| Networks, Cloud & Telecommunications (5G/6G) || Conference proceedings || Estimating Quality of Experience in Multicast Point Cloud Streaming over 5G Networks || https://doi.org/10.1109/BMSB65076.2025.11165660
|-
| Networks, Cloud & Telecommunications (5G/6G) || Conference proceedings || Mass Live and Interactive Multi-Cam VR360 Experiences as an Immersive Parallel Window to TV Broadcast Shows || https://doi.org/10.1109/BMSB65076.2025.11165554
|-
| Networks, Cloud & Telecommunications (5G/6G) || Conference proceedings || QoE-aware ML models based on network parameters for video streaming over 5G O-RAN architecture || https://doi.org/10.1109/BMSB65076.2025.11165626
|-
| Networks, Cloud & Telecommunications (5G/6G) || Peer reviewed articles || Utility-Aware Adaptive Streaming of Segmented Holographic Video Over Wireless Networks: A Knapsack-Theoretic Approach || https://doi.org/10.1109/JIOT.2025.3605830
|-
| Networks, Cloud & Telecommunications (5G/6G) || Peer reviewed articles || Quality and Energy-Aware Cooperative Distributed Solution for Mobile Rich Media Delivery in Wireless Heterogeneous Environments || https://doi.org/10.1109/TBC.2025.3609056
|-
| Networks, Cloud & Telecommunications (5G/6G) || Peer reviewed articles || A UAV-Centric Improved Soft Actor-Critic Algorithm for QoE-Focused Aerial Video Streaming || https://doi.org/10.1109/TVT.2024.3396349
|-
| Networks, Cloud & Telecommunications (5G/6G) || Peer reviewed articles || A Live Adaptive Streaming Solution for Enhancing Quality of Experience in Co-Created Opera || https://doi.org/10.1109/TBC.2025.3541875
|}
==== Technological assets ====
{| class="wikitable sortable"
! Title !! Type of Asset !! Link / DOI !! Description
|-
| EmoLoop || System / Software || https://doi.org/10.1109/IS264627.2025.11284604 || A bi-directional system designed for emotion-driven interaction between remote audiences and performers.
|}

Latest revision as of 13:24, 22 April 2026

HEAT Project

CORDIS Reference Start date End date Coordinator
https://cordis.europa.eu/project/id/101135637 01/06/2024 31/05/2027 UNIVERSITA DEGLI STUDI DI CAGLIARI / Italy

Project description

Hybrid Extended reAliTy (HEAT) is born to pave the way for the next-generation distributed experiences by addressing major challenges to make those experiences that up to now could only be in our imagination: being realistically immersed (holo-ported) within real captured omnidirectional and navigable hyper-realistic 3D spaces, feeling their atmosphere, and sharing these experiences with others, regardless of their location. The aim of this proposal is to integrate immersive media technologies such as point cloud/holographic imaging, multi-sensorial media, Social VR in a multi-user, feedback-enabled communication system to provide the construction of compelling context-aware and embodied experiences for innovative hybrid XR applications, where remote users can experience a real captured environment through immersive VR, while in presence users can visualise and interact with the holograms of remote users integrated in the real environment through holographic rendering. The system will aim at facilitating the exploitation of agile (multi-sensory) 3D data acquisition techniques, enhancing performance while reducing technology costs. It will create a scalable communication pipeline embodying either encapsulation of different media from classical audio-video to multisensorial to holographic video or a combination of them, providing means for efficient and scalable encoding, processing, storage, (real-time) streaming and rendering. In such a way, the system will allow users to provide/exploit the features according to the acquisition/rendering system available. The project will provide well designed and fully tested scenarios in real-world environments for enhanced XR experiences: a blended learning, a modern theatre act, a music festival and an opera show. All pilot actions will ensure that GDPR and ethics are addressed for end-users (privacy and ethics by design methodology).

Project outputs

Publications

Domain Type of output Title DOI URL
AI, Machine Learning & Data Science Conference proceedings Forecasting Ethereum Prices with Machine Learning, Deep Learning, and Explainable Artificial Intelligence Using Multi-source Market Articles and Hybrid Sentiment Analysis https://doi.org/10.1007/978-3-032-04339-9_12
AI, Machine Learning & Data Science Conference proceedings Pixelating to the Edge: Generative AI Art on Edge Devices https://doi.org/10.1109/BMSB65076.2025.11165718
Computer Vision, 3D Modeling & Rendering Peer reviewed articles Low-Complexity Patch-Based No-Reference Point Cloud Quality Metric Exploiting Weighted Structure and Texture Features https://doi.org/10.48550/ARXIV.2503.15001
Computer Vision, 3D Modeling & Rendering Peer reviewed articles CNN-Based 360° Scene Recognition for Automatic Generation of Omnidirectional Scent Effects https://doi.org/10.1109/TMM.2025.3618574
Ethics, Society, Arts & Culture Conference proceedings Proxemic Behavior in a Multi-User Virtual Reality Experience for Multilingual Cultural Heritage Education https://doi.org/10.1109/ISPA66905.2025.11259464
Extended Reality (VR/AR/MR) & HCI Conference proceedings Immersive Live Concert: A Multi-Sensory Experience Based on Real-Time Lyrics Detection from Spatial Audio Data https://doi.org/10.5753/IMXW.2025.1139
Extended Reality (VR/AR/MR) & HCI Conference proceedings Social eXtended Reality (XR) and Virtual Production: Toward New Engaging Immersive Experiences https://doi.org/10.1145/3706370.3731642
Extended Reality (VR/AR/MR) & HCI Conference proceedings Immersive Application for Real-Time Interactive Music Performances Using Spatial Audio https://doi.org/10.1109/BMSB65076.2025.11165553
Extended Reality (VR/AR/MR) & HCI Conference proceedings Live Feedback for Immersive Music Performances - A Case Study https://doi.org/10.5753/IMXW.2025.8520
Extended Reality (VR/AR/MR) & HCI Conference proceedings aDapT-XR: Adaptive Data Allocation and Prioritization for Synchronizing Real and Virtual Worlds in XR Digital Twins https://doi.org/10.1109/BMSB65076.2025.11165558
Extended Reality (VR/AR/MR) & HCI Conference proceedings EmoLoop: A Bi-Directional System for Emotion-Driven Interaction Between Remote Audiences and Live Performers https://doi.org/10.1109/IS264627.2025.11284604
Extended Reality (VR/AR/MR) & HCI Conference proceedings A Fast Volumetric Capture and Reconstruction Pipeline for Dynamic Point Clouds and Gaussian Splats https://doi.org/10.1145/3756863.3769713
Extended Reality (VR/AR/MR) & HCI Conference proceedings An Architecture for Real-Time Interactive Music Performances https://doi.org/10.1109/MEDCOMNET65822.2025.11103534
Extended Reality (VR/AR/MR) & HCI Conference proceedings Enhancing XR Theatre with Remote Scent Delivery Using Audio Speech Recognition and Convolutional Neural Network-Based Scene Detection https://doi.org/10.1109/IS264627.2025.11284558
Extended Reality (VR/AR/MR) & HCI Conference proceedings Social eXtended Reality (XR) and Virtual Production: Crafting New Frontiers in Immersive Storytelling https://doi.org/10.5753/IMXW.2025.1505
Extended Reality (VR/AR/MR) & HCI Conference proceedings Toward Interactive and Distributed Hybrid Extended Reality Experiences https://doi.org/10.1145/3672406.3672424
Extended Reality (VR/AR/MR) & HCI Conference proceedings Social VR with multi-user holo-portation: toward a new medium for rich interactive shared media consumption https://doi.org/10.1109/BMSB65076.2025.11165698
Extended Reality (VR/AR/MR) & HCI Peer reviewed articles Social VR With Holographic Comms: Enablers for New Engaging Experiences Within the TV / Video Consumption Landscape https://doi.org/10.1109/TBC.2025.3570869
Networks, Cloud & Telecommunications (5G/6G) Conference proceedings Collaborative 360-Degree Video Streaming: A Multi-User Synchronization Solution with Socket.IO https://doi.org/10.1109/BMSB65076.2025.11165535
Networks, Cloud & Telecommunications (5G/6G) Conference proceedings Estimating Quality of Experience in Multicast Point Cloud Streaming over 5G Networks https://doi.org/10.1109/BMSB65076.2025.11165660
Networks, Cloud & Telecommunications (5G/6G) Conference proceedings Mass Live and Interactive Multi-Cam VR360 Experiences as an Immersive Parallel Window to TV Broadcast Shows https://doi.org/10.1109/BMSB65076.2025.11165554
Networks, Cloud & Telecommunications (5G/6G) Conference proceedings QoE-aware ML models based on network parameters for video streaming over 5G O-RAN architecture https://doi.org/10.1109/BMSB65076.2025.11165626
Networks, Cloud & Telecommunications (5G/6G) Peer reviewed articles Utility-Aware Adaptive Streaming of Segmented Holographic Video Over Wireless Networks: A Knapsack-Theoretic Approach https://doi.org/10.1109/JIOT.2025.3605830
Networks, Cloud & Telecommunications (5G/6G) Peer reviewed articles Quality and Energy-Aware Cooperative Distributed Solution for Mobile Rich Media Delivery in Wireless Heterogeneous Environments https://doi.org/10.1109/TBC.2025.3609056
Networks, Cloud & Telecommunications (5G/6G) Peer reviewed articles A UAV-Centric Improved Soft Actor-Critic Algorithm for QoE-Focused Aerial Video Streaming https://doi.org/10.1109/TVT.2024.3396349
Networks, Cloud & Telecommunications (5G/6G) Peer reviewed articles A Live Adaptive Streaming Solution for Enhancing Quality of Experience in Co-Created Opera https://doi.org/10.1109/TBC.2025.3541875

Technological assets

Title Type of Asset Link / DOI Description
EmoLoop System / Software https://doi.org/10.1109/IS264627.2025.11284604 A bi-directional system designed for emotion-driven interaction between remote audiences and performers.
Retrieved from "https://wiki.open-verse.eu/index.php?title=HEAT&oldid=302"