XReco

XReco Project

CORDIS Reference	Start date	End date	Coordinator
https://cordis.europa.eu/project/id/101070250	01/09/2022	31/08/2025	DEUTSCHE WELLE / Germany

Project description

While media organisations increasingly support non-linear experiences for the consumer, those are still limited to single channels and media domains. Although several media organisations have recently succeeded in breaking data silos, data sharing is mostly limited to the organisation. Thus, there are challenges for producing content feeding multiple channels with different granularities and structures, mainly related to data discovery, management and (re-)use. XRECO will create a new data-driven ecosystem for the media industry, focusing on facilitating data sharing, search and discovery and supporting creation of news and entertainment content, in particular, the creation and (re-)use of location-related 2D and 3D assets and the creation of XR experiences. The ecosystem core, represented by a Neural Media Repository (NMR), will foster inter-organisation content sharing and provide increased access to content for media creators, considering novel data monetization and rights management policies. A set of AI-based media transformation services are built around the NMR to produce novel media- and XR experiences, including 3D neural reconstructions, neural based device localisation, image stitching, de-/re-lighting and holoportation. The developed technology will be validated in use case scenarios for (i) the news media for XR-based broadcasting and automatic and customized multitarget news publishing, and for (ii) location-based information and entertainment content, with applications in tourism and the automotive industry.

Project outputs

Publications

Domain	Type of output	Title	DOI URL
AI, Machine Learning & Data Science	Conference proceedings	Efficient Few-Shot Incremental Training for Landmark Recognition	https://doi.org/10.1145/3672406.3672414
Computer Vision, 3D Modeling & Rendering	Book chapters	Image Valuation in NeRF-Based 3D Reconstruction	https://doi.org/10.1007/978-3-032-04968-1_32
Computer Vision, 3D Modeling & Rendering	Book chapters	Complete Convolutional Neural Networks Environment for Computer Vision Problems With Nvidia Drive AGX Xavier	https://doi.org/10.1007/978-3-031-70248-8_7
Computer Vision, 3D Modeling & Rendering	Book chapters	Synthetic Football Sprite Animations Learned Across the Pitch	https://doi.org/10.1007/978-3-031-41774-0_48
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Descriptor Impact on Multimodal 3D Retrieval	https://doi.org/10.5281/ZENODO.11942398
Computer Vision, 3D Modeling & Rendering	Conference proceedings	A Dataset and Metric for Textual Video Content Description	https://doi.org/10.1145/3746027.3758224
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Urban Scene Removal and Completion	https://doi.org/10.3233/FAIA250603
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Analysis of Objective 3D Mesh Quality Metrics for Cultural Heritage	https://doi.org/10.1109/QOMEX65720.2025.11219903
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Untethered Real-Time Immersive Free Viewpoint Video	https://doi.org/10.1145/3652212.3652214
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Multimodal Understanding: Investigating the Capabilities of Large Multimodal Models for Object Detection in XR Applications	https://doi.org/10.1145/3688866.3689126
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Volumetric Video Reconstruction and Communications: Toward a New Era of Interactive and Immersive Social Virtual Reality (VR) Experiences	https://doi.org/10.1145/3672406.3672421
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Semi -Automated Digital Human Production for Enhanced Media Broadcasting	https://doi.org/10.1109/GEM61861.2024.10585601
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Multimedia Information Retrieval in XR	https://doi.org/10.1145/3664647.3689176
Computer Vision, 3D Modeling & Rendering	Conference proceedings	3DMSE: An Interactive 3D Media Search Engine	https://doi.org/10.1145/3652583.3657593
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Is Real-time Deep Learning-based Monocular Depth Estimation accurate for Multi-Camera Setups?	https://doi.org/10.1109/ICCT-EUROPE63283.2025.11157669
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Analysis and Development of Deep Learning Depth Estimation Techniques for Volumetric Capture and Free Viewpoint Video	https://doi.org/10.1145/3625468.3652913
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Real-Time Free Viewpoint Video for Immersive Videoconferencing	https://doi.org/10.1109/QOMEX61742.2024.10598259
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Multimodality in Media Retrieval	https://doi.org/10.1145/3652583.3657583
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Exploring Image Search on Quantum Computing Systems	https://doi.org/10.5220/0013562200004525
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Towards a Universal Query Representation for Multimodal Information Retreival	https://doi.org/10.1145/3746027.3758155
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Multimedia Retrieval in and for XR	https://doi.org/10.1145/3652583.3658421
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Enabling Domain Experts to Train Efficient Few-Shot Incremental Landmark Recognition	https://doi.org/10.1109/CBMI62980.2024.10859238
Computer Vision, 3D Modeling & Rendering	Conference proceedings	XReco Platform and RAI News Media Demonstrator	https://doi.org/10.1145/3746027.3761840
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Open-Source Multimedia Retrieval with vitrivr-engine	https://doi.org/10.1145/3746027.3756874
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Ubervvald: Advanced Object Detection Library for Optimizing Complex Convolutional Neural Networks (CNNs)	https://doi.org/10.1007/978-981-96-5887-9_14
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Nostrils and Mouth Detection for Drivers Using Convolutional Neural Networks with Automatically Generated Ground Truth Data	https://doi.org/10.1109/CSCI58124.2022.00265
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Ground Truth Data Generator in Automotive Infrared Sensor Vision Problems Using a Minimum Set of Operations	https://doi.org/10.1109/SYNASC61333.2023.00039
Computer Vision, 3D Modeling & Rendering	Conference proceedings	A new Retrieval Engine for vitrivr	https://doi.org/10.1007/978-3-031-53302-0_28
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Improving Query and Assessment Quality in Text-Based Video Retrieval Evaluation	https://doi.org/10.1145/3591106.3592281
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Real-Time Layered View Synthesis for Free-Viewpoint Video from Unreliable Depth Information	https://doi.org/10.1145/3592834.3592881
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Free-form Multi-Modal Multimedia Retrieval (4MR)	https://doi.org/10.1007/978-3-031-27077-2_58
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Multimedia Retrieval in Mixed Reality: Leveraging Live Queries for Immersive Experiences	https://doi.org/10.1109/AIxVR59861.2024.00048
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Mining Landmark Images for Scene Reconstruction from Weakly Annotated Video Collections	https://doi.org/10.1007/978-3-031-53302-0_12
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Synthetic Football Sprite Animations Learned Across the Pitch.	https://doi.org/10.1007/978-3-031-41774-0_48
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Subjective Evaluation of Dynamic Point Clouds: Impact of Compression and Exploration Behavior	https://doi.org/10.23919/EUSIPCO58844.2023.10290086
Computer Vision, 3D Modeling & Rendering	Conference proceedings	Multimodal 3D Object Retrieval	https://doi.org/10.5281/zenodo.10226588
Computer Vision, 3D Modeling & Rendering	Peer reviewed articles	Interactive Multimodal Video Search: An Extended Post-Evaluation for the VBS 2022 Competition	https://doi.org/10.1007/s13735-024-00325-9
Computer Vision, 3D Modeling & Rendering	Peer reviewed articles	Multimedia Systems	https://doi.org/10.5167/uzh-236035
Computer Vision, 3D Modeling & Rendering	Peer reviewed articles	An Assessment of the Stereo and Near-Infrared Camera Calibration Technique Using a Novel Real-Time Approach in the Context of Resource Efficiency	https://doi.org/10.3390/PR13041198
Cybersecurity, Privacy & Blockchain	Conference proceedings	Data as Remuneration in Digital Copyright Licensing: Some Reflections on the Concept of ‘Appropriate and Proportionate Remuneration’ Under Art. 18 EU Directive 2019/790 in the Data Era	https://doi.org/10.5281/ZENODO.18482948
Cybersecurity, Privacy & Blockchain	Conference proceedings	Secure, Dynamic and Uncomplicated Licensing of Movies on a Blockchain Infrastructure	https://doi.org/10.1109/ICOIN56518.2023.10049017
Extended Reality (VR/AR/MR) & HCI	Peer reviewed articles	IEEE Access	https://doi.org/10.5167/UZH-261490
Extended Reality (VR/AR/MR) & HCI	Peer reviewed articles	Delay Threshold for Social Interaction in Volumetric eXtended Reality Communication	https://doi.org/10.1145/3651164
Networks, Cloud & Telecommunications (5G/6G)	Conference proceedings	VERGE in CBMI2024	https://doi.org/10.5281/ZENODO.10652893
Networks, Cloud & Telecommunications (5G/6G)	Conference proceedings	VERGE in VBS 2024	https://doi.org/10.5281/zenodo.10652893

Technological assets

Title	Type of Asset	Link / DOI	Description
XR and Media Transformation APIs and Authoring Tools	APIs / Tools	https://xreco.eu/deliverables/#toc_D41_XR_and_Media_Transformation_Services_API_and	APIs integrating vertical technologies for XR media transformation and content creation.
Textual Video Content Dataset	Dataset / Metric	https://doi.org/10.1145/3746027.3758224	A dataset and corresponding metric created specifically for textual video content description.
vitrivr-engine	Open-Source Engine	https://doi.org/10.1145/3746027.3756874	An open-source multimedia retrieval engine for content and similarity searches.