Smart Multicodal Graphics Search

In the course of the last few years, driven by higher network bandwidths and the proliferation of powerful mobile devices, a rapid growth of worldwide image and video collections can be observed. In the course of these developments, the research field of content-based image and video search has emerged with the aim of locating images and videos in large databases based on similarity. Application fields of this technology range from mobile product searches and the exploitation of extensive image archives to the detection of copyright infringements or illegal image material.

At the same time, 3D modelling/reconstruction and real-time display of 3D objects are becoming more and more widespread with new APIs such as WebGL and application areas such as 3D printing and have now reached the consumer sector in addition to classic application fields in computer animation and CAD. Similar to image search, the retrieval of suitable 3D models and their exchange and integration across different application platforms is of great interest here.

However, an open question here is above all the linking of the two modalities "image" and "3D model":

• Can a given photo or screenshot from a set of 3D models be automatically matched to the one on which the query image is based?
• And conversely, can image databases or the internet be searched with a 3D query model for those images that have been created from the model?

Such a multicodal search, i.e. a search by image / 3D model for 3D model / image, is therefore the subject of the SMULGRAS project. Compared to a purely image-based search (whose images can differ significantly due to different perspectives, texturing and/or lighting), it offers the potential of higher accuracy. In addition, the topic opens up a range of innovative application possibilities in the context of the trend topic Industry 4.0, from the precise, image-based search in product catalogues to the detection of product piracy.