Latest Artificial Intelligence (AI) Research from Korea Open-Sources ‘Dr.3D’, a New 3D GAN Domain Adaptation Method for Drawings

It is essential for a wide range of computer graphics and vision applications that generative adversarial networks (GANs) learn to create realistic images. Notably, GANs allow exploration and editing of semantically meaningful images for real and synthetic images. Not surprisingly, the human face is one of the most frequently targeted image categories in computer vision and graphics by GAN algorithms. The awareness of GANs in 3D geometry has recently attracted a lot of attention, creating an exciting field of study for 3D GANs. By directly simulating 3D light transit between a camera and a target object, they solve the problem of learning the 3D-sensitive distribution of real images.

In addition to enabling semantically meaningful photo synthesis and manipulation, 3D GANs also take into account the geometry of the 3D scene. So far, the majority of 3D GAN demonstrations have been limited to real-world images, which are accurate recordings of real-world settings made with perspective cameras. In this study, they extend the ability of 3D GANs to handle drawing, a new but significant visual form. The drawings have a huge impact on human history as they reflect both real and fictional subjects with deliberate and unintended changes. By converting 2D GANs that have been pre-trained on real-world images into drawings, a process known as domain adaptation, existing 2D GAN approaches have been extended to handle drawings.

Figure 1: Examples of GAN inversion and semantic editing on a drawing of a person. By refining StyleNeRF and 𝜋-GAN on the portrait drawings, we perform naive domain adaptation to compare them. Then, using each 3D GAN model, we reconstruct the image and its shape in various camera poses after inverting the input image in (a) using a standard GAN inversion approach to a latent code. The results of (b) and (c) demonstrate how naive adaptations of 3D GAN are unable to handle input drawing. On the other hand, as shown in (d) and (e), our system can correctly recreate the input image and also support semantic editing (e). Image in (a): Portrait of a member of the Wedigh family

The adaptation technique takes advantage of the similarities between drawings and images, allowing us to apply the synthesis and editing capabilities of 2D GANs to the field of drawing. Unfortunately, as Figure 1 illustrates, the use of 3D GANs in the drawing domain is more difficult. The drawings have an inherent geometric ambiguity regarding the subject and the attitude of the camera, which is one of the main causes of this challenge. The creatively ambiguous drawings emerge from the deliberate or unintentional assumption of the artists about the non-deterministic geometry of the subjects from an imaginary point of view that diverges from the physical point of view. This makes learning a 3D compatible drawing image distribution even more difficult and makes it difficult to directly transfer previous domain adaptation techniques from 2D GAN methods to 3D GAN methods.

Figure 1 shows that using domain adaptation to apply state-of-the-art 3D GANs to drawings fails to produce accurate and consistent 3D images. This study proposes Dr.3D, a completely new 3D GAN domain adaptation technique for portrait drawings. Dr.3D uses three solutions to resolve fundamental geometric ambiguity in drawings. First, they provide a deformation-sensitive 3D synthesis network capable of learning various drawing shapes. To successfully reduce the learning complexity of ambiguous 3D geometries and camera poses in drawings, they also present an alternating adaptation technique for 3D image synthesis and posture estimation. Third, they impose geometric priors to facilitate stable domain matching of real images to drawings. The resulting domain adaptation technique, Dr.3D, is the first technique that can modify and synthesize 3D drawing images over time in a consistent manner. They carefully quantitatively and qualitatively evaluate the effectiveness of Dr. 3D. The PyTorch implementation will soon be available on GitHub.

Check Paper, Github linkand Project. All credit for this research goes to the researchers on this project. Also don’t forget to register. our Reddit page and discord channelwhere we share the latest AI research news, cool AI projects, and more.