Each set of four images contains (from left to right) a content input, a style input,
a standard style transfer output, and our proposed method's output.
Paper
Code
Demo
Talk
Bibtex
Abstract
Both geometry and texture are fundamental aspects of visual style. Existing style transfer methods,
however, primarily focus on texture, almost entirely ignoring geometry. We propose deformable style
transfer (DST), an optimization-based approach that jointly stylizes the texture and geometry of a
content image to better match a style image. Unlike previous geometry-aware stylization methods,
our approach is neither restricted to a particular domain (such as human faces), nor does it require
training sets of matching style/content pairs. We demonstrate our method on a diverse set of content
and style images including portraits, animals, objects, scenes, and paintings.
1 Minute Summary
Spatially Guided Style Transfer
The key idea of DST is to find a spatial deformation of the content image that
brings it into spatial alignment with the style image. This deformation is guided
by a set of matching keypoints, chosen to maximize the feature similarity between
paired keypoints of the two images. After roughly aligning the paired keypoints
with a rigid rotation and scaling, a simple L2 loss encourages warping our
output image in such a way that the keypoints become spatially aligned. This deformation
loss is regularized with a total variation penalty to reduce artifacts due to drastic
deformations, and combined with the more traditional style and content loss terms.
DST's joint, regularized objective simultaneously encourages preserving content,
minimizing the style loss, and obtaining the desired deformation, weighing these
goals against each other. This objective can be solved using standard iterative techniques.
Results
Each set of three images contains (from left to right) a content input, a style input, and our method's output.
Comment: A method for detecting facial landmarks and transferring geometric style of artistic portraits. We compare our DST results with results from this method.
Comment: A method for generating caricatures with geometric deformations given an input face photo. We compare our DST results with results from this method.
Comment: A geometry-aware style transfer method that represents geometry with features extracted from pre-trained convolutional neural networks.
Acknowledgements
This work is supported by the DARPA GARD award HR00112020003 and the University of Chicago CDAC Discovery Grant.
We would also like to thank Haochen Wang, Davis Gilton, Steven Basart, and members of the Perception and Learning Systems (PALS) at TTIC for their helpful comments.
