Each set of four images contains (from left to right) a content input, a style input,
a standard style transfer output, and our proposed method's output.
See the "Image Credits" section under subheading "Figure 1" for artist names and source links.
Paper
Code
Demo
Talk
Bibtex
Abstract
Both geometry and texture are fundamental aspects of visual style. Existing style transfer methods,
however, primarily focus on texture, almost entirely ignoring geometry. We propose deformable style
transfer (DST), an optimization-based approach that jointly stylizes the texture and geometry of a
content image to better match a style image. Unlike previous geometry-aware stylization methods,
our approach is neither restricted to a particular domain (such as human faces), nor does it require
training sets of matching style/content pairs. We demonstrate our method on a diverse set of content
and style images including portraits, animals, objects, scenes, and paintings.
1 Minute Summary
Spatially Guided Style Transfer
The key idea of DST is to find a spatial deformation of the content image that
brings it into spatial alignment with the style image. This deformation is guided
by a set of matching keypoints, chosen to maximize the feature similarity between
paired keypoints of the two images. After roughly aligning the paired keypoints
with a rigid rotation and scaling, a simple L2 loss encourages warping our
output image in such a way that the keypoints become spatially aligned. This deformation
loss is regularized with a total variation penalty to reduce artifacts due to drastic
deformations, and combined with the more traditional style and content loss terms.
DST's joint, regularized objective simultaneously encourages preserving content,
minimizing the style loss, and obtaining the desired deformation, weighing these
goals against each other. This objective can be solved using standard iterative techniques.
Results
Each set of three images contains (from left to right) a content input, a style input, and our method's output.
Comment: A method for detecting facial landmarks and transferring geometric style of artistic portraits. We compare our DST results with results from this method.
Comment: A method for generating caricatures with geometric deformations given an input face photo. We compare our DST results with results from this method.
Comment: A geometry-aware style transfer method that represents geometry with features extracted from pre-trained convolutional neural networks.
Acknowledgements
This work is supported by the DARPA GARD award HR00112020003 and the University of Chicago CDAC Discovery Grant.
We would also like to thank Haochen Wang, Davis Gilton, Steven Basart, and members of the Perception and Learning Systems (PALS) at TTIC for their helpful comments.
Finally, we thank authors of the artworks we used to develop and evaluate our method.
In the next section, we provide credits for the images used in the paper and this site.
The webpage template was adapted from this project page.
Original artist unknown: ‘Photo of Man with Shaven Head’. http://www.faculty.idc.ac.il/arik/site/foa/faceof-art.asp
Leger, Fernand: ‘La Botte de Navets’. (1951), https://www.flickr.com/photos/32215553@N02/45868933822
Figure 2 (Paper)
Original artist unknown: ‘Ray Kurzweil Photo’, https://www.raadfest.com/raykurzweil
Meckenem, Israhel van: ‘Double Portrait of Israhel van Meckenem and His Wife Ida’. (1490), https://www.nga.gov/collection/art-object-page.3311.html
Original artist unknown: ‘Photo of Man in Suit’, Unknown Source
Van Gogh, Vincent: ‘Self-Portrait’. (1889), https://www.nga.gov/collection/artobject-page.106382.html
Original artist unknown: ‘Closeup Photo of Blond Woman’, https://cgm.technion.ac.il/ComputerGraphics-Multimedia/Software/Contextual/resources/comparison/index.html
Original artist unknown: ‘Pig Photo’, https://foodsafetynewsfullservice.marlersites.com/files/2016/04/pig-big-ears406.jpg
Original artist unknown: ‘Poppy the Pig’, https://social-artworking.com/poppythe-pig
Figure 3 (Paper)
Original artist unknown: ‘Truck Photo’. https://marvel-b1-cdn.bc0a.com/f00000000067090/www.constructionequipment.com/sites/ce/files/styles/large/public/X11GM SH051%5B1%5D.jpg
Original artist unknown: ‘Closeup Photo of Woman with Dark Hair’. https://cgm.technion.ac.il/Computer-Graphics-Multimedia/Software/Contextual/resources/comparison/index.html
Picasso, Pablo: ‘Nude Woman With Necklace
Figure 5 (Paper)
Original artist unknown: ‘Ray Kurzweil Photo’, https://www.raadfest.com/raykurzweil
Van Gogh, Vincent: ‘The Zouave’. (June 1888), https://www.vangoghmuseum.nl/en/collection/s0067V1962
Original artist unknown: ‘Husky Puppy Photo’, Unknown source
Original artist unknown: ‘Photo of Woman with Wavy Hair’. http://www.faculty.idc.ac.il/arik/site/foa/face-of-art.asp
Modigliani, Amedeo: ‘Girl with a Polka-Dot Blouse’. (1919), https://collection.barnesfoundation.org/objects/5538/Girl-with-a-Polka-DotBlouse-(Jeune-fille-au-corsage-a-pois)/
Original artist unknown: ‘Running Shoes Photo’. http://www.stockphotos.ro/runningshoes-gym-floor-after-workout-image12693027.html
Van Gogh, Vincent: ‘Shoes’. (1888), https://www.metmuseum.org/art/collection/search/436533
Figure 6 (Paper)
Original artist unknown: ‘Peasant Woman Carrying A Pack On Her Back’. https://www.davidsongalleries.com/sections/modern/french-prints/additionalavailable-works/peasant-woman-carrying-a-pack-on-her-back/
Original artist unknown: ‘Truck Photo’. https://marvel-b1-cdn.bc0a.com/f00000000067090/www.constructionequipment.com/sites/ce/files/styles/large/public/X11GM SH051%5B1%5D.jpg
Original artist unknown: ‘Photo of Hilary Swank’, https://cs.nju.edu.cn/rl/WebCaricature.htm
Original artist unknown: ‘Caricature of Hilary Swank’, https://cs.nju.edu.cn/rl/WebCaricature.htm
Original artist unknown: ‘Photo of Rowan Atkinson’, https://cs.nju.edu.cn/rl/WebCaricature.htm
Original artist unknown: ‘Photo of Rowan Atkinson’, https://cs.nju.edu.cn/rl/WebCaricature.htm
Original artist unknown: ‘Photo of Cate Blanchett’, https://cs.nju.edu.cn/rl/WebCaricature.htm
Original artist unknown: ‘Caricature of Cate Blanchett’, https://cs.nju.edu.cn/rl/WebCaricature.htm
Figure 9 (Paper)
Original artist unknown: ‘Photo of Barred Owls’. https://1.bp.blogspot.com/-1weEwEzz2wk/Xb64zZC-2RI/AAAAAAADI-M/pTeSYN7v KAo2YK3FsPUt6SVe76A5pYaQCLcBGAsYHQ/s640/88l.jpg
Yun, Sungsil: ‘Korean Traditional Watercolor Owl’. https://www.saatchiart.com/art/PaintingKorean-traditional-watercolor-owl/1143968/4610489/view
Original artist unknown: ‘Surprised Boy Photo’. http://weknowyourdreams.com/single/surprise/surprise-01
Original artist unknown: ‘Maya the Bee Render’. https://www.independent.co.uk/artsentertainment/tv/news/netflix-penis-cartoon-penis-episode-maya-the-bee-a7958551.html
Original artist unknown: ‘Husky Photo’. Unknown Source
Original artist unknown: ‘Dog Caricature’. Unknown Source
Supplementary Figure 1 (Paper)
Original artist unknown: ‘Black Horse Photo’. Unknown Source
Bratby, John Randall: 'Kathy Wilkes, Fellow and Tutor in Philosophy' https://artuk.org/discover/artworks/kathy-wilkes-fellow-and-tutor-in-philosophy-223246
Original artist unknown: 'Keira Knightley Photo': http://wallpaperswide.com/keira_knightley_portrait_2-wallpapers.html
Matisse, Henri: 'Purple Robe and Anemones' https://www.matissepaintings.org/purple-robe-and-anemones/
Original artist unknown: 'Jeon Do-Yeon Photo' https://www.themoviedb.org/person/20737-jeon-do-yeon
Bratby, John Randall: 'Sir Arthur Bryan' https://artuk.org/discover/artworks/sir-arthur-bryan-20314
Animals and Objects (Website)
Original artist unknown: ‘Running Shoes Photo’. http://www.stockphotos.ro/runningshoes-gym-floor-after-workout-image12693027.html
Van Gogh, Vincent: ‘Shoes’. (1888), https://www.metmuseum.org/art/collection/search/436533
Original artist unknown: ‘Truck Photo’. https://marvel-b1-cdn.bc0a.com/f00000000067090/www.constructionequipment.com /sites/ce/files/styles/large/public/X11GM SH051%5B1%5D.jpg
Original artist unknown: ‘Peasant Woman Carrying A Pack On Her Back’. https://www.davidsongalleries.com/sections/modern/french-prints/additionalavailable-works/peasant-woman-carrying-a-pack-on-her-back/
Original artist unknown: 'Photo of Leonardo DiCaprio', https://www.atlantamagazine.com/news-culture-articles/leonardo-dicaprio-and-jonah-hill-set-to-star-in-film-about-richard-jewell/