Deformable Style Transfer

Sunnie S. Y. Kim1   Nicholas Kolkin1   Jason Salavon2   Gregory Shakhnarovich1
1Toyota Technological Institute at Chicago     2University of Chicago

Each set of four images contains (from left to right) a content input, a style input,
a standard style transfer output, and our proposed method's output.
See the "Image Credits" section under subheading "Figure 1" for artist names and source links.
 

Paper

Code

Demo

Talk

Bibtex

Abstract

Both geometry and texture are fundamental aspects of visual style. Existing style transfer methods, however, primarily focus on texture, almost entirely ignoring geometry. We propose deformable style transfer (DST), an optimization-based approach that jointly stylizes the texture and geometry of a content image to better match a style image. Unlike previous geometry-aware stylization methods, our approach is neither restricted to a particular domain (such as human faces), nor does it require training sets of matching style/content pairs. We demonstrate our method on a diverse set of content and style images including portraits, animals, objects, scenes, and paintings.

1 Minute Summary

Spatially Guided Style Transfer

The key idea of DST is to find a spatial deformation of the content image that brings it into spatial alignment with the style image. This deformation is guided by a set of matching keypoints, chosen to maximize the feature similarity between paired keypoints of the two images. After roughly aligning the paired keypoints with a rigid rotation and scaling, a simple L2 loss encourages warping our output image in such a way that the keypoints become spatially aligned. This deformation loss is regularized with a total variation penalty to reduce artifacts due to drastic deformations, and combined with the more traditional style and content loss terms. DST's joint, regularized objective simultaneously encourages preserving content, minimizing the style loss, and obtaining the desired deformation, weighing these goals against each other. This objective can be solved using standard iterative techniques.

Results

Each set of three images contains (from left to right) a content input, a style input, and our method's output.
 

Face Paintings

 

Animals and Objects

 

Face Caricatures

 

One Content to Multiple Styles

 

Multiple Contents to One Style

Related Work

Style Transfer by Relaxed Optimal Transport and Self-Similarity. Nicholas Kolkin, Jason Salavon and Gregory Shakhnarovich. CVPR 2019.
Comment: A one-shot, optimization-based style transfer method with which we demonstrate our DST framework.
 
Image Style Transfer Using Convolutional Neural Networks. Leon A. Gatys, Alexander S. Ecker and Matthias Bethge. CVPR 2016.
Comment: A one-shot, optimization-based style transfer method with which we demonstrate our DST framework.
 
Neural Best-Buddies: Sparse Cross-Domain Correspondence. Kfir Aberman, Jing Liao, Mingyi Shi, Dani Lischinski, Baoquan Chen and Daniel Cohen-Or. SIGGRAPH 2018.
Comment: A generic point matching method that we use to find correspondences between images and guide the spatial deformation.
 
The Face of Art: Landmark Detection and Geometric Style in Portraits. Jordan Yaniv, Yael Newman and Ariel Shamir. SIGGRAPH 2019
Comment: A method for detecting facial landmarks and transferring geometric style of artistic portraits. We compare our DST results with results from this method.
 
WarpGAN: Automatic Caricature Generation. Yichun Shi, Debayan Deb and Anil K. Jain. CVPR 2019.
Comment: A method for generating caricatures with geometric deformations given an input face photo. We compare our DST results with results from this method.
 
Geometric Style Transfer. Xiao-Chang Liu, Xuan-Yi Li, Ming-Ming Cheng and Peter Hall. arXiv 2020
Comment: A geometry-aware style transfer method that represents geometry with features extracted from pre-trained convolutional neural networks.

Acknowledgements

This work is supported by the DARPA GARD award HR00112020003 and the University of Chicago CDAC Discovery Grant. We would also like to thank Haochen Wang, Davis Gilton, Steven Basart, and members of the Perception and Learning Systems (PALS) at TTIC for their helpful comments. Finally, we thank authors of the artworks we used to develop and evaluate our method. In the next section, we provide credits for the images used in the paper and this site. The webpage template was adapted from this project page.

Image credits

Figure 1 (Paper, Website)

  1. Schoeller, Martin: ‘Meryl Streep’. (2018), http://www.artnet.com/artists/martinschoeller/meryl-streep-from-the-series-portraits-a-iuZXKIjNqXY2M9v6ih-p1A2
  2. Matisse, Henri: ‘Girl with a Black Cat’. (1910), https://www.wikiart.org/en/henrimatisse/girl-with-a-black-cat
  3. Spellman, Jim: ‘Jennifer Lawrence Photo’. (2015), https://www.gettyimages.com/detail/newsphoto/actress-jennifer-lawrence-attends-the-the-hunger-games-news-photo/497773150
  4. Snow, Rob: ‘Gillian Anderson Caricature’. https://www.juniqe.com/gilliananderson-premium-poster-portrait-990829.html
  5. Bernard, Jean: ‘Standing Bull Painting’. (c. 1775-1883), https://www.rawpixel.com/image/481523/free-illustration-image-bull-paintinganimal
  6. Picasso, Pablo: ‘Bull’. (1945), https://www.artyfactory.com/art appreciation/animals in art/pablo picasso.htm
  7. Turk, Greg. Levoy, Marc: ‘Stanford Bunny Render’. (1994), https://en.wikipedia.org/wiki/Stanford bunny
  8. Walter, Sonia: ‘Bunny Painting’. https://www.etsy.com/sg-en/listing/533084025/bunnypainting-rabbit-painting-childrens
  9. Original artist unknown: ‘Photo of Man with Shaven Head’. http://www.faculty.idc.ac.il/arik/site/foa/faceof-art.asp
  10. Leger, Fernand: ‘La Botte de Navets’. (1951), https://www.flickr.com/photos/32215553@N02/45868933822

Figure 2 (Paper)

  1. Original artist unknown: ‘Ray Kurzweil Photo’, https://www.raadfest.com/raykurzweil
  2. Meckenem, Israhel van: ‘Double Portrait of Israhel van Meckenem and His Wife Ida’. (1490), https://www.nga.gov/collection/art-object-page.3311.html
  3. Original artist unknown: ‘Photo of Man in Suit’, Unknown Source
  4. Van Gogh, Vincent: ‘Self-Portrait’. (1889), https://www.nga.gov/collection/artobject-page.106382.html
  5. Original artist unknown: ‘Closeup Photo of Blond Woman’, https://cgm.technion.ac.il/ComputerGraphics-Multimedia/Software/Contextual/resources/comparison/index.html
  6. Kisling, Moise: ‘Unidentified Portrait’, https://www.wikiart.org/en/moisekisling/not-identified-9/
  7. Original artist unknown: ‘Pig Photo’, https://foodsafetynewsfullservice.marlersites.com/files/2016/04/pig-big-ears406.jpg
  8. Original artist unknown: ‘Poppy the Pig’, https://social-artworking.com/poppythe-pig

Figure 3 (Paper)

  1. Original artist unknown: ‘Truck Photo’. https://marvel-b1-cdn.bc0a.com/f00000000067090/www.constructionequipment.com/sites/ce/files/styles/large/public/X11GM SH051%5B1%5D.jpg
  2. Galaich, George: ‘Distorted Truck’. https://pixels.com/featured/distortedtruck-george-galaich.html

Figure 4 (Paper)

  1. Original artist unknown: ‘Closeup Photo of Woman with Dark Hair’. https://cgm.technion.ac.il/Computer-Graphics-Multimedia/Software/Contextual/resources/comparison/index.html
  2. Picasso, Pablo: ‘Nude Woman With Necklace

Figure 5 (Paper)

  1. Original artist unknown: ‘Ray Kurzweil Photo’, https://www.raadfest.com/raykurzweil
  2. Van Gogh, Vincent: ‘The Zouave’. (June 1888), https://www.vangoghmuseum.nl/en/collection/s0067V1962
  3. Original artist unknown: ‘Husky Puppy Photo’, Unknown source
  4. Hannson, Stefan: ‘Husky Caricature’. (2017), https://www.artstation.com/artwork/ry8dm
  5. Original artist unknown: ‘Photo of Woman with Wavy Hair’. http://www.faculty.idc.ac.il/arik/site/foa/face-of-art.asp
  6. Modigliani, Amedeo: ‘Girl with a Polka-Dot Blouse’. (1919), https://collection.barnesfoundation.org/objects/5538/Girl-with-a-Polka-DotBlouse-(Jeune-fille-au-corsage-a-pois)/
  7. Original artist unknown: ‘Running Shoes Photo’. http://www.stockphotos.ro/runningshoes-gym-floor-after-workout-image12693027.html
  8. Van Gogh, Vincent: ‘Shoes’. (1888), https://www.metmuseum.org/art/collection/search/436533

Figure 6 (Paper)

  1. Original artist unknown: ‘Peasant Woman Carrying A Pack On Her Back’. https://www.davidsongalleries.com/sections/modern/french-prints/additionalavailable-works/peasant-woman-carrying-a-pack-on-her-back/
  2. Rivera, Diego: ‘Peasant Woman’. (1946), https://i2.wp.com/blog.dma.org/wpcontent/uploads/2017/02/1985 16 w.jpg
  3. Original artist unknown: ‘Truck Photo’. https://marvel-b1-cdn.bc0a.com/f00000000067090/www.constructionequipment.com/sites/ce/files/styles/large/public/X11GM SH051%5B1%5D.jpg
  4. Galaich, George: ‘Distorted Truck’. https://pixels.com/featured/distortedtruck-george-galaich.html
  5. Storm, Derek: ‘Ezra Miller Photo’. (2016), https://www.alamy.com/stockphoto-new-york-ny-usa-10th-nov-2016-ezra-miller-at-arrivals-for-fantastic-125688242.html
  6. Snow, Rob: ‘Benedict Cumberbatch Caricature’. https://cargocollective.com/robsnow creative/Caricatures/filter/human/CelebritySunday-Benedict-Cumberbatch
  7. Original artist unknown: ‘Horse Photo’. https://twitter.com/Bbelisle7/status/1215411108921724929/photo/1
  8. Sierrra, Dania: ‘Grey Mane’. https://www.saatchiart.com/art/Painting-GreyMane/561637/6472363/view

Figure 7 (Paper)

  1. Original artist unknown: ‘Photo of Woman with Wavy Hair’. http://www.faculty.idc.ac.il/arik/site/foa/face-of-art.asp
  2. Leger, Fernand: ‘La Femme et la fleur’. (1954), https://www.artsy.net/artwork/fernandleger-la-femme-et-la-fleur
  3. Original artist unknown: ‘Photo of Woman with Pulled Back Hair’. http://www.faculty.idc.ac.il/arik/site/foa/face-of-art.asp
  4. Modigliani, Amedeo: ‘Gypsy Woman with Baby’. (1919), https://www.nga.gov/collection/artobject-page.46649.html
  5. Original artist unknown: ‘Photo of Man with Short Hair’. http://www.faculty.idc.ac.il/arik/site/foa/face-of-art.asp
  6. Leger, Fernand: ‘Male Portrait, http://www.faculty.idc.ac.il/arik/site/foa/faceof-art.asp
  7. Original artist unknown: ‘Photo of Hilary Swank’, https://cs.nju.edu.cn/rl/WebCaricature.htm
  8. Original artist unknown: ‘Caricature of Hilary Swank’, https://cs.nju.edu.cn/rl/WebCaricature.htm
  9. Original artist unknown: ‘Photo of Rowan Atkinson’, https://cs.nju.edu.cn/rl/WebCaricature.htm
  10. Original artist unknown: ‘Photo of Rowan Atkinson’, https://cs.nju.edu.cn/rl/WebCaricature.htm
  11. Original artist unknown: ‘Photo of Cate Blanchett’, https://cs.nju.edu.cn/rl/WebCaricature.htm
  12. Original artist unknown: ‘Caricature of Cate Blanchett’, https://cs.nju.edu.cn/rl/WebCaricature.htm

Figure 9 (Paper)

  1. Original artist unknown: ‘Photo of Barred Owls’. https://1.bp.blogspot.com/-1weEwEzz2wk/Xb64zZC-2RI/AAAAAAADI-M/pTeSYN7v KAo2YK3FsPUt6SVe76A5pYaQCLcBGAsYHQ/s640/88l.jpg
  2. Yun, Sungsil: ‘Korean Traditional Watercolor Owl’. https://www.saatchiart.com/art/PaintingKorean-traditional-watercolor-owl/1143968/4610489/view
  3. Original artist unknown: ‘Surprised Boy Photo’. http://weknowyourdreams.com/single/surprise/surprise-01
  4. Original artist unknown: ‘Maya the Bee Render’. https://www.independent.co.uk/artsentertainment/tv/news/netflix-penis-cartoon-penis-episode-maya-the-bee-a7958551.html
  5. Original artist unknown: ‘Husky Photo’. Unknown Source
  6. Original artist unknown: ‘Dog Caricature’. Unknown Source

Supplementary Figure 1 (Paper)

  1. Original artist unknown: ‘Black Horse Photo’. Unknown Source
  2. Marc, Franz: ‘Blue Horse’. (1911), http://www.franzmarc.com/blue-horse/

Supplementary Figure 2 (Paper)

  1. Original artist unknown: ‘Photo of Woman with Wavy Hair’. http://www.faculty.idc.ac.il/arik/site/foa/face-of-art.asp
  2. Leger, Fernand: ‘La Femme et la fleur’. (1954), https://www.artsy.net/artwork/fernandleger-la-femme-et-la-fleur

Face Paintings (Website)

  1. Original artist unknown: ‘Ray Kurzweil Photo’, https://www.raadfest.com/raykurzweil
  2. Van Gogh, Vincent: ‘The Zouave’. (June 1888), https://www.vangoghmuseum.nl/en/collection/s0067V1962
  3. Original artist unknown: ‘Natalie Portman Photo’. https://www.wallpaperflare.com/actress-faces-natalie-portman-wallpaper-uygf
  4. Chagall, Marc: 'For Vava.' (1955), https://www.wikiart.org/en/marc-chagall/for-vava-1955
  5. Original artist unknown: 'Jason Momoa Photo' https://www.facebook.com/1JasonMomoaFans/photos/a.1069402176497396/2501478103289789/?type=3
  6. Dürer, Albrecht: 'Self-Portrait at Twenty-Eight': https://www.albrechtdurer.org/self-portrait/
  7. Original artist unknown: 'Joanna Kustra Photo': https://cherrydeck.com/joannakustra
  8. Matisse, Henri: 'The Green Stripe': https://www.henrimatisse.org/green-stripe.jsp
  9. Harrison, Frazer: 'Sandra Oh Photo': https://www.imdb.com/name/nm0644897/mediaviewer/rm2865150720/?ref_=nm_ov_ph
  10. Bratby, John Randall: 'Kathy Wilkes, Fellow and Tutor in Philosophy' https://artuk.org/discover/artworks/kathy-wilkes-fellow-and-tutor-in-philosophy-223246
  11. Original artist unknown: 'Keira Knightley Photo': http://wallpaperswide.com/keira_knightley_portrait_2-wallpapers.html
  12. Matisse, Henri: 'Purple Robe and Anemones' https://www.matissepaintings.org/purple-robe-and-anemones/
  13. Original artist unknown: 'Jeon Do-Yeon Photo' https://www.themoviedb.org/person/20737-jeon-do-yeon
  14. Kisling, Moise: ‘Unidentified Portrait’, https://www.wikiart.org/en/moisekisling/not-identified-9/
  15. Schoeller, Martin: ‘Meryl Streep’. (2018), http://www.artnet.com/artists/martinschoeller/meryl-streep-from-the-series-portraits-a-iuZXKIjNqXY2M9v6ih-p1A2
  16. Bratby, John Randall: 'Sir Arthur Bryan' https://artuk.org/discover/artworks/sir-arthur-bryan-20314

Animals and Objects (Website)

  1. Original artist unknown: ‘Running Shoes Photo’. http://www.stockphotos.ro/runningshoes-gym-floor-after-workout-image12693027.html
  2. Van Gogh, Vincent: ‘Shoes’. (1888), https://www.metmuseum.org/art/collection/search/436533
  3. Original artist unknown: ‘Truck Photo’. https://marvel-b1-cdn.bc0a.com/f00000000067090/www.constructionequipment.com /sites/ce/files/styles/large/public/X11GM SH051%5B1%5D.jpg
  4. Galaich, George: ‘Distorted Truck’. https://pixels.com/featured/distortedtruck-george-galaich.html
  5. Original artist unknown: ‘Peasant Woman Carrying A Pack On Her Back’. https://www.davidsongalleries.com/sections/modern/french-prints/additionalavailable-works/peasant-woman-carrying-a-pack-on-her-back/
  6. Rivera, Diego: ‘Peasant Woman’. (1946), https://i2.wp.com/blog.dma.org/wpcontent/uploads/2017/02/1985 16 w.jpg
  7. Original artist unknown: ‘Horse Photo’. https://twitter.com/Bbelisle7/status/1215411108921724929/photo/1
  8. Sierrra, Dania: ‘Grey Mane’. https://www.saatchiart.com/art/Painting-GreyMane/561637/6472363/view
  9. Original artist unknown: ‘Pig Photo’, https://foodsafetynewsfullservice.marlersites.com/files/2016/04/pig-big-ears406.jpg
  10. Original artist unknown: ‘Poppy the Pig’, https://social-artworking.com/poppythe-pig
  11. Turk, Greg. Levoy, Marc: ‘Stanford Bunny Render’. (1994), https://en.wikipedia.org/wiki/Stanford_bunny
  12. Walter, Sonia: ‘Bunny Painting’. https://www.etsy.com/sg-en/listing/533084025/bunnypainting-rabbit-painting-childrens
  13. Original artist unknown: 'Flower Vase Photo', https://www.baamboozle.com/game/155076
  14. Picasso, Pablo: 'Vase of Flowers'. (1943): https://www.wikiart.org/en/pablo-picasso/vase-with-flowers-1943
  15. Original artist unknown: 'Pineapple Photo,' https://muscatfoodmarket.com/pineapple-philipines-1-kg.html
  16. Drucker, Bobbi: 'Pineapple': https://fineartamerica.com/featured/pineapple-bobbi-drucker.html

Face Caricatures (Website)

  1. Original artist unknown: ‘Photo of Rowan Atkinson’, https://cs.nju.edu.cn/rl/WebCaricature.htm
  2. Original artist unknown: ‘Photo of Rowan Atkinson’, https://cs.nju.edu.cn/rl/WebCaricature.htm
  3. Original artist unknown: ‘Photo of Hilary Swank’, https://cs.nju.edu.cn/rl/WebCaricature.htm
  4. Original artist unknown: ‘Caricature of Hilary Swank’, https://cs.nju.edu.cn/rl/WebCaricature.htm
  5. Original artist unknown: 'Photo of Julianne Moore', https://www.businessinsider.com/julianne-moore-the-new-face-of-loreal-2012-11
  6. Snow, Rob: ‘Julianne Moore Caricature’. https://rob-snow.pixels.com/featured/celebrity-sunday-julianne-moore-rob-snow.html
  7. Original artist unknown: ‘Photo of Cate Blanchett’, https://cs.nju.edu.cn/rl/WebCaricature.htm
  8. Original artist unknown: ‘Caricature of Cate Blanchett’, https://cs.nju.edu.cn/rl/WebCaricature.htm
  9. Storm, Derek: ‘Ezra Miller Photo’. (2016), https://www.alamy.com/stockphoto-new-york-ny-usa-10th-nov-2016-ezra-miller-at-arrivals-for-fantastic-125688242.html
  10. Snow, Rob: ‘Benedict Cumberbatch Caricature’. https://rob-snow.pixels.com/featured/celebrity-sunday-benedict-cumberbatch-rob-snow.html
  11. Original artist unknown: 'Photo of Leonardo DiCaprio', https://www.atlantamagazine.com/news-culture-articles/leonardo-dicaprio-and-jonah-hill-set-to-star-in-film-about-richard-jewell/
  12. Snow, Rob: ‘Jack Nicholson Caricature’. https://rob-snow.pixels.com/featured/celebrity-sunday-jack-nicholson-rob-snow.html

One Content to Multiple Styles; Multiple Contents to One Style (Website)

Contact

Sunnie S. Y. Kim (sunnie@ttic.edu)