Information-Theoretic Segmentation by Inpainting Error Maximization [paper]

  • Pedro Savarese, Sunnie S. Y. Kim, Michael Maire, Greg Shakhnarovich, David McAllester. Accepted to CVPR 2021.
  • Abstract: We study image segmentation from an information-theoretic perspective, proposing a novel adversarial method that performs unsupervised segmentation by partitioning images into maximally independent sets. More specifically, we group image pixels into foreground and background, with the goal of minimizing predictability of one set from the other. An easily computed loss drives a greedy search process to maximize inpainting error over these partitions. Our method does not involve training deep networks, is computationally cheap, class-agnostic, and even applicable in isolation to a single unlabeled image. Experiments demonstrate that it achieves a new state-of-the-art in unsupervised segmentation quality, while being substantially faster and more general than competing approaches.


Fair Attribute Classification through Latent Space De-biasing [project page] [paper] [code]

  • Vikram V. Ramaswamy, Sunnie S. Y. Kim, Olga Russakovsky. Accepted to CVPR 2021.
  • Abstract: Fairness in visual recognition is becoming a prominent and critical topic of discussion as recognition systems are deployed at scale in the real world. Models trained from data in which target labels are correlated with protected attributes (e.g., gender, race) are known to learn and exploit those correlations. In this work, we introduce a method for training accurate target classifiers while mitigating biases that stem from these correlations. We use GANs to generate realistic-looking images, and perturb these images in the underlying latent space to generate training data that is balanced for each protected attribute. We augment the original dataset with this perturbed generated data, and empirically demonstrate that target classifiers trained on the augmented dataset exhibit a number of both quantitative and qualitative benefits. We conduct a thorough evaluation across multiple target labels and protected attributes in the CelebA dataset, and provide an in-depth analysis and comparison to existing literature in the space.
Training a visual classifier for an attribute (e.g., wearing hat) can be complicated by correlations in the training data. For example, the presence of hats can be correlated with the presence of glasses. We propose a dataset augmentation strategy using Generative Adversarial Networks (GANs) that successfully removes this correlation by adding or removing glasses from existing images, creating a balanced dataset.


Deformable Style Transfer [project page] [paper] [code] [demo] [1min video] [10min talk]

  • Sunnie S. Y. Kim, Nicholas Kolkin, Jason Salavon, Gregory Shakhnarovich. ECCV 2020.

  • Abstract: Geometry and shape are fundamental aspects of visual style. Existing style transfer methods focus on texture-like components of style, ignoring geometry. We propose deformable style transfer (DST), an optimization-based approach that integrates texture and geometry style transfer. Our method is the first to allow geometry-aware stylization not restricted to any domain and not requiring training sets of matching style/content pairs. We demonstrate our method on a diverse set of content and style images including portraits, animals, objects, scenes, and paintings.

Each set of four contains (from left to right) a content input, a style input, a standard style transfer output, and our method's output.


2018 Environmental Performance Index [website] [report & data] [discussion at WEF18] [news 1] [news 2]

  • A biennial report produced by researchers at Yale and Columbia, in collaboration the World Economic Forum. I built the data pipeline and led the data analysis work, under the guidance of Prof. Jay Emerson and Dr. Zach Wendling.

  • Abstract: Careful measurement of environmental trends and progress provides a foundation for effective policymaking. The 2018 Environmental Performance Index (EPI) ranks 180 countries on 24 performance indicators across ten issue categories covering environmental health and ecosystem vitality. These metrics provide a gauge at a national scale of how close countries are to established environmental policy goals. The EPI thus offers a scorecard that highlights leaders and laggards in environmental performance, gives insight on best practices, and provides guidance for countries that aspire to be leaders in sustainability.