Generating Compositional Color Representations from Text
release_iqjwbngzs5hdpbry65tqntidz4
by
Paridhi Maheshwari, Nihal Jain, Praneetha Vaddamanu, Dhananjay Raut, Shraiysh Vaishay, Vishwa Vinay
2021
Abstract
We consider the cross-modal task of producing color representations for text
phrases. Motivated by the fact that a significant fraction of user queries on
an image search engine follow an (attribute, object) structure, we propose a
generative adversarial network that generates color profiles for such bigrams.
We design our pipeline to learn composition - the ability to combine seen
attributes and objects to unseen pairs. We propose a novel dataset curation
pipeline from existing public sources. We describe how a set of phrases of
interest can be compiled using a graph propagation technique, and then mapped
to images. While this dataset is specialized for our investigations on color,
the method can be extended to other visual dimensions where composition is of
interest. We provide detailed ablation studies that test the behavior of our
GAN architecture with loss functions from the contrastive learning literature.
We show that the generative model achieves lower Frechet Inception Distance
than discriminative ones, and therefore predicts color profiles that better
match those from real images. Finally, we demonstrate improved performance in
image retrieval and classification, indicating the crucial role that color
plays in these downstream tasks.
In text/plain
format
Archived Files and Locations
application/pdf 10.5 MB
file_v365n7dwbvgqxm6sqa7eo3zcka
|
arxiv.org (repository) web.archive.org (webarchive) |
2109.10477v1
access all versions, variants, and formats of this works (eg, pre-prints)