Discriminative Class Tokens for Text-to-Image Diffusion Models
Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review
Standard
Discriminative Class Tokens for Text-to-Image Diffusion Models. / Schwartz, Idan; Snæbjarnarson, Vésteinn; Chefer, Hila; Belongie, Serge; Wolf, Lior; Benaim, Sagie.
Proceedings - 2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023. IEEE, 2023. p. 22668-22678.Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - GEN
T1 - Discriminative Class Tokens for Text-to-Image Diffusion Models
AU - Schwartz, Idan
AU - Snæbjarnarson, Vésteinn
AU - Chefer, Hila
AU - Belongie, Serge
AU - Wolf, Lior
AU - Benaim, Sagie
N1 - Funding Information: This project was supported by a grant from the Tel Aviv University Center for AI and Data Science (TAD). VS, SB, and SB are supported by the Pioneer Centre for AI, DNRF grant number P1. Publisher Copyright: © 2023 IEEE.
PY - 2023
Y1 - 2023
N2 - Recent advances in text-to-image diffusion models have enabled the generation of diverse and high-quality images. While impressive, the images often fall short of depicting subtle details and are susceptible to errors due to ambiguity in the input text. One way of alleviating these issues is to train diffusion models on class-labeled datasets. This approach has two disadvantages: (i) supervised datasets are generally small compared to large-scale scraped text-image datasets on which text-to-image models are trained, affecting the quality and diversity of the generated images, or (ii) the input is a hard-coded label, as opposed to free-form text, limiting the control over the generated images.In this work, we propose a non-invasive fine-tuning technique that capitalizes on the expressive potential of freeform text while achieving high accuracy through discriminative signals from a pretrained classifier. This is done by iteratively modifying the embedding of an added input token of a text-to-image diffusion model, by steering generated images toward a given target class according to a classifier. Our method is fast compared to prior fine-tuning methods and does not require a collection of in-class images or retraining of a noise-tolerant classifier. We evaluate our method extensively, showing that the generated images are: (i) more accurate and of higher quality than standard diffusion models, (ii) can be used to augment training data in a low-resource setting, and (iii) reveal information about the data used to train the guiding classifier. The code is available at https://github.com/idansc/discriminative-class-tokens.
AB - Recent advances in text-to-image diffusion models have enabled the generation of diverse and high-quality images. While impressive, the images often fall short of depicting subtle details and are susceptible to errors due to ambiguity in the input text. One way of alleviating these issues is to train diffusion models on class-labeled datasets. This approach has two disadvantages: (i) supervised datasets are generally small compared to large-scale scraped text-image datasets on which text-to-image models are trained, affecting the quality and diversity of the generated images, or (ii) the input is a hard-coded label, as opposed to free-form text, limiting the control over the generated images.In this work, we propose a non-invasive fine-tuning technique that capitalizes on the expressive potential of freeform text while achieving high accuracy through discriminative signals from a pretrained classifier. This is done by iteratively modifying the embedding of an added input token of a text-to-image diffusion model, by steering generated images toward a given target class according to a classifier. Our method is fast compared to prior fine-tuning methods and does not require a collection of in-class images or retraining of a noise-tolerant classifier. We evaluate our method extensively, showing that the generated images are: (i) more accurate and of higher quality than standard diffusion models, (ii) can be used to augment training data in a low-resource setting, and (iii) reveal information about the data used to train the guiding classifier. The code is available at https://github.com/idansc/discriminative-class-tokens.
U2 - 10.1109/ICCV51070.2023.02077
DO - 10.1109/ICCV51070.2023.02077
M3 - Article in proceedings
AN - SCOPUS:85185873615
SP - 22668
EP - 22678
BT - Proceedings - 2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023
PB - IEEE
T2 - 2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023
Y2 - 2 October 2023 through 6 October 2023
ER -
ID: 389306742