In the Style of Alex Katz
February 19, 2023
Summary
Moving on from the abstract expressionism and Joan Mitchell, I am moving to Alex Katz’s figurative paintings.
Similar to the prior work, I manually created a small dataset of Alex Katz’s work and descriptions (image-text pairs). Then I fine-tuned Stable Diffusion 2 for 100 steps on the dataset. The model is available here. The model is not very stable so would probably benefit from more training data or more training steps.
Change in Style
Unsurprisingly, the model picked up painting style from the fine-tuning dataset in Alex Katz’s style.
| Prompt | StabilityAI Stable Diffusion 2 | Alex Katz Portraiter |
| A close up image of woman in big white hat. | 
|
|
| A very close up image of a woman in big white hat. No bangs. | 
|
|
| A very close up image of woman in big white hat, looking towards the right. Blonde. Dark blue background. | 
|
|
More Examples
A woman with black straight hair and a thin golden necklace, red lips.
A man in suit.
Two women side by side. The woman on the left has straight blonde hair in black blouse. The woman on the right has curly blond hair, wearing a white sweater and a thin gold necklace.
A woman with blue eyes. In black shirt.
A woman with blue eyes. In black shirt.
It generalizes to non-figurative paintings well too.
A painting of trees in the style of Alex Katz.
Application
One can imagine an apple where the model is converted to Core ML to run on Apple Silicon. Then an app can be created for people to craft their own portrait with Alex Katz’s style with natural language. I built a simple Hugging Face space for the idea. But it takes way to long to run with the free-tier CPU. Feel free to fork and give it more resources.
Ethical Considerations
This work continues questions asked in AEM on originality of creation. Please use responsibly.
In addition, there is excellent research from Ben Y. Zhao’s group: GLAZE: Protecting Artists from Style Mimicry by Text-to-Image Models. It develops a cloaking technique to prevent learning the style from fine-tuning (as demonstrated in this work). And has many excellent pointers on the issues that such fine-tuning create in the artistic community.