The Abstract Expression Machine

January 29, 2023

Summary

I fine-tuned Stable Diffusion 2 on a small Joan Mitchell dataset (11 image-text pairs) and created what I call the Abstract Expression Machine (AEM). There are two versions available, AEM30 (fine-tuned 30 steps) and AEM100 (fine-tuned 100 steps).

Both AEM30 and AEM100 produces abstract images better than the baseline (Stable Diffusion 2). Although personally I find AEM100 is more tuned towards Joan Mitchell’s personal style, while AEM30 seems to be able to generalize to other abstract expression artists’ styles (e.g., Franz Kline) to a decent amount. Scaling up the images also adds beautiful drawing/analog texture.

The work is also meant to further push the boundary on the discussion of human v.s. AI creativity and originality. Who is the artist? Is this original or plagiarism?

Please use the dataset and the models responsibly.

Abstraction and Emotions

I grew up in a culture where we don’t like to express emotions directly. However, people manage to express intense emotions through art forms such as poems and paintings. I presume that’s why I was so attracted to abstract expressionism when I first discovered it — Mark Rothko, Franz Kline, Joan Mitchell, etc.

If impressionists like Monet captured ambiance, abstract expressionists distilled feelings or memories into forms and colors. I liked the idea of transforming emotions and memories into visual representations, and naturally I wonder if the state-of-art image models could do it.

Increased Abstraction

I created a small dataset of 11 image-text pairs of Joan Mitchell and fine-tuned Stable Diffusion 2 on this dataset.

The first observation is the fine-tuned models have increased abstraction. As the number of fine-tuning steps increase (from 0, 30, 100), the model’s output becomes less figurative (e.g., a winter landscape) to more abstract.

The second observation is when prompted to draw in the style of certain artist, the fine-tuned model increasingly learns the style of the said artist. In this case, Joan Mitchell.

Prompt	StabilityAI Stable Diffusion 2	Abstract Expression Machine 30	Abstract Expression Machine 100
a winter landscape, with feelings of hope and loneliness	image	image	image
a winter landscape, with feelings of hope and loneliness in the style of joan mitchell	image	image	image
feelings of scared, intense anger, and vulnerability in the style of joan mitchell	image	image	image
a girl standing by the river, feeling calm in the style of joan mitchell	image	image	image
feeling wanting to be held by someone in the style of joan mitchell	image	image	image
summer garden sunset, full of melancholy and some sadness in the style of joan mitchell	image	image	image

Generalization

The second interesting finding is the fine-tuned models seem to be able to generalize, to certain degree, to other abstract expressionism painters (e.g., Mark Rothko, Franz Klein).

To my eyes, the AEM30 (fine-tuned 30 steps) seems to generalize better to other artists styles, while the AEM100 (fine-tuned 100 steps) seems to be more “heavily influenced” by Joan Mitchell’s style.

Prompt	StabilityAI Stable Diffusion 2	Abstract Expression Machine 30	Abstract Expression Machine 100
a winter landscape, with feelings of hope and loneliness in the style of mark rothko	image	image	image
a winter landscape, with feelings of hope and loneliness in the style of franz kline	image	image	image

Monet - Mitchell

In Oct 2022, I saw the Monet - Mitchell exhibit at Foundation Louis Vuitton in Paris, and liked how the curators put Monet and Mitchell’s work side by side.

In addition to similarity in colors among the two, Monet, especially late Monet begins to paint increasingly abstractly. So I tested how the models would perform on a prompt in Monet’s style. The increased abstraction pattern holds, and I also think the AEM30’s output is more similar to Monet’s style than AEM100.

Prompt	StabilityAI Stable Diffusion 2	Abstract Expression Machine 30	Abstract Expression Machine 100
a winter landscape, with feelings of hope and loneliness in the style of claude monet	image	image	image

Upscaling Adds Texture

The generated images look great but when enlarged we can notice the texture of the images are flat. For paintings, one of the most interesting thing is the texture of the paint to me. And it’s the texture and analog feeling that I feel a lot of the digital art (including my own) lacks.

Interestingly, I found when I upscaled the images with stabilityai/stable-diffusion-x4-upscaler, the images gain some beautiful texture.

Abstract Expression Machine 30	Abstract Expression Machine 100
image	image

Final Outputs

Putting everything together, here are some of my favorite creations from the AEM100 with upscaling.

Three-panel painting of spring, by AEM100

River flowing 1, by AEM100

River flowing 2, by AEM100

River flowing 3, by AEM100

Art and Originality

This work is meant to facilitate discussions on human v.s machine creativity and originality, and to show my admiration for Joan Mitchell and a few other abstract expressionism artists.

AI-generated art has been causing controversy over creativity, ownership, and originality. In particular, when AI-generated art won an art competition, artists are very upset. There is also the anti-AI movement and report of plagiarism.

In my opinion, there are merits to both sides of the debate. On one hand, we should develop tools to protect and artists opt-out of training data. But what about fine-tuning? On the other hand, it’s not the first time in history that we have heard the argument that — that’s not art (Duchamp’s readymades, photography v.s. painting, even digital art itself).

Technical Notes

I manually created a small dataset of Joan Mitchell’s work and descriptions (image-text pairs). Texts are from the Collection in the Foundation Louis Vuitton. Images are from the Joan Mitchell Foundation. And it’s also available as on Hugging Face as maximalmargin/mitchell.