{"id":45,"date":"2022-06-27T04:46:47","date_gmt":"2022-06-27T04:46:47","guid":{"rendered":"https:\/\/formatswap.com\/blog\/?p=45"},"modified":"2023-05-28T22:14:10","modified_gmt":"2023-05-28T22:14:10","slug":"how-to-generate-images-from-text-prompts-with-vqgan-clip-python-and-tensorflow","status":"publish","type":"post","link":"https:\/\/formatswap.com\/blog\/machine-learning-tutorials\/how-to-generate-images-from-text-prompts-with-vqgan-clip-python-and-tensorflow\/","title":{"rendered":"How to Generate Images From Text Prompts with Python and TensorFlow"},"content":{"rendered":"\n
\n
\n
\n
\"example
Example #1<\/figcaption><\/figure>\n\n\n\n
\"example
Example #2<\/figcaption><\/figure>\n\n\n\n
\"example
Example #3<\/figcaption><\/figure>\n\n\n\n
\"example
Example #4<\/figcaption><\/figure>\n\n\n\n
\"example
Example #5<\/figcaption><\/figure>\n\n\n\n
\"a
Example #6<\/figcaption><\/figure>\n<\/figure>\n<\/div>\n<\/div>\n\n\n\n

In this tutorial you will learn how to generate images from text prompts using Python, VQGAN, and neural networks. You can create some very interesting machine learning generated artwork using this software. The possibility are limitless when it comes to the different types of images you can generate. This article will be using Ubuntu 22.04 for the operating system. However any Ubuntu based version of Linux should work. Keep in mind you will want to make sure to have a graphics card with at least 6GB of VRAM. If not you will have to generate lower resolution images. If creating machine learning generated artwork sounds interesting to you continue reading this tutorial below.<\/p>\n\n\n\n

Step 1: Install Required Dependencies<\/h2>\n\n\n\n

The first thing to do is install Anaconda. You can find the latest version and instructions HERE<\/a>. This tutorial has been tested on Ubuntu 16.04, Ubuntu 18.04, Ubuntu 20.04, and Ubuntu 22.04.<\/p>\n\n\n\n

Step 2: Create the Conda Environment<\/h2>\n\n\n\n
conda create --name vqgan python=3.9\nconda activate vqgan<\/code><\/pre>\n\n\n\n
Next you will set up the Conda environment. This is where you will run VQGAN from.<\/p>\n\n\n\n
Step 3: Install Pytorch in the new environment:<\/h2>\n\n\n\n
Note: This installs the CUDA version of Pytorch for Nvidia graphics cards. If you are using an AMD graphics card, read our AMD section at the bottom of the article.<\/p>\n\n\n\n
pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https:\/\/download.pytorch.org\/whl\/torch_stable.html<\/code><\/pre>\n\n\n\n
Step 4: Install other required Python packages:<\/h2>\n\n\n\n
pip install ftfy regex tqdm omegaconf pytorch-lightning IPython kornia imageio imageio-ffmpeg einops torch_optimizer<\/code><\/pre>\n\n\n\n
Step 5: Clone the required VQGAN repositories: <\/h2>\n\n\n\n
git clone 'https:\/\/github.com\/nerdyrodent\/VQGAN-CLIP'\ncd VQGAN-CLIP\ngit clone 'https:\/\/github.com\/openai\/CLIP'\ngit clone 'https:\/\/github.com\/CompVis\/taming-transformers'\npip install taming-transformers && pip install CLIP\npip install setuptools==59.5.0<\/code><\/pre>\n\n\n\n
Now you need to clone the git repository’s. After cloning install taming-transformers and CLIP using pip. Finally install setuptools==59.5.0. This is required as the latest version of setuptools will not work with VQGAN.<\/p>\n\n\n\n
Step 6: Download at least 1 VQGAN pretrained model<\/h2>\n\n\n\n
mkdir checkpoints\n\ncurl -L -o checkpoints\/vqgan_imagenet_f16_16384.yaml -C - 'https:\/\/heibox.uni-heidelberg.de\/d\/a7530b09fed84f80a887\/files\/?p=%2Fconfigs%2Fmodel.yaml&dl=1' #ImageNet 16384\ncurl -L -o checkpoints\/vqgan_imagenet_f16_16384.ckpt -C - 'https:\/\/heibox.uni-heidelberg.de\/d\/a7530b09fed84f80a887\/files\/?p=%2Fckpts%2Flast.ckpt&dl=1' #ImageNet 16384<\/code><\/pre>\n\n\n\n
Choose at least one of the above pretrained models, and download it using the curl command. I recommend downloading both of the models so that you can try each to see which works best. <\/p>\n\n\n\n
Pretrained Model Information<\/h3>\n\n\n\n
Visit https:\/\/github.com\/CompVis\/taming-transformers#overview-of-pretrained-models<\/a> to learn more about VQGAN pre-trained models, including download links. The model .yaml and .ckpt files need to be the checkpoints<\/code> directory.<\/p>\n\n\n\n
Generating VQGAN image from basic text prompt<\/h2>\n\n\n\n
You have now finished installing VQGAN. You are ready to begin generating images. To generate an image from text, specify your text prompt as shown in the example below.<\/p>\n\n\n\n
python generate.py -p \"A illustration of a pineapple in a fruit bowl\"<\/code><\/pre>\n\n\n
\n
\"pineapple<\/figure>\n<\/div>\n\n\n
Generating VQGAN image from multiple prompts <\/h2>\n\n\n\n
You are also able to generate images from split text prompts. In the below example you can see I am using four different descriptors to generate the image.<\/p>\n\n\n\n
python generate.py -p \"A painting of a pineapple in a fruit bowl | psychedelic | surreal:0.5 | weird:0.25\"<\/code><\/pre>\n\n\n
\n
\"A<\/figure>\n<\/div>\n\n\n
Additionally you can use an input image as one of your split prompt inputs. This will use the supplied image as a sample during the generation process.<\/p>\n\n\n\n
python generate.py -p \"A picture of a bathroom with a portrait of Van Gogh\" -ip \"samples\/VanGogh.jpg | samples\/Bedroom.png\"<\/code><\/pre>\n\n\n
\n
\"A<\/figure>\n<\/div>\n\n\n
Generating VQGAN “story mode” images<\/h2>\n\n\n\n
Story mode allows you to create a story from multiple text prompts using the carrot symbol. This will generate you a .mp4 video file. For example:<\/p>\n\n\n\n
python generate.py -p \"A painting of a apple|photo:-1 ^ a painting of a banana ^ a painting of a grape ^ a painting of a watermelon ^ a photograph of strawberry\" -cpe 1500 -zvid -i 6000 -zse 10 -vl 20 -zsc 1.005 -opt Adagrad -lr 0.15 -se 6000<\/code><\/pre>\n\n\n
\n
\"A<\/figure>\n<\/div>\n\n\n
VQGAN Feedback loop animation’s<\/h2>\n\n\n\n
You are able to generate multiple images into a video. The script makes slight changes to each image creating a warping effect. The “150” at the end of the line is the number of frames. “blackhole.png” is your output filename. This will produce a .mp4 file named “video.mp4” by default. Example below:<\/p>\n\n\n\n
.\/zoom.sh \"A painting of a green firetruck spinning through a black hole\" blackhole.png 150<\/code><\/pre>\n\n\n\n
The ImageMagick package is requred to generate mp4 animated loop videos. If you don’t have it installed, you can install it using the below command.<\/p>\n\n\n\n
sudo apt install imagemagick<\/code><\/pre>\n\n\n
\n
\"green<\/figure>\n<\/div>\n\n\n
Generating multiple random images<\/h2>\n\n\n\n
You can also supply multiple phrases to be used at random to generate multiple images. There is a random list of words in the “.\/zoom.sh” file. Edit this file if you want to change the words used in the random phrase generator. Here is an example of the random images put into a collage.<\/p>\n\n\n\n
chmod +x .\/zoom.sh\n.\/zoom.sh<\/code><\/pre>\n\n\n
\n
\"multiple
Left to right #1-#9<\/figcaption><\/figure>\n<\/div>\n\n\n