$1.$ Finetuning with LoRA adapters on custom dataset $.$

LoRA adapters can be applied for finetuning large diffusion models, trained on diverse mix of data to improve image generation quality or text$-$image alignment on a specific domain.

Your task is to take pretrained Stable Diffusion model $($or any other model at your preference$)$ and finetune in on one of the two tasks listed below:

lambdalabs$/$naruto$-$blip$-$captions

Your notebook should contain the entire dataset preprocessing and training pipeline.

You may use any functionality from the diffusers, datasets, transformers, peft and accelerate libraries. You may adopt pieces of code, but not the entire training$\_$script.

Use of the models hosted on the hub and trained on this dataset is strictly prohibited!

Data loading and visualization

Load the dataset and visualize some samples from it$.$ How many samples are there? Split the dataset into train$/$test set in proportion $75\%/25\%$ at random. Fix all seeds to $42($torch$,$ numpy, random$).$

LoRA preparation

Define LoRA config using the peft library. Which layers can you apply LoRA to$?$

Training

Train the model and save the adapter of the final model. Since there is no reasonable quantative metric for the quality of images, visualize $20$ samples generated on the prompts from the hold$-$out set.

LoRA ablation

Experiment with different ranks of LoRA adapters. How does the quality $/$ aligment to the target domain depend on the rank of adapter? Try $3-5$ options for LoRA rank that yield noticeable difference in performance. Apply LoRA to different layers. What layers are better for LoRA application?

Impact of CFG scale

For a single model and a $2-3$ prompts generate images with different classifier$-$free guidance scale $($from $1$ to some large value, like $20).$ What do you observe?