Fine-tuning Pipeline for Hand Image Generation Using Diffusion Model

The hand images generated by the image generative model may suffer distortions, such as stable diffusion. To solve this issue, we introduce a hand image fine-tuning pipeline consisting of three stages: hand detection, object masking, and image inpainting. First, a hand detection model is trained to identify flawed hands using bounding boxes (Bbox). Then, these Bbox regions are masked in conjunction with Mediapipe landmarks. Finally, a ControlNet model is trained for inpainting the masked areas, and the targeted LoRA is also trained to minimize boundary fragmentation. The results indicate that our method achieves better anatomical accuracy in hand reconstruction compared to the original diffusion model. Furthermore, the introduction of the directional LoRA model further enhances the evaluation outcomes.