Image forgery classification and localization through vision transformers

Due to the easy availability of software over the Internet, any naive user can tamper the images for entertainment purposes or to defame a personality by circulating over social media networks. The practice of image tampering is a serious issue and can attract legal action if proven guilty. Forensic researchers employ various methods to detect and localize image forgeries. In this research, we use a Vision transformer (ViT) as a method for binary classification of images distinguishing forged and unforged images. Further, we use a pre-trained Segment Anything Model(SAM) which is fine-tuned with custom data to adaptively recognize patterns indicating forged regions within the images. SAM can localize these forged areas and is leveraged to create templates by extracting the identified regions. The proposed method is rigorously tested across various datasets, including CASIA v1.0, CASIA v2.0, MICC-F2000, MICC-F600, and Columbia. Through comprehensive experimentation, our approach showcases considerable promise yielding accuracy in image forgery classification and localization. Our model’s robustness and adaptability make it an attractive tool for forensic analysis in diverse scenarios, contributing to the advancement of multimedia forensics security research.