Classification of Deepfake Images Using a Novel Explanatory Hybrid Model
DOI:
https://doi.org/10.21512/commit.v17i2.8761Keywords:
Image Classification, Deepfake Images, Explanatory Artificial Intelligence, Hybrid ModelAbstract
In court, criminal investigations and identity management tools, like check-in and payment logins, face videos, and photos, are used as evidence more frequently. Although deeply falsified information may be found using deep learning classifiers, block-box decisionmaking makes forensic investigation in criminal trials more challenging. Therefore, the research suggests a three-step classification technique to classify the deceptive deepfake image content. The research examines the visual assessments of an EfficientNet and Shifted Window Transformer (SWinT) hybrid model based on Convolutional Neural Network (CNN) and Transformer architectures. The classifier generality is improved in the first stage using a different augmentation. Then, the hybrid model is developed in the second step by combining the EfficientNet and Shifted Window Transformer architectures. Next, the GradCAM approach for assessing human understanding demonstrates deepfake visual interpretation. In 14,204 images for the validation set, there are 7,096 fake photos and 7,108 real images. In contrast to focusing only on a few discrete face parts, the research shows that the entire deepfake image should be investigated. On a custom dataset of real, Generative Adversarial Networks (GAN)-generated, and human-altered web photos, the proposed method achieves an accuracy of 98.45%, a recall of 99.12%, and a loss of 0.11125. The proposed method successfully distinguishes between real and manipulated images. Moreover, the presented approach can assist investigators in clarifying the composition of the artificially produced material.
Plum Analytics
References
H. Murfi, N. Rosaline, and N. Hariadi, “Deep autoencoder-based fuzzy c-means for topic detection,” Array, vol. 13, pp. 1–9, 2022.
A. Kammoun, R. Slama, H. Tabia, T. Ouni, and M. Abid, “Generative adversarial networks for face generation: A survey,” ACM Computing Surveys, vol. 55, no. 5, pp. 1–37, 2022.
A. Sharma, V. Sharma, M. Jaiswal, H. C. Wang, D. N. K. Jayakody, C. M. W. Basnayaka, and A. Muthanna, “Recent trends in AI-based intelligent sensing,” Electronics, vol. 11, no. 10, pp. 1–39, 2022.
S. Li, V. Dutta, X. He, and T. Matsumaru, “Deep learning based one-class detection system for fake faces generated by GAN network,” Sensors, vol. 22, no. 20, pp. 1–23, 2022.
T. Boll´e, E. Casey, and M. Jacquet, “The role of evaluations in reaching decisions using automated systems supporting forensic analysis,” Forensic Science International: Digital Investigation, vol. 34, pp. 1–13, 2020.
T. T. Nguyen, Q. V. H. Nguyen, D. T. Nguyen, D. T. Nguyen, T. Huynh-The, S. Nahavandi, T. T. Nguyen, Q. V. Pham, and C. M. Nguyen, “Deep learning for deepfakes creation and detection: A survey,” Computer Vision and Image Understanding, vol. 223, 2022.
L. Li, J. Bao, T. Zhang, H. Yang, D. Chen, F. Wen, and B. Guo, “Face x-ray for more general face forgery detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Virtual, June 14–19, 2020, pp. 5001–5010.
S. A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “iCaRL: Incremental classifier and representation learning,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA: IEEE, July 21–26, 2017, pp. 5533–5542.
P. S. Sisodiya, “DeepFake detection using various deep learning techniques,” Ph.D. dissertation, Delhi Technological University, 2022.
H. Zhao, W. Zhou, D. Chen, W. Zhang, and N. Yu, “Self-supervised transformer for deepfake detection,” 2022. [Online]. Available: https://arxiv.org/abs/2203.01265
H. Guo, S. Hu, X. Wang, M. C. Chang, and S. Lyu, “Robust attentive deep neural network for detecting GAN-generated faces,” IEEE Access, vol. 10, pp. 32 574–32 583, 2022.
T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of GANs for improved quality, stability, and variation,” in ICLR 2018, Vancouver, Canada, April 30–May 3, 2018.
L. Ma, K. Huang, D. Wei, Z. Y. Ming, and H. Shen, “FDA-GAN: Flow-based dual attention GAN for human pose transfer,” IEEE Transactions on Multimedia, vol. 25, pp. 930–941, 2021.
Y. Li, X. Yang, P. Sun, H. Qi, and S. Lyu, “Celebdf: A large-scale challenging dataset for deepfake forensics,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Virtual, June 14–19, 2020, pp. 3207–3216.
P. Korshunov and S. Marcel, “Deepfakes: A new threat to face recognition? assessment and detection,” 2018. [Online]. Available: https: //arxiv.org/abs/1812.08685
A. Rossler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Nießner, “FaceForensics++: Learning to detect manipulated facial images,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea, Oct. 27–Nov.2, 2019, pp. 1–11.
B. Dolhansky, J. Bitton, B. Pflaum, J. Lu, R. Howes, M. Wang, and C. C. Ferrer, “The DeepFake Detection Challenge (DFDC) dataset,” 2020. [Online]. Available: https://arxiv.org/abs/2006.07397
L. Jiang, R. Li, W. Wu, C. Qian, and C. C. Loy, “DeeperForensics-1.0: A large-scale dataset for real-world face forgery detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Virtual, June 14–19, 2020, pp. 2889–2898.
NVlabs, “Flickr-Faces-HQ Dataset (FFHQ),” 2019. [Online]. Available: https://archive.org/details/ffhq-dataset
Computational Intelligence and Photography Lab, Yonsei University, “real-and-fake-face-detection,” 2019. [Online]. Available: https://archive.org/details/real-and-fake-face-detection
F. Marra, D. Gragnaniello, L. Verdoliva, and G. Poggi, “Do GANs leave artificial fingerprints?” in 2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR). San Jose, CA, USA: IEEE, March 28–30, 2019, pp. 506–511.
H. Mo, B. Chen, andW. Luo, “Fake faces identification via convolutional neural network,” in Proceedings of the 6th ACM Workshop on Information Hiding and Multimedia Security, Innsbruck, Austria, June 20–22, 2018, pp. 43–47.
N. Hulzebosch, S. Ibrahimi, and M. Worring, “Detecting CNN-generated facial images in real-world scenarios,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Virtual, June 14–19, 2020, pp. 642–643.
G. Tang, L. Sun, X. Mao, S. Guo, H. Zhang, and X. Wang, “Detection of GAN-synthesized image based on discrete wavelet transform,” Security and Communication Networks, vol. 2021, pp. 1–10, 2021.
X. Yang, Y. Li, H. Qi, and S. Lyu, “Exposing GAN-synthesized faces using landmark locations,” in Proceedings of the ACM Workshop on Information Hiding and Multimedia Security, Paris, France, July 3–5, 2019, pp. 113–118.
I. Demir and U. A. Ciftci, “Where do deep fakes look? Synthetic face detection via gaze tracking,” in ACM symposium on eye tracking research and applications, Virtual, May 25–27, 2021, pp. 1–11.
N. T. Do, I. S. Na, and S. H. Kim, “Forensics face detection from GANs using convolutional neural network,” ISITC, vol. 2018, pp. 376–379, 2018.
L. Nataraj, T. M. Mohammed, S. Chandrasekaran, A. Flenner, J. H. Bappy, A. K. Roy-Chowdhury, and B. S. Manjunath, “Detecting GAN generated fake images using co-occurrence matrices,” 2019. [Online]. Available: https://arxiv.org/abs/1903.06836
H. Mansourifar and W. Shi, “One-shot GAN generated fake face detection,” 2020. [Online]. Available: https://arxiv.org/abs/2003.12244
R. Wang, F. Juefei-Xu, L. Ma, X. Xie, Y. Huang, J. Wang, and Y. Liu, “FakeSpotter: A simple yet robust baseline for spotting AI-synthesized fake faces,” 2019. [Online]. Available: https://arxiv.org/abs/1909.06122
H. Li, B. Li, S. Tan, and J. Huang, “Identification of deep network generated images using disparities in color components,” Signal Processing, vol. 174, 2020.
B. Chen, W. Tan, Y. Wang, and G. Zhao, “Distinguishing between natural and GAN-generated face images by combining global and local features,” Chinese Journal of Electronics, vol. 31, no. 1, pp. 59–67, 2022.
S. Hu, Y. Li, and S. Lyu, “Exposing GANgenerated faces using inconsistent corneal specular highlights,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Toronto, ON, Canada: IEEE, June 6–11, 2021, pp. 2500–2504.
F. Matern, C. Riess, and M. Stamminger, “Exploiting visual artifacts to expose deepfakes and face manipulations,” in 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW). Waikoloa, HI, USA: IEEE, Jan. 7–11, 2019, pp. 83–92.
M. Taeb and H. Chi, “Comparison of deepfake detection techniques through deep learning,” Journal of Cybersecurity and Privacy, vol. 2, no. 1, pp. 89–106, 2022.
S. S. Khalil, S. M. Youssef, and S. N. Saleh, “iCaps-Dfake: An integrated capsule-based model for deepfake image and video detection,” Future Internet, vol. 13, no. 4, pp. 1–19, 2021.
D. Gong, Y. J. Kumar, O. S. Goh, Z. Ye, and W. Chi, “Deepfakenet, an efficient deepfake detection method,” International Journal of Advanced Computer Science and Applications, vol. 12, no. 6, pp. 201–207, 2021.
H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “Mixup: Beyond empirical risk minimization,” 2017. [Online]. Available: https: //arxiv.org/abs/1710.09412
S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo, “Cutmix: Regularization strategy to train strong classifiers with localizable features,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea, Oct. 27–Nov.2, 2019, pp. 6023–6032.
J. Guo, K. Han, H. Wu, Y. Tang, X. Chen, Y. Wang, and C. Xu, “CMT: Convolutional neural networks meet vision transformers,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), New Orleans, Louisiana, June 19–24, 2022, pp. 12 175–12 185.
M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in Proceedings of the 36th International Conference on Machine Learning, PMLR. Long Beach, California, USA: PMLR, June 9–15, 2019, pp. 6105–6114.
Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin Transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Virtual, Oct. 11–17, 2021, pp. 10 012–10 022.
R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-CAM: Visual explanations from deep networks via gradientbased localization,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017, pp. 618–626.
Downloads
Published
Issue
Section
License
Copyright (c) 2023 Sudarshana Kerenalli, Vamsidhar Yendapalli, Mylarareddy Chinnaiah
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with this journal agree to the following terms:
a. Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License - Share Alike that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
b. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
c. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
USER RIGHTS
All articles published Open Access will be immediately and permanently free for everyone to read and download. We are continuously working with our author communities to select the best choice of license options, currently being defined for this journal as follows: Creative Commons Attribution-Share Alike (CC BY-SA)
