Walk-Forward Evaluation of ROI-Based Decision Thresholds for Random Forest Loan Default Prediction
DOI:
https://doi.org/10.21512/emacsjournal.v8i2.16455Keywords:
Lending Club, loan default prediction, random forest, return on investment, right-censoring, walk-forward validationAbstract
Default prediction models for peer-to-peer lending are often tested on random data splits and judged by accuracy, which can hide temporal leakage and says little about investor returns. This study evaluates a random forest on 1,020,768 resolved 36-month Lending Club loans issued from 2007 to 2018, using 30 walk-forward windows with a 36-month gap between validation and test periods and only information available when a loan is funded. A decision threshold chosen by validation return on investment (ROI) was compared with the default threshold of 0.5, lending to every loan a grade-based rule and logistic regression, with full and downsampled training data, five random seeds, and paired tests across windows. The random forest reached a mean AUC of 63.7–64.6%, close to logistic regression (64.2–65.0%). Overall windows, the ROI-based threshold raised mean window ROI from 4.04% (lend-all) to 4.97% (difference 0.92 percentage points, 95% CI −0.29 to 2.23) while lending 51% of the capital; no difference remained significant after Holm adjustment. The share of test loans resolved by the end of the data fell from over 99% in windows 1–19 to 7% in window 30, so the negative returns of late windows reflect censoring. In the 19 matured windows, lend-all earned 9.08% on average and the random forest did not improve on it: ROI-based thresholds lowered mean ROI by 0.92–1.15 percentage points and profit by 40–54%. Return-based evaluations of default models therefore need time-ordered tests that account for unresolved loans.
References
Brown, I., & Mues, C. (2012). An experimental comparison of classification algorithms for imbalanced credit scoring data sets. Expert Systems with Applications, 39(3), 3446–3453. https://doi.org/10.1016/j.eswa.2011.09.033
Byanjankar, A., Heikkilä, M., & Mezei, J. (2015). Predicting credit risk in peer-to-peer lending: A neural network approach. 2015 IEEE Symposium Series on Computational Intelligence, 719–725. https://doi.org/10.1109/SSCI.2015.109
Chang, S., Kim, S. D., & Kondo, G. (2015). Predicting default risk of lending club loans. Machine Learning, 1–5.
Chen, Y., Calabrese, R., & Martin-Barragan, B. (2024). Interpretable machine learning for imbalanced credit scoring datasets. European Journal of Operational Research, 312(1), 357–372. https://doi.org/10.1016/j.ejor.2023.06.036
Emekter, R., Tu, Y., Jirasakuldech, B., & Lu, M. (2015). Evaluating credit risk and loan performance in online Peer-to-Peer (P2P) lending. Applied Economics, 47(1), 54–70. https://doi.org/10.1080/00036846.2014.962222
Falessi, D., Huang, J., Narayana, L., Thai, J. F., & Turhan, B. (2020). On the need of preserving order of data when validating within-project defect classifiers. Empirical Software Engineering, 25, 4805–4830. https://doi.org/10.1007/s10664-020-09868-x
Khushbu, S., Ganatra, A., & Thacker, C. (2024). Ensemble Approach for Stroke Prediction Using Machine Learning. 2024 Parul International Conference on Engineering and Technology (PICET), 1–7. https://ieeexplore.ieee.org/document/10819558/
Lessmann, S., Baesens, B., Seow, H.-V., & Thomas, L. C. (2015). Benchmarking state-of-the-art classification algorithms for credit scoring: An update of research. European Journal of Operational Research, 247(1), 124–136. https://doi.org/10.1016/j.ejor.2015.05.030
Liu, Z. (2025). Evaluate Lending Club Loan Status by Machine Learning. Advances in Economics, Management and Political Sciences, 154, 51–57. https://doi.org/10.54254/2754-1169/2024.19542
Lyócsa, Š., Vašaničová, P., Hadji Misheva, B., & Vateha, M. D. (2022). Default or profit scoring credit systems? Evidence from European and US peer-to-peer lending markets. Financial Innovation, 8(1), 32. https://doi.org/10.1186/s40854-022-00338-5
Malekipirbazari, M., & Aksakalli, V. (2015). Risk assessment in social lending via random forests. Expert Systems with Applications, 42(10), 4621–4631. https://doi.org/10.1016/j.eswa.2015.02.001
Namvar, A., Siami, M., Rabhi, F., & Naderpour, M. (2018). Credit risk prediction in an imbalanced social lending environment. International Journal of Computational Intelligence Systems, 11(1), 925–935. https://doi.org/10.2991/ijcis.11.1.70
Nikolaidis, D., & Doumpos, M. (2022). Credit scoring with drift adaptation Using Local Regions of competence. Operations Research Forum, 3(4), 67. https://doi.org/10.1007/s43069-022-00177-1
Núñez Mora, J. A., Moncayo, P., Franco, C., Madrazo-Lemarroy, P., & Beltrán, J. (2023). Loan Default Prediction: A Complete Revision of LendingClub. Revista Mexicana de Economía y Finanzas, 18(3). https://doi.org/10.21919/remef.v18i3.886
Óskarsdóttir, M., Bravo, C., Sarraute, C., Vanthienen, J., & Baesens, B. (2019). The value of big data for credit scoring: Enhancing financial inclusion using mobile phone data and social network analytics. Applied Soft Computing, 74, 26–39. https://doi.org/10.1016/j.asoc.2018.10.004
Serrano-Cinca, C., Gutiérrez-Nieto, B., & López-Palacios, L. (2015). Determinants of default in P2P lending. PloS One, 10(10), e0139427. https://doi.org/10.1371/journal.pone.0139427
Shan, Q., & Nilsson, M. (2018). Credit risk analysis with machine learning techniques in peer-to-peer lending market. Stockholm Business School.
Turiel, J. D., & Aste, T. (2020). Peer-to-peer loan acceptance and default prediction with artificial intelligence. Royal Society Open Science, 7(6), 191649. https://doi.org/10.1098/rsos.191649
Verbraken, T., Bravo, C., Weber, R., & Baesens, B. (2014). Development and application of consumer credit scoring models using profit-based classification measures. European Journal of Operational Research, 238(2), 505–513. https://doi.org/10.1016/j.ejor.2014.04.001
Yang, C., Dong, Y., Lu, J., & Peng, Z. (2022). Solving imbalanced data in credit risk prediction: a comparison of resampling strategies for different machine learning classification algorithms, taking threshold tuning into account. Proceedings of the 2022 5th International Conference on Machine Learning and Machine Intelligence, 30–40. https://doi.org/10.1145/3568199.3568204
Zhang, Z., Liu, N., & Xu, X. (2019). Research on Estate Mortgage Risk Control Based on RF-XGboost Model. Journal of Physics: Conference Series, 1345(4), 042016. https://doi.org/10.1088/1742-6596/1345/4/042016
Zhu, L., Qiu, D., Ergu, D., Ying, C., & Liu, K. (2019). A study on predicting loan default based on the random forest algorithm. Procedia Computer Science, 162, 503–513. https://doi.org/10.1016/j.procs.2019.12.017
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Henry Lucky, Anderies
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License - Share Alike that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
USER RIGHTS
All articles published Open Access will be immediately and permanently free for everyone to read and download. We are continuously working with our author communities to select the best choice of license options, currently being defined for this journal as follows: Creative Commons Attribution-Share Alike (CC BY-SA)
