Comparative Evaluation and Comprehensive Analysis of Machine Learning Models for Regression Problems

Artificial intelligence (AI) and machine learning (ML) models have overachievement in the ability to create relations between the variables (attributes) and the observations (instances) for different kinds of tasks such as classification and regression. Unlike classification applications, in which samples are assigned to certain labels, regression applications aim to draw a regression line or plane on all samples with the least error. In several studies, researchers considered the real-valued outputs as bucketed outputs and converted regression problems into the classification domain if possible [1]. This led more implementations of ML models on classification than regression tasks that aimed to predict real-valued infinite outputs. However, the deployments of regression studies also affected multidisciplinary fields such as the healthcare sector [2, 3], education [4], price predictions [5], sports [6], and finance [7].

The primary concern of the ML applications is determining the model suitable for the dataset used for the particular application. This has vital importance since determining the optimal ML model for all kinds of applications in both problem domains is almost impossible because of the different characteristics of datasets and the ability of the models [1]. In addition, the characteristics of the considered datasets can increase the complexity of the studies in both domains. Based on the attribute properties, datasets can be structured-unstructured datasets, numeric-categorical, and combined datasets. However, they can most importantly be defined as linear/nonlinear or high/low correlated when the relations between attributes and outputs are considered—these cause ML models to produce different results in applications on datasets with different characteristics. Therefore, the analysis of the success of an ML model should be performed on datasets containing different and varied conditions.

Besides the characteristics of the datasets, training of the ML models (validation techniques) and evaluation techniques differ in most of the studies. The hold-out method, which splits the train and test data using different ratios (60:40, 70:30, 75:25, 80:20, etc.), is common in AI and ML implementations [2, 4, 6]. The other commonly used training and validation method for ML models is k-fold cross-validation, which is used for hyperparameter tuning and evaluating final results [6]. The main drawback of the hold-out method is that the samples are only used in training or testing, and the actual prediction abilities of the models differ according to the training and testing samples. Instead, k-fold cross-validation, which finds the average result by dividing the data into k equal parts and training models k times, produces more accurate results [8] because of the consideration of all data both in testing and training. However, obtaining results using all data samples could affect the obtained results, negatively or positively, slightly or significantly. The number of training data and the validation of methods significantly affect the model trained with random data selection and do not determined by data mining. The recent regression studies differ in how they were implemented in terms of model selection, evaluation, and training. However, the implementation of multiple models and the consideration of several evaluation metrics are common in these studies, even though the training strategies are different [2, 6, 9, 10, 11].

The use of hold-out ratios or cross-validation method does not have a standard implementation even in recent research. The reasons for this are the number of instances in the dataset, the researchers' preferences, and the responses of the models. However, in ML, it is known that even a small change in the trained data can significantly affect the results in a positive or negative direction, especially in datasets with different characteristics or in big data. Therefore, it is crucial to investigate the effect of cross-validation as well as the hold-out ratios on models and results using varied datasets [8].

Although most machine learning studies included the comparative analysis in determining the optimal model, a few direct comparison studies were also performed. Huang et al. [12] compared the three ML models (backpropagation neural network, support vector regression, and extreme learning machine) for the regression problems using a total of four datasets. The evaluation was performed using mean squared error (MSE), mean absolute percentage error (MAPE), and coefficient of determination (R² score). In addition, 20-fold cross-validation was considered to estimate the prediction errors and the authors concluded that the use of integrated models produced superior scores than single models. Bratsas et al. [13] compared four models, namely multilayer perceptron, linear regression, random forest, and support vector regression, to predict the traffic status of Thessaloniki, Greece. The comparison was performed using three scenarios created from a single dataset. The evaluation of the models was performed using Root Mean Square Error (RMSE). It was concluded that the performances of Neural Networks (NN) and Supoort Vector Regression (SVR) models outperformed both Random Forest (RF) and Linear Regression (LR). Recent comparative research showed that considering multiple and varied datasets, models, and validation techniques were of crucial importance to analyzing the models' abilities and the effect of training strategies.

Automated Machine Learning (AutoML) has been started to be implemented recently, and besides finding the model that produces superior results among different ML models, it aims to achieve the best result with the ensemble method. However, although AutoML provides great advantages to users in terms of ease of implementation, its' computational costs and computer crashes, even in relatively small datasets, are the major disadvantages of the AutoML.

This study aims to compare ML models for regression tasks with different scenarios that have not been studied together in recent studies. An excessive number of multi-character datasets such as time-series, multivariate, high instance, and high attribute, are considered with varied validation strategies to analyze the response of the models to the different numbers of training data, datasets, and the effect of hold-out and cross-validation on the regression tasks. Finally, it aims to achieve the primary goal of the study, which is to determine starting points for future regression studies to minimize model and validation strategy selection procedure.

For this purpose, in this paper, ten benchmark ML models were selected to be included in the comparisons due to their frequency of use and the formation of the basis for other models.

Linear Regression is still one of the most frequently used statistical models in regression tasks, particularly for the data with a linear relationship. Decision Tree (DT) is another common ML model for regression problems and also forms the basis for other tree-ensemble models such as Random Forest (RF), Gradient Boosting (GradBoost), and Extreme Gradient Boosting (XGBoost). RF, GradBoost, and XGBoost minimize the error obtained by the DT either with bagging or boosting strategies and have increasing popularity in regression tasks. Even though the XGBoost and GradBoost do not have higher popularity as much as RF, their implementation in regression tasks is becoming common. The superior results of the support vector machine in classification problems lead the implementation of Support Vector Regression (SVR) for the regression problems, and the use of SVR also spreads. However, the different kernel functions make it difficult to implement all of them, and in this study, Support Vector Regression with Radial Basis Function (RBF) kernel (SVRBF), Support Vector Regression with the linear kernel (SVRL) are considered. The neural networks, which are the primary tools for obtaining reasonable results, particularly in nonlinear data, are of significant importance to regression studies. For this reason, two artificial neural networks with shallow and deep versions (NN and DNN) and a special type of recurrent neural network which is significant importance to regression tasks, deep Long-Short Term Memory Neural Network (deep LSTM), are implemented.

A total of 680 experiments were performed on 17 considered datasets to perform a comprehensive evaluation and comparison. The obtained results were analyzed using three common evaluation metrics for regression problems: MSE, MAE, and R² Scores. The analysis of considering the hold-out method with different ratios was performed. The effect of increment or decrement in training data was analyzed for each model. The data dependency of the models was determined. The obtained hold-out results were compared to the five-fold cross-validation method, and the effect of cross-validation was demonstrated. In addition, the fold analysis in cross-validation was performed to present the changes for each model in each fold with statistical descriptions. The model-based evaluation was performed, and the advantages and disadvantages of the models were presented. Finally, the recommendations for models and validation strategies are presented.

A total of 17 regression datasets from different real-life application areas such as environmental sciences, social sciences, civil engineering, finance and sales sector, and energy consumption were selected in our study to compare machine learning models for different application fields and obtain more generalizable results. The datasets consisted of a varied number of attributes and instances to analyze the ability of the models on different data. In addition, the datasets are selected to analyze the performances of the models with different data, such as time-series and multivariate.

Air Quality [14], Wine Quality [15], Combined Cycle Power Plant (CCPP) [16], Behavior of the urban traffic of the city of Sao Paulo in Brazil Dataset (SPB) [17], Real Estate (RE) Valuation [18], Concrete Compressive Strength (CON) Data Set [19], Daily Demand Forecasting Orders Data (DDFO) Set [20], two Student Performance (SP) datasets [21] and three Power consumption of Tetouan city (TCPC) datasets [22] each of which is for a single zone, were used to analyze and evaluate the considered machine learning models. Table 1 shows the number of instances and attributes included in the datasets used in this study.

Numerical representation of the attributes makes it difficult for humans to observe the relationships of data and the characteristics of the dataset. Figure 1 shows the correlation analysis of the datasets. The highest correlation between the attributes was observed in AQ and DDFO datasets (Figure 1 (d) and (h)), and the lowest can be listed in WQR, WQW, RE, and STP datasets (Figure 1 (a-c) and (j)). The correlation provided by the last three attributes of the STM dataset has been eliminated by removing the two attributes from the STM dataset, and a more challenging dataset has been obtained in STP, as mentioned above.

The following section summarizes the basic principles of the mentioned ML models.

Backpropagation is the most frequently considered neural network for optimization, regression, and classification problems. Interconnections of neurons which are the weights, are updated by considering the actual response of the neural network and expected or observed data. Gradient-descent is the algorithm to calculate the weight change and update each interconnection. It is still one of the most implemented algorithms for comparative studies of neural networks and machine learning models [23]. In this study, the shallow version was used with a single hidden layer, and the deep version was implemented using four hidden layers.

Linear Regression is a statistical method that draws the best-fitting regression through the real points. It is frequently and successfully used in regression problems, especially on datasets whose attributes have a linear correlation [24].

Support Vector Regression (SVR) was improved to get real-valued outputs instead of binary numbers for regression problems [25]. Error is minimized while the hyperplane margin is maximized, which provides an efficient distinguishment of data from each other [26, 27]. Different kernel functions can be used to project data into higher dimensions, and in this study, Linear and Radial-Basis Function kernels were considered in the comparisons.

LSTM is an effective special version of recurrent networks and can be used for both classification and regression problems [28]. Cell, input gate, output gate, and forget gate are the four major components of its architecture. It uses gradients to update weights; however, it remembers previous errors, which improves the netw'rk's error minimization in minimized iterations [29].

Decision Trees are tree-structured algorithms with an initial root node, decision nodes, and leaf nodes. They are using the divide-and-conquer strategy, which brings several advantages and disadvantages for them [8]. The simplicity and speed are the main advantages of decision trees; however, the determination of the initial root or the sequence of nodes is the main drawback.

Random forests are a kind of tree-based ensemble learning and can be used for both classification and regression [30, 31]. It constructs several decision trees during the training and optimizes the mean regression of the individual trees.

Gradient Boosting is another tree-based ensemble machine learning algorithm [32]. It aims to optimize the outputs by minimizing the loss obtained by the constructed weak learners, which are decision trees. The loss is calculated, and then a new or modified tree is added to reduce the total loss using a gradient descent algorithm. The mo'el's output is modified after adding each tree to the model, and different stopping criteria such as no decrement in loss, adding a fixed number of trees, etc., can be applied to obtain the final output of the model.

Similar to Gradient Boosting, Extreme Gradient Boosting [33] is also an ensemble tree method and applies the principle of boosting weak learners using the gradient descent algorithm. However, XGBoost includes some enhancements to minimize the used resources and to improve the obtained results. Different regularization models (i.e., LASSO) are used to overcome overfitting problems during the learning. The built-in cross-validation is applied in each iteration to determine the exact number of iterations on a single run.

Three commonly used evaluation metrics were considered to compare the obtained results: Mean Squared Error (MSE), Mean Absolute Error (MAE), and coefficient of determination (R² score).

MSE takes the square of error before averaging them, and this provides a relatively higher weight to significant errors (outliers). This supports researchers in observing the errors of the datasets with larger values. However, the frequency of the errors has a significant effect on MSE results, and the repetition of the error causes the increment of MSE.

However, reaching minimum error does not always show that the predictions will be more accurate than other models. Particularly, some significant errors within the dataset may increase the error. Therefore, it causes overestimating mod'ls' errors due to the higher value of MSE. This is because the nature of MSE that considers outliers more than other evaluation metrics. On the other hand, small errors between predicted and actual data may cause underestimating the error. Thus, it is required to consider other evaluation criteria and consider all of these during the evaluation of the models.

The other metric that is used to evaluate the ability of regression models is the MAE, which is the mean of absolute errors. MAE focuses on the magnitude of the errors between predicted and actual outputs and does not consider the direction of the error. It is assumed that more stable results could be obtained using the MAE.

The R² score, which is strongly related to MSE, is used to measure the correlation level of predicted and observed values within the considered dataset. This provides scaled evaluation results for the models and allows researchers to perform a more robust evaluation between them.

The design of experiments was based on four varied training of the considered ML models to obtain results with different training ratios and k-fold cross-validation.

Models were trained by three hold-out ratios, 60:40, 70:30, and 80:20 of the considered datasets, and scores were obtained from the untrained (test) data separately. Furthermore, the ML models were trained using five-fold cross-validation to provide a more accurate and robust evaluation and analysis of the ML models. The results of five-fold cross-validation experiments were obtained by taking the mean of the fold results.

The results obtained by the five-fold cross-validation also provided the analysis of how the hold-out and k-fold cross-validation strategies positively or negatively affect the obtained scores. This analysis was performed using the results obtained by all hold-out ratios and five-fold cross-validation and individual fold results obtained in five-fold cross-validation. During the training, the architectures of neural network models, NN, DNN, and deep LSTM, were fixed; however, parameters were tuned depending on the dataset performances of the related model.

The experiments used fixed training and testing data for each hold-out ratio. Therefore, the effect of change in the training data in the results has been eliminated. In cross-validation experiments, each fold was fixed, and it was ensured that each model performed training with the same data in each fold.

Deep Neural Network (DNN) was implemented with four hidden layers, 500 neurons within each hidden layer, and the Sigmoid function was used as an activation function for each layer. ‘Adam’ and MSE were used as optimizers and loss functions, respectively. A shallow Neural Network (NN) was used with a single hidden layer and 500 neurons. The other parameters were set as the same as DNN.

Deep Long-Short Term Memory Neural Network was used as four-layered, and the maximum iteration number was determined based on the highest scores obtained. The experiments were repeated in DNN, NN, and LSTM experiments with different iterations for each dataset to obtain superior results.

The grid search was applied to obtain optimized scores in Support Vector Regression, Random Forest, GradBoost, and XGBoost. The best parameters are used to train each fold in five-fold cross-validation experiments and all ratios of hold-out experiments. Mean Squared Error was used to build a decision tree regressor structure.

This section summarizes the results obtained in the experiments and compares the models at different training ratios by presenting quantitative results. All models produced fluctuating results at different learning rates and datasets. The results are compared and discussed in the following sections in detail. S1 Table, S2 Table, and S3 Table present the obtained MSE, MAE, and R² score results for all datasets in all experiments. Figure 2 demonstrates the visualization of the obtained R² scores of the datasets for each hold-out ratio and five-fold cross-validation experiments. Bold values within the tables indicate superior results.

The analysis of the effect of hold-out ratios on the prediction performance of the models should be performed in two stages.

The first stage analyseszes the increment or decrement in the number of training data for the models. The second stage is to determine the impact of different training ratios and cross-validation on the performance of a particular model. This would yield to determine the data dependency and the sensitivity of the models for the change in the number of training data.

In this stage, the R² scores were used to analyze the obtained results since the R² score is scaled results and provides a more effective evaluation. Therefore, the analysis was performed using the number of highest R² scores obtained in the experiments and the statistical descriptions obtained using the R² scores for each dataset and model individually.

NN, DT, and RF produced fluctuated and similar results when the hold-out results were considered. Their lowest and highest R² scores were obtained in the 70:30 and 80:20 hold-out ratios.

On the other hand, GradBoost and XGBoost produced fluctuated results. Although their lowest R² scores were obtained in the 70:30 hold-out ratio, they achieved their highest scores in 60:40 experiments.

Models in which the increment of training data linearly affected the performances negatively were LSTM, SVRL, SVRBF, and LR. While the highest R² scores of these models were obtained at 60:40 ratios, the produced results decreased as the training data increased, and the lowest results were obtained in 80:20 experiments.

The most positively affected model by the increase in the number of training data was DNN. DNN, which could not produce the highest R² scores for any dataset with the lowest training ratio (60:40), achieved higher results at 70:30 and 80:20 ratios. The experiments in which DNN was most successful were the 80:20 hold-out experiments.

However, considering the five-fold cross-validation experiments, the obtained results significantly changed for NN, DNN, LR, DT, GradBoost, and XGBoost. Performing experiments using five-fold cross-validation provided these models to improve their performances and achieve the highest number of superior results in experiments. Nevertheless, it should be noticed that the training of SVRL, SVRBF, and RF using five-fold cross-validation did not positively impact these models.

Determining the highest scores obtained in different hold-out ratios is valuable for analyzing models. However, the statistical descriptions, such as mean, median, minimum, and maximum R² scores, standard deviation between the obtained results, 25%, and 75% quartiles, are also effectively used in analyzing the order and the change in results. Furthermore, they provide information to determine the models' sensitivity for the number of training data and adaptation of training data. Therefore, we provided the maximum and minimum standard deviation of the results obtained for each model using a dataset with different training ratios.

The obtained mean, median, quartile, and standard deviation results demonstrated that different models could achieve superior, fluctuated, or decreased prediction levels based on the number of training data. However, the deep LSTM achieved superior results in all descriptions for all experiment scenarios. The SVRL and LR produced the highest standard deviation and lowest mean and median results for different hold-out and cross-validation experiments which are the worst results in this study. Table 2 presents the statistical descriptions for the hold-out and cross-validation experiments, Figure 3 shows the model-based distribution of the results for all experiments, and Figure 4 presents the hold-out and cross-validation results per model independently.

Additionally, we provided the maximum and minimum standard deviation of the results obtained for each model using a dataset with different training ratios. The close minimum and maximum standard deviations were calculated in the obtained R² scores of NN and DNN. While'NN's minimum and maximum standard deviations were 0.0006 and 0.0612, it was calculated as 0.0003 and 0.0641 for DNN. These obtained maximum standard deviation results were the lowest maximum standard deviations among all experiments.

The model that followed the NN and DNN models was LR with a 0.0797 maximum standard deviation. LR produced more stable R² scores for all datasets, and the lowest average standard deviation was calculated (0.0215), even though it could not produce superior results in most of the experiments.

There were no significant differences in the maximum standard deviations of the R² scores of SVRBF, SVRLR, and DT. Therefore, the average standard deviations of these models were calculated as 0.0154, 0.0173, and 0.0275. Results showed that the SVR models had the lowest average standard deviation.

Tree ensemble models achieved more fluctuated results than other models except for deep LSTM. The maximum standard deviations calculated for RF, GradBoost, and XGBoost were 0.1384, 0.1569, and 0.1461. The average standard deviations of these models were 0.0305, 0.0322, and 0.0331, respectively.

The largest and most significant changes in standard deviations of R² scores obtained in different holdout and five-fold cross-validation experiments were calculated in deep LSTM. The maximum and average standard deviations for deep LSTM were calculated as 0.3467 and 0.0629. These results were the highest standard deviation values calculated within the considered models. Table 3 shows the minimum, maximum, and average standard deviations for all models calculated using the R² scores obtained in the hold-out and five-fold cross-validation experiments.

Five-fold cross-validation experiments were performed to analyze the effect of changing training data on the learning and performance rates of the models. Therefore, it was tried to show how much the changing data will affect the results produced by the models in the experiments performed with hold-out ratios. We used the Average θ value, where θ is the difference between the highest and the lowest R² scores obtained in the folds, to indicate the general change value produced by the models.

While presenting the results in this section, AQ and CCPP were not considered because the R² scores produced in each fold by each model were at the highest level. Furthermore, the changes between folds were tolerable since the obtained values were less than 0.001 on average.

NN obtained the highest change (θ) between folds in the SPB dataset with 0.48. In this dataset, the highest R² score reached in a fold was 0.81, while the lowest was 0.32. NN produced a minor change between fold results in WQW, CON, DDFO, and STM datasets, giving more stable results. For these three experiments, θ was calculated between 0.07 and 0.10. More fluctuations occurred between folds in WQR, RE, and STP datasets, and θ values were calculated as 0.17, 0.21, and 0.20.

In the DNN model, close results were obtained with NN. The differences observed between these two models were that DNN produced more stable results in WQR, CON, STM, and STP datasets. The most remarkable change among the fold results occurred in the SPB dataset (θ = 0.47) as in NN. The average θ value of the DNN model was calculated as 0.16.

Since the LR model achieved total success with each fold in the DDFO dataset, no change was calculated between the folds. However, LR could not establish any correlation between attributes and instances in a fold in the SPB dataset and could not produce results. This caused the θ value to be 0.74. The most stable results obtained by the LR model, except for DDFO data, was the WQW dataset (θ = 0.15). The average θ value of LR, which produces fluctuating results in other experiments, was 0.26.

SVRL and SVRBF models produced close θ values for WQR, WQW, RE, SPB, DDFO, and STM datasets. The highest θ between the two models occurred in the CON (SVRL = 0.51 and SVRBF = 0.44) and STP (SVRL = 0.26 and SVRBF = 0.30) datasets. However, both SVRL and SVRBF produced rather variable and different results in SPB data in terms of θ as observed in other models (SVRL = 0.73 and SVRBF = 0.73). The average θ value for both models was calculated as 0.29.

Although deep LSTM produced superior results in most experiments, the model produced the most fluctuated and data-dependent results in five-fold cross-validation experiments considering DDFO, STM, and STP data. While deep LSTM showed the most negligible variation between folds in other datasets, it produced more stable results in the SPB dataset than in other models. However, in STM and STP datasets, it caused the θ to increase to 0.99 by making inaccurate and most successful predictions in different folds. A more significant fluctuation was observed in the DDFO dataset compared to other models. While the average θ value of deep LSTM was calculated as 0.36, it was marked as the most sensitive model in this study.

Since DT could not produce any results in any fold of the five-fold cross-validation experiments of the STP dataset, the θ for STP was 0. Therefore, the average θ value was calculated by ignoring the STP data. Similarly, WQR and WQW data failed to produce results in a fold. This caused an increase in θ values in these experiments (0.13 and 0.21, respectively). In the SPB dataset, where other models (NN, DNN, SVRL, SVRBF) produced fluctuating results, DT could produce a more stable result, although it could not produce a superior result (θ = 0.23). However, in the RE dataset, the highest and lowest R² scores produced at different folds were calculated as 0.69 and 0.32, resulting in the highest θ value for DT (0.37). The average θ value is calculated as 0.22.

Although RF produced more reasonable and stable results than DT, it could not produce results in one fold of CON and SPB datasets experiments. High R² scores obtained in other folds caused the θ value to increase significantly. As a result, θ values for both CON and SPB datasets were calculated as 0.83. This, in turn, affected the overall stability of the model, and the average θ was calculated as 0.33.

Although the general results of the GradBoost model were similar to RF, GradBoost produced more stable results in CON and SPB datasets resulting in the average θ value being calculated as 0.17 and observed as one of the most stable models.

XGBoost produced the lowest θ value for the WQW and STP experiments (0.01 and 0.11, respectively). The experiments in which XGBoost produced the highest θ value were RE and SPB datasets (0.30 and 0.29, respectively). Although it could not produce superior results in other datasets, it generally produced the most stable results. This produced the lowest average θ value (0.15). The dataset-based plot of change in R² scores that show the minimum, maximum, and average R² scores obtained in the folds is shown in Figure 5. Table 4 presents the θ values obtained by each model for each dataset and the average θ obtained for a particular model.

Several points should be discussed by considering the obtained results. We separately analyzed the performances of the models in terms of R² scores and error minimization for different kinds of datasets, the effects of training ratios of the hold-out strategy, and the impact of cross-validation on the models learning to deduce general opinions. In addition, the data dependency of the models was analyzed by combining all the obtained results and performing a fold analysis. This also provided us to analyze the consistency and the stability of the models.

When the results of varied hold-out ratios are compared without considering five-fold cross-validation results, it is challenging to make a consistent analysis due to the responses to the new data added at increasing training ratios, even if the same samples were considered. When we analyzed the results for all experiments, fluctuations were observed regardless of the training ratios. However, it is debatable whether there were very significant changes in the results produced by the models on a ratio basis.

The SVRBF and SVRL were the least affected models by the number of training data, indicating that the projection of the data to another plane reduces the effect of the number of training data. SVRBF was the most successful model at the point, as the model with the lowest average standard deviation value, followed by the SVRL. However, the increase in the number of training data in both models affected the success rates negatively. It has been once shown by the obtained results that SVR models could produce better results with less training data.

NN and DNN showed that the neural-based models also minimized the changes due to the number of data. Although SVRBF, SVRL, NN, and DNN are less interpretable than other models, which complicates the analysis of results, neural-based models could achieve more stable results since they are capable of effective convergence because of hidden layers and neurons in these layers. However, fewer hidden layers and neurons might result in more inconsistent results, making it difficult to determine the proper training data ratio.

On the other hand, in DNN with more hidden layers and neurons, the success rate increased as the number of training data increased. Although the changes were not at significant levels, it was observed that the number of data needed increases as the neural network architecture gets deeper. The results showed the disadvantages of using a large number of processing elements with a minimized number of training data.

Determining the node sequences and decision leaf by DT is the most crucial factor in this'model's success. The results showed that DT produced fluctuating results depending on the training ratios. However, the need for DT to achieve better results using more training data has been reduced in the RF model and eliminated in the GradBoost and XGBoost models.

The standard deviations of the RF, GradBoost and XGBoost models were higher than the other models. It has been observed that these models produced more sensitive responses to the changing number of training data. Still, the applied processes provided better results by minimizing the errors obtained in DT and reducing their dependence on the training data number. Accordingly, GradBoost and XGBoost achieved higher results with fewer training data.

Deep LSTM was the model with the highest sensitivity to the number of training data. In terms of produced R² scores and the standard deviation between these scores, the most fluctuated results of this study were provided by the deep LSTM. Although the fixed number of LSTM layers might lead to these results, the states of input sequences as forgotten and remembered in the LSTM layers could be considered one of the most significant factors both in the success and fluctuated results of this model. On the other hand, the deep LSTM model produced the highest results with the lowest training ratio, even though it was used with a large amount of LSTM layers. The increment in training ratios did not positively affect the models' performance.

When we consider all the results obtained for all models, the ratio of 70:30 was found as a training ratio that causes the minimum success of the models. On the other hand, although higher results were obtained in the 80:20 experiments, the 60:40 ratio was observed as the training ratio with the highest results.

If cross-validation was not considered in regression studies, the consideration of 60:40 and 80:20 for specific models based on the above-mentioned information would help researchers decrease experimental costs and achieve superior results.

Cross-validation was considered in many studies for hyper-parameter tuning or final performance evaluation of the models. The results obtained in our study showed that cross-validation is vital to determining the general abilities of the models. In this study, the consideration of the five-fold cross-validation provided the training of the models with an 80:20 ratio using five different datasets in five folds. Therefore, the analyses of he models could be performed using all data.

The obtained average fold results showed that NN, DNN, LR, DT, GradBoost, and XGBoost achieved their highest R² scores in experiments using five-fold cross-validation (S3 Table). These results showed that the use of cross-validation in studies where these models will be considered would be more consistent, more reliable, and more successful. However, SVRBF, SVRL, and LSTM did not make a remarkable difference in experiments with five-fold cross-validation compared to training with other ratios.

On the other hand, when the R² score differences (θ) obtained between the folds were considered, a more complex relationship was observed (Table 4). The obtained θ values showed the response of the models to the training data in the learning process and the data dependency of the models, and the achieved results demonstrated that the most successful model in this regard was XGBoost (min θ=0.01, max θ=0.31, average θ=0.113).

XGBoost successfully minimized the error while adding new trees to the created ones and reduced the dependency on new and different data. This caused the model to produce more stable results between

The models that followed XGBoost based on the θ values were GradBoost, DNN, and NN. Although GradBoost and DNN are completely different classifiers in structure, their average θ was calculated as equal (0.17).

GradBoost, which has similar features to XGBoost, creates an ensemble model by adding new trees by minimizing the error between the gradient descent algorithm and weak trees. This caused the model to produce similar results with XGBoost and low data dependency.

On the other hand, the DNN used in this study showed that higher numbers of hidden layers and neurons could produce more stable results, although not superior. This property increased the impact of DNN on learning the changing data in cross-validation.

However, although NN with a single hidden layer produced more fluctuating results than DNN when cross-validation was used, the change in R² scores between the folds was minimal. This showed that increasing the number of hidden layers could produce more stable results and reduce the number of experiments to be performed but would not significantly change the overall results.

DT and LR produced close results considering θ values. These two models are successful on datasets with more linear relationships than other models, and that increased the importance of the data they fed in the training process and the test data. However, LR produced more stable results in highly correlated data, but DT produced more significant differences in terms of the R² scores it produced between the folds in these datasets. The creation of a single tree and the node sequences were the most significant factors at this point.

The SVRBF and SVRL produced reasonably similar results in this analysis. θ was calculated as 0.276 and 0.251, respectively. High variation between the folds was caused by the data projection, which reduces the dependence on the number of data and is unpredictable when applied to the data containing different information. While the data in one fold can be projected to draw the regression line with a minimum error, the data in the other fold might not be appropriately projected to provide a best-fitted regression line. This created highly variable results for the two models.

The most unexpected results in this analysis of the study were produced by the RF and LSTM models. As an ensemble model, RF was expected to produce more stable results between the folds, but it was observed that the effect of the created trees and the number of trees used in the experiments on the model was very high. Although it produced successful results, RF has become one of the most data-dependent models.

In the deep LSTM model, although generally stable results were produced, the extreme results obtained in the STP and STM dataset experiments determined the average θ as the highest. The data dependency of the LSTM appeared to be in the upper level; however, it is also analyzed along with the analysis of the characteristic features of the dataset in the next section.

The dataset-based analysis was performed to evaluate the general responses of the models under different conditions. It is common knowledge that providing a large number of instances to a model with a high number of attributes facilitates learning (i.e., the AQ Dataset). However, the relevancy of these instances and the information they feed the model have a vital role in the learning process of the models (i.e., the WQW Dataset).

Neural network models, which were expected to be more accurate in solving nonlinear problems with fewer relationships between their attributes, outperform or lag behind other models with a slight difference in these datasets.

If the results for NN and DNN have been interpreted on the STM and STP dataset, removing two highly correlated attributes from the training data severely reduced the success of NN and DNN, similar to other models. In addition, although these two models produced high results for highly correlated datasets (i.e., DDFO), they did not achieve the prediction rate of LR. At this point, the achievement of better results than LR in the highly correlated AQ dataset by these two models showed that the minimized number of instances used during the training process had a significant and negative effect on the performances of NN and DNN.

Although SVRBF and SVRL models achieved results close to NN and DNN, it was observed that they could achieve superior results in the highly correlated datasets with a minimal number of attributes and instances (i.e., DDFO). However, SVR models produced lower results than the NN and DNN models in other datasets with higher numbers of attributes and instances, even though the correlation between attributes was high. This showed once again that the use of a more informative and limited number of attributes and instances for data projection would increase the success of SVR models.

The DT model produced fluctuating results depending on the dataset characters, and it could not achieve successful results in general. Even it was unable to make any predictions in the high attribute, low instance, and low correlation STP dataset, DT succeeded in outperforming the RF, XGBoost, and GradBoost models in the highly correlated DDFO dataset. It was observed that the success rate of DT decreased significantly at low-instance high-attribute datasets. As with all other analyses, the success of DT in identifying the starting nodes and the sequences of subsequent nodes proved to be vital to the model.

The obtained results showed that the tree-based ensemble models achieved more successful results in datasets with a high number of instances. In AQ, CCPP, and WQW datasets, RF, GradBoost, and XGBoost models generally achieved superior results than other models except for LSTM. The use of a high number of instances provided successful results in these models since the creation or addition of different trees allows them to perform more meaningful information connections systematically.

LSTM achieved results that outperformed all other models regardless of the correlation between attributes when trained with many instances. LSTM, which produces highly fluctuating and variable results in STM, STP, and DDFO datasets, was the model most affected by the dataset's number of instances and attributes. Training the deep LSTM with a low number of data makes it challenging to achieve high prediction rates.

In 57 out of a total of 68 experiments, deep LSTM produced the highest R² scores and superior error minimization results, achieving a very high success rate compared to other models. The model that followed the deep LSTM was GradBoost, which produced two times the superior and three times the second-highest results. XGBoost, which achieved the superior result once and the second-highest result ten times, was another model that produced stable results. Besides, LR produced superior results in an experiment and the DT and RF models. On the other hand, NN, DNN, and SVR models could not achieve optimal results in any of the experiments.

Although the increase in the number of hidden layers and neurons produced more stable results than NN, DNN could not have the superior or even the second-superior score in any experiment. This caused neural-based models not to outperform other models (i.e., deep LSTM, GradBoost, XGBoost) in this study. The obtained results limit the success rate of the neural-based models in general; however, considering the fixed architecture and parameters in this study could be one of the reasons for this. In NN, although the use of a lower number of hidden layers and neurons did not significantly reduce the success rate to DNN, it had a negative effect on the stability of the model. This instability also reduced the reliability of the model.

SVR models also produced similar results to neural-based models but slightly lower overall. As mentioned above, although higher and more stable results were obtained compared to NN and DNN in datasets with low instances and attributes, they lagged behind LSTM and tree-ensemble models in general.

DT could not achieve high results as tree-based ensemble models. It was seen as an inevitable improvement that optimizing many trees to be created could yield superior results than a single tree, which was the main aim of proposing tree-based ensemble models. Although the DT can be easily implemented and more responsive model, the development of models such as RF, GradBoost, and XGBoost has overshadowed the success of DT. The main limitation of DT, which is the sequence of the attributes, proved once again that ensembling multiple trees achieve superior results.

LR, the most basic regression model, proved its success in linear problems once again and showed that it could correlate between correctly selected data and nonlinear test data. However, since the ability to establish nonlinear correlations between all datasets is very limited, further experiments are required to conclude this. LR attracted attention again as a model that must be considered among datasets containing linearly related attributes.

RF lagged slightly behind the other two models (GradBoost and XGBoost) among tree-based models. By optimizing the regression results of the constructed trees, RF produced slightly more fluctuating and inconsistent results than the performance of GradBoost and XGBoost, which considers the losses of weak models. This is caused by randomly sampling (bagging) the instances while creating individual trees in RF; since GradBoost and XGBoost consider weak samples in the re-creation of trees (boosting). While bagging made RF more sensitive to overfitting, the boosting models are more advantageous in tree-based ensemble methods.

GradBoost and XGBoost models have proven to be models that should be considered in terms of stability, data dependency, and their results in regression problems. Considering that the optimization of the fixed parameters could further increase the performance of these models, they stand out as the models that should be preferred in the first place.

However, the deep LSTM model, which achieved the highest results in almost all experiments and produced high results even in datasets where other models failed to make predictions, showed that it is one of the top ML models that should be considered for regression problems. However, the most significant disadvantage of deep LSTM is the data dependency, unstable response to changing training data, and inconsistent results in some conditions. Even between folds, it was observed that while it could predict all the data with high scores in one fold, it could not properly predict any data in the other fold. In LSTM cells, the dependence of forgetting states on previous data and the effect on data from later sequences significantly affected the consistency of the LSTM. However, the fixed structure of LSTM might cause fluctuations and require further experiments for this model to generalize its' characteristics.

Considering that the data in all experiments in this study were randomly selected and then fixed, it can be considered that the fluctuations and instability in many models were acceptable. The most significant limitation of the LSTM model was not completing the learning process in datasets with a low number of instances and the high number of attributes (STM and STP datasets). Therefore, it could be concluded that the deep LSTM would continue to achieve high-level results among regression models with appropriate and sufficient data selection.

The outcomes and recommendations of the study are listed below:

• Long-Short Term Memory Neural Network, Extreme Gradient Boosting, and Gradient Boosting were the superior models in the overall experiments. It is recommended that these models be used more widely in regression problems.

• Random Forest, Support Vector Regression Models, and LSTM were the most data-dependent models. Therefore, cross-validation in studies involving these models is required to evaluate the prediction abilities of the models.

• Since which superior hold-out ratio in ML training varies depending on the data and the model, the characteristics of the models should be considered in the experiments where hold-out will be used. However, the 60:40 hold-out was the ratio with the highest results.

• The XGBoost was the model with the least dependence on the number of training and changing data. GradBoost followed this model. Therefore, any hold-out or cross-validation would not significantly change the results of these models.

• DNN and NN were not affected much by data changes, but cross-validation should be continued to be implemented for these models' parameter and structure determination.

• Once again, it has been shown that as the number of data (instances and/or attributes) increases, the deeper neural network produces higher results than the shallow neural network.

• LSTM achieved superior results than other models in datasets of different types and sizes. However, the determination of its' structure could lead to inconsistent results.

• Even though the XGBoost and GradBoost could not achieve results as high as LSTM, they produced more stable and data-independent results with low computational cost.

• It has been determined that cross-validation could affect the results of all models on average 0.01-0.20 R2 score, even though the XGBoost and GradBoost are data-independent. Therefore, cross-validation is essential for the models to obtain more consistent and comparable results.

• It was observed that the data feed without selection to SVR models caused not produce higher results than other considered models. Data minimization might increase the ability of SVR.

• LR is still one of the superior methods for data with a linear relationship. However, the success of the model decreases drastically in complicated datasets.

AutoML approaches, which would become more widespread in parallel with the use of ML in every field of life, would be able to offer more effective uses considering the characteristics of the models in parallel with the results obtained in this study and to reduce the computational cost.

This study has limitations. Using two kernels for the Support Vector Regression, implementing fixed structures for deep LSTM, NN, and DNN, and optimizing the mentioned points could increase the obtained results by the models. Furthermore, considering the various k numbers in k-fold cross-validation technique might provide additional information for the analysis of the models' behaviours.

Comparing machine learning models is a challenging and complicated task due to a large number of models, excessively varied datasets, and different kinds of training strategies.

In this study, we performed a total of 680 experiments using 15 different datasets. Ten benchmark machine learning models were trained with varied training strategies using three different train/test ratios of the hold-out method and five-fold cross-validation.

The results obtained in this study have shown that each model has its unique weaknesses and strengths that could be considered in the regression implementations in order to determine the optimal model for a particular application.

Although it has a significant data dependency to produce consistent and optimal results, the recurrent neural network, deep LSTM, significantly outperformed other considered models almost in all experiments even though it is high-sensitive to the change of training data.

Linear Regression is still an essential model for regression implementations, especially with highly correlated data. Tree-based ensemble models, particularly Gradient Boosting and Extreme Gradient Boosting, are the models that could achieve reliable and consistent results which are low-sensitive to the change of training data. On the other hand, neural-based models produced stable results but were not superior, and more training data is required for the deeper architectures. The support vector regression models achieved their superior results in the minimized number of attributes and instances; however, the data-dependency of the models complicated the implementation of the models.

It was observed that the different hold-out ratios do not significantly affect the model performances, and using a 60:40 hold-out is more beneficial to the models. However, training of the models with cross-validation has a considerable impact on the prediction abilities, and the analysis should be performed not by considering fixed and randomized training ratios.

The data that support the findings of this study are openly available in UCI Machine Learning Repository at https://archive.ics.uci.edu/ [14–22].

Boran Sekeroglu ([email protected]): Conceptualization, Methodology, Software, Writing— original draft preparation, Writing—review and editing.

Yoney Kirsal Ever ([email protected]): Conceptualization, Project administration, Methodology, Writing—original draft preparation, Writing—review and editing.

Kamil Dimililer ([email protected]): Conceptualization, Methodology, Software, Writing—original draft preparation, Writing—review and editing.

Fadi Alturjman ([email protected]): Conceptualization, Methodology, Writing—review, and editing.