Increasing the explainability and success in classification: many-objective classification rule mining based on chaos integrated SPEA2

Multi/Many objective optimization

Multi/many objective optimization (MOO) deals with problem solutions that aim to optimize multiple conflicting goals simultaneously. The basic principle in MOO is to identify a solution set created by the trade-off between different targeted objectives. The number of conflicting objectives allows the relevant problem to be classified as multi- or many-objective. If the number of objectives is two or three, the problem is called multi-objective, and for more objectives, it is called many-objective. A plethora of methodologies have been proposed in the literature for the resolution of MOO problems (Stewart, Palmer & DuPont, 2021; Taha, 2020). Classical solution techniques, such as weighted sum, lexicographic and ɛ-Constraint, evaluate the objectives of the problem by placing them according to certain weights or in order. In these methodologies, which are referred to as prior approaches, decision makers are involved in the solution process at the outset. Decision makers can prioritize goals or reduce multiple goals to a single goal using certain ratios. Conversely, the involvement of decision makers in these methods introduces subjectivity and limitations to the exploration process. While the subjective preferences of the decision maker may result in suboptimal or biased solutions, these preferences may also lead to a narrow focus on a specific region of the search space. Another approach is the posteriori approach, also known as the Pareto-based approach. In such approaches, the objectives are treated equally, and a solution set is presented that includes non-dominated results that do not have absolute superiority over each other. The decision maker then evaluates this solution set and reaches a conclusion. These approaches are suitable for problems where the decision maker does not have to have information about the goals or where the desired goals are of equal importance. The non-domination criterion between candidates is of significant importance in the creation of the Pareto-front. In Pareto-based approaches, the domination criterion is of decisive importance. In a maximization problem such as the one presented in this study, the domination criterion is expressed as in Eq. (1). If this criterion is met, the solution vector s 1 → is said to dominate the solution vector s 2 →. In this context, the variable k represents the number of objectives, the variable i represents the relevant objective index in the solution vector, and the function f . represents the fitness function that determines the fitness value for the candidate. (1)∀ i ∈ 1 , 2 , 3 , … , k , F s 1 i ≥ f s 2 i ∩ ∃ i ∈ 1 , 2 , 3 , … , k : f s 1 i > f s 2 i .

In addition to prior and posterior methods, there are also progressive methods in which the decision maker or designer can intervene throughout the current iteration and hybrid approaches where all these methods are used together (Luo et al., 2022). The choice of method is determined by the type of problem, the decision maker/designer and the importance of the targeted goals. However, in NP-hard (non deterministic polynomial time) optimization problems where deterministic solutions are inadequate, the increase in the number of objectives will be an important factor in determining the type of method to be chosen. In such complex problems, posteriori approaches may be preferred instead of priori approaches due to the subjectivity problem. In particular, posteriori methods using metaheuristic mechanisms are very effective in solving such NP-hard problems. Algorithms such as SPEA2 (Zitzler & Thiele, 1999; Zitzler, Laumanns & Thiele, 2001), NSGA-II/III (Non-dominated Sorting Genetic Algorithm-II/III) (Deb et al., 2002; Deb & Jain, 2014) and MOPSO (multi-objective particle swarm optimization) (Junjie et al., 2009) have been applied to a wide range of MOO problems, demonstrating their effectiveness in addressing the challenges posed by these complex problems.

In this study, the posteriori solution type was selected in order to exclude user influence in rule extraction. Furthermore, the extensive feature space employed and the high number of objectives led us to the conclusion that posteriori solutions utilizing metaheuristic mechanisms were the most appropriate. Literature review and previous experience of the authors show that SPEA2 algorithm has some advantages in this type of problems. For example, SPEA2 has better resource utilization cost since it does not perform multiple Pareto management like NSGA-II and III. It also provides a more successful diversity since it uses k-nearest based density estimation instead of the grid based distance criterion in MOGWO (Many Objective Grey Wolf Optimization) (Yildirim, 2022).

Strength Pareto Evolutionary Algorithm 2 (SPEA2)

The classical SPEA (Zitzler & Thiele, 1999) is a state-of-the-art MOO algorithm that identifies Pareto solution sets for problems containing multiple conflicting objectives. The classical SPEA, which employs evolutionary mechanisms, always considers the dominance and density factors. The dominance factor controls the superiority of the candidates over each other, while the density factor observes the distribution of the solutions found in the solution space and helps to create a balanced Pareto front. However, classical SPEA may require more generations to reach optimal solutions, which may result in slower convergence. Furthermore, the simple distance-based density estimation it employs may also diminish the efficacy of Pareto-front formation. Consequently, SPEA developers have proposed the SPEA2 algorithm, which represents an enhanced iteration of this algorithm (Zitzler, Laumanns & Thiele, 2001). While SPEA2 offers faster convergence than SPEA, utilizing elitism and tournament mechanisms, it also enables more successful Pareto-front formation with an advanced density estimation method.

SPEA2 has an initial population (P₀) and an empty archive (P ¯ 0) in the first iteration (t = 0). These two entities have a dynamic structure and are updated at each iteration according to dominance evaluations and algorithm mechanisms (P_t and P ¯ t). Generation control is carried out according to the fitness values of both population and archive individuals. The fact that individuals dominated by the same archive individuals have similar fitness values negatively affects density. To prevent this, SPEA-2 takes into account both the dominance and being dominated statuses of each candidate. The number of solutions dominated by the ith individual in P_t and P ¯ t is its Strength value (S i) and is calculated with Eq. (2). Here, the symbol . represents cardinality, the symbol +represents multiset union, and the symbol ≻ represents Pareto dominance. (2)S i = j j ∈ P t + P ¯ t ⋀ i ≻ j .

The S i value is employed to calculate the raw fitness value, denoted by (R i), which elucidates the dominance status of the ith individual. The greater the value of R i determined by Eq. (3), the more the related candidate is dominated. If R i isequal to zero, the candidate is considered to be non-dominated. (3)R i = ∑ j ∈ P t + P ¯ t , j ≻ i S j .

The raw fitness value provides an indication of the dominance status of the candidate, but may be insufficient for accurate evaluation if the number of non-dominated candidates is high. Therefore, density information is also used in addition to the raw fitness value. The k -th nearest neighbour technique is used for density estimation (Silverman, 1986). Consequently, the intensity at a given point is a decreasing function of the distance to the k-th nearest data point. For each ith individual, the distances of all other individuals in the objective space to the j-th individual are calculated both in the archive and in the population and stored in a list. In the list organized in ascending order, the k-th value gives the desired distance and is denoted by σ i k. The k value depends on the population size (N) and the archive size (N ¯) and is calculated with k = N + N ¯. The density value of a candidate is inversely proportional to σ i k and is found by Eq. (4). Here, θ is a constant used to ensure that the denominator value is positive, and generally θ = 2. Consequently, a candidate’s fitness value (F i isdetermined by Eq. (5) according to raw-fitness (R i) and density values D(i)). (4)D i = 1 σ i k + θ (5)F i = R i + D i .

The next generation is created (environmental selection) according to the F i values of the candidates. Initially, non-dominated individuals with fitness values less than 1 are copied to the next generation’s archive, as illustrated in Eq. (6). During this process, the archive size of the next generation is taken into account. If P ¯ t + 1 = N ¯, the environmental selection process is completed. In the case of P ¯ t + 1 < N ¯, the previous archive and the best N ¯ − P ¯ t + 1 non-dominated individual in the population are added to the next generation archive. In the case of P ¯ t + 1 > N ¯, the candidate is removed from the next generation archive until P ¯ t + 1 = N ¯ is achieved. (6)P ¯ t + 1 = i i ∈ P t + P ¯ t ⋀ F i < 1 .

The candidate extraction process considers the distance (σ i k) between individual i in set P ¯ t + 1 and its k-th nearest neighbour. The individual with the shortest distance is selected, as demonstrated in Eq. (7). In the event that there are multiple candidates with the same distance, the equality is broken by taking the second smallest distances into account. (7)i ≤ d j ⇔ ∀ 0 < k < P ¯ t + 1 : σ i k = σ j k ∨ ∃ 0 < k < P ¯ t + 1 : ∀ 0 < l < k : σ i l = σ j l ⋀ σ i k < σ j k .

The Simulated Binary Crossover (SBX) operator was employed as crossover operator (Deb & Agrawal, 1994). SBX generates two new individuals according to the probability distributions of the parent individuals (pr₁, pr₂). The polynomial probability distribution that ensures that the newly produced individuals are related to their parents is a function of the spread factor (β) and is expressed as in Eq. (8). The similarity of the new individuals to the parent individuals is determined by the constant φ (φ ∈ R⁺) . A high φ value increases the similarity of the new individuals to the parent individuals. The spread factor value for each variable h of the candidate solution vector is distinct (β_h) and is determined by Eq. (9). Here, ϑ_c, represents the random constant between 0 and 1 generated for the variable h. At the conclusion of the crossover process, both newly produced individuals (sv₁, sv₂) are identified by Eqs. (10)–(11).

(8)P β = 0 . 5 φ + 1 β φ , i f β ≤ 1 , 0 . 5 φ + 1 1 β φ + 2 , o t h e r w i s e (9)β h = 2 ϑ c 1 φ + 1 , i f ϑ c ≤ 0 . 5 , 1 2 1 − ϑ c 1 φ + 1 , o t h e r w i s e (10)s v 1 = 0 . 5 1 + β h p r 1 + 1 − β h p r 2 (11)s v 2 = 0 . 5 1 − β h p r 1 + 1 + β h p r 2 .

The mutation operation is performed for all vector variables of the selected individual. The amount of change in the solution vector in each variable is calculated with (Δc_h) Eq. (12) and added to the relevant variable. The amount of change depends on the predefined mutation coefficient (γ) and a random number (ϑ m h) generated for the variable h . (12)Δ c h = 2 ϑ m h 1 γ + 1 − 1 , i f ϑ m h ≤ 0 . 5 , 1 − 2 ϑ m h 1 γ + 1 , o t h e r w i s e .

Many objective chaotic rule-based SPEA2 and evaluation principles

This study proposes a novel many-objective metaheuristic method for machine learning-based rule extraction. To this end, the SPEA2 algorithm has been adapted to address the rule inference problem. This necessitates two fundamental changes to the SPEA2 algorithm. The first is to design an appropriate representation form for candidate solutions. The second is to determine how to evaluate the objective values according to the rule compliance. In the metaheuristic-based rule extraction technique, the representation forms and search technique of the candidate solutions to be used are of significant importance. In the proposed method, the candidate solution employs a representation format comprising three distinct vectors. Consequently, candidate solution Vi comprises sub vectors V → i b , V → i l a n d V → i u (i = 1, 2, …, n). V i b = v 1 b , v 2 b , … , v j b , … , v n b is a binary vector, indicating which attributes will be utilized in the rule that the candidate will develop. If the value of v j b is greater than the predefined λ threshold value, as indicated in Eq. (13), the j-th attribute is included in the rule that the candidate will develop. (13)v j = 1 F e a t u r e j i s u s e d i n t h e r u l e i f v i b > λ , λ ϵ 0 , 1 0 o t h e r w i s e .

Throughout the iterations, candidates obtain two values, one lower and one upper, for each attribute of the search space. A candidate keeps the upper/lower values found for each attribute in V i l = v 1 l , v 2 l , . . v j l , . . v n l a n d V i u = v 1 u , v 2 u , . . v j u , . . v n u vectors (v j l , v j u ϵ R). In an iteration, the upper and lower values found for the jth attribute must fall between the minimum (L^l and maximum (U^u values in the search space of the relevant attribute. Therefore, condition L j l ≤ v j l < v j u ≤ U j u must be constantly checked. At each iteration, candidates are presented with a single rule, which is updated throughout the process. The v j b value plays a pivotal role in the formation of these rules, while v j b and v j b provide context and meaning. To illustrate, consider a search for a class “C” that includes attributes 4., 7., and 10. in the ith iteration v 4 b , v 7 b , and v 10 b > λ . In this case, the candidate’s rule expression for the data set attributes (F₁, F₂, …, F_n) will be as in Eq. (14). This type of rule inference has two important contributions to increasing interpretability. The first is to show which attributes contribute to success and in what ranges. The second is that no discretization mechanism is needed to determine the lower and upper limits. Thus, there is no loss of information. (14)i f v 4 l ≤ F 4 ≤ v 4 u a n d v 7 l ≤ F 7 ≤ v 7 u a n d v 10 l ≤ F 10 ≤ v 10 u t h e n C .

All candidates assess the rules they have derived throughout the iterations in accordance with their performance in the search space. A candidate’s performance is contingent upon the degree of consistency exhibited by the rule they have derived for each class of data. In the evaluation of rules, the “if” and “then” components of the rule produced by the candidate solution V_i are taken into account. The attribute values and class of the data being compared determine which components of the rule are deemed to be consistent with them. Following the comparison in accordance with Table 1, the true positive (TP), true negative (TN), false positive (FP) and false negative (FN) metrics of the candidate solution are updated. These metrics are then employed in order to calculate the candidate’s goal values.

In multi-objective optimization methods, the contradiction between the selected objectives is of significant importance. The trade-offs resulting from this contradiction help to form the Pareto-front. Pareto solutions assist decision makers in interpreting trade-offs between objectives. Given that this study is a data mining optimization problem, it is of paramount importance to select the most appropriate metrics. In data mining, precision and recall are metrics that may conflict with one another. Precision represents the ratio of true positive predictors to all positive predictions. Recall is defined as the ratio of true positive predictions among all actual positive predictions. Consequently, in certain instances, an increase in one of these metrics may result in a decrease in the other. Two additional metrics that may conflict are accuracy and F1-score. Accuracy is a measure of the overall accuracy of the model’s predictions, whereas F1-score represents the harmonic average of the precision and recall metrics. The conflict between these two metrics is particularly evident in unbalanced data sets. In this case, it would be misleading to consider only the accuracy metric. On the other hand, since F1-score combines precision and recall into a single metric, the trade-off between these two metrics cannot be seen. In data mining, it is possible to present Pareto solutions that show all trade-offs between these metrics to the decision maker with many-objective optimization. Consequently, in this study, Pareto fronts were constructed on the basis of the trade-offs between these four metrics.

The objective values for a candidate solution are calculated with the previously obtained total TP, TN, FP and FN values at the end of each iteration. Equations (15)–(18) illustrate the calculation of the Accuracy (Acc), Precision (Pre), Recall (Rec) and F1-score (F1) objectives, respectively. The i-th candidate solution maintains these four metrics in the objective vector O s → = A c c , P r e , R e c , F 1. The dominance relationship between candidates is determined by comparing these objective vectors

(15)A c c = T P + T N T P + T N + F P + F N (16)P r e = T P T P + F P (17)R e c a l l = T P T P + F N (18)F 1 = 2 ∗ P r e ∗ R e c P r e + R e c .

The randomness mechanism is of paramount importance in the exploration and exploitation processes of metaheuristic methods. Some traditional random number generation functions may exhibit statistical weaknesses, such as poor distribution properties, insufficient uniformity, or correlations between generated values. These weaknesses can lead to biased results, especially in simulations and modelling. In order to address this issue, researchers have examined chaotic maps due to their deterministic behaviour, longer periods and advanced statistical properties. In Yildirim et al. (2021), the authors of this paper tested the performance of chaotic functions in metaheuristic approaches. As a result of the study, it was observed that the tent function can be partially more resistant to local minima (or maxima). For this reason, tent function is also preferred in random process management in this study. It is well documented in the literature that metaheuristic algorithms employing chaotic maps yield competitive outcomes and offer flexibility, particularly during the exploration phase. This study opted to utilize chaotic maps in the randomness mechanism of SPEA2 due to the flexibility they afford, especially during the exploration process. Given the promising results demonstrated by the authors in their previous studies, the tent chaotic map (Li et al., 2017), whose mathematical expression is given in Eq. (19), was employed. This adapted version was designated as chaotic SPEA2 (CRb-SPEA2). (19)X n + 1 = ∂ X n , X n < 1 2 ∂ 1 − X n , 1 2 ≤ X n .

The chaotic map is dependent on the real ∂ constant and initial (X₀) values. By setting these two coefficients correctly, the sequence produced exhibits chaotic behaviour. In CRb-SPEA2, the V → i b , V → i l a n d V → i u initial vectors of the candidate solution, ϑ_c and ϑ m h values in crossover and mutation operations are generated by the Tent chaotic map function. While the pseudo code demonstrating all these processes of the rule extraction method with CRb-SPEA2 is presented in Algorithm 1 , the basic mechanisms of the algorithm are shown in Fig. 2.