Supervised methods¶

In this practice we are going to use the classic iris data set, trying to classify different varieties of the iris flower according to the length and width of its petals and sepals. We will try to optimize different metrics and see how the different models classify the points and with which we obtain greater precision.

The practice is structured as follows (in which the score for each part is detailed).

0. dice loading
1. Exploratory data analysis
2. k nearest neighbors
3. Support vector machines
4. decision tree
5. Random forest
6. Neural networks

Important: Each exercise can take several minutes to execute, so the delivery must be done in notebook and html format, where the code and results can be seen, along with the comments for each exercise.

0. Load from data set¶

1. Exploratory data analysis¶

We will explore our data set. To do this, we will carry out the following inspections:

• We will look at the size of the dataset and see if there are null values
• We will calculate the main statistics of the dataset (that is, number of records, mean value, standard deviation and quartiles)
• We will see the distribution of the classes (i.e., if the dataset is balanced)
• We will make some visualizations to get an idea.

We have put in comment form the analyzes that you would have to do

Univariate analysis is the simplest way to analyze data. It does not deal with causes or relationships (unlike regression) and its main purpose is to describe and find patterns in the data.

To do this we are going to do what is known as Distribution Plots (or histograms). Distribution plots are used to visually evaluate how data points are distributed with respect to their frequency. Typically, data points are grouped into bins and the height of the bars indicates the number of data points (frequency of occurrence).

• If we use PetalLengthCm we can separate the iris-setosa species.
• We cannot use SepalLengthCm or SepalWidthCm because everything is mixed and we cannot separate the flowers.
• PetalWidthCm is also not separated correctly.

The only conclusion is that with PetalLengthCm we can separate the iris-setosa species.

A box plot (box plot) is a standardized way of displaying the distribution of data based on a summary of five numbers ("minimum", first quartile (Q1), median, third quartile (Q3), and "maximum"). The box plots They tell us about outliers and what their values are. It can also tell us if the data is symmetrical, clustered, and skewed. To do this we can use the function boxplot of seaborn.

The Violin Plot is a method to visualize the distribution of numerical data of different variables. It is similar to the box plot (box plot) but with a rotated plot on each side that provides more information about the density estimate on the y-axis. The density is reflected and flipped and the resulting shape is filled creating an image that resembles a violin. The advantage of a violin plot is that it can show nuances in the distribution that are not noticeable in a box plot. On the other hand, the boxplot more clearly shows the outliers in the data. Violin plots typically contain more information than box plots although they are less popular.

Now let's plot the violin plots for our iris data set. For this we can use the function violinplot of seaborn . For its interpretation, let us take into account that the rectangle that appears in the violin plot is equivalent to the information given to us by the box plot and that the white circle tells us where the 50th percentile is.

Finally we will carry out a small study using a pair-plot to visualize possible relationships between our variables (pairwise).

In this case we will use the function pairplot from the bookstore seaborn.

If we look at the dispersion graphs that relate the characteristics of the sepal field we will see how they are distributed almost uniformly (especially those corresponding to Iris setosa), while those corresponding to versicolor and virginica have somewhat similar qualities so they sometimes overlap.

On the other hand, if we compare the petal, it is a much more uniform distribution compared to the sepal.

If we analyze the violin plot it shows that Iris Virginica has a higher mean value in petal length, petal width and sepal length compared to Versicolor and Setosa. In another sense, Iris Setosa has the highest mean value of sepal width. We can also see a significant difference between the length and width of Setosa's sepal versus the length and width of its petals. That difference is smaller in Versicolor and Virginica. The violin diagram also indicates that the weight of Virginica sepal width and petal width are highly concentrated around the median.

Regarding the box plot, the isolated points that can be seen are the outliers in the data. Since these are very few in number, it would not have any significant impact on our analysis.

Model application¶

Before applying any model, we have to separate the data between sets of train and test. We will always work on the set of train and we will evaluate the results in the set of test.

It is important to keep in mind that our target variable is categorical. The classifier KNeighborsClassifier does not accept type tags string, so we must transform these labels into numbers (this is what we know as Label encoding).

To do this we will divide the dataset into two arrays: X (characteristics) and Y (labels) and we will apply the following correspondence:

• Iris-setosa corresponds to 0
• Iris-versicolor corresponds to 1
• Iris.virginica corresponds to 2

Throughout the exercises we will learn to graphically visualize the decision boundaries that the different models return to us. For this purpose we will use the function defined below (which will help us draw the respective decision boundaries throughout the entire PEC), which follows the following steps:

• Create a meshgrid with the minimum and maximum values of x and y.
• Train the classifier with the values of the meshgrid.
• Makes a reshape of the data to obtain the correct format.

After this process, we can now make the graph of the decision boundaries and add the real points. This way we will see the areas in which the model considers to be of one class and those in which it considers to be of the other. By putting the real points on top we will see if it classifies them correctly.

1. k nearest neighbors ¶

The first algorithm that we will use to classify the points is k-nn. In this exercise we will adjust two hyperparameters to try to obtain greater precision:

• k: the number of neighbors considered to classify a new example. We will try all values between 1 and 10.
• pesos: importance given to each neighbor. In this case we will try two options: uniform weights, where all neighbors are considered equal; and weights according to distance, where the closest neighbors have more weight than the most distant neighbors.

To decide the optimal hyperparameters we will use the technique of grid search, which consists of training a model for each possible combination of hyperparameters and we will evaluate it using cross validation with 4 stratified partitions. Subsequently, we will choose the combination of hyperparameters that has obtained the best results.

To solve the first part you can use the modules GridSearchCV and KNeighborsClassifier of sklearn. For viewing the heatmap you can use the function pivot that the library allows Pandas.

• The best solution has been given with a value k = 9 and the weights calculated with uniform. These results may vary if the training and test set is modified, that is, these results vary by execution of the cell that generates the division.
• The minimum value is 74.17 and the maximum value is 81.67, that is, the difference is 7 percentage points, with standard deviations of the order of 0.5 percentage points we can affirm that there are options that are clearly better than others.
• I have observed that except for k=1 the weights do not matter, for the rest it is significant. At k=1 the new examples are classified with the nearest neighbor class.
• It seems that the precision depends more on k than on the type of weight. Although we observe that the weights with 'uniform' have a lower standard deviation.

The results have not been very good. The decision border does not seem clear, there are many areas that have not been properly classified.

The resulting decision boundary when we have used the information based on the petals seems more accurate and accurate than the information provided with the sepal. When we have used sépal, "islands" appear and the boundaries between classes are not clear.

The advantages of this algorithm are the following:

• Non-parametric. It makes no explicit assumptions about the functional form of the data, avoiding the dangers of the underlying distribution of the data.

• Simple algorithm. To explain, understand and interpret.

• High precision (relative). It is quite high but not competitive compared to better supervised learning models.

• Insensitive to outliers. Accuracy can be affected by noise or irrelevant features.

The disadvantages of this algorithm are:

• Instance based. The algorithm does not explicitly learn a model, instead choosing to memorize the training instances which are later used as knowledge for the prediction phase. Concretely, this means that only when a query is made to our database, that is, when we ask it to predict a label given an input, will the algorithm use the training instances to spit out an answer.

• Computationally expensive. Because the algorithm stores all the training data.

• High memory requirement. Stores all (or almost all) training data.

2.Support Vector Machine ¶

In this second exercise we will classify the points using the SVM algorithm with different types of kernel. In this case we will use a kernel radial, a kernel linear and a kernel polynomial of degree 3. We will again use a grid search (grid search) for the optimization of hyperparameters.

In this case the hyperparameters to optimize are:

• C: is the regularization, that is, the penalty value of classification errors. We will try the values: 0.01, 0.1, 1, 10, 50, 100 and 200.
• gamma: coefficient that multiplies the distance between two points on the kernel. A roughly, the smaller the gamma, the more influence two nearby points have. We will try the values: 0.001, 0.01, 0.1, 1 and 10.

As in the previous case, to validate the performance of the algorithm we will use cross validation (cross-validation) with 4 stratified partitions. In this case we will only do it for the height-width characteristics of the petal.

Additional material that may help you:

• Introduction to Statistical Learning. Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani

• Support Vector Machines Succinctly. Alexander Kowalczyk

• A Practical Guide to Support Vector Classification. Chih-Wei Hsu, Chih-Chung Chang, and Chih-Jen Lin

• Tutorial on Support Vector Machines (SVM). Enrique J. Carmona Suárez

• A Gentle Introduction to Support Vector Machines in Biomedicine. Alexander Statnikov, Douglas Hardin, Isabelle Guyon, Constantin F. Aliferis

You can use the modules GridSearchCV and svm of sklearn. Analyze what influence the hyperparameters C and gamma once the best hyperparameters have been calculated. For each type of kernel, make predictions for each of them, calculate their confusion matrix and finally draw their decision boundaries.

• The parameters that have obtained the best results have been C = 100 and gamma = 0.01 for the executed iteration. Each iteration may vary because the training and test sets are modified.
• We find a maximum difference of 5 percentage points, that is, there is a considerable difference and there are notable variations with the standard deviations. This makes it clear that some combinations are better than others.
• The best solutions are given with gammas between 0.001 and 1. It can be seen that in the best solutions C is quite variable, so we can conclude that the gamma parameter has more weight. It has also been detected that the solution gamma = 10 provides worse results.

The decision boundaries are very fluid and defined. The three well-differentiated classes are observed with only two errors.

3. Decision trees ¶

In this third exercise we will draw the decision boundaries of the two types of attributes (sepals and petals). We will see what precision we obtain with the decision trees. We will map the tree and analyze it.

To draw the tree we will need to install the library graphviz. To do this from terminal we will write the following command:

sudo apt-get install graphviz

If anyone uses the Conda environment, it can also be installed from this environment.

Decision trees are a method used in different disciplines as a prediction model. These are similar to flowcharts, in which we arrive at points where decisions are made according to a rule.

In the field of machine learning there are different ways to obtain decision trees, the one we will use this time is known as CART: Classification And Regression Trees. This is a supervised learning technique. We have a target variable (dependent) and our goal is to obtain a function that allows us to predict, from predictor variables (independent), the value of the target variable for unknown cases.

As the name indicates, CART is a technique with which classification and regression trees can be obtained. We use classification when our target variable is discrete, while we use regression when it is continuous. We will have a discrete variable, so we will do classification.

In general, what this algorithm does is find the independent variable that best separates our data into groups, which correspond to the categories of the target variable. This best separation is expressed with a rule. Each rule corresponds to a node.

To do this, we must make sure that we have the library installed in our environment. graphviz.

The tree interpretation of this decision tree would be: if the petal width is less than 0.8 centimeters, then the iris flower belongs to the iris-setosa variety. If, on the other hand, the length of the petal is greater than 0.8 centimeters and greater than 1.75 it belongs to Iris.virginica. If it is greater than 0.8 centimeters and less than 1.75, we will look at the length of the petal. If the length is greater than 5.45, it is directly Iris.virginica. If the length is less than 5.45 cm and the width is less than 1.65 then it is Iris-versicolor and if it is greater than 1.65 cm Iris.virginica

• Gini index: It is used for attributes with continuous values (price of a house). This cost function measures the “degree of impurity” of the nodes, that is, how disordered or mixed the nodes are once divided. Gini is a measure of impurity. When Gini is 0, it means that that node is totally pure, that is, this should be our goal.

Among other data mining methods, decision trees have several advantages:

• Easy to understand and interpret. People are able to understand decision tree models after a brief explanation.
• Requires little data preparation. Other techniques often require data normalization, dummy variables need to be created and blank values ​​need to be removed.
• Able to handle both numerical and categorized data. Other techniques are generally specialized in analyzing data sets that have only one type of variable. (For example, ratio rules can only be used with nominal variables, while neural networks can be used only with numerical variables.)
• Use a white box model. If a given situation is observable in a model then the condition is easily explained by Boolean logic. (An example of a black box model is an artificial neural network since the explanation of the results is difficult to understand.)
• It is possible to validate a model using statistical tests. This makes it possible to take into account the reliability of the model.
• Robust. It performs well even if its assumptions are violated by the true model from which the data was generated.
• Works well with large data sets. Large amounts of data can be analyzed using standard computing resources in a reasonable time frame.

Disadvantages

• Overfitting
• Loss of information when categorizing continuous variables
• Instability: A small change in data can greatly modify the structure of the tree. Therefore the interpretation is not as direct as it seems.

Decision boundaries are fluid and defined. The three well-differentiated classes are observed with only two errors.

4.Random forest ¶

In this fourth section we will classify the points using a Random forest. We will use, as in the previous cases, a grid search (grid search) stop adjusting the hyperparameters.

In this case, the hyperparameters that we must adjust are:

• max_depth: The maximum depth of the tree. We will explore values between 6 and 12.
• n_estimators: name of trees. We will explore the values: 10, 50, 100 and 200.

As in the previous case, we will use cross validation (cross-validation) with 4 stratified partitions to validate the performance of the algorithm with each combination of hyperparameters.

You can use the modules GridSearchCV and RandomForestClassifier of sklearn.

• We have obtained the best result with max_depth = 12 and n_estimators = 10 (it may vary depending on the distribution of the training and test set)
• The differences are almost 2 percentage points at most, that is, it is quite small.
• I have not observed in this iteration any relationship that makes me see a greater influence. I expected that the larger the n_estimators, the greater the precision, but this has not been the case.

In the previous practice we were studying the influence of some of the parameters of the Random forest, among them the parameter max_depth and how too large a tree depth could cause what we know as overtraining (overfitting).

In this section we are going to use the interactive capabilities that the library offers us. plotly to view effects of overfitting and how the decision boundary changes because of it. To do this, we will create graphs of two Random Forest classifiers, the first with a reasonable tree depth (max_depth=4) and the second presenting a clear overfitting (e.g. max_depth=300).

Invoke the code Plotly It is very similar to that of Matplotlib to generate the decision boundary. We will need a mesh Numpy to form the basis of our surface graphs, as well as the method predict of the learning model to populate our frontier with data.

Remember that you will have to install the release Plotly. This can be done by:

pip install plotly

Note: You don't need to do it for all four characteristics (height-width of the petal and height-width of the sepal), doing it for two would be enough.

Advantages of Random Forest

• There are very few assumptions and therefore data preparation is minimal.
• It can handle up to thousands of input variables and identify the most significant ones. Dimensionality reduction method.
• One of the outputs of the model is the importance of variables.
• Incorporates effective methods to estimate missing values.
• It is possible to use it as an unsupervised method (clustering) and outlier detection.

Disadvantages of Random Forest

• Loss of interpretation
• Good for classification, not so much for regression. The predictions are not continuous in nature.
• In regression, you cannot predict beyond the range of values of the training set.
• Little control over what the model does (black box model for statistical modelers)

5. Neural networks ¶

In this last part of the PEC we are going to use the library Keras. To do this we will compare the networks with the regular dense layer (regular Dense Layer) with a different number of nodes, using as activation function softmax and as an optimizer Adam.

To do this we will have to make sure we have the libraries Tensorflow and Keras installed.

To do this from terminal we will write the following command:

pip install tensorflow

pip install keras

If anyone uses the Conda environment, it can also be installed from this environment.

On the other hand, this is the most complicated section of the entire practice and the one with which you are least familiar. For this reason, throughout the section we will give you a series of links to concepts and examples that will help you better understand what we are doing. It is highly recommended Carefully read the links (marked in blue) and references indicated and understand the theoretical explanations and code examples provided.

To prepare the data, we will simply use the OneHotEncoder to encode the entire features in a One-hot vector and we will use a StandardScaler to remove the mean and scale the features to unit variance. Finally, we will use train_test_split to compare our results later.

We configure our neural network models¶

To do this, we define a function that will be responsible for creating our models (in this specific case we are going to create three models that we will call Model1, Model2 and Model3). We will use as activation function the ReLu function and how loss function the categorical_crossentropy function

For more depth on the study of the activation functions:

• https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html
• https://towardsdatascience.com/activation-functions-neural-networks-1cbd9f8d91d6
• https://medium.com/the-theory-of-everything/understanding-activation-functions-in-neural-networks-9491262884e0

For greater depth in the study of the loss functions:

• https://machinelearningmastery.com/how-to-choose-loss-functions-when-training-deep-learning-neural-networks/
• https://www.machinecurve.com/index.php/2019/10/22/how-to-use-binary-categorical-crossentropy-with-keras/

Training the models¶

Now we move on to training. For this we will use TensorBoard Callback1, 2, 3, 4 so you can explore the model and outputs in detail.

Viewing results¶

Knowing that the ROC graph compares the false positive rate with the true positive rate, I have decided to use this value to see which model has obtained better results. The results obtained have been very good, in all models we have obtained values ​​close to 1. I think that model 3 is the best of all.

Furthermore, we calculate for each model the Area Under the Curve (AUC), where auc = 1 is a perfect classification and auc = 0.5 is a random guess and, in this case, we obtain results very close to 1.