CS360 Lab 7: sklearn

Due: Wednesday, Nov 4 Friday, Nov 6 at 11:59pm

Grace period: will be collected Thursday Saturday at midnight

Overview

The goals of this lab:

Optional engagement:

Log in and check for libraries

First make sure you can log in to the GPU lab machines (see Piazza for which machines have GPUs and for instructions about using the Haverford VPN).

ssh <username>@<machine-name>.cs.haverford.edu

After logging on to the lab machine, open your .bashrc file using

emacs .bashrc

There should be a lot of stuff already there. At the end, add the following lines:

export PATH=/packages/cs/python3.7.7/bin:/usr/local/cuda-10.1/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-10.1/lib64:/usr/local/cuda/extras/CUPTI/lib64:/packages/cs/python3.7.7/lib

Then use Ctrl-x Ctrl-s to save and Ctrl-x Ctrl-c to close the file. Then log out and log back in.

Finally, check for the following libraries and make sure all import with no errors.

$ python3
Python 3.7.7 (default, Jul 14 2020, 14:39:22)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> import sklearn
>>> import tensorflow

Please post on Piazza if you have any issues doing this part! If you cannot access the lab machines easily, make sure you have the above libraries installed on your own machine:

pip3 install numpy
pip3 install sklearn
pip3 install tensorflow


Practice using sklearn

Put the following code in run_pipeline.py. At the end of this step you should have an accuracy for KNN and an accuracy for Random Forests on the cancer dataset. In your README, briefly comment on which method was better. For this short exercise we will use the Wisconsin Breast Cancer dataset.

When importing sklearn modules, avoid importing with the * - this imports everything in the library (and these functions can be confused with user-defined functions). Instead, import functions and classes directly. For example:

from sklearn.datasets import fetch_openml, load_breast_cancer

Next we will look at the data:

data = load_breast_cancer()

X = data['data']
y = data['target']
print(X.shape)
print(y.shape)

which outputs 569 examples with 30 features each:

(569, 30)
(569,)

Now we fill fit two classifiers to this data (KNN and Random Forests) and compare the accuracy. Throughout this part I’ll assume we take the first 400 datasets to be “training” and the remaining to be “testing” (create variables to save the train and test). In your code, I would recommend creating a function that takes in a classifier and a dataset and returns an accuracy, but as long as this part works and makes sense, I don’t mind what the code looks like.

K-Nearest Neighbors

First import the KNeighborsClassifier and check out the documentation. Then create an instance of this classifier with K=3.

from sklearn.neighbors import KNeighborsClassifier

knn_clf = KNeighborsClassifier(n_neighbors=3)

Then fit the classifier to the training data and predict the labels of the test data:

knn_clf.fit(X_train,y_train)
knn_clf.predict(X_test)

Random Forests

We will now do the same steps for the RandomForestClassifier using the default parameters:

from sklearn.ensemble import RandomForestClassifier

rf_clf = RandomForestClassifier()

Then fit and predict as before:

rf_clf.fit(X_train,y_train)
rf_clf.predict(X_test)

Finally, compare the predictions to the true values and report accuracy for both methods (in your README). Optional: look into changing some of the hyper-parameters for each classifier and see what produces the best results!

(OPTIONAL STARTING HERE) Extra Datasets and Command line arguments

Throughout the lab, consult the sklearn documentation frequently to find the appropriate methods for this lab. Reading and using documentation is a very important skill that we will practice in Lab 7, Lab 8, and the final project. Make sure you really understand each line of code you’re writing and each method you’re using - there are fewer lines of code for this lab, but each line is doing a lot!

Rather than parse and process data sets, you will use sklearn’s pre-defined data sets. Details can be found here. At a minimum, your experiments will require using the MNIST and 20 Newsgroup datasets. Both are multi-class tasks (10 and 20 classes, respectively).

The MNIST dataset is very large and takes a lot of time to run, so you can randomly select 1000 examples; you should also normalize the pixel values between 0 and 1 (instead of 0 and 255):

data = fetch_openml('mnist_784', data_home="/home/smathieson/Public/cs360/sklearn-data/")
X = data['data']
y = data['target']
X,y = utils.shuffle(X,y) # shuffle the rows (utils is from sklearn)
X = X[:1000] # only keep 1000 examples
y = y[:1000]
X = X/255 # normalize the feature values

The newsgroup dataset in vector form (i.e., bag of words) is obtained using:

data = fetch_20newsgroups_vectorized(subset='all', data_home="/home/smathieson/Public/cs360/sklearn-data/")

No normalization is required; I also suggest randomly sampling 1000 examples for this dataset as well. The data object also contains headers and target information which you should examine for understanding. For your analysis, it may be helpful to know the number of features, their types, and what classes are being predicted.

Coding Requirements

You will have or create the following files:

The coding portion is flexible - the goal is to be able to execute the experiments below. However, you should keep these requirements in mind:

$ python3 run_pipeline.py -d cancer
$ python3 run_pipeline.py -d mnist
$ python3 run_pipeline.py -d news

You may have cases for each one (since they need to be treated differently). If the user does not enter a dataset, rely on optparse to print a helpful message.

knn_clf = KNeighborsClassifier()
parameters = {"weights": ["uniform", "distance"], "n_neighbors": [1, 5, 11]}
test_results = runTuneTest(clf, parameters, X, y)

Note that the hyper-parameters match the API for KNeighborsClassifier. In the dictionary, the key is the name of the hyper-parameter and the value is a list of values to try.

Experiment 1: Random Forest vs SVM Generalization Error

Using run_pipeline.py, you will run both Random Forests and SVMs and compare which does better in terms of estimated generalization error.

Coding Details

Your program should read in the dataset using the command line, as discussed above. You should specify your parameters and classifier and call runTuneTest (see the above example), which follows this sequence of steps:

  1. Divide the data into training/test splits using StratifiedKFold. Stratified folds balance the number of examples from each class in each fold. Follow the example in the documentation to create a for-loop for each fold. Set the parameters to shuffle the data and use 5 folds. Set the random_state to a fixed integer value (e.g., 42) so the folds are consistent for both algorithms.

  2. For each fold, tune the hyper-parameters using GridSearchCV, which is a wrapper for your base learning algorithms; it automates the search over multiple hyper-parameter combinations. Use the default value of 3-fold 5-fold tuning (so we are essentially performing a cross-validation within a cross-validation).

  3. After creating a GridSearchCV classifier, fit it using your training data. Report the training score using the score method.

  4. Get the test-set accuracy by running the score method on the fold’s test data.

  5. Return a list of test accuracy scores for each fold.

In main(), you should print the test accuracies for all 5 folds for both classifiers (pair up the accuracies for each fold for ease of comparison). The classifiers/hyper-parameters are defined as follows:

Code incrementally, and be sure to examine the results of your tuning (what were the best hyper-parameter settings? what were the scores across each parameter?) to ensure you have the pipeline correct. Since the analysis below is dependent on your results, I cannot provide sample output for this task. However, this is what is generated if I change my classifier to K-Nearest Neighbors using the parameters listed in the previous section (you can try to replicate this using a random_state of 42). Update: in sklearn version 23 the default cv changed, so these results are updated below:

$ python3 run_pipeline.py -d cancer

-------------
KNN
-------------

Fold 1:
{'n_neighbors': 11, 'weights': 'uniform'}
Training Score: 0.9362637362637363
Fold 2:
{'n_neighbors': 11, 'weights': 'uniform'}
Training Score: 0.9494505494505494
Fold 3:
{'n_neighbors': 11, 'weights': 'distance'}
Training Score: 1.0
Fold 4:
{'n_neighbors': 11, 'weights': 'uniform'}
Training Score: 0.9318681318681319
Fold 5:
{'n_neighbors': 5, 'weights': 'uniform'}
Training Score: 0.9473684210526315

Fold, Test Accuracy
1, 0.9385964912280702
2, 0.9122807017543859
3, 0.9298245614035088
4, 0.9649122807017544
5, 0.9557522123893806

Analysis

In Part 1 of your writeup (must be a PDF), you will analyze your results. At a minimum, your submission should include the following type of analysis:

You do not need to go into exhaustive detail, but your analysis should be written as if it were the results section of a scientific report/paper.

Experiment 2: Learning Curves

Using generate_curves.py, you will generate learning curves for the above two classifiers. We will vary one of the hyper-parameters and see how the train and test error accuracy changes.

Coding Requirements

$ python3 generate_curves.py -d cancer
Neighbors,Train Accuracy,Test Accuracy
 1  1.0000  0.9051
 3  0.9561  0.9209
 5  0.9473  0.9227
 7  0.9438  0.9191
 9  0.9429  0.9315
11  0.9385  0.9315
13  0.9385  0.9280
15  0.9376  0.9245
17  0.9367  0.9174
19  0.9323  0.9174
21  0.9306  0.9157

Analysis

Analyze your results for experiment 2. At a minimum, you should have:

Optional Extensions

  1. For the SVM method, do some investigation into the support vectors (the SVC class in sklearn has some attributes that allow you to see the support vectors). How many support vectors are typical for these datasets? How can you determine the size of the margin? Does there seem to be a relationship between the size of the margin and the quality of the test results?

  2. There are many other parameters we did not tune in these methods, and many values we did not consider. Expand your analysis. Some suggestions: entropy vs. Gini for Random Forests, max tree depth for Random Forests, and other kernels for SVMs.

Acknowledgements: modified from materials by Ameet Soni