CS360 Lab 6: Support Vector Machines

Due: Monday, Mar 25 at 11:59pm

Overview

The goals of this lab:

Log in and check for libraries

First make sure you can log in to the GPU lab machines (see Piazza for which machines have GPUs and for instructions about using the Haverford VPN).

ssh <username>@<machine-name>.cs.haverford.edu

After logging on to the lab machine, open your .bashrc file using

emacs .bashrc

There should be a lot of stuff already there. At the end, add the following lines:

export PATH=/packages/cs/python3.7.7/bin:/usr/local/cuda-10.1/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-10.1/lib64:/usr/local/cuda/extras/CUPTI/lib64:/packages/cs/python3.7.7/lib

Then use Ctrl-x Ctrl-s to save and Ctrl-x Ctrl-c to close the file. Then log out and log back in.

Finally, check for the following libraries and make sure all import with no errors.

$ python3
Python 3.7.7 (default, Jul 14 2020, 14:39:22)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> import sklearn
>>> import tensorflow

Please post on Piazza if you have any issues doing this part! If you cannot access the lab machines easily, make sure you have the above libraries installed on your own machine:

pip3 install numpy
pip3 install sklearn
pip3 install tensorflow

or

python3 -m pip install numpy
python3 -m pip install sklearn
python3 -m pip install tensorflow


Introduction

You will have or create the following files:


Datasets and Command line arguments

Throughout the lab, consult the documentation frequently to find the appropriate methods for this lab. Reading and using documentation is a very important skill that we will practice in Lab 7, Lab 8, and the final project. Make sure you really understand each line of code you’re writing and each method you’re using - there are fewer lines of code for this lab, but each line is doing a lot!

When importing sklearn modules, avoid importing with the * - this imports everything in the library (and these functions can be confused with user-defined functions). Instead, import functions and classes directly. For example:

from sklearn.datasets import fetch_openml, load_breast_cancer

Imports should be sorted and grouped by type (for example, usually I group imports from python libraries vs. my own libraries).

Rather than parse and process data sets, you will use sklearn’s pre-defined data sets. Details can be found here. At a minimum, your experiments will require using the MNIST and 20 Newsgroup datasets. (Edit: links have been updated.) Both are multi-class tasks (10 and 20 classes, respectively). Note that both of these are large and take time to run, so I recommend developing using the Wisconsin Breast Cancer dataset (example below).

For this lab, your run_pipeline.py file should take in one command line argument (using the optparse library), the dataset name. This does not refer to a file, but allows the program to import the correct dataset (options: cancer, mnist, and news). Here is an example:

if opts.dataset == "cancer":
    data = load_breast_cancer()

X = data['data']
y = data['target']
print(X.shape)
print(y.shape)

which outputs 569 examples with 30 features each:

(569, 30)
(569,)

The MNIST dataset is very large and takes a lot of time to run, so you can randomly select 1000 examples; you should also normalize the pixel values between 0 and 1 (instead of 0 and 255):

data = fetch_openml('mnist_784', data_home="/home/smathieson/Public/cs360/sklearn-data/")
X = data['data']
y = data['target']
X,y = utils.shuffle(X,y) # shuffle the rows (utils is from sklearn)
X = X[:1000] # only keep 1000 examples
y = y[:1000]
X = X/255 # normalize the feature values

The newsgroup dataset in vector form (i.e., bag of words) is obtained using:

from sklearn.datasets import fetch_20newsgroups_vectorized
data = fetch_20newsgroups_vectorized(subset='all', data_home="/home/smathieson/Public/cs360/sklearn-data/")

No normalization is required; I also suggest randomly sampling 1000 examples for this dataset as well. The data object also contains headers and target information which you should examine for understanding. For your analysis, it may be helpful to know the number of features, their types, and what classes are being predicted.

Coding Requirements

The coding portion is flexible - the goal is to be able to execute the experiments below. However, you should keep these requirements in mind:

$ python3 run_pipeline.py -d cancer
$ python3 run_pipeline.py -d mnist
$ python3 run_pipeline.py -d news

You may have cases for each one (since they need to be treated differently). If the user does not enter a dataset, rely on optparse to print a helpful message.

knn_clf = KNeighborsClassifier()
parameters = {"weights": ["uniform", "distance"], "n_neighbors": [1, 5, 11]}
test_results = runTuneTest(clf, parameters, X, y)

Note that the hyper-parameters match the API for KNeighborsClassifier. In the dictionary, the key is the name of the hyper-parameter and the value is a list of values to try.

Experiment 1: Random Forest vs SVM Generalization Error

Using run_pipeline.py, you will run both Random Forests and SVMs and compare which does better in terms of estimated generalization error.

Coding Details

Your program should read in the dataset using the command line, as discussed above. You should specify your parameters and classifier and call runTuneTest (see the above example), which follows this sequence of steps:

  1. Divide the data into training/test splits using StratifiedKFold. Stratified folds balance the number of examples from each class in each fold. Follow the example in the documentation to create a for-loop for each fold. Set the parameters to shuffle the data and use 5 folds. Set the random_state to a fixed integer value (e.g., 42) so the folds are consistent for both algorithms.

  2. For each fold, tune the hyper-parameters using GridSearchCV, which is a wrapper for your base learning algorithms; it automates the search over multiple hyper-parameter combinations. Use the default value of 3-fold 5-fold tuning (so we are essentially performing a cross-validation within a cross-validation).

  3. After creating a GridSearchCV classifier, fit it using your training data. Report the training score using the score method.

  4. Get the test-set accuracy by running the score method on the fold’s test data.

  5. Return a list of test accuracy scores for each fold.

In main(), you should print the test accuracies for all 5 folds for both classifiers (pair up the accuracies for each fold for ease of comparison). The classifiers/hyper-parameters are defined as follows:

Code incrementally, and be sure to examine the results of your tuning (what were the best hyper-parameter settings? what were the scores across each parameter?) to ensure you have the pipeline correct. Since the analysis below is dependent on your results, I cannot provide sample output for this task. However, this is what is generated if I change my classifier to K-Nearest Neighbors using the parameters listed in the previous section (you can try to replicate this using a random_state of 42). Update: in sklearn version 23 the default cv changed, so these results are updated below:

$ python3 run_pipeline.py -d cancer

-------------
KNN
-------------

Fold 1:
{'n_neighbors': 11, 'weights': 'uniform'}
Training Score: 0.9362637362637363
Fold 2:
{'n_neighbors': 11, 'weights': 'uniform'}
Training Score: 0.9494505494505494
Fold 3:
{'n_neighbors': 11, 'weights': 'distance'}
Training Score: 1.0
Fold 4:
{'n_neighbors': 11, 'weights': 'uniform'}
Training Score: 0.9318681318681319
Fold 5:
{'n_neighbors': 5, 'weights': 'uniform'}
Training Score: 0.9473684210526315

Fold, Test Accuracy
1, 0.9385964912280702
2, 0.9122807017543859
3, 0.9298245614035088
4, 0.9649122807017544
5, 0.9557522123893806

Analysis

In Part 1 of your writeup (must be a PDF), you will analyze your results. At a minimum, your submission should include the following type of analysis:

You do not need to go into exhaustive detail, but your analysis should be written as if it were the results section of a scientific report/paper.

Experiment 2: Learning Curves

Using generate_curves.py, you will generate learning curves for the above two classifiers. We will vary one of the hyper-parameters and see how the train and test error accuracy changes.

Coding Requirements

$ python3 generate_curves.py -d cancer
Neighbors,Train Accuracy,Test Accuracy
 1  1.0000  0.9051
 3  0.9539  0.9191
 5  0.9473  0.9279
 7  0.9420  0.9262
 9  0.9420  0.9315
11  0.9398  0.9297
13  0.9385  0.9332
15  0.9380  0.9297
17  0.9367  0.9297
19  0.9337  0.9297
21  0.9306  0.9298

Analysis

Analyze your results for experiment 2. At a minimum, you should have:

Optional Extensions

  1. For the SVM method, do some investigation into the support vectors (the SVC class in sklearn has some attributes that allow you to see the support vectors). How many support vectors are typical for these datasets? How can you determine the size of the margin? Does there seem to be a relationship between the size of the margin and the quality of the test results?

  2. There are many other parameters we did not tune in these methods, and many values we did not consider. Expand your analysis. Some suggestions: entropy vs. Gini for Random Forests, max tree depth for Random Forests, and other kernels for SVMs.

Acknowledgements: modified from materials by Ameet Soni