Full example of multiple-class SVM with cross-validation using Matlab

Question

I am currently confusing about implementing SVM with cross-validation using Matlab now. There are many post on stackoverflow that mentioned pieces of information about SVM and its cross-validation; however, there is no full example even using the simplest 'fisheriris' data sets.

I have summarised those post with the problems as follows:

a. Binary and multiple-class SVM: Answered by support vector machines in matlab but without example of cross-validation.

b. Cross validation using SVM: Example of 10-fold SVM classification in MATLAB but without example of multiple-class SVM.

c. One-against-one and one-against-all SVM: 1-against-1 can be found at support vector machines in matlab 1-against-all can be found at Multi-class classification in libsvm Multi-Class SVM( one versus all) No example of cross-validation

d. libSVM and Matlab build-in SVM (statistical toolbox) A partly full example using libSVM can be found at 10 fold cross-validation in one-against-all SVM (using LibSVM)

e. Parameter optimization Retraining after Cross Validation with libsvm

But things are really complicated for one to learn and eventually deploy SVM for their real problem, and it is problematic and fallible by just looking at these previous posts. At least I am stupid to solve the jigsaw problem.

Why don't we together build a easily understandable codes for SVM with following features?

A. Just use 'fisheriris' data.

B. Can be used for binary and multiple-class problem (fisheriris can be chosen to be binary).

C. Implement the cross validation.

D. Implement both One-against-one and one-against-all.

E. Two versions using libSVM and Matlab build-in SVM respectively. As the svmtrain is the same name for both package, I suggest we change it to libsvmtrain and MEX it before use. We can then compare these two methods as well.

F. Currently, due to the training/testing data separation, the results are not always reproducible. Can we fix this?

F. (Optional) Add parameter optimization.

G. (Optional) Add ROC analysis.

My start is some codes like:

#

% libSVM version_1
clc; clear all;

load fisheriris
[~,~,labels]            = unique(species);              % Labels: 1/2/3
data                    = zscore(meas);                 % Scale features
numInst                 = size(data,1);
numLabels               = max(labels);

%# Split training/testing
idx                     = randperm(numInst);
numTrain                = 100; 
numTest                 = numInst - numTrain;

trainData               = data(idx(1:numTrain),:);  
testData                = data(idx(numTrain+1:end),:);

trainLabel              = labels(idx(1:numTrain)); 
testLabel               = labels(idx(numTrain+1:end));

%# Train one-against-all models
model                   = cell(numLabels,1);
for k=1:numLabels
    model{k}                = libsvmtrain(double(trainLabel==k), trainData, '-c 1 -g 0.2 -b 1');
end

%# Get probability estimates of test instances using each model
prob                    = zeros(numTest,numLabels);
for k=1:numLabels
    [~,~,p]                 = svmpredict(double(testLabel==k), testData, model{k}, '-b 1');
    prob(:,k)               = p(:,model{k}.Label==1);    % Probability of class==k
end

% Predict the class with the highest probability
[~,pred]                = max(prob,[],2);
acc                     = sum(pred == testLabel) ./ numel(testLabel);    % Accuracy
C                       = confusionmat(testLabel, pred);                 % Confusion matrix

#

% Matlab build-in SVM version_1
clc; clear all;

load fisheriris
[g, gn]                 = grp2idx(species);     % Nominal class to numeric

% Split training and testing sets
[trainIdx, testIdx]     = crossvalind('HoldOut', species, 1/3);

pairwise                = nchoosek(1:length(gn),2);            % 1-vs-1 pairwise models
svmModel                = cell(size(pairwise,1),1);            % Store binary-classifers
predTest                = zeros(sum(testIdx),numel(svmModel)); % Store binary predictions

%# classify using one-against-one approach, SVM with 3rd degree poly kernel
for k=1:numel(svmModel)
    %# get only training instances belonging to this pair
    idx                     = trainIdx & any( bsxfun(@eq, g, pairwise(k,:)) , 2 );

    %# train
    svmModel{k}             = svmtrain(meas(idx,:), g(idx), ...
        'BoxConstraint',2e-1, 'Kernel_Function','polynomial', 'Polyorder',3);

    %# test
    predTest(:,k)           = svmclassify(svmModel{k}, meas(testIdx,:));
end
pred = mode(predTest,2);   % Voting: clasify as the class receiving most votes

%# performance
cmat                        = confusionmat(g(testIdx),pred);
acc                         = 100*sum(diag(cmat))./sum(cmat(:));
fprintf('SVM (1-against-1):\naccuracy = %.2f%%\n', acc);
fprintf('Confusion Matrix:\n'), disp(cmat)

Please feel free to add yours until we finish all the missions. Someone can also create a google code project for us to finish this.

Thanks so much.