VLBenchmarks

• VLBenchmarks
  • License
• Tutorials
• Documentation

VLBenchmarks is a MATLAB framework for testing image feature detectors and descriptors. The latest version can be downloaded here.

If you use this project in your work, please cite it as:

    @misc{lenc12vlbenchmarks,
    Author = {K. Lenc and V. Gulshan and A. Vedaldi},
    Title = {VLBenchmkars},
    Year  = {2011},
    Howpublished = {\url{http://www.vlfeat.org/benchmarks/}xsxs}
    }
  

This project is sponsored by the PASCAL Harvest programme as part of this project. VLBenchmkar is a sibling of the VLFeat Library, which it uses as suppot, but is otherwise independent of it.

The authors would like to thank Andrew Zisserman, Jiri Matas, Krystian Mikolajczyk, Tinne Tuytelaars, and Cordelia Schmid for helpful discussion and supports.

• Overview
  • Changes
• Download and install
• Quick guide
• References

Overview

VLBechmarks is a MATLAB framework to evaluate feature detector and descriptors automatically. Benchmarking your own features is as simple as writing a single wrapper class. Then VLBenchmarks takes care of downloading the required benchmarking data from the Internet and running the evaluation(s). The framework ships with wrappers for a number of publicly available features to enable comparing to them easily. VLBenchmarks has a number of functionalities, such as caching of intermediate results, that allow running benchmarks efficiently.

The current version of VLBechmarks implements:

• The feature extractor repeatability of [1].
• The descriptor matching score of [1].
• A new retrieval-based test based on the retrieval method of [2].

The code is distributed under the permissive MIT license.

Changes

1.0-beta

(7/10/2012) Initial release.

Download and install

• MATLAB/C code for Mac, Linux, and Windows.
• Git repository.

The archive linked above contains the complete implementation of the VLBenchmarks suite, including a copy of this documentation. The minimal required version of MATLAB is R2010a (7.10) or later (due to extensive use of the new class type introduced in that release). To install simple unpack the archive, start MATLAB, change thea directory just created, and run:

  >> install

This script downloads and installs a copy of VLFeat as the only dependency, and it compiles a number of MEX files. To successfully compile the code, you will need to be able to compile MEX files in your MATLAB environment, for detail see MATLAB documentation.

To test the benchmark installation run:

  >> benchmarksDemo

Note that this script will download and install the VGG Affine Dataset.

Quick guide

Here is a script for computing image feature extractor repeatability [1]. This script gets pair of images from the 'Graffiti' dataset, run the feature extractor and compute there repeatability:

dataset = datasets.VggAffineDataset('Category','graf'); % Test data
repBenchmark = benchmarks.RepeatabilityBenchmark(); % Benchmark object
sift = localFeatures.VlFeatSift(); % Tested feature extractor

[repeatability numCorresp] = ...
  repBenchmark.testFeatureExtractor(sift,...
                                    dataset.getTransformation(2),...
                                    dataset.getImagePath(1),...
                                    dataset.getImagePath(2));

With a result:

repeatability = 0.6589       numCorresp = 825

Running this script once again you will get the results immediately as they are being loaded from cache. Now let's play with the descriptor parameters. For example we can test what would happen if we compute features of up-sampled images (by setting vl_sift parameter 'FirstOctave' to -1):

usSift = localFeatures.VlFeatSift('FirstOctave',-1); % Up-sampling SIFT
[repeatability numCorresp] = ...
  repBenchmark.testFeatureExtractor(usSift,...
                                    dataset.getTransformation(2),...
                                    dataset.getImagePath(1),...
                                    dataset.getImagePath(2))

Few moments later we get the results:

repeatability =  0.5838       numCorresp =  2435

Now let's run your custom feature extractor instead. Suppose you have a command line utility called "DETECTORX" that implements it. Start by copying a template wrapper located in 'localFeatures/TemplateWrapper.m' and edit it according to your feature extractor:

classdef DetectorX < localFeatures.GenericLocalFeatureExtractor
% localFeatures.DetectorX X feature frames detector.
%   localFeatures.DetectorX('OptionName', optionValue) Construct new
%   wrapper of X Features detector.
%
% See also: localFeatures.SID
  properties (Constant)
    BinPath = fullfile('data','software','detx','detectorx');
  end
  methods
    function obj = ExampleLocalFeatureExtractor(varargin)
      obj.Name = 'Detector X'; % Name of the wrapper
      obj.DetectorName = obj.Name; % Name of feature detection algorithm
    end
    
    function frames = extractFeatures(obj, imagePath)
      frames = obj.loadFeatures(imagePath,nargout > 1); % Check cache
      if numel(frames) > 0; return; end;
      
      obj.info('Computing frames of image %s.',getFileName(imagePath));
      featFile = [tempname '.frames'];
      system(sprintf('./%s %s %s', obj.BinPath, imagePath, featFile));
      frames = localFeatures.helpers.readFramesFile(featuresFile);
      delete(featuresFile);
      
      obj.storeFeatures(imagePath, frames, []); % Cache the results
    end
    
    function signature = getSignature(obj)
      signature = helpers.fileSignature(obj.BinPath); % Unique signature
    end
  end
end

Let's compute matching score of your feature extractor [1]. Because created feature extractor is not able to compute descriptors we will use SIFT descriptors instead.

matchBenchmark = ...
  benchmarks.RepeatabilityBenchmark('MatchFrameDescriptors',true);
xDet = localFeature.DetectorX();
% User xDet for feature detection and sift for description
xDetSiftDesc = localFeatures.DescriptorAdapter(xDet, sift);
[matchingScore numMatches] = ...
  matchBenchmark.testFeatureExtractor(xDetSiftDesc,...
                                      dataset.getTransformation(2),...
                                      dataset.getImagePath(1),...
                                      dataset.getImagePath(2))

Retrieval benchmark puts the feature extractor into a simple image retrieval system and calculates its mean Average Precision. Interface of the retrieval benchmark is similar to the repeatability benchmark. This minimal example shows how to calculate the Mean Average Precision of an image retrieval system which uses VLFeat SIFT feature extractor.

dataset = dataset.VggRetrievalDataset('Category','oxbuild');

retBenchmark = benchmarks.RetrievalBenchmark();
mAP = retBenchmark.testFeatureExtractor(sift, dataset);

References

1. K. Mikolajczyk, T. Tuytelaars, C. Schmid, A. Zisserman, J. Matas, F. Schaffalitzky, T. Kadir, and L. Van Gool. A comparison of affine region detectors. IJCV, 1(65):43–72, 2005.
2. H. Jegou, M. Douze, and C. Schmid. Exploiting descriptor distances for precise image search. Technical Report 7656, INRIA, 2011.