• VLBenchmarks
• Tutorials
• Documentation

Documentation

• Framework structure
  • Image feature extractors
  • Datasets
  • Benchmarks
• Parallelisation
• Helpers
  • DataCache
  • Logger
• References

Framework structure

VLBenchmarks is organised into four parts, corresponding to an equal number of MATLAB packages (namespaces):

• Image feature extractors (localFeatures). This package contains wrappers for features detectors and descriptors. Add your own wrapper here to evaluate your features.
• Datasets (datasets) This package contains code that manages (downloads and reads) benchmark data. The most common use is to adopt one of the supported standard benchmarks, but you may want to add a wrapper to your own dataset here.
• Feature benchmarks. (benchmarks). This package contains the benchmarking code.
• Supporting functions and classes (helpers).

Options of functions and of class objects are passed as optional function/constructor arguments in the following format:

  function(obligatoryParams,'OptionName',OptionValue,...)

Available options are listed in help string of functions or classes. It is not possible to change options of already constructed object.

Feature objects

All classes from package localFeatures are subclasses of GenericLocalFeaturesExtractor. In VLBenchmarks those feature extractors wrappers are already implemented:

Benchmarking your own feature extractor

The VLBenchmarks framework is easily extensible with your own image feature extraction algorithm. Your feature extractor wrapper has to be a subclass of GenericLocalFeaturesExtractor class and has to implement method extractFeatures(imgPath) and/or extractDescriptors(imgPath, frames) depending on your algorithm abilities.

You can use existing infrastructure of the benchmark. For example, by inheriting from GenericLocalFeatureExtractor you also inherit helpers.Logger class which implements simple logger. See Logging for details.

Another helper class used with the most of the built in detectors is helpers.GenericInstaller. This class handles installation process and supports to define class dependencies on web-located archives, mex files and other classes.

This example shows a feature extractor which supports both feature frame detection and descriptor calculation. This class supports downloading and compiling mex source code of the detector.

classdef ExtractorY < localFeatures.GenericLocalFeatureExtractor & ...
  helpers.GenericInstaller
% localFeatures.DetectorY Y feature frames detector.

%   localFeatures.DetectorY('OptionName', optionValue) Construct new wrapper
%   of Y image features extractor. This algorithm is able to both detect image
%   features and compute their descriptors.
%
%   This class support caching and automatic installation. The sources are
%   downloaded from:
%
%   http://supercoolstuff.com/dreamdetector.zip
%
% Options:
%   DetOption:: 0
%     Serious configuration stuff...
%
%   AutoInstall:: true
%     Install dependencies during object construction.
%
%   This class accepts helpers.Logger options.
%
% See also: localFeatures.DetectorX helpers.Logger
  properties (SetAccess=private, GetAccess=public)
    Opts = struct('detOption',0); % Default option value
  end
  properties (Constant)
    % Location of your code
    Dir = fullfile('data','software','exy');
    % URL with the source codes
    Url = 'http://supercoolstuff.com/dreamdetector.zip';
    % MEX files needed to be compiled for the detector
    SrcFiles = {fullfile(localFeatures.ExtractorY.Dir,'y_algorithm.c')};
    % MEX files needed to be compiled for the detector
    MexFiles = ...
      {fullfile(localFeatures.ExtractorY.Dir,['y_algorithm.',mexext])};
  end
  methods
    function obj = ExampleLocalFeatureExtractor(varargin)
      obj.Name = 'Detector Y'; % Name of the wrapper
      varargin = obj.configureLogger(obj.Name,varargin); % Configure logger
      varargin = obj.checkInstall(varargin); % Check whether installed
      obj.Opts = vl_argparse(obj.Opts,varargin); % Parse object options
      obj.setup(); % Setup the paths
    end

    function setup(obj)
      % setup Setup extractorY paths.
      addpath(obj.Dir);
    end
    
    function [frames descriptors] = extractFeatures(obj, imagePath)
      % extractFeatures Extract features from an image
      %   FRAMES = extractFeatures(IMG_PATH) Detect features in image IMG_PATH
      %   using the Y algorithm.
      %
      %   [FRAMES DESCRIPTORS] = extractFeatures(IMG_PATH) Detect frames and
      %   compute their descriptors using the Y algorithm.
      frames = obj.loadFeatures(imagePath,nargout > 1); % Check cache
      if numel(frames) > 0; return; end;
      descriptors = [];
      obj.info('Computing frames of image %s.',getFileName(imagePath));
      if nargout == 1
        frames = y_algorithm(imagePath, obj.Opts{:});
        obj.debug('Frames computed in %gs',toc(startTime));
      elseif nargout == 2
        [frames descriptors] = y_algorithm(imagePath, obj.Opts{:});
        obj.debug('Frames and descriptors computed in %gs',toc(startTime));
      end
      obj.storeFeatures(imagePath, frames, descriptors); % Cache the results
    end

    function [frames descriptors] = extractDescriptors(obj, imagePath, frames)
      % extractDescriptors Extract descriptors of given frames
      startTime = tic;
      obj.info('Computing descriptors of image %s.',getFileName(imagePath));
      [frames, descriptors] = y_algorithm(imagePath,frames,obj.Opts{:});
      obj.debug('Descriptors computed in %gs',toc(startTime));
    end

    function signature = getSignature(obj)
      signature = [helpers.struct2str(obj.Opts),';',...
        helpers.fileSignature(obj.MexFiles)]; % Detector unique signature
    end
  end
  methods (Access=protected)
    % Define which archives should be downloaded and where extracted
    function [urls dstPaths] = getTarballsList(obj)
      import localFeatures.*;
      urls = {DetectorY.Url};
      dstPaths = {DetectorY.Dir};
    end
    % Define which mex files should be compiled
    function [srclist flags] = getMexSources(obj)
      import helpers.*;
      srclist = obj.SrcFiles;
      % Flags passed to mex command
      flags = {};
    end
  end
end

Method extractFeatures(imgPath) can be called with one output argument when only feature frames need to be detected. When called with two output arguments, it extracts both feature frames and descriptors. This may seem to be dual to the extractDescriptors() method however some detectors does not support computation of descriptors of given frames.

To cache your detected features, you can use loadFeatures() or obj.storeFeatures() methods. However these methods need to implement method obj.getSignature() which generates unique string signature of the detector parameters. It is also useful to include signature of the detector binary file (helpers.fileSignature). Caching can be enabled/disables with methods obj.enableCaching(), obj.disableCaching()

To see details about logging, class options and installation framework, see the localFeatures.ExampleLocalFeatureExtractor class.

Datasets

All datasets are subclasses of abstract GenericDataset which defines method getImagePath(imageNumber) to access dataset images. Number of images in the dataset can be obtained from object property obj.NumImages.

Repeatability benchmark needs datasets which inherits from class GenericTransfDataset. This benchmark needs method getTransformation(imageNumber) which returns homography between a dataset image and a reference image (first image).

VggAffineDataset

Wrapper of the datasets from [2] is implemented in class VggAffineDataset and is a subclass of GenericTransfDataset. Datasets data are available and downloaded from VGG website.

Available categories (accessible through option 'Category') are:

VggRetrievalDataset

For the retrieval benchmark VggRetrievalDataset wraps around datasets introduced in [4], [5] These datasets provide both a set of images and a set of queries. Each query is specified by the query image, a bounding box and three subset of images:

• Good A nice, clear picture of the object/building
• Ok More than 25% of the object is clearly visible.
• Junk Less than 25% of the object is visible, or there are very high levels of occlusion or distortion.
• Bad Object not present.

A query is accessible through method getQuery(queryNum) which returns a structure with the following format:

>> dst = datasets.VggRetrievalDataset();
(INFO)  VggRetrievalDataset:  Loading dataset VggRetrievalDataset.
(DEBUG) VggRetrievalDataset:  Size of the images subset: 945
>> query = dst.getQuery(1)

query = 

         name: 'all_souls_1'        % Query name
    imageName: 'all_souls_000013'   % Name of the query image
      imageId: 8                    % Query image id
          box: [4x1 double]         % Query bounding box [xmin ymin xmax ymax]
         good: [1x24 double]
           ok: [1x54 double]
         junk: [1x33 double]

The image subsets are numeric arrays with image IDs. To obtain a path of a certain image from these subsets, you can simply call:

>> dst.getImagePath(query.junk(1))

ans =

data/datasets/vggRetrievalDataset/oxbuild/all_souls_000220.jpg

Number of all images in the dataset is stored in property obj.NumQueries.

Currently, two dataset categories are available:

These datasets usually contain thousands of images. It is possible to work only with its subset specified by these constructor parameters:

    GoodImagesNum
      Number of 'Good' images preserved in the database. When inf, all
      images preserved.
 
    OkImagesNum
      Number of 'ok' images preserved in the database. When inf, all
      images preserved.
 
    JunkImagesNum
      Number of 'junk' images preserved in the database. When inf, all
      images preserved.
 
    BadImagesNum
      Number of 'junk' images preserved in the database. When inf, all
      images preserved.
 
    SamplingSeed
      Seed of the random number generator used for sampling the image
      dataset.

Subsets are sampled used uniform random sampling. You can change the particular sampling by changing the seed of random number generator with parameter SamplingSeed.

Benchmarks

All benchmark classes are subclasses of abstract class GenericBenchmark. Currently three benchmarks are implemented.

RepeatabilityBenchmark is based on tests introduced in [2]. For details about this test see the help string of the RetrievalBenchmark class or see repeatability tutorial. Because this test is mostly reimplementing the original test, wrapper of the original benchmark IjcvOriginalBenchmark is also available. This class directly calls the repeatability.m script.

RetrievalBenchmark class implements simple retrieval benchmark of image features detectors. For details see help string of the class or see image retrieval benchmark tutorial.

Currently the image retrieval benchmark depends on the Yael library which is not available for Microsoft Windows platforms.

Parallelisation

The way how to parallelise is different for the repeatability and retrieval benchmark but both approaches use Matlab Parallel Computing Toolbox. For details about this toolbox, see its documentation.

Parallelisation of the repeatability benchmark can be done by interchanging for with parfor in any level, e.g.:

% Prevent cache to clear the last recently used items
helpers.DataCache.disableAutoclear();
for di = 1:numDetectors
  % Extract features from the first image to avoid race condition
  % as several processes may try to write the same file.
  % Features does not have to be stored in any variable as they are
  % cached.
  detectors{di}.extractFeatures(dataset.getImagePath(1));
  parfor imageIdx = 2:numImages
    % Run the repeatability benchmark
  end
end
helpers.DataCache.enableAutoclear();

This is possible thanks to the fact that the cache system depends only on the file system context and does not share any variables. However it does not implement any synchronisation mechanisms. If the loops are properly designed, each process works with different data. However cache autoclear has to be disabled as it can produce situations when two processes are trying to delete the same files.

The image retrieval benchmark is already parallelised and the class RetrievalBenchmark uses several parfor loops for both features extraction and KNN search.

In the case of KNN search you can advantage both from symmetric processing and distributed computing as the Yael KNN uses OpenMP to run its algorithms in multiple threads.

Helpers

Package helpers contains several classes and functions which are used in the rest of the project. Logger and DataCache classes are used almost in all classes therefore are described more in the following text.

Caching

This project extensively caches its results using key-value caching system. The key string is usually created based on the properties of input data and processing algorithm. Data itself can be any Matlab data object and are stored as *.mat files in directory ./data/cache. The target filename is created as MD5 sum of the key.

Binary data are usually distinguished by a signature of the source file. File signature contains the file name and last modification date of the file. This is also used for algorithm binaries. Among that, the signature usually contain algorithm parameters as their values affect the results.

DataCache methods

Caching is implemented in class helpers.DataCache and has only static methods, in order to be able to use it in parallel where no memory is shared among processes. This class implements the following methods:

• data = DataCache.getData(key) Get data identified by a string key. If the data has not been found, return [].
• storeData(data,key) Store data identified by key.
• res = hasData(key) Check whether data are cached.
• removeData(key) Remove particular data from the cache.
• clearCache() Delete the last recently used data to limit the overall size of the cached data to DataCache.maxDataSize limit.
• deleteAllCachedData() Delete all cached data.
• disableAutoClear() Temporarily disable the autoclear function. Cannot be called in parallel function as it creates a lock file.
• enableAutoClear()Enable autoClear after disableAutoClear. Cannot be called in parallel part of code as it deletes a lock file.

DataCache properties

The caching properties can be changed only in the helpers/DataCache.m source code by changing the class constant properties:

• maxDataSize Maximal storage space occupied by cached data in Bytes.
• dataPath Directory where to store cached data
• dataFileVersion Version of the .mat file used for data storage. See 'help save' for details.
• autoClear Check whether storage size has not exceeded the allowed size after each storeData method call. If so, the last recently used data are removed
• disable Globally disable caching.

Please note that the standard Matlab behaviour is to set these values only once when the class file is loaded for the first time. Therefore to apply the options you must call:

>> clear all;

Auto clear functionality

Caching system implements a way how to limit the overall size of the cached data, further denoted as an autoclear function. It checks the size of the files in the cache directory and removes the last recently used items (based on the file modification date) to limit the storage usage. To globally disable autoclear set the class property helpers.DataCache.autoClear to false. You can also disable it temporarily by calling method helpers.DataCache.disableAutoclear() which creates a lock file. It is recommended to call this method before running parallel code to prevent two processes to delete the same files.

Disable caching

Cache can be disabled globally by setting the constant property helpers.DataCache.autoClear to true. Also several framework classes implements methods disableCaching() and enableCaching() to control the caching behaviour of a single class.

Cache does not support data invalidation. However to invalidate cached data of a certain detector you can use a simple trick to change the detector binary modification date:

>> helpers.touch('./data/software/det_binary');

DataCache limits

As the cache depends heavily on a file system it is basically limited by a number of files in a single directory which depends on your operating system. Big directory structures can lead to slow file operations and therefore the cache operations can take longer. Thus it is recommended to clear the cache once a while (specially when using network file system).

Logger

Several classes in the framework uses logging implemented in class helper.Logger class. This class supports both sending the events to std output or writing them to a log file. Following events, inspired by the apache Log4j framework, are supported:

The verbose level can be set both globally editing the constant properties of class helpers.Logger or locally for each object which inherits the Logger class by providing the 'OptionName',OptionValue parameters to the class constructor call. Supported options are:

   VerboseLevel:: [helpers.Logger.DEBUG]
    Maximal verbosity level of messages sent to stdout. If set to OFF all
    messages are ignored, even errors would not stop the program execution.

  FileVerboseLevel:: [helpers.Logger.OFF]
    Maximal verbosity level of messages which are written to a log file.

  LogFile :: [./data/log]
     Path to a log file.

If you want to change the format of the log output, just adjust the methods helpers.Logger.displayLog and helpers.Logger.logToFile according to your aesthetic preferences.

Using Logger in your own classes

You can also use logger on your class as well simply by specifying the helpers.Logger as a superclass. Here is a simple template how to do it and how to properly configure the logger:

  classdef MyClass < helpers.Logger
    methods
      function obj = RepeatabilityBenchmark(varargin)
        varargin = obj.configureLogger('MyClassName',varargin);
        % Now you can process varargin for your arguments
        obj.info('Constructed.');
      end
    end
  end

And constructing your object you will get:

>> MyClass();
(INFO)  MyClassName:  Constructed.

Logging to a file and distributed computing

When your computation are being computed with several processes, possibly on several computers, having a single logging file may lead to inconsistent data. That is why we would like to create separate files for each Matlab process. This is generally a difficult task as the parfor environment does not include any information about the process.

But simple trick can be used. Matlab creates default values for object properties only once when the file is loaded for the first time. And in classic matlabpool each lab has got its own MATLAB process, therefore a log file name in the helpers.Logger can be set for example to:

LogFile = fullfile('data',['log-',helpers.hostname(),...
  randsample(char(97:122),5)]);

To test we can call the following code:

>> matlabpool 5;
>> import helpers.Logger;
>> parfor i=1:3
  for j=1:3
    obj = Logger();
    fprintf('Proc. #%d Obj. #%d: %s\n',i,j,obj.LogFile);
  end;
end;

Proc. #1 Obj. #1: data/log-comp02xcdqv
Proc. #1 Obj. #2: data/log-comp02xcdqv
Proc. #1 Obj. #3: data/log-comp02xcdqv
Proc. #2 Obj. #1: data/log-comp02lwmxr
Proc. #3 Obj. #1: data/log-comp02scirh
Proc. #3 Obj. #2: data/log-comp02scirh
Proc. #2 Obj. #2: data/log-comp02lwmxr
Proc. #2 Obj. #3: data/log-comp02lwmxr
Proc. #3 Obj. #3: data/log-comp02scirh

You can see that each process generated the random string only once. If the parfor would be distributed on more computers, the hostname() value would change as well.

References

1. A. Vedaldi and B. Fulkerson. VLFeat: An Open and Portable Library of Computer Vision Algorithms, vlfeat.org, 2008.
2. K. Mikolajczyk, T. Tuytelaars, C. Schmid, A. Zisserman, J. Matas, F. Schaffalitzky, T. Kadir, and L. Van Gool. A comparison of affine region detectors. IJCV, 1(65):43–72, 2005.
3. Perdoch, M. and Chum, O. and Matas, J. Efficient Representation of Local Geometry for Large Scale Object Retrieval. In proceedings of CVPR09, 2009.
4. J. Philbin, O. Chum, M. Isard, J. Sivic and A. Zisserman. Object retrieval with large vocabularies and fast spatial matching CVPR, 2007
5. J. Philbin, O. Chum, M. Isard, J. Sivic and A. Zisserman., Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases (2008)