The VLFeat C library is designed to integrate seamlessly with MATLAB. Binary compatibility is simplified by the use of the C language (rather than C++). In addition, the library design follows certain restrictions that make it compatible with the MATLAB MEX interface.
The main issue in calling a library function from a MATLAB MEX function is that MATLAB can abort the execution of the MEX function at any point, either due to an error, or directly upon a user request (Ctrl-C) (empirically, however, a MEX function seems to be incorruptible only during the invocation of certain functions of the MEX API such as mexErrMsgTxt).
When a MEX function is interrupted, resources (memory blocks or objects) whose ownership was transferred from VLFeat to the MEX function may be leaked. Notice that interrupting a MEX function would similarly leak any memory block allocated within the MEX function. To solve this issue, MATLAB provides his own memory manager (mxMalloc, mxRealloc, ...). When a MEX file is interrupted or ends, all memory blocks allocated by using one of such functions are released, preventing leakage.
In order to integrate VLFeat with this model in the most seamless way, VLFeat memory allocation functions (vl_malloc, vl_realloc, vl_calloc) are mapped to the corresponding MEX memory allocation functions. Such functions automatically dispose of all the memory allocated by a MEX function when the function ends (even because of an exception). Because of the restrictions of the library design illustrated in Memory and resource handling, this operation is safe and correctly dispose of VLFeat local state. As a consequence, it is possible to call mexErrMsgTxt at any point in the MEX function without worrying about leaking resources.
This however comes at the price of some limitations. Beyond the restrictions illustrated in Memory and resource handling, here we note that no VLFeat local resource (memory blocks or objects) can persist across MEX file invocations. This implies that any result produced by a VLFeat MEX function must be converted back to a MATLAB object such as a vector or a structure. In particular, there is no direct way of creating an object within a MEX file, returning it to MATLAB, and passing it again to another MEX file.