KWIVER v1.1.0

Release Summary

This is a minor release of KWIVER that provides both new functionality and
fixes over the previous v1.0.0 release. There are numerous changes in this
release that have been motivated primarily by three different applications
of KWIVER. One is the use of KWIVER in MAP-Tk to provide 3D reconstruction
algorithms to the MAP-Tk tools and TeleSculptor GUI application. This
work has been largely sponsored by the US Air Force Research Laboratories.
A second driving factor is the use of KWIVER as an evaluation framework for
video surveillance applications in the upcoming IARPA DIVA program. The
third driving factor for improvements in this release is the use of KWIVER
as the basis for the VIAME toolkit for underwater image processing, supported
by NOAA. These project needs have improved the general processing framework
of KWIVER as well as specific algorithm improvements in 3D reconstruction
and object detection in video.

Vital, the core framework of KWIVER, has been updated in several ways in
this release. Previously there were multiple plugin loader mechanisms
stemming from the origins of KWIVER as the fusion of multiple projects.
In this release there is now a unified plugin loader framework that provides
a common mechanism for loading algorithm plugins (Arrows), process and cluster
plugins (Sprokit), loggers (Vital), and so on. There is also a single tool,
plugin_explorer, that allows introspection of these plugins of various types.
This release also has a new multi-threading API in Vital that provides an
interface for a thread pool with multiple back ends. The thread pool API
allows KWIVER algorithms to more easily leverage parallel processing without
being tied to a single thread management library. Additional back ends,
like Intel TBB, will be added in future releases. C and Python bindings for
Vital have also been extended considerably in this release.

Some other changes in Vital enhance the way image data is loaded and stored.
This release generalizes the image data structure to support images with
various pixel types. Previously Vital only supported 8-bit images. It now
supports many pixel types including 16 and 32 bit integers and single and
double precision floating point types. Also new in this release is a
video input API for reading image sequences directly from video sources.
There are implementations of this API in Arrows for reading video files
directly as well as lists of images. The video API also supports reading
video metadata that is encoded with the video.

In support of the MAP-Tk TeleSculptor application, many abstract algorithms
in Vital and their implementations in Arrows now support callbacks. These
callbacks allow the algorithms to report intermediate results or to signal the
algorithm to terminate early. These callback are used in GUI applications
to get live updates of processing results and to allow users to cancel
processing. This release also adds binary serialization for feature detectors
and descriptors, which allows algorithms to cache features and descriptors
on disk and load them back in later. The algorithms for matching features
have also been improved for better handling of very large images. Large images
are best matched by comparing features at a coarse scale before matching fine
scale features.

In support of the DIVA program, the Sprokit pipeline framework now supports
frame dropping and processing time-outs. These changes help prevent the
pipeline from falling behind in applications that need to keep up with real
team processing requirements. A pipeline can be configured to drop frames
if needed to catch up.

In support of the VIAME toolkit, this release adds new object detection
APIs and provides and implementation of those APIs use the Darknet YOLO
(You Only Look Once) network architecture.

There are many other smaller changes in this release. These are detailed in
the change log below.

Updates since v1.0.0

Arrows

• Updated all algorithms to use the new plugin loader approach.

Arrows - Ceres

• Added a Ceres implementation of the optimize_cameras abstract algorithm.
  This implementation is very similar to the bundle_adjust algorithm but
  does not modify the landmarks.

• Refactored common config options and helper functions between
  optimize_cameras and bundle_adjust into new shared classes in options.h.
  This makes it much easer to create a new Ceres solver algorithm and
  reuse the same solver options or camera options.

• Implemented callbacks for the Ceres bundle_adjust to emit the state of
  the optimization after each iteration if a callback is registered.

• Added two forms of camera path regularization to the bundle_adjust and
  optimize_cameras algorithms. The first adds a cost that favors smooth
  camera paths; the camera location at time t should be close to the
  average of locations at t-1 and t+1. The second adds cost for cameras
  moving along the principal ray direction; this allows optimization to
  favor focal length change (zoom) over forward motion.

Arrows - Core

• Added an option to the PCA canonical transform estimator to shift the ground
  plane vertically to place a configurable percentile of the landmarks above
  the ground.

• Implemented callbacks for the core initialize_cameras_landmarks algorithm
  to emit the set of cameras and landmarks after each camera is added if a
  callback is registered.

• Added logic to compute changes in ground sample distance (GSD) as images are
  added in the camera_landmark_initialization algorithm. If GSD changes are
  above a threshold then assume the camera intrinsics have changed and create
  a new shared intrinsic model. This allows for shared intrinsics when the
  camera does not zoom but varying intrinsics when zoom is detected.

• Changed the order of when Necker reversal correction is attempted in the
  initialize_cameras_landmarks algorithm. Previously it was attempted at the
  end after adding all cameras, often too late. Now it is attempted the
  first time the ratio of reversed to non-reversed RMSE is within a threshold.
  Necker reversal is only attempted once. It is either needed or not.

• Added default version of dynamic_configuration which always returns
  an empty config block.

• Added a core implementation of "filter_tracks" which applies filters based
  on track length or based on importance of the track in covering a match matrix.

• Added a core implementation of "filter_features" which filters features
  based on key point scale.

• Changed "match_features_homography" to apply feature filtering to both the
  source and destination frames. This seems to be more robust when the number
  of features is very large. Also adjusted the homography inlier scale with
  respect to the average scale of the features.

• Added multi-threaded operation to close_loops_keyframe and
  close_loops_exhaustive algorithms. Frame matching tasks are added to the
  thread pool.

• Added a video_input algorithm for image lists that treats a sequence of
  images files like a video, using an image_io algorithm to load each frame.

• Added a video_input algorithm for POS files that produces a metadata-only
  stream out of POS files associated with a list of images. Each image file
  name is matched to a POS file with the same base name in a specified metadata
  directory.

• Added a video_input algorithm that splits imagery and metadata between two
  other video_input sources. For example this implementation can combine the
  images from an image_list video_input and metadata from a POS video_input.

• Added a video_input filter algorithm that reads data from another
  video_input instance and reduces the frame range or adjusts frame timing.

• Updated hierarchical_bundle_adjust and initialize_cameras_landmarks to
  accept an optional metadata_map and pass it along to nested algorithms.

• Generalized the track_features_core algorithm to handle tracking on frames
  that may come out of order and frames that have already been tracked. Also
  removed the caching of the next frame id in the algorithm and now retrieve it
  from the existing track set instead. These changes are to support the
  MAP-Tk GUI which allows users to stop tracking, change the active frame,
  and start a new tracker.

• Loop closure is now an optional component algorithm of the core feature
  tracker. Leaving the loop closure algorithm unspecified will cause the
  tracker to skip loop closure.

• Updated the core tracking algorithm to read and write features and
  descriptors to disk if a feature_descriptor_io algorithm is specified.

• Changed the meaning of the final_reproj_thresh parameter of the core
  initialize_cameras_landmarks algorithm to be relative to the final median
  reprojection error rather than an absolute measure. This adaptive
  threshold is more robust.

Arrows - OpenCV

• Updated the OpenCV image container functions to handle conversions between
  vital::image and cv::Mat for all supported pixel types and memory layouts.

• Updated the ocv::image_io algorithm to load images of various pixel types
  and to explicitly apply BGR to RGB conversions instead of hiding those
  color conversions elsewhere (see fixes below).

• Added implementation for draw_detected_object_set.

Arrows - VXL

• Updated the VXL image container functions to handle conversions between
  vital::image and vil_image_view_base_sptr for all supported pixel types and
  memory layouts.

• Generalized the vxl::image_io algorithm to read and write images with
  different pixel formats (bool, uint8, uint16, float, etc.). Added a
  "force_byte" option to always load or save as u...

KWIVER v1.0.0

Release Summary

This is the first formal release of KWIVER. The features in this
initial release are summarized below. Future release notes will
describe modifications since the previous release.

Vital is a result of refactoring the common support functionality,
basic types and abstract algorithms from the MAP-Tk package combined
with some core capabilities from the Vibrant package.

Library Features

Package Configuration, Build System, & Documentation

• Build system uses CMake as a meta build system for easy build configuration,
  cross-platform compiling, and dashboard regression testing.

• A unit testing framework using CTest is provided in the 'test' directory.
  This makes writing unit tests easy.

• CMake scripts are provided to make it very easy to build the Vital code
  documentation using Doxygen. Just turn on the "VITAL_ENABLE_DOCS" option
  in CMake to create a documentation build target (if you have Doxygen).
  Turn on "VITAL_INSTALL_DOCS" to have to built documentation installed.

Common Support Functionality

• The Logger supports a general purpose logging facility that
  interfaces to some well known logging packages.

• Config management converts config files to internal configuration
  structures. These config blocks are used to specify the behavior of
  algorithms and other software components.

• Video metadata parsing can be used to decode the metadata stream
  in video files.

• Algorithm Plugin loader loads and makes algorithms packaged as
  shared objects available to be instantiated and executed.

Common algorithm data types

This is a collection of data types used by the algorithms and
tools. These data types are needed to interface with the
algorithms. There are also C and python bindings for some of these
types.

Abstract Algorithms

The abstract algorithms provide an interface with which to interact
with the concrete algorithms.

KWIVER v0.2

End to End Video Analytics

Release corresponding January 2016 Kitware Source Article An End to End Video Analytics Toolkit