Background Subtraction - II

R. Venkatesh Babu

SE 263 R. Venkatesh Babu

Running Gaussian Average

• Pfinder(Wren, Azarbayejani, Darrell, Pentland, 1997):– fitting one Gaussian distribution (µ,σ)over the

histogram: this gives the background PDF

– background PDF update: running average:

– In test | F -µ| > Th, Th can be chosen as kσ

– It does not cope with multimodal backgrounds()

SE 263 R. Venkatesh Babu

Need for Adaptive Systems

• Consider the pixel process at (x0,y0)

SE 263 R. Venkatesh Babu

Mixture of Gaussian(Stauffer and Grimson, CVPR’99, PAMI’00)

• This model copes with multimodal background distributions.

• Issues:

– the number of modes is arbitrarily pre-defined (usually from 3 to 5)

– how to initialize the Gaussians?

– how to update them over time?

SE 263 R. Venkatesh Babu

Mixture Model

• Training:

• Model the pixel intensity variation as:

– The weight of a particular Gaussian is a measure of how good it represents the background.

SE 263 R. Venkatesh Babu

Updation

• If none of the K distributions match the current pixel value:

– Replace the least probable dist with a dist with the current value as its mean, having high variance and low prior weight

• Update Prior Weights:

Where, ɑ is the learning rate and Mk,t =1 for matched Gaussian model and zerofor unmatched Gaussian

SE 263 R. Venkatesh Babu

MoG: Algorithm• All weights ωi are updated (updated and/or normalised) at

every new frame

• At every new frame, some of the Gaussians “match” the current value (those at a distance < 2.5 σi ): for them, µi,σi are updated by the running average:

Where, learning factor

SE 263 R. Venkatesh Babu

Background Models

• The mixture of Gaussians actually models both the foreground and the background: how to pick only the distributions modeling the background?: – All distributions are ranked according to their ωi /σi and the first ‘b’ are

chosen as “background”

SE 263 R. Venkatesh Babu

Example

SE 263 R. Venkatesh Babu

OpenCV Gaussian Mixture Based Background subtraction

• Class to be used: BackgroundSubtractorMOG

• Background model update function: Computes a foreground mask from the given input frame “image” and returns it in “fgmask”.operator()(InputArray image, OutputArray fgmask, double learningRate=0);

• Background computing function: Computes the current background and returns it in “backgroundImage”getBackgroundImage(OutputArray backgroundImage);

• The class implements the algorithm described in P. KadewTraKuPong and R. Bowden, An improved adaptive background mixture model for real-time tracking with shadow detection, Proc. 2nd European Workshop on Advanced Video-Based Surveillance Systems, 2001:http://personal.ee.surrey.ac.uk/Personal/R.Bowden/publications/avbs01/avbs01.pdf

• Other background modelling classes available in OpenCV: BackgroundSubtractor, BackgroundSubtractorMOG2.More info: http://opencv.itseez.com/modules/video/doc/motion_analysis_and_object_tracking.html#backgroundsubtractor http://opencv.itseez.com/modules/video/doc/motion_analysis_and_object_tracking.html#backgroundsubtractormog http://opencv.itseez.com/modules/video/doc/motion_analysis_and_object_tracking.html#backgroundsubtractormog2

SE 263 R. Venkatesh Babu

OpenCV Code Snippet(C++)……cv::Mat NewFrame, FGMask, BG; //Image matrices for newframe, foreground mask and the Background.

cv::VideoCapture VideoFeed(“test.avi"); //Create class instance to handle video capture.BackgroundSubtractorMOG BGModel; //Create class instance for Background modelling.

cv::namedWindow(“Input Video"); //Create Windows.cv::namedWindow(“ForeGround Mask");cv::namedWindow(“BackGround");

while ( VideoFeed.isOpened()) // Returns true if video capturing has been initialized already.{

if(VideoFeed.grab()) // Grabs the next frame from video file or capturing device and return true (non-zero) in the // case of success.

{VideoFeed.retrieve(NewFrame); // Decodes and returns the grabbed video frame.

BGModel.operator()(NewFrame, FGMask, 0); //Update Background model and return foreground mask.BGModel.getBackgroundImage(BG ); //Returns the current Background image.

cv::imshow(“Input Video", NewFrame); //Display video frames.cv::imshow(“ForeGround Mask", FGMask);cv::imshow(“BackGround", BG);

}cv::waitKey(3);

}cvDestroyWindow(“Input Video“); //Delete Window and free the memory used by it.cvDestroyWindow(“ForeGround Mask");cvDestroyWindow(“BackGround");……

SE 263 R. Venkatesh Babu

Non Parametric Model for Background(Ahmed Elgammal ,David Harwood, Larry Davis, ECCV ’00)

• It is a non – parametric model ie., distribution is derived solely from the observed samples.

• Say we have samples of data for a pixel {Xi}, i= 1 to nthe probability density of an intensity ‘x’ is given by

• ‘K’ is the kernel function, typically a Gaussian.

Reference Paper - Ahmed Elgammal ,David Harwood, Larry Davis.“Non Parametric Model for Background Subtraction”

SE 263 R. Venkatesh Babu

Pixel Intensity over time

SE 263 R. Venkatesh Babu

Histogram

SE 263 R. Venkatesh Babu

KDE - Example

• data=[randn(100,1);randn(100,1)*2+35 ;randn(100,1)+55];

• kde(data,2^14,min(data)-5,max(data)+5);

• hold on;plot(data,0,'+');

SE 263 R. Venkatesh Babu

Choice of kernel bandwidth

SE 263 R. Venkatesh Babu

Kernel Bandwidth

• Bandwidth decides the smoothness.

• Small bandwidth leads to spurious data artifacts.

• Large bandwidth may result in over smoothing

• Bandwidth has to be chosen optimally.

SE 263 R. Venkatesh Babu

Finding the bandwidth

• Find the median absolute deviation for every 2 consecutive samples for each color channel.

– Median m of |Xi+1 –Xi| for every 2 consecutive samples (Xi+1, Xi) .

– BW =

SE 263 R. Venkatesh Babu

Kernel

• K = N(0,Σ), where, Σ represents kernel BW

• KDE:

– Assuming:

– Then, KDE=

KDE using Gaussian Kernel Generalization of GMM where each sample is considered N(0, Σ).

SE 263 R. Venkatesh Babu

Result

SE 263 R. Venkatesh Babu

Suppression of false Detection:Pixel displacement distribution

• If then we say that particular value is a foreground value. Otherwise it is background.

• To make sure that the detected pixel is a foreground pixel, we define a Pixel displacement distribution given by

• It is a measure of the maximum probability of the pixel being the background pixel of a nearby pixel.

SE 263 R. Venkatesh Babu

Component displacement distribution

Further constraints :

• Foreground and background intensities may be similar.

• So we check if the entire detected object has displaced from nearby pixels.

• For this we define Component displacement distribution given by

SE 263 R. Venkatesh Babu

Component displacement distribution

• Detected Pixel ‘x’ is BG iff

• Neigborhood 5 dia circular neighborhood

SE 263 R. Venkatesh Babu

Results

SE 263 R. Venkatesh Babu

KDE - Results

SE 263 R. Venkatesh Babu

Model Update

• Selective Update:

• Blind Update

SE 263 R. Venkatesh Babu

ViBE = Visual Background Extractor

ViBe: A Universal Background Subtraction Algorithm for Video Sequences,

Olivier Barnich and Marc Van Droogenbroeck (ICASSP ’09 & IEEE Trans. IP ‘11)

SE 263 R. Venkatesh Babu

Model

• Classify a new pixel value with respect to its immediate neighborhood

• Model each background pixel with a set of samples instead of with an explicit pixel model.

– No estimation of pdfs

SE 263 R. Venkatesh Babu

BG Pixel Classification

• Each background pixel xis modeled by a collection of Nbackground sample values

• New pixel value v(x) is BG iff:

> #min

SE 263 R. Venkatesh Babu

ViBE: Parameters

• Radius of Sphere R

– Set at 20 for Monochrome Images

• #min:

– Set as 2

• These parameters are fixed for all pixels/frames

• Adjust the sensitivity by changing the ratio: #min /N

SE 263 R. Venkatesh Babu

Model Initialization

• From a single frame populate the pixel models with values found in the spatial neighborhood of each pixel.

– Values randomly taken in their neighborhood

– Neighborhood chosen – 8 connected

SE 263 R. Venkatesh Babu

Background model initialization

• The Model is initialized as follows:

M0(x) = { v0 (y) | y ∈ NG (x) }

Where,

Vo(y) = Intensity at pixel ‘y’ at t=0

NG(x) = Neighborhood of pixel ‘x’

M0(x) = Background model for pixel ‘x’ at t=0

SE 263 R. Venkatesh Babu

Background model initialization

Problem with this kind of initialization :

Foreground objects being present in the first frame itself – presence of ‘ghosts’ in the background model.

SE 263 R. Venkatesh Babu

Is a pixel Background or Foreground?

Now we have the background model for a pixel ‘x’ as

M(x) = { v1 , v2 , v3 … vN }

Say v(x) is the observed intensity value of pixel ‘x’.

Steps:

• Define a sphere of radius ‘R’ centered at the intensity value v(x).

• Check how many points from M(x) are inside this sphere.

• If more than nthresh values from M(x) are inside the sphere, v(x) is a background value, otherwise not.

SE 263 R. Venkatesh Babu

Is a pixel Background or Foreground?

SE 263 R. Venkatesh Babu

Updating the background model over time

• Background model of a pixel is updated randomly !

SE 263 R. Venkatesh Babu

Updating the background model over time

Conservative Update :• Update the model only if a pixel value is declared as

a background. If it is foreground, do not update at all.

• Consider a parked car that begins to move suddenly. According to conservative update policy, the region uncovered by the car will not be updated. This leads to permanent presence of the car’s “ghost”.

• But in ViBE, this problem is overcome because neighboring pixels can update the background model of a given pixel.

SE 263 R. Venkatesh Babu

Updating the background model over time

Memoryless Update:

• Which value is discarded from the model M(x) is determined randomly.

• Every value in M(x) has equal probability of being discarded after a fixed time.

• The past has no effect in the update process.

SE 263 R. Venkatesh Babu

Model Update

SE 263 R. Venkatesh Babu

Updating the background model over time

Random time Subsampling:• Update the background model randomly.• Say a pixel has been declared as background some ‘k’

times, we update the background model of that pixel only once, out of these ‘k’ times, that too randomly.– K =16 (used) a background pixel value has 1 chance in 16 of

being selected to update its pixel model.

• Advantage: The old model is not discarded immediately.• Disadvantage: Failure whenever there are fast moving

objects.

SE 263 R. Venkatesh Babu

Updating the background model over time

Updating the neighboring pixels’ background values:

If the current pixel value v(x) has been declared a background, we randomly choose a pixel from the neighborhood of x and update its background value.

Assumption: The pixels are spatially correlated.

SE 263 R. Venkatesh Babu

Results

SE 263 R. Venkatesh Babu

Results

SE 263 R. Venkatesh Babu

ViBE software download

http://www2.ulg.ac.be/telecom/research/vibe/

(Pre-compiled)

SE 263 R. Venkatesh Babu

Eigen Backgrounds

• Take N sample images and compute mean and covariance matrix (µb,Cb)

• Diagonalize Cb

• Reduce Dimension by keeping only M Eigen vectors corresponding to M largest Eigen values.

• Project each input image Ii onto the space expanded by the eigen background images

• Detect Moving objects:

“A Bayesian Computer Vision System for Modeling Human Interactions” Nuria M. Oliver, Barbara Rosario, and Alex P. Pentland, IEEE Trans. PAMI, 2000

Eigen Vector Matrix

Where,