Poisson&Mang& - Visualization Lab | UC...

37
Poisson Ma)ng Jian Sun, Jiaya Jia, ChiKeung Tang, heungYeung Shum Presenter: Vera Dadok Discussant: Eileen Bai

Transcript of Poisson&Mang& - Visualization Lab | UC...

Page 1: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Poisson  Ma)ng  Jian  Sun,  Jiaya  Jia,  Chi-­‐Keung  Tang,  heung-­‐Yeung  Shum  

 Presenter:  Vera  Dadok  Discussant:  Eileen  Bai  

Page 2: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Recall:  Ma)ng  •  Ma)ng  equaEon  over  image  

 –  F  –  foreground  –  B  –  background  –  α  –  alpha  maKe    

ResulEng  alpha  maKe  Image  I  Source:  hKp://www.cse.cuhk.edu.hk/~leojia/all_project_webpages/Poisson%20ma)ng/poisson_ma)ng.html  

BFI )1( αα −+=

Page 3: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Recall:  Ma)ng  •  Ma)ng  equaEon  over  image  

•  Under-­‐constrained  equaEon:  –  7  unknown  variables  

•  α    •  R,G,B  channels  of  F  •  R,G,B  channels  of  B  

–  3  known  variables  •  R,G,B  channels  of  I  

•  AddiEonal  informaEon  –  User-­‐defined  Trimap  

Trimap  

Image  I  

BFI )1( αα −+=

Page 4: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Poisson  Ma)ng  Another  approach  to  solving  the  ma)ng  problem  Approach:    

–  Use  gradient  of  image  to  calculate  gradient  of  maKe  –  Reconstruct  maKe  from  gradient  of  maKe  

Key  addi.on:    –  Enables  user  interacEon  to  locally  improve  results  

Page 5: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Poisson  Ma)ng  •  Ma)ng  equaEon  

•  Gradient  of  ma)ng  equaEon  

•  Gradient  rewriKen  

•  Once  gradient  of  α  is              obtained,  we  can  create              maKe  

 

BFBFI ∇−+∇+∇−=∇ )1()( ααα

BFI )1( αα −+=

A = 1F −B

D =α∇F − (1−α)∇B))1((1 BFI

BF∇−−∇−∇

−=∇ ααα

Page 6: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Poisson  Ma)ng  •  Ma)ng  equaEon  

•  Gradient  of  ma)ng  equaEon  

•  Gradient  rewriKen  

•  Once  gradient  of  α  is              obtained,  we  can  create              maKe  

 

)( DIA −∇=∇α

BFBFI ∇−+∇+∇−=∇ )1()( ααα

BFI )1( αα −+=

A = 1F −B

D =α∇F − (1−α)∇B

Page 7: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Poisson  Ma)ng:  Two  steps  

•  Step  1:    Global              (required)  –  Solve  global  Poisson  ma)ng  problem,  assumes  

•  Step  2:    Local                (opEonal)    –  Adjust  local  regions  with  local  Poisson  ma)ng  operaEons  

(1)  Global  Poisson  Ma)ng   (2)  Local  Poisson  Ma)ng  

)( DIA −∇=∇α

D = 0

Page 8: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Step  1:  Global  Poisson  Ma)ng  

Global  soluEon  assumes  D  =  0  in  gradient  equaEon  

–  i.e.  assumes  “smooth”  foreground  and  background  

IntuiEvely  – MaKe  gradient  is  approximately  proporEonal  to  image  gradient  

)( DIA −∇=∇α

A = 1F −B

D =α∇F − (1−α)∇B

∇α ≈∇IF −B

D =α∇F − (1−α)∇B = 0

General  Case   Global  Case  with  AssumpEon  

Page 9: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

What  does  ‘smooth’  mean?  D =α∇F − (1−α)∇B = 0

BFBFI ∇−+∇+∇−=∇ )1()( ααα

Color  Image   Grayscale  I  

Solve  for  alpha  using  smooth  assump.on,  D=0  What  should  the  ma>e  edge  be?  

Page 10: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

What  does  ‘smooth’  mean?  D =α∇F − (1−α)∇B = 0

BFBFI ∇−+∇+∇−=∇ )1()( ααα

Grayscale  I  

Gradient  α  (in  x)  Gradient  I  (in  x)  

α  

∇α ≈∇IF −B

Page 11: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

What  does  ‘smooth’  mean?  D =α∇F − (1−α)∇B = 0

BFBFI ∇−+∇+∇−=∇ )1()( ααα

F  

α  

B  Does  this  seem  like  a  good  ma>e?    Was  this  “smooth”?  

Page 12: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

What  does  ‘smooth’  mean?  

Color  Image   Grayscale  I  

D =α∇F − (1−α)∇B = 0BFBFI ∇−+∇+∇−=∇ )1()( ααα

What  should  the  ma>e  edge  be?  

Page 13: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

What  does  ‘smooth’  mean?  

Color  Image  

Does  this  seem  like  a  good  ma>e?  Was  this  “smooth”  assump.on  good?      

F  

α  

B  

D =α∇F − (1−α)∇B = 0BFBFI ∇−+∇+∇−=∇ )1()( ααα

Page 14: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Step  1:  Global  SoluEon  method  This  simplified  equaEon  is  solved  in  grayscale  channel  

–  2  known  variables:    •  gradients  of  I  

–  3  unknown  variables:    •  (F-­‐B),  treated  as  a  single  variable  •  gradients  of  α    

∇α ≈∇IF −B

+  AddiEonal  knowledge  •  Boundary  condiEons  from  Trimap  

Page 15: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Step  1:  Global  SoluEon  method  Minimize  error  in  

In  the  sense  of  least  squares    

This  has  an  associated  Poisson  equaEon  with  Dirichlet  boundary  condiEons  (due  to  Trimap),  which  we  know  how  to  solve  

∇α ≈∇IF −B

This  is  solved  iteraEvely,  approximaEng  F-­‐B,  then  solving  for  α,  then  repeat    

Page 16: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Global  Poisson  Ma)ng  

Results  good  for  smooth  F  and  B    (thus  D≈0)  

Source:  hKp://www.cse.cuhk.edu.hk/~leojia/all_project_webpages/Poisson%20ma)ng/poisson_ma)ng.html  

Page 17: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Why  do  we  need  local  operaEons?  

Global  Poisson  maFng  assumed  D=0  

Exact  gradient  equaEon  

)( DIA −∇=∇αA = 1

F −BD =α∇F − (1−α)∇B

Page 18: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Step  2:  Local  OperaEons  

Do  not  assume  D=0  •  Goal:  change  A,  D  to  produce  beKer  approximaEon  of  α

–  ‘Fix’ problematic areas in global result  •  Two  categories  

–  Channel  SelecEon  –  Local  filters  (brushes)  

 

Δα = div(A(∇I −D))

)( DIA −∇=∇α

•  Aner  local  α-­‐gradient  changes  solve  local  Poisson  equaEon  with  Dirichlet  condiEons  –  Key:  local  changes  in  α-­‐gradient  do  not  produce  

arEfacts  in  final  maKe  

Page 19: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Local  OperaEons:  Channel  SelecEon  •  Create  new  weighted  channel  of  R,  G,  B  to  minimize  |D|    

–  Instead  of  grayscale  channel  –  Reduces  error  in  D  

•  Process  –  User  selects  background  samples  –  AutomaEcally  calculates  channel  weights  to  minimize  variance  of  foreground  and  background  colors  

)( DIA −∇=∇α

grayscale    

new    weighted  channel  

α  maKe  channel  

Page 20: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Local  OperaEons:  High-­‐pass  filter  

•  Low  pass  filter  on  D  •  High-­‐pass  on  α  due  to            term  – D  is  esEmated  using  low-­‐frequencies  of  gradient  

before  

aner  

)( DIA −∇=∇α

(∇I −D)

MaKe  gradient   MaKe  

Page 21: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Local  OperaEons:  BoosEng  brush  

•  Increase  or  decrease  sharpness    

•  Gaussian  filter  on  A  

before  

aner  

)( DIA −∇=∇α

MaKe  gradient   MaKe  

Page 22: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Local  OperaEons:  Diffusion  filtering  

•  Remove  JPEG  arEfacts  

before  

aner  

)( DIA −∇=∇α

MaKe  gradient   MaKe  

Page 23: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Local  OperaEons:  Clone  Brush  

•  Exactly  copy  gradient  )( DIA −∇=∇α

MaKe  gradient   MaKe  

Erase  brush    Inverse  brush    

Page 24: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Full  algorithm  summary  0:    Create  Trimap                    (user  defined)  1:    Global  Poisson  ma)ng            (automaEc)  2:    Apply  local  Poisson  operators          (user  defined)  

 (i)    Apply  channel  selecEon  (smoother  background  or  foreground,  reduces  |D|)    (ii)    Apply  highpass  filtering  to  obtain  approximaEon  of  D    (iii)    Apply  boosEng  brush  to  change  A    (iv)  Apply  clone  brush  to  disEnguish  gradients  

   Note:  the  order  of  these  steps  is  important!    Erase  brush  and  inverse  brush  can    be  applied  any  Eme  

Page 25: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Video  break  

Global  SoluEon:  1:10  Local  SoluEons:  1:30    

Page 26: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Full  algorithm  summary  

Source:  hKp://www.cse.cuhk.edu.hk/~leojia/all_project_webpages/Poisson%20ma)ng/poisson_ma)ng.html  

User  (Trimap)  

Page 27: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Comparison  to  other  methods  

•  Via  picture  

Page 28: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Comparison  to  other  methods  

•  Via  picture  

Page 29: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Improvements  over  other  methods  

•  Natural  image  ma)ng  including  Bayesian  Ma)ng  – Not  easy  to  adjust  output  of  algorithms  if  mistakes  are  made  in  alpha  maKe  

•  Difference  ma)ng  and  triangulaEon  ma)ng  – Need  mulEple  input  images  

Page 30: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

ApplicaEons  

•  Image  composiEng  

Page 31: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Other  applicaEons  

•  De-­‐fogging  

Page 32: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Other  ApplicaEons  

•  MulE-­‐background  

Page 33: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Thank  you!  

QuesEons?  

Page 34: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Discussion  

Page 35: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

LimitaEons  1.)  User-­‐defined  local  operaEons  

l  Relies  on  the  user  for  input  l  Good  enough?  l  Background  knowledge  of  brushes  

2.)  Specific  order  of  operaEons  l  Restricts  freedom/flexibility  

   3.)  Similar  foreground/background  elements  

l  Hard  to  disEnguish  underlying  structure  of  maKe  l  Difficult  for  computers  as  well  as  humans  

 

*First  two  points  could  be  argued  as  posiEve  qualiEes  as  well.  

4.)  MulEple  disEnct  foreground  elements  l  Requires  complicated  trimap  l  Bayesian  could  solve  relaEvely  simply  

   

Page 36: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Improvements  

l  Ideas?  l  Separate  “hair”  from  “fur”  -  These  two  come  up  the  most  onen  -  Have  very  different  properEes  

Page 37: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung

Future  Work  

l  Extends  naturally  to...      Video  ma)ng!  -  “Real-­‐Time  Video  Ma)ng  Using  MulEchannel  Poisson  EquaEons”  -­‐  2010  

-  AutomaEcally  generated  trimaps    

l  CombinaEon  of  Poisson  and  Bayesian  Ma)ng  -  Combine  user-­‐defined  local  operaEons  with  the  pixel  by  pixel  

esEmaEon  in  Bayes'