A Linear Least-Squares Solution to Elastic Shape-From-Template

download A Linear Least-Squares Solution to Elastic Shape-From-Template

of 9

  • date post

  • Category


  • view

  • download


Embed Size (px)

Transcript of A Linear Least-Squares Solution to Elastic Shape-From-Template

  • A Linear Least-Squares Solution to Elastic Shape-from-Template

    Abed Malti1 Adrien Bartoli2 Richard Hartley31 Fluminance/INRIA, Rennes, France

    2 ALCoV/ISIT, UMR 6284 CNRS/Universite dAuvergne, Clermont-Ferrand, France3Australian National University and NICTA, Canberra, Australia


    We cast SfT (Shape-from-Template) as the search of avector field (X, X), composed of the pose X and the dis-placement X that produces the deformation. We proposethe first fully linear least-squares SfT method modeling elas-tic deformations. It relies on a set of Solid Boundary Con-straints (SBC) to position the template at X in the deformedframe. The displacement is mapped by the stiffness matrixto minimize the amount of force responsible for the defor-mation. This linear minimization is subjected to the Repro-jection Boundary Constraints (RBC) of the deformed shapeX + X on the deformed image. Compared to state-of-the-art methods, this new formulation allows us to obtainaccurate results at a low computation cost.

    1. IntroductionThe SfT problem is to reconstruct a 3D deformed surface

    from its image and a 3D shape template. Most SfT methodsrely on point correspondences between the template and theimage, defining Reprojection Boundary Constraints (RBC).These constrain 3D points to lie on the sightlines from theimage, leaving an infinite number of possible 3D deforma-tions. Physics-based or statistics-based priors are then usedas additional constraints. In physics-based priors, the ge-ometric and mechanical priors are the most used. Whilegeometric priors tend to use invariant measures betweendeformations, mechanical priors represent the physics thatrules the deformation. In parallel to an extensive study ofisometry-based priors [3, 5, 17, 15, 18, 19], mechanical pri-ors [1, 2, 10, 7] have recently become a line of investigationin SfT. They have potential for real applications since theyaddress a wider class of deformable objects such as organs,tissues and other elastic objects. In comparison, isometricSfT is limited to paper and some types of clothes.

    Current geometric/mechanical SfT methods suffer froma certain number of caveats that are summarized in table1. First, the types of deformations: geometric approaches[3, 9] may not behave well when the prior is not fulfilled bythe deformation, while mechanical methods [10, 1] cover alarger range of deformations (such as linear elasticity). Sec-ond, the tradeoff between accuracy and speed of estimation.

    Analytical solutions [3] are fast but with lower accuracy.Non-linear optimization methods [9, 10] are more accuratebut at a higher computation cost. Moreover, their accuracydepends on the initialization and on some hyper-parameters.Methods based on Kalman filtering [1] also require an ini-tialization step besides the fact that errors may accumulateover the considered time frame. In summary, physics-basedapproaches lack a method which is (i) able to exploit themechanical constraints to cover a large deformation range,(ii) robust to noise, and (iii) both accurate and fast. Devel-oping such a method is the main goal of this work. Theanswer we propose is to use mechanical constraints into alinear least-squares estimation framework.

    We use finite elements (FEM) to represent the surfaceand the deformation. This is particularly adapted to SfT andfits the problem we want to solve. Indeed, only the finitediscrete set of point correspondences has a natural bound-ary condition through the RBC. Any other point of the sur-face is free from boundary conditions and is only subject tothe physical prior. Thus, we use the finite set of correspon-dences as nodes of the elements (triangles). Each elementis subject to the mechanical laws that rule the deformation.We use a weak formulation of these laws to derive a linearrelation (via the stiffness matrix) between the displacementof the nodes and the external deforming forces. The assem-bly of all elementary matrices reconstruct the global stiff-ness matrix that links the global external deforming forceto the global deformation X. Minimizing this force sub-ject to the RBC is a linear least-squares problem that can besolved very efficiently.

    2. State-of-the-ArtStatistical constraints usually represent the deformation

    as a linear combination of basis vectors, which can belearned online either for face reconstruction [12] or forgeneric shapes [17, 15, 19]. Non-linear learning methodswere applied in human tracking [16] and then extended tomore generic surfaces [18].

    Physical constraints include spatial and temporal sur-face priors. The isometry constraint [5] requires that anygeodesic distance is preserved by the deformation. Thisapproach has proven its accuracy for paper-like surfacesand was extended to conformal deformations [9, 3]. [3]


  • Method Deformation Time Acc Noise Init/params

    Isometric Conformal Elastic

    analytic-GbM-[3] + + + +iterative-GbM-[9] + + + +

    iterative-MbM-[10] + + + + + sequential-MbM-[1] + + + + +

    linear-MbM-Proposed + + + + + + +

    Table 1. Strong (+) and weak (-) points of state-of-the-art physics-based SfT methods. GbM stands for geometric-based methodand MbM for mechanics-based methods. Time is the computa-tion cost. Acc is the accuracy of the method to recover the de-formed shape. Noise is the sensitivity of the method to imagenoise. Init/params tells if the method requires an initial shape orhyper-parameters (-) or not (+). Isometric SfT methods are limitedto isometric deformations and are not mentioned.




    unknown non-rigid


    rigid mapping


    from state-of-the-art



    2D-3D warp

    solid boundary


    rigid positioning

    of template

    Figure 1. New formalization of the physics-based SfT. The dis-placement field at the solid boundary point is the zero 3-vector.

    studied the well-posedness of both the isometric and con-formal cases. The first SLAM (Simultaneous LocalizationAnd Mapping) method for elastic surfaces was attemptedwith fixed boundary conditions [1] and later extended tofree boundary conditions [2]. This approach relies on theNavier-Stokes fluid-flow equation. It makes use of FEM tomodel the surface and approximates the deformation forceswith Gaussians. Both the surface and the forces are es-timated using an EKF (Extended Kalman Filter). It hasto be noticed that SLAM and SfTare conceptually differ-ent. In the former a time sequential smoother is embed-ded in a Bayesian a-posteriori estimator. At the initializa-tion the scene has to be static. Recently [10] proposed anSfTmethod with mechanical priors. The approach relies onstretching energy minimization under reprojection bound-ary conditions. [7] tested a similar approach with differentmechanical models and orthographic projection. Thoughthis has proven effective, it has the drawbacks of non-lineariterative methods: the need for an initialization (that is dif-ficult to find) and for hyper-parameter setting to obtain anoptimal accuracy.

    FEM techniques with mechanical priors have been ex-tensively used in medical imaging to solve organ segmen-tation from 3D volume scanning modalities [13, 20]. How-ever these methods do not address the registration problemas is usually done in SfT.

    3. Mathematical FormulationLet n be the number of point correspondences. We pro-

    pose to formalize SfT as follows: find the 3D displacementfield (X, X) R3n R3n, where: (i) X minimizes thenorm of the applied forces for the deformation, (ii) X + Xsatisfies the RBC, and (iii) X is a rigid positioning of thetemplate shape in the deformed frame thanks to the SBC.See figure 1 for an illustration. Formally speaking, finding(X, X) involves two main steps:

    1. Find X from X0 with a PnP method applied to thesolid boundary points of SBC. X0 is the template posein the world coordinate frame.

    2. Find X such that:



    2 K X 2 s.t.

    {P X = b RBCS X = 0 SBC


    Matrix K is the stiffness matrix of size 3n3n. Note thatf = K X is thus the vector of external forces to be mini-mized. P is an n-block diagonal matrix of dimension 2 3per block. P and b enforce the fact that the n points mustlie on the sightlines that pass through the n correspondingpoints in the deformed image. S is an m 3n sparse ma-trix. It adds m depth displacement constraints which to-gether with the corresponding 2m, among 2n, equations ofRBC set the m solid boundary points. The constructionof matrix P and vector b are straightforward. Considera point X + X on the deformed shape and its reprojec-tion =

    (x y

    )>in the image. The RBC for this point

    provides two non-homogeneous equations in the unknownpoint displacement X =

    (u v w

    )>. We use perspec-

    tive projection and assume that the effect of the intrinsicparameters were undone on the image coordinates, whichgives:

    U + u = x(W + w)

    V + v = y(W + w)(2)

    or in matrix form as:(1 0 x0 1 y




    = (xW UyW V



    , (3)

    whereX =(U V W

    )>. The global matrix P is written


    P =

    P 1 0 00 P i 00 0 Pn

    (4)and b is written as:

    b =(b1 bi bn


    The next sections are dedicated to the construction of matrixK with the related assumptions of deformation modeling.


  • xyv

    ux y plane y z plane

    z z =@w




    x z planex

    u y =@w











    Figure 2. Local in-plane and out-of-plane deformations. To sim-plify the representation, we do not consider transverse shear in thisexample.

    4. Mechanical ModelingWe adapt the plate theor