GAUGED FLOER HOMOLOGY FOR HAMILTONIAN ISOTOPIES I: DEFINITION …guangbox/VHF1.pdf ·...

of 47/47
GAUGED FLOER HOMOLOGY FOR HAMILTONIAN ISOTOPIES I: DEFINITION OF THE FLOER HOMOLOGY GROUPS GUANGBO XU Abstract. We construct the vortex Floer homology group VHF (M,μ; H) for an aspherical Hamil- tonian G-manifold (M,ω) with moment map μ and a class of G-invariant Hamiltonian loop Ht , following the proposal of [3]. This is a substitute for the ordinary Hamiltonian Floer homology of the symplectic quotient of M. We achieve the transversality of the moduli space by the classical perturbation argument instead of the virtual technique, so the homology can be defined over Z or Z2. Contents 1. Introduction 1 2. Basic setup and outline of the construction 7 3. Asymptotic behavior of the connecting orbits 14 4. Fredholm theory 21 5. Compactness of the moduli space 28 6. Floer homology 31 Appendix A. Transversality by perturbing the almost complex structure 37 References 46 1. Introduction 1.1. Background. Floer homology, introduced by Andreas Floer (see [8], [9]), has been a great triumph of J -holomorphic curve technique invented by Gromov [17] in many areas of mathemat- ics. Hamiltonian Floer homology gives new invariants of symplectic manifolds and its Lagrangian submanifolds and has been the most important approach towards the solution to the celebrated Arnold conjecture initiated in the theory of Hamiltonian dynamics; the Lagrangian intersection Floer homology is the basic language in defining the Fukaya category of a symplectic manifold and stating Kontsevich’s homological mirror symmetry conjecture; several Floer-type homology theory, including the instanton Floer homology ([7], [4]), Heegaard-Floer theory ([29]), Seiberg- Witten Floer homology ([22]), ECH theory ([20], [21]), has become tools of understanding lower dimensional topology. All these different types of Floer theory, are all certain infinite dimensional Morse theory, whose constructions essentially apply Witten’s point of view ([34]). Basically, if f : X R is certain Date : December 24, 2013. 1
  • date post

    12-Mar-2020
  • Category

    Documents

  • view

    2
  • download

    0

Embed Size (px)

Transcript of GAUGED FLOER HOMOLOGY FOR HAMILTONIAN ISOTOPIES I: DEFINITION …guangbox/VHF1.pdf ·...

  • GAUGED FLOER HOMOLOGY FOR HAMILTONIAN ISOTOPIES I:

    DEFINITION OF THE FLOER HOMOLOGY GROUPS

    GUANGBO XU

    Abstract. We construct the vortex Floer homology group V HF (M,µ;H) for an aspherical Hamil-

    tonian G-manifold (M,ω) with moment map µ and a class of G-invariant Hamiltonian loop Ht,

    following the proposal of [3]. This is a substitute for the ordinary Hamiltonian Floer homology of

    the symplectic quotient of M . We achieve the transversality of the moduli space by the classical

    perturbation argument instead of the virtual technique, so the homology can be defined over Z orZ2.

    Contents

    1. Introduction 1

    2. Basic setup and outline of the construction 7

    3. Asymptotic behavior of the connecting orbits 14

    4. Fredholm theory 21

    5. Compactness of the moduli space 28

    6. Floer homology 31

    Appendix A. Transversality by perturbing the almost complex structure 37

    References 46

    1. Introduction

    1.1. Background. Floer homology, introduced by Andreas Floer (see [8], [9]), has been a great

    triumph of J-holomorphic curve technique invented by Gromov [17] in many areas of mathemat-

    ics. Hamiltonian Floer homology gives new invariants of symplectic manifolds and its Lagrangian

    submanifolds and has been the most important approach towards the solution to the celebrated

    Arnold conjecture initiated in the theory of Hamiltonian dynamics; the Lagrangian intersection

    Floer homology is the basic language in defining the Fukaya category of a symplectic manifold

    and stating Kontsevich’s homological mirror symmetry conjecture; several Floer-type homology

    theory, including the instanton Floer homology ([7], [4]), Heegaard-Floer theory ([29]), Seiberg-

    Witten Floer homology ([22]), ECH theory ([20], [21]), has become tools of understanding lower

    dimensional topology.

    All these different types of Floer theory, are all certain infinite dimensional Morse theory, whose

    constructions essentially apply Witten’s point of view ([34]). Basically, if f : X → R is certain

    Date: December 24, 2013.

    1

  • 2 GUANGBO XU

    smooth functional on manifold X (which could be infinite dimensional), then with an appropriate

    choice of metric on X, we can study the equation of negative gradient flow of f , of the form

    x′(t) +∇f(x(t)) = 0, t ∈ (−∞,+∞). (1.1)

    If some natural energy functional defined for maps from R to X is finite for a solution to the aboveequation, then x(t) will converges to a critical point of f . Assuming that all critical points of f is

    nondegenerate, then usually we can define a Morse-type index (or relative indices) λf : Critf →Z. Then for a given pair of critical points a−, a+ ∈ Critf , the moduli space of solutions to thenegative gradient flow equation which are asymptotic to a± as t → ±∞, denoted by M(a−, a+),has dimension equal to λf (a−)−λf (a+), if f and the metric are perturbed generically. If λf (a−)−λf (a+) = 1, because of the translation invariance of (1.1), we expect to have only finitely many

    geometrically different solutions connecting a− and a+. In many cases (which we call the oriented

    case), we can also associate a sign to each such solutions.

    On the other hand, we define a chain complex over Z2 (and over Z in the oriented case), spannedby critical points of f and graded by the index λf ; the boundary operator ∂ is defined by the

    (signed) counting of geometrically different trajectories of solutions to (1.1) connecting two critical

    points with adjacent indices. We expect a nontrivial fact that ∂ ◦ ∂ = 0. So a homology group isderived.

    1.2. Hamiltonian Floer homology and the transversality issue. In Hamiltonian Floer the-

    ory, we have a compact symplectic manifold (X,ω) and a time-dependent Hamiltonian Ht ∈C∞(X), t ∈ [0, 1]. We can define an action functional AH on a covering space L̃X of the con-tractible loop space of X. The space L̃X consists of pairs (x,w) where x : S1 → X is a contractibleloop and w : D→ X with w|∂D = x; the action functional is defined as

    AH(x,w) = −∫Dw∗ω −

    ∫S1Ht(x(t))dt. (1.2)

    The Hamiltonian Floer homology is formally the Morse homology of the pair(L̃X,AH

    ).

    The critical points are pairs (x,w) where x : S1 → X satisfying x′(t) = XHt(x(t)), where XHt isthe Hamiltonian vector field associated to Ht; these loops are 1-periodic orbits of the Hamiltonian

    isotopy generated by Ht. Then, choosing a smooth S1-family of ω-compatible almost complex

    structures Jt on X which induces an L2-metric on the loop space of X, (1.1) is written as the Floer

    equation for a map u from the infinite cylinder Θ = R× S1 to X, as∂u

    ∂s+ Jt

    (∂u

    ∂t−XHt(u)

    )= 0. (1.3)

    Here (s, t) is the standard coordinates on Θ. This is a perturbed Cauchy-Riemann equation, so

    Gromov’s theory of pseudoholomorphic curves is adopted in Floer’s theory.

    There is always an issue of perturbing the equation in order to make the moduli spaces transverse,

    so that the ambiguity of counting of solutions doesn’t affect the resulting homology. Floer originally

    defined the Floer homology in the monotone case, which was soon extended by Hofer-Salamon

    ([19]) and Ono ([28]) to the semi-positive case. Finally by applying the “virtual technique”, Floer

    homology is defined for general compact symplectic manifold by Fukaya-Ono ([15]) and Liu-Tian

    ([23]).

  • GAUGED FLOER HOMOLOGY 3

    1.3. Hamiltonian Floer theory in gauged σ-model. In this paper, we consider a new type of

    Floer homology theory proposed in [3] and motivated from Dostoglou-Salamon’s study of Atiyah-

    Floer conjecture (see [6]). The main analytical object is the symplectic vortex equation, which was

    also independently studied initially in [3] and by Ignasi Mundet in [25].

    The symplectic vortex equation is a natural elliptic system appearing in the physics theory

    “2-dimensional gauged σ-model”. Its basic setup contains the following ingredients:

    (1) The target space is a triple (M,ω, µ), where (M,ω) is a symplectic manifold with a Hamil-

    tonian G-action, and µ is a moment map of the action. We also choose a G-invariant,

    ω-compatible almost complex structure J on M .

    (2) The domain is a triple (P,Σ,Ω), where Σ is a Riemann surface, P → Σ is a smooth G-bundle, and Ω is an area form on Σ.

    (3) The “fields” are pairs (A, u), where A is a smooth G-connection on P , and u : Σ→ P ×GMis a smooth section of the associated bundle.

    Then we can write the system of equation on (A, u):

    {∂Au = 0;

    ∗FA + µ(u) = 0.(1.4)

    Here ∂Au is the (0, 1)-part of the covariant derivative of u with respect to A; FA is the curvature

    2-form of A; ∗ is the Hodge star operator associated to the conformal metric on Σ with area formΩ; µ(u) is the composition of µ with u, which, after choosing a biinvariant metric on g, is identified

    with a section of adP → Σ. This equation contains a symmetry under gauge transformations onP . Moreover, its solutions are minimizers of the Yang-Mills-Higgs functional:

    YMH(A, u) := 12

    (‖dAu‖2L2 + ‖FA‖

    2L2 + ‖µ(u)‖

    2L2

    )(1.5)

    which generalizes the Yang-Mills functional in gauge theory and the Dirichlet energy in harmonic

    map theory.

    Now, similar to Hamiltonian Floer theory, consider the following action functional on a covering

    space of the space of contractible loops in M × g. Let H : M × S1 → R be an S1-family of G-invariant Hamiltonians; for any contractible loop x̃ := (x, f) : S1 → M × g with a homotopy classof extensions of x : S1 →M , represented by w : D→M , the action functional (given first in [3]) is

    ÃH(x, f, w) := −∫Dw∗ω +

    ∫S1

    (µ(x(t)) · f(t)−Ht(x(t))) dt. (1.6)

    The critical loops of ÃH corresponds to periodic orbits of the induced Hamiltonian on the symplecticquotient M := µ−1(0)/G. The equation of negative gradient flows of ÃH , is just the symplecticvortex equation on the trivial bundle G × Θ, with the standard area form Ω = ds ∧ dt, and theconnection A is in temporal gauge (i.e., A has no ds component). If choosing an S1-family of

    G-invariant, ω-compatible almost complex structures Jt, then the equation is written as a system

  • 4 GUANGBO XU

    of (u,Ψ) : Θ→M × g: ∂u

    ∂s+ Jt

    (∂u

    ∂t+XΨ(u)− YHt

    )= 0;

    ∂Ψ

    ∂s+ µ(u) = 0.

    (1.7)

    Solutions with finite energy are asymptotic to loops in CritÃH . Then the moduli space of suchtrajectories, especially those zero-dimensional ones, gives the definition of the boundary operator in

    the Floer chain complex, and hence the Floer homology group. We call these homology theory the

    vortex Floer homology. We have to use certain Novikov ring Λ, which will be defined in Section

    2, as the coefficient ring, and the vortex Floer homology will be denoted by V HF (M,µ;H,J ; Λ).

    The main part of this paper is devoted to the analysis about (1.7) and its moduli space, in order

    to define V HF (M,µ;H,J ; Λ).

    1.4. Lagrange multipliers. The action functional (1.6) seems to be already complicated, not to

    mention its gradient flow equation (1.7). However, the action functional (1.6) is just a Lagrange

    multiplier of the action functional (1.2). Indeed there is a much simpler situation in the case of

    the Morse theory of a finite-dimensional Lagrange multiplier function, which is worth mentioning

    in this introduction as a model.

    Suppose X is a Riemannian manifold and µ : X → R is a smooth function, with 0 a regularvalue. Then consider a function f : X → R whose restriction to X = µ−1(0) is Morse. Thencritical points of f |X are the same as critical points of the Lagrange multiplier F : X × R → Rdefined by F (x, η) = f(x) + ηµ(x), and the Morse index as a critical point of f |X is one less thanthe index as a critical point of F . Then instead of considering the Morse-Smale-Witten complex

    of f |X , we can consider that of F . In generic situation, these two chain complexes have the samehomology (with a grading shifting), and a concrete correspondence can be constructed through the

    “adiabatic limit” (for details, see [33]).

    Indeed, the vortex Floer homology proposed by Cieliebak-Gaio-Salamon and studied in this

    paper is an infinite-dimensional and equivariant generalization of this Lagrange multiplier technique.

    Therefore, the vortex Floer homology is expected to coincide with the ordinary Hamiltonian Floer

    homology of the symplectic quotient (the proof of this correspondence will be treated in separate

    work).

    1.5. Advantage in achieving transversality. It seems that by considering the complicatd equa-

    tion (1.7) and the moduli spaces we can only recover what we have known of the Hamiltonian Floer

    theory of the symplectic quotient. But the trade-off is that the most crucial and sophisticated

    step–transversality of the moduli space–can be achieved more easily. The advantage of lifting to

    gauged σ-model is because, in many cases, M has simpler topology than M . So the issue caused by

    spheres with negative Chern numbers is ruled out by topological reason. This phenomenon allows

    us to achieve transversality of the moduli space by using the traditional “concrete perturbation” to

    the equation. Moreover, when using virtual technique, the Floer homology group of the symplectic

    quotient can only be defined over Q but here it can be defined over Z or Z2.

  • GAUGED FLOER HOMOLOGY 5

    1.6. Computation of the Floer homology group and adiabatic limits. The ordinary Hamil-

    tonian Floer homology HF (M,H) of a compact symplectic manifold can be shown to be canon-

    ically independent of the Hamiltonian H and to be isomorphic to the singular homology of M .

    This correspondence plays significant role in proving the Arnold conjecture. To prove this isomor-

    phism, basically two methods have been used. One is to use a time-independent Morse function

    as the Hamiltonian, and try to prove that when the function is very small in C2-norm, there is no

    “quantum contribution” when defining the boundary operator in the Floer chain complex; this was

    also Floer’s original argument. Another is via the Piunikhin-Salamon-Schwarz (PSS) construction,

    introduced in [30].

    For the case of the vortex Floer homology, it has been well-expected to be isomorphic to the

    singular homology of the symplectic quotient M . To prove this isomorphism we can try the similar

    methods as for the ordinary Hamiltonian Floer homology (which we will discuss in Section 6.3), as

    well as the adiabatic limit method, which we discuss here.

    Indeed, for any λ > 0, we consider a variation of (1.7)∂u

    ∂s+ Jt

    (∂u

    ∂t+XΨ(u)− YHt

    )= 0;

    ∂Ψ

    ∂s+ λ2µ(u) = 0.

    (1.8)

    This can be viewed as the symplectic vortex equation over the cylinder R × S1 with area formreplaced by λ2dsdt. The moduli space of solutions to the above equation also defines a Floer

    homology group, and by continuation method we can show that this homology is (canonically)

    independent of λ.

    Then we would like to let λ approach to ∞. By a simple energy estimate, solutions of (1.8) will“sink” into the symplectic quotient M and become Floer trajectors of the induced pair (H, J); at

    isolated points there will be energy blow up, and certain “affine vortices” will appear, which are

    finite energy solutions to the symplectic vortex equation over the complex plane C. In the Gromov-Witten setting, the work of Gaio-Salamon [16] shows that (in special cases), the Hamiltonian-

    Gromov-Witten invariants with low degree insertions coincide with the Gromov-Witten invariants

    of the symplectic quotient, via the Kirwan map κ : H∗G(M) → H∗(M). The high degree partshall be corrected, by the contribution from the affine vortices. This leads to the definition of the

    “quantum Kirwan map” (see [38], [35]).

    In the case of vortex Floer homology, as long as we can carefully analyze the contribution of

    affine vortices (maybe with similar restriction on M as in [16]), we could prove that V HF (M,µ;H)

    is isomorphic to HF (M ;H), with appropriate changes of coefficients.

    It is an interesting topic to consider the reversed limit λ→ 0, and it actually motivated the workof the author with S. Schecter [33], where they considered the nonequivariant, finite dimensional

    Morse homology. In [33] it was shown that, the Morse-Smale trajectories, as λ→ 0, will converge tocertain “fast-slow” trajectories, and the counting of such trajectories defines a new chain complex,

    which also computes the same homology.

    1.7. Gauged Floer theory for Lagrangian intersections. In Frauenfelder’s thesis and [12], he

    used the symplectic vortex equation on the strip R× [0, 1] to define the “moment Floer homology”

  • 6 GUANGBO XU

    for certain types of pairs of Lagrangians (L0, L1) in M . The Lagrangians are not G-invariant

    in general, but their intersections with µ−1(0) reduce to a pair of Lagrangians (L0, L1) in the

    symplectic quotient M . Then by the calculation in the Morse-Bott case, he managed to prove the

    Arnold-Givental conjecture with certain topological assumption on M .

    Woodward also defined a version of gauged Floer theory in [36], where he considered a pair of

    Lagrangians L0, L1 in the symplectic quotient M . They lift to a pair of G-invariant Lagrangians

    L0, L1 ⊂ µ−1(0) ⊂M . Then his equation for connecting orbits is the naive limit of the symplecticvortex equation on the strip R × [0, 1], by setting the area form to be zero. Since the strip iscontractible, the equation is just the J-holomorphic equation on the strip, and two solutions are

    regarded equivalent if they differ by a constant gauge transformation. Then he applied this Floer

    theory to the fibres of the toric moment map for any toric orbifold and showed the relation between

    the nondisplacibility of toric fibres and the Hori-Vafa potential, which reproduces and extends the

    results of Fukaya et. al. [13] [14].

    Both of the above take advantage of the simpler topology of M than the symplectic quotient, as

    we mentioned above, to avoid certain virtual technique. Further work are expected to relate the

    Hamiltonian gauged Floer theory we studied here and the Lagrangian versions, for example, by

    constructing the so-called “open-close map”.

    1.8. Organization and conventions of this paper. In Section 2 we give the basic setup, includ-

    ing the action functional, the definition of the Floer chain complex and the equation of connecting

    orbits. In Section 3 we proved that each finite energy solution is asymptotic to critical loops of the

    action functional. In Section 4 we study the Fredholm theory of the equation of connecting orbits

    (modulo gauge transformations); we show that the linearized operator is a Fredholm operator whose

    index is equal to the difference of Conley-Zehnder indices of the two ends of the connecting orbit. In

    Section 5 we prove that our moduli space is compact up to breaking, if assuming the nonexistence

    of nontrivial holomorphic spheres. In Section 6 we summarize the previous constructions and give

    the definition of the vortex Floer homology (where we postpone the proof of transversality). We

    also prove the invariance of the homology group by using continuation method. In the final Section

    we give some discussions on our further work along this line.

    In the appendices we provide detailed proof of several technical theorems. Most importantly,

    we showed that by using concrete perturbation of the almost complex structure, we can achieve

    transversality of the moduli space, which allows us to avoid the more sophisticated virtual technique.

    We use Θ to denote the infinite cylinder R×S1, with the axial coordinate s and angular coordinatet. We denote Θ+ = [0,+∞)× S1 and Θ− = (−∞, 0]× S1.G is a connected compact Lie group, with Lie algebra g. Any G-bundle over Θ is trivial, and we

    just consider the trivial bundle P = G×Θ. Any connection A can be written as a g-valued 1-formon Θ. We always use Φ to denote its ds component and Ψ to denote its dt component.

    There is a small � > 0 such that for the �-ball g∗� ⊂ g∗ centered at the origin of g∗, U� := µ−1(g∗� )can be identified with µ−1(0)× g∗� . We denote by πµ : U� → µ−1(0) the projection on the the firstcomponent, and by πµ : U� →M the composition with the projection µ−1(0)→M .

  • GAUGED FLOER HOMOLOGY 7

    1.9. Acknowledgments. The author would like to thank his PhD advisor Gang Tian for introduc-

    ing him to this field and the support and encouragement. He would like to thank Urs Frauenfelder,

    Kenji Fukaya, Chris Woodward, and Weiwei Wu for many helpful discussions and encouragement.

    2. Basic setup and outline of the construction

    Let (M,ω) be a symplectic manifold. We assume that it is aspherical, i.e., for any smooth map

    f : S2 → M ,∫S2f∗ω = 0. This implies that for any ω-compatible almost complex structure J on

    M , there is no nonconstant J-holomorphic spheres.

    Let G be a connected compact Lie group which acts on M smoothly. The infinitesimal action g 3ξ 7→ Xξ ∈ Γ(TM) is an anti-homomorphism of Lie algebra. We assume the action is Hamiltonian,which means that there exists a smooth function µ : M → g∗ satisfying

    µ(gx) = µ(x) ◦Ad−1g , ∀ξ ∈ g, d (µ · ξ) = ιXξω. (2.1)

    Suppose we have a G-invariant, time-dependent Hamiltonian H ∈ C∞c(M × S1

    )with compact

    support. For each t ∈ S1, the associated Hamiltonian vector field YHt ∈ Γ(TM) is determined by

    ω(YHt , ·) = dHt ∈ Ω1(M). (2.2)

    The flow of YHt is a one-parameter family of diffeomorphisms

    φHt : M →M,dφHt (x)

    dt= YHt

    (φHt (x)

    )(2.3)

    which we call a Hamiltonian path.

    To achieve transversality, we put the following restriction on Ht. It will be used only in the

    appendix.

    Hypothesis 2.1. There exists a nonempty interval I ⊂ S1 such that Ht ≡ 0 for t ∈ I.

    On the other hand, we need to put several assumptions to the given structures, which are still

    general enough to include the most important cases (e.g., toric manifolds as symplectic quotients

    of Euclidean spaces).

    Hypothesis 2.2. We assume that µ : M → g∗ is proper, 0 ∈ g∗ is a regular value of µ and theG-action restricted to µ−1(0) is free.

    With this hypothesis, µ−1(0) is a smooth submanifold of M and the symplectic quotient M :=

    µ−1(0)/G is a symplectic manifold, which has a canonically induced symplectic form ω. Also, the

    Hamiltonian function Ht descends to a time-dependent Hamiltonian

    Ht : M → R (2.4)

    by the G-invariance of Ht. It is easy to check that YHt is tangent to µ−1(0) and the projection

    µ−1(0)→M pushes YHt forward to YHt . Then we assume

    Hypothesis 2.3. The induced Hamiltonian Ht : M → R is nondegenerate in the usual sense.

    Finally, in the case when M is noncompact, we need the convexity assumption (cf. [2, Definition

    2.6]).

  • 8 GUANGBO XU

    Hypothesis 2.4. There exists a pair (f, J), where f : M → [0,+∞) is a G-invariant and properfunction, and J is a G-invariant, ω-compatible almost complex structure on M , such that there

    exists a constant c0 > 0 with

    f(x) ≥ c0 =⇒ 〈∇ξ∇f(x), ξ〉+ 〈∇Jξ∇f(x), Jξ〉 ≥ 0, df(x) · JXµ(x) ≥ 0, ∀ξ ∈ TxM. (2.5)

    In this paper, to achieve transversality, we need to perturb J near µ−1(0) (see the appendix).

    The above condition is only about the behavior “near infinity”, so such perturbations don’t break

    the hypothesis.

    2.1. Equivariant topology.

    2.1.1. Equivariant spherical classes. Recall that the Borel construction of M acted by G is MG :=

    EG ×G M , where EG → BG is a universal G-bundle over the classifying space BG. Then theequivariant (co)homology of M is defined to be the ordinary (co)homology of MG, denoted by

    HG∗ (M) for homology and H∗G(M) for cohomology.

    On the other hand, for any smooth manifold N , we denote by S2(N) to be the image of the

    Hurwitz map π2(N) → H2(N ;Z), and classes in S2(N) are called spherical classes. We define theequivariant spherical homology of M to be SG2 (M) := S2(MG).

    Geometrically, any generator of SG2 (M) can be represented by the following object: a smooth

    principal G-bundle P → S2 and a smooth section φ : S2 → P ×GM . We denote the class of thepair (P, φ) to be [P, φ] ∈ SG2 (M).

    2.1.2. Equivariant symplectic form and equivariant Chern numbers. The equivariant cohomology

    of M can also be computed using the equivariant de Rham complex(Ω∗(M)G, dG

    ). In Ω2(M)G,

    there is a distinguished closed form ω−µ, called the equivariant symplectic form, which representsan equivariant cohomology class.

    We are interested in the pairing 〈[ω − µ], [P, u]〉 ∈ R. It can be computed in the following way.Choose any smooth connection A on P . Then there exists an associated closed 2-form ωA on

    P ×G M , called the minimal coupling form. If we trivialize P locally over a subset U ⊂ S2,such that A = d+ α, α ∈ Ω1(U, g) with respect to this trivialization, then ωA can be written as

    ωA = π∗ω − d(µ · α) ∈ Ω2(U ×M). (2.6)

    Then we have

    〈[ω − µ], [P, u]〉 =∫S2u∗ωA (2.7)

    which is independent of the choice of A.

    On the other hand, any G-invariant almost complex structure J on X makes TX an equivariant

    complex vector bundle. So we have the equivariant first Chern class

    cG1 := cG1 (TM) ∈ H2G(M ;Z). (2.8)

    This is independent of the choice of J .

  • GAUGED FLOER HOMOLOGY 9

    2.1.3. Kirwan maps. The cohomological Kirwan map is a map

    κ : H∗(M ;R)→ H∗(M ;R). (2.9)

    Here we take R-coefficients for simplicity. It is easy to check that

    κ([ω − µ]) = [ω] ∈ H2(M ;R), κ(cG1 ) = c1(TM) ∈ H2(M ;R). (2.10)

    We define

    NG2 (M) = ker[ω − µ] ∩ kercG1 ⊂ SG2 (M), N2(M) = ker[ω] ∩ kerc1(TM) ⊂ S2(M), (2.11)

    and

    Γ := SG2 (M)/NG2 (M). (2.12)

    2.2. The spaces of loops and equivalence classes. Let P̃ be the space of smooth contractibleparametrized loops in M × g and a general element of P̃ is denoted by

    x̃ := (x, f) : S1 →M × g. (2.13)

    Let P̃ be a covering space of P̃, consisting of triples x := (x, f, [w]) where x̃ = (x, f) ∈ P and[w] is an equivalence class of smooth extensions of x to the disk D. The equivalence relation isdescribed as follows. For each pair w1, w2 : D → M both bounding x : S1 → M , we have thecontinuous map

    w12 := w1#(−w2) : S2 →M (2.14)

    by gluing them along the boundary x. We define

    w1 ∼ w2 ⇐⇒ [w12] = 0 ∈ SG2 (M). (2.15)

    Denote by LG := L∞G := C∞(S1, G) the smooth free loop group of G. Then for any point

    x0 ∈M , we have the homomorphism

    l(x0) : π1(G)→ π1(M,x0) (2.16)

    which is induced by mapping a loop t 7→ γ(t) ∈ G to a loop t 7→ γ(t)x0 ∈ M . For differentx1 ∈M and a homotopy class of paths connecting x0 and x1, we have an isomorphism π1(M,x0) 'π1(M,x1); it is easy to see that l(x0) and l(x1) are compatible with this isomorphism. This means

    kerl(x0) ⊂ π1(G) is independent of x0. Then we define

    LMG :={γ : S1 → G | [γ] ∈ kerl(x0) ⊂ π1(G)

    }. (2.17)

    Let L0G ⊂ LMG be the subgroup of contractible loops in G.It is easy to see that LMG acts on P̃ (on the right) by

    P̃ × LMG → P̃((x, f), h) 7→ h∗(x, f)(t) =

    (h(t)−1x(t),Ad−1h(t)(f(t)) + h(t)

    −1∂th(t)).

    (2.18)

    Here the action on the second component can be viewed as the gauge transformation on the space

    of G-connections on the trivial bundle S1×G. (For short, we denote by d log h the g-valued 1-formh−1dh, which is the pull-back by h of the left-invariant Maurer-Cartan form on G.)

  • 10 GUANGBO XU

    But LMG doesn’t act on P̃ naturally; only the subgroup L0G does: for a contractible loop

    h : S1 → G, extend h arbitrarily to h : D→ G. The homotopy class of extensions is unique becauseπ2(G) = 0 for any connected compact Lie group ([1]). Then the class of (h

    −1x, h∗f, [h−1w]) in P̃

    is independent of the extension. It is easy to see that the covering map P̃→ P̃ is equivariant withrespect to the inclusion L0G→ LMG. Hence it induces a covering

    P̃/L0G→ P̃/LMG. (2.19)

    2.3. The deck transformation and the action functional. We now define an action of SG2 (M)

    on P̃/L0G. Take a class A ∈ SG2 (M) represented by a pair (P, u), where P → S2 is a principalG-bundle and u : S2 → P ×GM is a section of the associated bundle.

    Consider Un ' C∗ ∪ {∞} ⊂ S2 as the complement of the south pole 0 ∈ S2. Take an arbitrarytrivialization φ : P |Un → Un ×G, which induces a trivialization

    φ : P ×GM |Un → Un ×M. (2.20)

    Then φ ◦ u is a map from Un to M and there exists a loop h : S1 → G and x ∈M such that

    limr→0

    φ ◦ u(reiθ) = h(θ)x. (2.21)

    Note that the homotopy class of h is independent of the trivialization φ and the choice of x.

    Now, for any element (x, f, [w]) ∈ P̃, find a smooth path γ : [0, 1] → M such that γ(0) = w(0)and γ(1) = x. Then define γh : S

    1 × [0, 1]→M by γ(θ, t) = h(θ)γ(t).On the other hand, view D \ {0} ' (−∞, 0]× S1. Consider the map

    wh(r, θ) = h(θ)w(r, θ)

    and the “connected sum”:

    u#̃w := (φ ◦ u) #γh#wh : D→M (2.22)

    which extends the loop

    xh(θ) = h(θ)x(θ). (2.23)

    Denote fh := Adhf − ∂th · h−1. Then we define the action by

    A#[x, f, [w]] =[xh, fh, [u#̃w],

    ], ∀A ∈ SG2 (M). (2.24)

    On the other hand, there exists a morphism

    SG2 (M)→ kerl(x0) ⊂ π1(G) (2.25)

    which sends the homotopy class of [P, u] to the homotopy class of h : S1 → G where h is the onein (2.21). Then it is easy to see the following.

    Lemma 2.5. The action (2.24) is well-defined (i.e., independent of the representatives and choices)

    and is the deck transformation of the covering P̃/L0G→ P̃/LMG.

  • GAUGED FLOER HOMOLOGY 11

    Now by this lemma, we denote P :=(P̃/L0G

    )/NG2 , which is again a covering of P := P̃/LMG,

    with the group of deck transformations isomorphic to Γ. We will use x to denote an element in P̃

    or P̃/L0G and use [x] an element in P.

    We can define a 1-form B̃H on P̃ by

    T(x,f)P̃ 3 (ξ, h) 7→∫S1{ω (ẋ(t) +Xf − YHt , ξ(t)) + 〈µ(x(t)), h(t)〉} dt ∈ R. (2.26)

    Its pull-back to P̃ is exact and one of the primitives is the following action functional on P̃:

    ÃH(x, f, [w]) := −∫Bw∗ω +

    ∫S1

    (µ(x(t)) · f(t)−Ht(x(t))) dt. (2.27)

    The zero set of the one-form B̃H consists of pairs (x, f) such that

    µ(x(t)) ≡ 0, ẋ(t) +Xf(t)(x(t))− YHt(x(t)) = 0. (2.28)

    The critical point set of ÃH are just the preimage of ZeroB̃H under the covering P̃→ P̃.

    Lemma 2.6. ÃH is L0G-invariant and B̃H is LMG-invariant.

    Proof. Take any h : S1 → G, extend it smoothly to some h : D→ G. Then we see that

    (h−1w)∗ω = ω(∂x(h

    −1w), ∂y(h−1w)

    )dxdy = w∗ω + d

    (µ(h−1w) · d log h

    ). (2.29)

    Also, we see that

    µ(h−1(t)x(t)) ·(

    Ad−1h(t)f(t) + h(t)−1h′(t)

    )= µ(x(t)) · f(t) +

    (µ(h−1w) · d log h

    )∣∣∂D . (2.30)

    By Stokes’ theorem and the G-invariance of Ht, we obtain the invariance of ÃH . The LMG-invariance of B̃H follows from equivariance of the involved terms in a similar way. �

    So we have the induced action functional ÃH : P̃/L0G→ R and it satisfies the following.

    Lemma 2.7. For any [x] = [x, f, [w]] ∈ P̃/L0G and any A ∈ SG2 (M), we have

    ÃH (A#[x]) = ÃH ([x])− 〈[ω − µ], A〉. (2.31)

    Proof. Use the same notation as we define the action A#[x], we see that∫D\{0}

    w∗hω =

    ∫Dw∗ω +

    ∫ 0−∞

    ds

    ∫S1dtω (h∗∂sw, h∗X∂t log h(w))

    =

    ∫Dw∗ω −

    ∫ 0−∞

    ∫ 10d (µ(w) · d log h) =

    ∫Dw∗ω −

    ∫S1

    (µ(x(t))− µ(w(0)))d log h. (2.32)

    Also ∫S1×[0,1]

    γ∗hω = −∫S1

    (µ(w(0))− µ(x)) d log h. (2.33)

    In the same way we can calculate∫S2\{0}

    (φ ◦ u)∗ ω = 〈[ω − µ], [P, u]〉 −∫S1µ(x)d log h. (2.34)

  • 12 GUANGBO XU

    So we have

    ÃH (A#[x]) = −∫D

    (u#̃w

    )∗ω +

    ∫S1

    {〈µ(h(t)x(t)),Adh(t)f(t)− h′(t)h(t)−1

    〉−Ht(h(t)x(t))

    }dt

    = −〈[ω − µ], A〉 −∫Dw∗ω +

    ∫S1〈µ(x(t)), f(t)〉 −Ht(x(t))dt = −〈[ω − µ], A〉+ ÃH ([x]) (2.35)

    This lemma implies that ÃH descends to a well-defined function

    AH : P→ R. (2.36)

    Our Floer theory will be formally a Morse theory of the pair (P,AH).Before we move on to the chain complex, we see that AH is a Lagrange multiplier function

    associated to the action functional AH of the induced Hamiltonian H on the symplectic quotientM . Let P̃M be the space of contractible loops in M and let PM be pairs (x, [w]) where x ∈ P̃Mand w : D→M extends x; [w] = [w′] if (−w)#w′ is annihilated by both ω and c1(TM). Then forany (x, [w]) ∈ PM , we can pull-back the principal G-bundle µ−1(0)→ M to D. Any trivialization(or equivalently a section s) of this bundle over D induces a map ws : D→ µ−1(0) whose boundaryrestriction, denoted by x : S1 → µ−1(0), lifts x. Now, if (x, [w]) ∈ CritAH , i.e.

    0 = x′(t)− YHt(x(t)) = (πµ)∗(x′(t)− YHt(x(t))

    )(2.37)

    there exists a smooth function, fs : S1 → g such that

    x′(t) +Xfs(t)(x(t))− YHt(x(t)) = 0. (2.38)

    Then this gives a map

    ι : P̃M → P̃/L0G(x, [w]) 7→ [xs, fs, [ws]]

    (2.39)

    By the correspondence of the symplectic forms and Chern classes between upstairs and downstairs,

    we have

    Proposition 2.8. The class [xs, fs, [ws]] is independent of the choice of the section s and only

    depends on the homotopy class of w. Moreover, it induces a map

    ι :(PM ,CritAH

    )→ (P,CritAH) (2.40)

    2.4. The Floer chain complex. For R = Z2, Z or Q, we consider the Novikov ring of formalpower series over a base ring R to be the downward completion of R[Γ]:

    ΛR := Λ↓R :=

    {∑B∈Γ

    λBqB

    ∣∣∣∣∣ ∀K > 0,# {B ∈ Γ | 〈[ω − µ], B〉 > K, λB 6= 0}

  • GAUGED FLOER HOMOLOGY 13

    2.5. Gradient flow and symplectic vortex equation. Now we choose an S1-family of G-

    invariant, ω-compatible almost complex structures J on M , we assume that

    f(x) ≥ c0 =⇒ Jt(x) = J(x) (2.43)

    where (f, J) is the convex structure which we assume to exist in Hypothesis 2.4. Then ω and Jt

    defines a Riemannian metric on M , which induces an L2-metric g̃ on the loop space LM . On the

    other hand, we fix a biinvariant metric on the Lie algebra g which induces a metric g2 on Lg; it

    also identifies g with g∗ and we use this identification everywhere in this paper without mentioning

    it. These choices induce a metric on P.

    Then, it is easy to see that formally, the equation for the negative gradient flow line of AH isthe following equation for a pair (u,Ψ) : Θ→M × g

    ∂u

    ∂s+ Jt

    (∂u

    ∂t+XΨ − YHt

    )= 0,

    ∂Ψ

    ∂s+ µ(u) = 0.

    (2.44)

    The equation is invariant under the action of LG. The action is defined by

    LG×Map (Θ,M × g) → Map (Θ,M × g)g∗(u,Ψ)(s, t) =

    (g(t)−1u(s, t),Ad−1g(t)Ψ(s, t) + g(t)

    −1∂tg(t)) (2.45)

    Definition 2.9. The energy for a flow line ũ (or its Yang-Mills-Higgs functional) is defined to be

    E (ũ) := E (u,Ψ) :=1

    2‖du+ (XΨ − YHt)⊗ dt‖

    2L2 +

    1

    2

    ∥∥∥∥∂Ψ∂s∥∥∥∥2L2

    +1

    2‖µ(u)‖2L2 . (2.46)

    The second equation of (2.44) is actually the symplectic vortex equation on the triple (P, u,Ψdt),

    where P → Θ is the trivial G-bundle, u is a section of P×GM and Ψdt corresponds to the covariantderivative d+ Ψdt. (For a detailed introduction to the symplectic vortex equation, see [3] or [25]).

    This is why we name our theory the vortex Floer homology.

    The connection d+ Ψdt has already been put in the temporal gauge, i.e., it has no ds compo-

    nent. A more general equation on pairs (u, α), with α = Φds+ Ψdt ∈ Ω1(Θ)⊗ g is∂u

    ∂s+XΦ + Jt

    (∂u

    ∂t+XΨ − YHt

    )= 0,

    ∂Ψ

    ∂s− ∂Φ∂t

    + [Φ,Ψ] + µ(u) = 0.

    (2.47)

    We write the object (u, α) in the form of a triple ũ = (u,Φ,Ψ). The above equation is invariant

    under the action by GΘ := C∞ (Θ, G), which is defined by

    GΘ ×Map (Θ,M × g× g) → Map (Θ,M × g× g)

    g∗

    uΦΨ

    (s, t) = g(s, t)

    −1u(s, t)

    Ad−1g(s,t)Φ(s, t) + g(s, t)−1∂sg(s, t)

    Ad−1g(s,t)Ψ(s, t) + g(s, t)−1∂tg(s, t)

    (2.48)

  • 14 GUANGBO XU

    A solution ũ = (u,Φ,Ψ) to (2.47) is called a generalized flow line. The energy of a generalized

    flow line is

    E(u,Φ,Ψ) =1

    2‖du+XΦ ⊗ ds+ (XΨ − YHt)⊗ dt‖L2 +

    1

    2‖µ(u)‖2L2 +

    1

    2‖∂sΨ− ∂tΦ + [Φ,Ψ]‖2L2 .

    (2.49)

    It is easy to see that any smooth generalized flow line is gauge equivalent via a gauge transformation

    in GΘ to a smooth flow line in temporal gauge; and the energy is gauge independent.

    2.6. Moduli space and the formal definition of the vortex Floer homology. We focus on

    solutions to (2.47) with finite energy. In Section 3 we will show that, any such solution is gauge

    equivalent to a solution ũ = (u,Φ,Ψ) such that there exists a pair x̃± = (x±, f±) ∈ ZeroB̃H with

    lims→±∞

    Φ(s, t) = 0, lims→±∞

    (u(s, ·),Ψ(s, ·)) = x̃±. (2.50)

    Hence for any pair [x±] ∈ CritAH , we can consider solutions which “connect” them. We denote by

    M ([x−], [x+]; J,H) (2.51)

    the moduli space of all such solutions, modulo gauge transformation.

    In the appendix we will show that, if we assume that Ht vanishes for t in a nonempty open

    subset I ⊂ S1, and J a generic, “admissible” family of almost complex structures, then the spaceM ([x−], [x+]; J,H) is a smooth manifold, whose dimension is equal to the difference of the Conley-Zehnder indices of [x±]. Moreover, assuming that the manifold M cannot have any nonconstant

    pseudoholomorphic spheres, we show thatM ([x−], [x+]; J,H) is compact modulo breakings. Finally,there exist coherent orientations on different moduli spaces, which is similar to the case of ordinary

    Hamiltonian Floer theory (see [10]). Then, the signed counting of isolated gauged equivalence

    classes of trajectories in our case has exactly the same nature as in finite-dimensional Morse-Smale-

    Witten theory, which defines a boundary operator

    δJ : V CF∗(M,µ;H; ΛZ)→ V CF∗−1(M,µ;H; ΛZ). (2.52)

    The vortex Floer homology is then defined

    V HF∗ (M,µ; J,H; ΛZ) = H (V CF∗(M,µ;H; ΛZ), δJ) . (2.53)

    Moreover, for a different choice of the pair (J ′, H ′), we can use continuation principle to prove

    that the chain complex (V CF∗(M,µ;H′; ΛZ), δJ ′) is chain homotopic to (V CF∗(M,µ;H; ΛZ); δJ).

    There is a canonical isomorphism between the homologies, and we denote the common homology

    group by V HF∗(M,µ; ΛZ). The details are given in Section 6 and the appendix.

    3. Asymptotic behavior of the connecting orbits

    In this section we analyze the asymptotic behavior of solutions ũ = (u,Φ,Ψ) to (2.47) which has

    finite energy and for which u(Θ) has compact closure in M . We call such a solution a bounded

    solution. We denote the space of bounded solutions by M̃bΘ. We can also consider the equationon the half cylinder Θ+ or Θ− and denote the spaces of bounded solutions over Θ± by M̃bΘ± .

    The main theorem of this section is

  • GAUGED FLOER HOMOLOGY 15

    Theorem 3.1. (1) Any (u,Φ,Ψ) ∈ M̃bΘ± is gauge equivalent (via a smooth gauge transforma-tion g : Θ± → G) to a solution (u′,Φ′,Ψ′) ∈ M̃bΘ± such that there exist x̃± = (x±, f±) ∈ZeroB̃H and

    lims→±∞

    (u′(s, ·),Ψ′(s, ·)

    )= x̃±, lim

    s→±∞Φ(s, ·) = 0 (3.1)

    uniformly for t ∈ S1.(2) There exists a compact subset KH ⊂M such that for any (u,Φ,Ψ) ∈ M̃bΘ, we have u(Θ) ⊂

    KH .

    We will prove (1) for ũ ∈ M̃bΘ+ in temporal gauge, i.e., Φ ≡ 0 and the case for Θ− is the same.Then (2) follows from a maximum principle argument, given at the end of this section. The proof

    is based on estimates on the energy density, which has been given by others in several different

    settings (see [2], [16]). The only possibly new ingredient is that we have a nonzero Hamiltonian

    here.

    3.1. Covariant derivatives. The S1-family of metrics gt := ω(·, Jt·) induces a metric connection∇ on the bundle u∗TM . Moreover, we define

    ∇A,sξ = ∇sξ +∇ξXΦ, ∇A,tξ = ∇tξ +∇ξXΨ. (3.2)

    Also, on the trivial bundle Θ× g, define the covariant derivative

    ∇A,sθ = ∇sθ + [Φ, θ], ∇A,tθ = ∇tθ + [Ψ, θ]. (3.3)

    We denote by ∇A the direct sum connection on u∗TM × g. Note that it is compatible with respectto the natural metric on this bundle.

    Define the g-valued 2-form ρt on M by

    〈ρ(ξ1, ξ2), η〉t = 〈∇ξ1Xη, ξ2〉t = −〈∇ξ2Xη, ξ1〉t, ξi ∈ TM, η ∈ g. (3.4)

    We list several useful identities of this covariant derivative. The reader may refer to [16] for

    details.

    Lemma 3.2. For ũ in temporal gauge, we have the following equalities

    ∇A,sXη −X∇A,sη = ∇∂suXη, ∇A,tXη −X∇A,tη = ∇∂tu+XΨXη. (3.5)

    ∇A,s (∂tu+XΨ)−∇A,t∂su = X∂tΨ. (3.6)

    ∇A,s (dµ · Jξ)− dµ · J (∇A,sξ) = ρ(∂su, ξ), ∇A,t (dµ · Jξ)− dµ · J (∇A,tξ) = ρ(∂tu+XΨ, ξ).(3.7)

    On the other hand, since our equation is perturbed by a Hamiltonian H, we consider another

    covariant derivative which takes H into account. Let φtH be the Hamiltonian isotopy defined by Ht

    and

    JHt :=(dφtH

    )−1 ◦ Jt ◦ dφtH , t ∈ Rbe the 1-parameter family of almost complex structures. They are still compatible with ω and

    hence defines a family of metric gHt , with induced inner product denoted by 〈·, ·〉H,t. We denotethe induced metric connection on u∗TM by ∇H .

  • 16 GUANGBO XU

    We think (u,Ψ) as a map from R× R→M × g, periodic in the second variable. Then we havea well-defined connection D on u∗TM → R× R, given by

    (Dsξ) (s, t) =(∇Hs ξ

    )(s, t), (Dtξ) (s, t) = ∇Ht ξ +∇Hξ XΨ −∇Hξ YHt . (3.8)

    Lemma 3.3. D is compatible with the inner product 〈·, ·〉H,t on u∗TM .

    Proof. By direct calculation, we see for ξ, η ∈ Γ(u∗TM),

    ∂s〈ξ, η〉H,t = 〈Dsξ, η〉H,t + 〈ξ,Dsη〉H,t,

    ∂t〈ξ, η〉H,t =dgtdt

    (ξ, η) +〈∇gtt ξ +∇

    gtξ XF , η

    〉H,t

    +〈ξ,∇gtt η +∇gtη XF

    〉Ht

    = −(LYHtgt

    )(ξ, η) +

    〈∇gtt ξ +∇

    gtξ XF , η

    〉H,t

    +〈ξ,∇gtt η +∇gtη XF

    〉Ht

    = 〈Dtξ, η〉H,t + 〈ξ,Dtη〉H,t .

    3.2. Estimate of the energy density.

    Lemma 3.4. There exist positive constants c1 and c2 depending only on (X,ω, J, µ,Ht) and the

    subset Ku, such that for any flow line (u, 0,Ψ) in temporal gauge, we have

    ∆(|∂su|2H,t

    )≥ −c1|∂su|4H,t − c2. (3.9)

    Proof. First we see that

    1

    2∆ |∂su|2H,t =

    (∂2s + ∂

    2t

    )|∂su|2H,t = ∂s 〈Ds∂su, ∂su〉H,t + ∂t〈Dt∂su, ∂su〉H,t

    = |Ds∂su|2H,t + |Dt∂su|2H,t +

    〈(D2s +D

    2t

    )∂su, ∂su

    〉H,t

    .(3.10)

    We denote vs = ∂su ∈ Γ(R2, u∗TM) and vt = ∂tu+XΨ − YHt ∈ Γ(R2, u∗TM). Hence vs = −Jtvt.Then we have the following computation

    Dsvt −Dtvs=∇Hs (∂tu+XΨ − YHt)−∇Ht ∂su+∇H∂su (−XΨ + YHt) = X∂sΨ = −Xµ(u).

    (3.11)

    Dsvs +Dtvt

    =Ds(−Jtvt) +Dt(Jtvs)

    =−∇Hs (Jtvt) +∇Ht (Jtvs) +∇HJvs(XΨ − YHt)

    =−∇Hs (Jtvt) +∇Ht (Jtvs) + [Jtvs, XΨ − YHt ] +∇HXΨ−YHt (Jtvs)

    =−∇Hs (Jvt) +∇Hvt(Jtvs) + J̇tvs + [Jtvs, XΨ − YHt ]

    =− (∇HvsJt)vt − Jt∇Hs vt + (∇HvtJt)vs + Jt∇

    Hvtvs + J̇tvs + Jt [vs, XΨ] + [YHt , Jtvs]

    =− (∇HvsJt)vt + (∇HvtJt)vs − Jt[vs, vt]− JX∂sΨ + J̇tvs + Jt[vs, XΨ] + Jt [YHt , vs] + (LYHtJt)vs

    =− (∇HvsJt)vt + (∇HvtJt)vs + (LYHtJt)vs + JtXµ + J̇tvs.

    (3.12)

  • GAUGED FLOER HOMOLOGY 17

    On the other hand, for any (s, t) ∈ Σ, any ξ ∈ Tu(s,t)M , we extend ξ and vs(s, t) to be G-invariantvector fields locally. Then for the Riemann curvature tensor associated to JHt , we have

    RH(vs, XΨ)ξ = ∇Hvs∇HXΨξ −∇HXΨ∇

    Hvsξ −∇

    H[vs,XΨ]

    ξ = ∇Hvs∇Hξ XΨ −∇H∇HvsξXΨ. (3.13)

    Hence

    (DsDt −DtDs) ξ

    = ∇Hs(∇Ht ξ +∇Hξ (XΨ − YHt)

    )−∇Ht (∇Hs ξ)−∇H∇Hs ξ(XΨ − YHt)

    = RH(vs, ∂tu)ξ +∇Hs ∇Hξ (XΨ − YHt)−∇H∇Hs ξ(XΨ − YHt)−(d

    dt∇Hs)ξ

    = RH(vs, ∂tu+XΨ)ξ −∇Hs ∇Hξ YHt +∇H∇Hs ξYHt +∇Hξ X∂sΨ −

    (d

    dt∇Hs)ξ

    = RH(vs, vt + YHt)ξ −∇Hs ∇Hξ YHt +∇H∇Hs ξYHt −∇Hξ Xµ −

    (d

    dt∇Hs)ξ. (3.14)

    The third equality above uses (3.13).

    Then we denote

    D2svs +D2t vs = Ds (Dsvs +Dtvt) + (DtDs −DsDt) vt −Dt (Dsvt −Dtvs) =: Q1 +Q2 +Q3.

    By (3.12),

    〈Q1, vs〉H,t =〈∇Hs

    (−(∇HvsJt

    )vt +

    (∇HvtJt

    )vs +

    (LYHtJt

    )vs + JtXµ + J̇tvs

    ), vs

    〉H,t

    ≥ −C1(|vs|3H,t + |vs|2H,t + |vs|4H,t + |vs|2H,t

    ∣∣∇Hs vs∣∣H,t + |vs|H,t ∣∣∇Hs vs∣∣H,t) (3.15)for some C1 > 0. Here we used the fact that µ and dµ are uniformly bounded because u(Θ+) has

    compact closure.

    By (3.14), for some C2 > 0, we have

    〈Q2, vs〉Ht =〈−RH(vs, vt + YH,t)vt +∇Hs ∇HvtYHt −∇

    H∇Hs vt

    YHt +∇HvtXµ +(d

    dt∇Hs)vt, vs

    〉H,t

    ≥ −C2(|vs|4H,t + |vs|3H,t + |vs|2H,t + |vs|H,t

    ∣∣∇Hs vs∣∣H,t) . (3.16)By (3.11), for some C3 > 0, we have

    〈Q3, vs〉H,t= 〈DtXµ, vs〉H,t

    =〈∇Ht Xµ +∇HXµ(XΨ − YHt), vs

    〉H,t

    =〈Xdµ·∂tu +∇HvtXµ + [Xµ, XΨ − YHt ], vs

    〉=〈Xdµ·∂tu +∇HvtXµ −X[µ,Ψ], vs

    〉=〈Xdµ·vt −Xdµ·XΨ +∇

    HvtXµ −X[µ,Ψ], vs

    〉H,t

    =〈Xdµ·vt +∇HvtXµ, vs〉H,t ≥ −C3|vs|2H,t.

    (3.17)

  • 18 GUANGBO XU

    Hence for some C4 > 0 and c1, c2 > 0, we have

    1

    2∆|vs|2H = |Dsvs|2H + |Dtvt|2H + 〈Q1 +Q2 +Q3, vs〉H ≥ |∇gts vs|2H + 〈Q1 +Q2 +Q3, vs〉H

    ≥ |∇gts vs|2H − C4

    (4∑i=2

    |vs|iH +2∑i=1

    |vs|iH |∇gts vs|H

    )≥ −c1|vs|4H − c2. (3.18)

    Now we consider the other part of the energy density. We have the following calculation:

    1

    2∆ |µ(u)|2 = |∇A,sµ|2 + |∇A,tµ|2 + 〈∇A,s∇A,sµ+∇A,t∇A,tµ, µ〉. (3.19)

    And (see [16, Lemma C.2])

    ∇A,s∇A,sµ(u) +∇A,t∇A,tµ(u)

    =∇A,sdµ · vs +∇A,tdµ · vt = ∇A,t (dµ · Jvs)−∇A,s (dµ · Jvt)

    =− 2ρ(vs, vt) + dµ · Jt (∇A,tvs −∇A,svt) + dµ(J̇tvs

    )=dµ ·

    (JtXµ + J̇tvs

    )− 2ρ(vs, vt).

    (3.20)

    Since u(Θ) has compact closure, we may assume that supu(Θ) |ρ| ≤ cu. Hence there exists c3, c4 > 0such that

    ∆ |µ(u)|2 ≥ −c3 − c4 |vs|4 . (3.21)

    3.3. Decay of energy density. To proceed, we quote the following lemma (cf. [32, Page 12]).

    Lemma 3.5. Let Ω ⊂ R2 be an open subset containing the origin and e is defined over Ω. Supposeit satisfies

    ∆e ≥ −A−Be2,

    then ∫Br(0)

    e ≤ π16B

    =⇒ e(0) ≤ 8πr2

    ∫Br(0)

    e+Ar2

    4. (3.22)

    Apply the above lemma to the function |vs|2H,t+|µ(u)|2 and note that different norms are actuallyequivalent, we see

    Proposition 3.6. If (u,Ψ) satisfies the equation (2.44) such that u(Θ) has compact closure and

    E(u,Ψ)

  • GAUGED FLOER HOMOLOGY 19

    3.4. Approaching to periodic orbits.

    Proposition 3.7. Any (u, 0,Ψ) ∈ M̃bΘ+ is gauge equivalent to a solution to (2.47) (v,Φ,Ψ) onΘ+ with the following properties

    (1) (v,Ψ)|{s}×S1 converges in C0 to an element of ZeroB̃H as s→ +∞;

    (2) lims→+∞

    Φ(s, ·) = 0 uniformly in t.

    Proof. By the above proposition, we know that finite energy implies that µ(u(s, ·))→ 0 as s→ +∞.Let g∗� ⊂ g∗ be the �-open ball of the origin. Then there is an equivariant symplectic diffeomorphism

    U� := µ−1(g∗� )→ µ−1(0)× g∗� . (3.23)

    Then µ|U� is just the projection of the right hand side of (3.23) onto the second factor (see forexample [18]). Let πµ be the projection on to the first factor, then we define the almost complex

    structure

    J0,t := π∗µ

    (Jt∣∣µ−1(0)

    )and the vector field

    Y0,Ht := π∗µ

    (YHt

    ∣∣µ−1(0)

    ).

    Then there exists K1 > 0 depending on (M,ω, J, µ,Ht) such that

    ‖J0,t(x)− Jt(x)‖ ≤ K1|µ(x)|, ‖Y0,Ht(x)− YHt(x)‖ ≤ K1|µ(x)|, ∀x ∈ U�.

    We denote πµ : µ−1(g∗� ) → M the composition of πµ with the projection µ−1(0) → M . For

    (u, 0,Ψ) ∈ M̃bΘ+ , we have proved that µ(u) converges to 0 uniformly as s → +∞. Hence forN := N(�) sufficiently large, u(s, ·) maps ΘN+ := [N,+∞)× S1 into U�. So on ΘN+ , we have

    ∂su+ J0,t (∂tu+XΨ(u)− Y0,Ht(u)) = (J0,t − Jt) (∂tu+XΨ(u)− Y0,Ht) + Jt (YHt − Y0,Ht) .(3.24)

    Denoting u := πµ ◦ u : ΘN+ →M and applying (πµ)∗ to the above equality, we see on ΘN+ ,∥∥∂su+ J t (∂tu− YHt(u))∥∥ ≤ K2� (3.25)

    for some constant K2. Here J t is the induced almost complex structure on M . Hence for s ≥ N ,the family of loops u(s, ·) in the quotient will be close (in C0) to some 1-periodic orbits γ : S1 →Mof YHt .

    We take a lift p ∈ µ−1(0) with ππµ(p) = γ(0). We see that there exists a unique gp ∈ G suchthat

    φ1Hp = gpp. (3.26)

    Suppose gp = exp ξp, ξp ∈ g. It is easy to see that the loop

    (x(t), f(t)) :=(exp(−tξp)φtH(p), ξp

    )(3.27)

    is an element of ZeroB̃H . We will construct a gauge transformation g̃ on Θ+ and show thatg̃∗(u, 0,Ψ) satisfies the condition stated in this proposition.

  • 20 GUANGBO XU

    Take a local slice of the G-action near p. That is, an embedding i : B2n−2kδ → µ−1(0) where

    B2n−2kδ is the δ-ball in R2n−2k such that i(0) = p and (y, g) 7→ g(i(y)) is a diffeomorphism from

    B2n−2kδ ×G onto its image.Denote u(s, t) = (v(s, t), ξ(s, t)) ∈ µ−1(0) × g∗ with respect to the decomposition (3.23). Then

    for s large enough, there exists a unique g(s) ∈ G such that

    g(s)v(s, 0) ∈ i(B2n−2kδ

    ), g(s)v(s, 0)→ p. (3.28)

    Moreover, by the fact that |∂su| converges to zero, we see

    lims→+∞

    ∣∣g(s)−1ġ(s)∣∣ = 0. (3.29)Define hs(t) ∈ G by

    hs(0) = 1, hs(t)−1∂hs(t)

    ∂t= Ψ(s, t).

    Then by the fact that lims→+∞ |∂sΨ| = 0 we see that

    lims→∞

    |∂s log hs(t)| = 0. (3.30)

    Thus we have

    d(gpp, g(s)hs(1)g(s)

    −1p)

    ≤ d(gpp, φ

    1Hg(s)v(s, 0)

    )+ d

    (φ1Hg(s)v(s, 0), g(s)hs(1)v(s, 0)

    )+ d

    (g(s)hs(1)v(s, 0), g(s)hs(1)g(s)

    −1p)

    = d(gpp, φ

    1Hg(s)v(s, 0)

    )+ d

    (φ1Hv(s, 0), hs(1)v(s, 0)

    )+ d (g(s)v(s, 0), p)

    =: d1(s) + d2(s) + d3(s).

    (3.31)

    Here d is the G-invariant distance function induced by an invariant Riemannian metric. By (3.26)

    and (3.28), we have d1(s) + d3(s)→ 0. By the decay of energy density, i.e.,

    lims→+∞

    supt|∂tu+XΨ − YHt | = 0,

    we have d2(s)→ 0. Hence we have

    lims→+∞

    d(gpp, g(s)hs(1)g(s)

    −1p)

    = 0. (3.32)

    Since the G-action on µ−1(0) is free, we have

    lims→+∞

    g(s)hs(1)g(s)−1 = gp (3.33)

    Then by (3.33), there exists a continuous curve ξ(s) ∈ g defined for large s, such that

    g(s)hs(1)g(s)−1 = exp ξ(s), lim

    s→+∞ξ(s) = ξp. (3.34)

    Then apply the gauge transformation

    g̃(s, t) = hs(t)−1g(s)−1 exp(tξ(s)) (3.35)

    to the pair (u, 0,Ψ), we see

    (g̃∗u) (s, 0) = g̃(s, 0)−1(u(s, 0)) = g(s)u(s, 0)→ p, g̃∗ (Ψdt) = ξ(s)dt+ η(s, t)ds (3.36)

  • GAUGED FLOER HOMOLOGY 21

    where

    η = ∂s log g̃ = g̃−1∂g̃

    ∂s.

    The fact that lims→+∞ ‖η‖ = 0 follows from (3.29) and (3.30).Hence

    lims→+∞

    (g̃∗u)|{s}×S1 =(exp(−tξp)φtHp, ξp

    )∈ ZeroB̃H .

    Definition 3.8. Let x̃± := (x±, f±) ∈ ZeroB̃H . We denote

    M̃ (x̃−, x̃+) := M̃ (x̃−, x̃+; J,H) :={

    (u,Φ,Ψ) ∈ M̃bΘ | lims→±∞(u,Φ,Ψ)|{s}×S1 = (x̃±, 0)}. (3.37)

    For x± = (x̃±, [w±]) ∈ CritÃH which projects to x̃± via CritÃH → ZeroB̃H , we define

    M̃ (x−, x+) := M̃ (x−, x+; J,H) :={

    (u,Φ,Ψ) ∈ M̃(x̃−, x̃+) | [u#w−] = [w+]}. (3.38)

    Then it is easy to deduce the following energy identity for which we omit the proof.

    Proposition 3.9. Let x± ∈ CritÃH . Then for any (u,Φ,Ψ) ∈ M̃ (x−, x+), we have

    E (u,Φ,Ψ) = ÃH (x−)− ÃH (x+) . (3.39)

    3.5. Convexity and uniform bound on flow lines. We will show in this subsection the following

    Proposition 3.10. There exists a compact subset KH ⊂ M such that for any (u,Φ,Ψ) ∈ M̃bΘ,u(Θ) ⊂ KH .

    Proof. The proof is to use maximum principle as in [2, Subsection 2.5]. We claim this proposition

    is true for

    KH = SuppH ∪ f−1 ([0, c1])

    where

    c1 = max

    {c0, sup|µ(x)|≤1

    f(x)

    }(3.40)

    where c0 is the one in Hypothesis 2.4.

    Suppose the statement is not true. Then there exists a solution ũ = (u,Φ,Ψ) ∈ M̃bΘ whichviolates this condition and (s0, t0) ∈ Θ such that u(s0, t0) /∈ SuppH and f(u(s0, t0)) > c1. Onthe other hand, by the previous results, we know that lim

    s→±∞µ(u(s, t)) = 0 so lim

    s→±∞f(u(s, t)) ≤ c1.

    Hence f(u) achieves its maximum at some point of Θ. As in the proof of [2, Lemma 2.7], we see that

    f(u) is subharmonic on u−1 (M \KH) and hence f(u) must be constant. However, this contradictswith the fact that lim

    s→±∞f(u(s, t)) ≤ c1. �

    4. Fredholm theory

    In this section we investigate the infinitesimal deformation theory of solutions to our equation

    (modulo gauge). For a similar treatment of a relevant situation, the reader may refer to [4].

  • 22 GUANGBO XU

    4.1. Banach manifolds, bundles, and local slices. First we fix two loops x̃± ∈ ZeroB̃H . Forany k ≥ 1, p > 2, we consider the space of W k,ploc -maps ũ := (u,Φ,Ψ) : Θ → M × g × g, such thatΦ ∈ W k,p (Θ, g) and (u,Ψ) is asymptotic to x̃± = (x±, f±) at ±∞ in W k,p-sense. Then this is aBanach manifold, denoted by

    B̃k,p := B̃k,p(x̃−, x̃+). (4.1)

    The tangent space at any element ũ ∈ B̃k,p is the Sobolev space

    TũB̃k,p = W k,p (Θ, u∗TM ⊕ g⊕ g) . (4.2)

    We denote by ẽxpt the exponential map of M×g×g, where the Riemannian metric on M is ω(·, Jt·)which is t-dependent. Then the map ξ̃ 7→ ẽxptũξ̃ is a local diffeomorphism from a neighborhood of0 ∈ TũB̃k,p and a neighborhood of ũ in B̃k,p.

    Then consider a pair x± = (x±, f±, [w±]) ∈ CritÃH with x̃± = (x±, f±) ∈ ZeroB̃H . We define

    B̃k,p(x−, x+) :={ũ = (u,Φ,Ψ) ∈ B̃k,p(x̃−, x̃+) | [w−#u] = [w+]

    }. (4.3)

    Let Gk+1,p0 be the space of Wk+1,ploc -maps g : Θ → G which is asymptotic to the identity of G at

    ±∞. Then this is a Banach Lie group. The gauge transformation extends to a free Gk+1,p0 -actionon B̃k,p(x̃−, x̃+) (resp. B̃k,p(x−, x+)), because the symplectic quotient M is a free quotient. Thenthis makes the quotient

    Bk,p(x̃−, x̃+) := B̃k,p(x̃−, x̃+)/Gk+1,p(

    resp. Bk,p(x−, x+) := B̃k,p(x−, x+)/Gk+1,p0)

    (4.4)

    a Banach manifold. Indeed, to see this we have to construct local slices of the Gk+1,p-action. Forany ũ ∈ B̃k,p (whose image in Bk,p is denoted by [ũ]), consider the operator

    d∗0 : TũB̃k,p → W k−1,p (Θ, g)(ξ, φ, ψ) 7→ −dµ(Jtξ)− ∂sφ− [Φ, φ]− ∂tψ − [Ψ, ψ],

    (4.5)

    which is the formal adjoint of the infinitesimal Gk+1,p-action. Then as in gauge theory, we have anatural identification

    T[ũ]Bk,p ' kerd∗0 (4.6)

    (where the orthogonal complement is taken with respect to the L2-inner product) and the expo-

    nential map ẽxpt induces a local diffeomorphism

    kerd∗0 3 ξ̃ 7→[ẽxptũξ̃

    ]∈ Bk,p. (4.7)

    If we have g± ∈ L0G, and x′± = g∗±x± ∈ CritÃH , then the pair (g−, g+) extends to a smoothgauge transformation on Θ which identifies B̃k,p(x−, x+) with B̃k,p(x′−, x′+). Then, with abuse ofnotation, if x± ∈ CritÃH/L0G, then we can denote by Bk,p(x−, x+) to be the common quotientspace. Finally, for two pairs [x±] ∈ CritAH , we define

    Bk,p([x−], [x+]) :=⋃

    y±∈CritÃH/L0G, [y±]=[x±]

    Bk,p(y−, y+), (4.8)

    which is a discrete union of Banach manifolds.

  • GAUGED FLOER HOMOLOGY 23

    Over B̃k,p(x−, x+), we have the smooth Banach space bundle Ẽk−1,p(x−, x+), whose fibre over ũ isthe Sobolev space

    Ẽk−1,pũ := Wk−1,p (u∗TM ⊕ g) . (4.9)

    The Gk+1,p0 -action makes Ẽk−1,p an equivariant bundle, hence descends to a Banach space bundle

    Ek−1,p(x−, x+)→ Bk,p(x−, x+)(

    or Ek−1,p ([x−], [x+])→ Bk,p ([x−], [x+])). (4.10)

    Moreover, the H-perturbed vortex equation (2.47) gives a section

    F̃ : B̃k,p (x−, x+)→ Ẽk−1,p(x−, x+) (4.11)

    which is Gk+1,p-equivariant. So it descends to a section

    F : Bk,p ([x−], [x+])→ Ek−1,p ([x−], [x+]) .

    Then we see that M̃ (x−, x+; J,H) is the intersection of F̃−1(0) with smooth objects. We define

    M ([x−], [x+]; J,H) = F−1(0), (4.12)

    whose elements, by the standard regularity theory about symplectic vortex equation (see [2, The-

    orem 3.1]), all have smooth representatives. Therefore M ([x−], [x+]; J,H) is independent of k, p.The linearization of F̃ at ũ is

    D̃ũ := dF̃ũ : (ξ, φ, ψ) 7→

    (∇A,sξ + (∇ξJt) (∂tu+XΨ − YHt) + Jt (∇A,tξ −∇ξYHt) +Xφ + JXψ

    ∂sψ + [Φ, ψ]− ∂tφ− [Ψ, φ] + dµ(ξ)

    )(4.13)

    Hence the linearization of F , under the isomorphism (4.6), is the restriction of D̃ũ to (kerd∗0)⊥.

    We define the augmented linearized operator

    Dũ := D̃ũ ⊕ d∗0 : TũB̃k,p → Ẽk−1,pũ ⊕W

    k−1,p (Θ, g) . (4.14)

    It is a standard result that the Fredholm property of dF[ũ] is equivalent to that of Dũ for anyrepresentative ũ. Hence in the remaining of the section we will study the Fredholm property of the

    augmented operator.

    4.2. Asymptotic behavior of Dũ. Up to an L0G-action we can choose representatives x± =

    (x±, f±, [w±]) such that f± are constants θ± ∈ g. Take any ũ = (u,Φ,Ψ) ∈ B̃k,p(x−, x+).For ξ̃ := (ξ, ψ, φ) ∈ TũB̃k,p(x−, x+), define J̃(ξ, ψ, φ) = (Jξ,−φ, ψ). Here ψ is the variation of Ψ

    and φ is the variation of Φ. Then

    Dũ

    ξψφ

    = ∇A,s ξψ

    φ

    +J̃ ∇A,tξ −∇ξYH,t∇A,tψ

    ∇A,tφ

    + 0 JLu Ludµ 0 0

    L∗u 0 0

    ξψ

    φ

    +q(s, t) ξψ

    φ

    =:

    ∇s∂s

    ∂s

    + J̃ ∇t∂t

    ∂t

    +R(s, t) + q(s, t) ξψ

    φ

    . (4.15)

  • 24 GUANGBO XU

    Here q(s, t) is a linear operator such that lims→±∞

    q(s, t) = 0; and

    Rx̃±(t) := lims→±∞R(s, t) =

    Jt∇(Xθ± − YHt) JtLu Ludµ 0 −adθ±L∗u adθ± 0

    . (4.16)Here Lu : g→ u∗TM is the infinitesimal action along the image of u and L∗u is its dual.

    The following result is implied by Hypothesis 2.3.

    Proposition 4.1. x̃ := (x, θ) ∈ ZeroB̃H , then for all t-dependent, G-invariant, ω-compatible almostcomplex structure Jt, the self-adjoint operator

    L2(S1, x∗TM ⊕ g⊕ g

    )→ L2

    (S1, x∗TM ⊕ g⊕ g

    ) ξψφ

    7→J̃ ∇t∂t

    ∂t

    +Rx̃(t) ξψ

    φ

    . (4.17)has zero kernel.

    Proof. Suppose (ξ, ψ, φ)T is in the kernel, which means

    Jt∇tξ + Jt∇ξ (Xθ − YHt) +Xφ + JtXψ = 0, (4.18)

    −dφdt

    + dµ(ξ)− [θ, φ] = 0, (4.19)

    dt+ dµ(Jtξ) + [θ, ψ] = 0. (4.20)

    Apply dµ ◦ Jt to (4.18), we get

    dµ(JtXφ) = dµ(∇tξ +∇ξ(Xθ − YHt)).

    Hence for any η ∈ g,d

    dt〈dµ(ξ), η〉g =

    d

    dtω(Xη, ξ)

    =ω([YHt −Xθ, Xη], ξ) + ω(Xη,∇tξ −∇ξ(YHt −Xθ))

    =ω(X[θ,η], ξ) + 〈dµ(JXφ), η〉g=〈dµ(ξ), [θ, η]〉g + 〈dµ(JXφ), η〉g=〈dµ(JXφ)− [θ, dµ(ξ)], η〉g.

    (4.21)

    Therefore,

    d

    dtdµ(ξ) = dµ(JXφ)− [θ, dµ(ξ)]. (4.22)

    Then by (4.19),

    dµ(JtXφ)

    =d

    dtdµ(ξ) + [θ, dµ(ξ)]

    =d

    dt

    (dφ

    dt+ [θ, φ]

    )+

    [θ,dφ

    dt+ [θ, φ]

    ]=φ′′ + 2

    [θ, φ′

    ]+ [θ, [θ, φ]].

    (4.23)

  • GAUGED FLOER HOMOLOGY 25

    Suppose ‖φ‖ takes its maximum at t = t0 ∈ S1. Then for t ∈ (t0 − �, t0 + �), define φ̃(t) =Ade(t−t0)θφ(t). Then the right hand side of (4.23) is equal to Ade(t0−t)θ φ̃

    ′′(t). Hence at t = t0,

    0 ≥12

    d2

    dt2‖φ‖2 = 1

    2

    d2

    dt2

    ∥∥∥φ̃∥∥∥2 = 〈φ̃′′, φ̃〉+ ∥∥∥φ̃′∥∥∥2g

    =〈dµ(JtXφ(t0)), φ(t0)

    〉+∥∥∥φ̃′(t0)∥∥∥2 = ∥∥Xφ(t0)∥∥2 + ∥∥∥φ̃′(t0)∥∥∥2 . (4.24)

    Hence Xφ ≡ 0, which implies φ ≡ 0 and by (4.19), ξ is tangent to µ−1(0).Now x∗Tµ−1(0) = Et⊕Lxg, where Et = (Lxg)⊥ ∩x∗Tµ−1(0) and the orthogonal complement is

    taken with respect to the Riemannian metric g = ω(·, Jt·). Then with respect to this (G-invariant)decomposition, write

    ξ = ξ⊥(t) +Xη(t).

    Then take the Et-component of (4.18), and use the nondegeneracy assumption on the induced

    Hamiltonian Ht on the symplectic quotient, we see that ξ⊥ ≡ 0. The only equations left is ∇tXη(t) +∇Xη(t)(Xθ − YHt) +Xψ = 0,ψ′ + [θ, ψ] + dµ(JXη) = 0.

    The first equation is equivalent to

    η′ + [θ, η(t)] + ψ = 0.

    Hence

    η′′(t) + 2[θ, η′(t)

    ]+ [θ, [θ, η]] = dµ(JtXη).

    This can be treated similarly as (4.23), using maximum principle, which shows that η ≡ 0 andhence ψ ≡ 0. �

    Corollary 4.2. Under Hypothesis 2.3, for any x̃± ∈ ZeroB̃H and any ũ ∈ B̃k,p (x̃−, x̃+), theaugmented linearized operator Dũ is Fredholm for any k ≥ 1, p ≥ 2.

    Corollary 4.3. There exists δ = δ(x̃±) > 0 such that, for any ũ = (u,Φ,Ψ) ∈ M̃ (x̃−, x̃+; J,H)and any ξ̃ ∈ kerDũ, there exists c > 0 such that∣∣∣ξ̃(s, t)∣∣∣ ≤ ce−δ|s|. (4.25)In particular, if (u, 0,Ψ) ∈ M̃bΘ, then |∂su| and |∂sΨ| decay exponentially.

    Proof. The first part is standard, see for example [32, Lemma 2.11]. For a solution ũ = (u, 0,Ψ)

    in temporal gauge, by the translation invariance of the equation (2.44), we see that ξ̃ = (∂su, βs =

    0, βt = ∂sF ) ∈ kerdF̃ũ. Moreover,

    −d∗0ξ̃ =− d∗0(∂su, 0, ∂sΨ) = ∂t∂sΨ + L∗u(∂su) + [Ψ, ∂sΨ]

    =− ∂t(µ(u)) + dµ(Jt∂su) + [Ψ, ∂sΨ]

    =− dµ(∂tu) + dµ(Jt∂su) + [Ψ, ∂sΨ]

    = dµ (XΨ − YHt)− [Ψ, µ] = 0.

    (4.26)

    This implies that ξ̃ ∈ kerDũ. Choose a smooth gauge transformation g : Θ → G such thatg∗ũ satisfies the asymptotic condition of Proposition 3.7. Then g∗ξ̃ ∈ kerDg∗ũ, which decaysexponentially. So does ξ̃. �

  • 26 GUANGBO XU

    4.3. The Conley-Zehnder indices. In this subsection we define a grading on the set CritAH ,which is analogous to the Conley-Zehnder index in usual Hamiltonian Floer theory, and we will

    call it the same name.

    For the induced Hamiltonian system on the symplectic quotient M , we have the usual Conley-

    Zehnder index

    CZ : CritAH → Z. (4.27)

    We prove the following theorem

    Theorem 4.4. There exists a function

    CZ : CritAH → Z (4.28)

    satisfying the following properties

    (1) For the embedding ι : CritAH → CritAH , we have

    CZ ◦ ι = CZ; (4.29)

    (2) For any B ∈ Γ and [x] ∈ CritAH we have

    CZ (B# [x]) = CZ ([x])− 2cG1 (B). (4.30)

    (3) For [x±] = [x±, f±, [w±]] ∈ CritAH and [ũ] ∈ Bk,p ([x−], [x+]) with [ũ#w−] = [w+], we have

    ind(dF[ũ]

    )= CZ ([x−])− CZ ([x+]) . (4.31)

    We first review the notion of Conley-Zehnder index in Hamiltonian Floer homology. Let A :

    [0, 1]→ Sp(2n) be a continuous path of symplectic matrices such that

    A(0) = I2n, det (A(1)− I2n) 6= 0. (4.32)

    We can associate an integer CZ(A) to A, called the Conley-Zehnder index. We list some properties

    of the Conley-Zehnder index below which will be used here (see for example [31]).

    (1) For any path B : [0, 1]→ Sp(2n), we have CZ(BAB−1) = CZ(A);(2) CZ is homotopy invariant;

    (3) If for t > 0, A(t) has no eigenvalue on the unit circle, then CZ(A) = 0;

    (4) If Ai : [0, 1]→ Sp(2ni) for n = 1, 2, then CZ(A1 ⊕A2) = CZ(A1) + CZ(A2);(5) If Φ : [0, 1]→ Sp(2n) is a loop with Φ(0) = Φ(1) = Id, then

    CZ(ΦA) = CZ(A) + 2µM (Φ) (4.33)

    where µM (Φ) is the Maslov index of the loop Φ.

    With this algebraic notion, in the usual Hamiltonian Floer theory one can define the Conley-

    Zehnder indices for nondegenerate Hamiltonian periodic orbits. In our case, the induced Hamil-

    tonian Ht : M → R has the usual Conley-Zehnder index

    CZ : CritAH → Z. (4.34)

    Then, for each x = (x, f, [w]) ∈ CritÃH , the homotopy class of extensions [w] induces a homotopyclass of trivializations of x∗TM over S1. With respect to this class of trivialization, the operator

    (4.17) is equivalent to an operator J0∂t + A(t), which defines a symplectic path. We define the

  • GAUGED FLOER HOMOLOGY 27

    Conley-Zehnder index of x to be the Conley-Zehnder index of this symplectic path. By the second

    and fifth axioms listed above, this index induces a well-defined function

    CZ : CritAH → Z (4.35)

    which satisfies (2) and (3) of Theorem 4.4.

    Now we prove (1). For any contractible periodic orbits x : S1 → M of YHt and any extensionw : D→M of x, we can lift the pair (x,w) to a tuple x = (x, f, [w]) ∈ CritÃH .

    Proposition 4.5. If ι : CritAH → CritAH is the inclusion we described in Proposition 2.8, then

    CZ ◦ ι = CZ. (4.36)

    Proof. Since the Conley-Zehnder index is homotopy invariant, and the space of G-invariant ω-

    compatible almost complex structures is connected, we will compute the Conley-Zehnder index

    using a special type of almost complex structures, and modify the Hamiltonian H.

    Starting with any almost complex structure J on M and a G-connection on µ−1(0)→M , J liftsto the horizontal distribution defined by the connection. On the other hand, the biinvariant metric

    on g gives an identification g ' g∗. We denote by η∗ ∈ g∗ the metric dual of η ∈ g. Recall that wehave a symplectomorphism

    µ−1 (g∗� ) ' µ−1(0)× g∗� . (4.37)

    For η ∈ g, we define JXη = η∗ ∈ g∗, as a vector field on µ−1(0)× g∗. Then this gives a G-invariantalmost complex structure on TM |µ−1(0), compatible with ω. Then we pullback J by the projectionµ−1(0)× g∗� → µ−1(0) and denote the pullback by J .

    We also modify Ht by requiring that Ht(x, η) = Ht(x) for (x, η) ∈ µ−1(0)×g∗� . Then the modifiedHt can be continuously deformed to the original one, and it doesn’t change H hence doesn’t change

    CritAH . Moreover, it is easy to check that for the modified pair (J,H),(LYHtJ

    )Xη =

    [LYHt , JXη

    ]= 0. (4.38)

    Now for any (x,w) ∈ CritAH , we lift it to (x, f, [w]) ∈ CritÃH with w : D → µ−1(0) and fbeing a constant θ ∈ g. Then any symplectic trivialization of x̃∗TM → S1 induces a symplectictrivialization

    φ : x∗TM ' S1 ×[R2n−2k ⊕ (g⊕ g)

    ](4.39)

    such that φ(Xη, JXζ) = (0, η, ζ). Then we see, with respect to φ, the operator (4.17) restricted to

    g4 is η1

    ψ

    η2

    φ

    7→ J̃ ddt

    η1

    ψ

    η2

    φ

    +φ− [θ, η2]η2 − [θ, φ]ψ + [θ, η1]

    η1 + [θ, ψ]

    =:(J̃d

    dt+ S

    )η1

    ψ

    η2

    φ

    . (4.40)Here we used the property (4.38) and

    J̃ :=

    [0 −Idg⊕g

    Idg⊕g 0

    ], S =

    [0 Idg⊕g − adθ

    Idg⊕g + adθ 0

    ](4.41)

  • 28 GUANGBO XU

    Moreover, the operator (4.17) respect the decomposition in (4.39). Hence by the fourth axiom of

    Conley-Zehnder indices we listed above, we have

    CZ (x, θ, [w]) = CZ(x,w) + CZ(eJ̃St

    ). (4.42)

    As we have shown in the proof of Proposition 4.1 that for any θ the operator (4.17) is an isomor-

    phism, we can deform θ to zero and compute instead CZ(eJ̃S0t) for

    S0 =

    [0 Idg⊕g

    Idg⊕g 0

    ], (4.43)

    thanks to the homotopy invariance property. Then we see that

    eJ̃S0t =

    [e−t 0

    0 et

    ](4.44)

    which has no eigenvalue on the unit circle for t > 0. Therefore by the third axiom, CZ(eJ̃S0t) =

    0. �

    5. Compactness of the moduli space

    For a general Hamiltonian G-manifold, the failure of compactness of the moduli space of con-

    necting orbits comes from two phenomenon. The first is the breaking of connecting orbits, which is

    essentially the same thing happened in finite dimensional Morse-Smale-Witten theory. The second

    is the blow-up of the energy density, which results in sphere bubbling. Since here we have assumed

    that there exists no nontrivial holomorphic sphere in M , so we only have to consider the breakings.

    5.1. Moduli space of stable connecting orbits and its topology. Let’s fix a pair [x±] ∈CritAH . Denote by M̂([x−], [x+]) := M̂([x−], [x+]; J,H) =M([x−], [x+]; J,H)/R the quotient of themoduli space by the translation in the s-direction. We denote by {ũ} the R-orbit in M̂([x−], [x+])of [ũ] ∈M([x−], [x+]; J,H) and call it a trajectory from [x−] to [x+].

    Definition 5.1. A broken trajectory from [x−] to [x+] is a collection

    ũ :=({ũ(α)

    })α=1,...,m

    :=({u(α),Φ(α),Ψ(α)

    })α=1,...,m

    (5.1)

    where for each α,{ũ(α)

    }∈ M̂ ([xα−1] , [xα]) and E(ũ(α)) 6= 0. Here {[xα]}α=0,...,m is a sequence of

    critical points of AH and[x0] = [x−] , [xm] = [x+] .

    We regard the domain of ũ as the disjoint union

    ∪mα=1Θ

    and let Θ(α) ⊂ ∪mα=1Θ the α-th cylinder.

    We denote by

    M ([x−] , [x+]) (5.2)

    the space of all broken trajectories from [x−] to [x+]. Then naturally we have inclusion

    M̂ ([x−] , [x+])→M ([x−] , [x+]) . (5.3)

  • GAUGED FLOER HOMOLOGY 29

    Definition 5.2. We say that a sequence of trajectories {ũi} = {ui,Φi,Ψi} ∈ M̂ ([x−] , [x+]) from[x−] to [x+] converges to a broken trajectory

    ũ :=({ũ(α)

    })α=1,...,m

    if: for each i, there exists sequences of numbers s(1)i < s

    (2)i < · · · < s

    (m)i and gauge transformations

    g(α)i ∈ GΘ such that for each α (

    g(α)i

    )∗ (s

    (α)i

    )∗(ui,Φi,Ψi) (5.4)

    converges to(u(α),Φ(α),Ψ(α)

    )on any compact subset of Θ and such that for any sequence of (si, ti)

    with

    limi→∞|si − s(α)i | =∞, ∀α

    we have

    limi→∞

    e (ũi) (si, ti) = 0.

    It is easy to see that this convergence is well-defined and independent of the choices of represen-

    tatives of the trajectories. We can also extend this notion to sequences of broken connecting orbits.

    We omit that for simplicity.

    The main theorem of this section is

    Theorem 5.3 (Compactness of the moduli space of stable connecting orbits). The space

    M ([x−] , [x+]) is a compact Hausdorff space with respect to the topology defined in Definition 5.2.

    Indeed the proof is routine and it has been carried out in many literature for general symplectic

    vortex equations, for example [25], [2], [26], [37]. Since bubbling is ruled out, the proof is almost the

    same as that for finite dimensional Morse theory, while the gauge symmetry is the only additional

    ingredient.

    5.2. Local compactness with uniform bounded energy density. For any compact subset

    K ⊂ Θ, consider a sequence of solutions ũi := (ui,Φi,Ψi) such that the image of ui is contained inthe compact subset KH ⊂M and such that

    lim supi→∞

    E(ũi)

  • 30 GUANGBO XU

    5.3. Energy quantization. To prove the compactness of the moduli space, we need the following

    energy quantization property.

    Proposition 5.5. There exists �0 := �0(J,H) > 0, such that for any connecting orbit ũ ∈ M̃bΘ, wehave E(ũ) ≥ �0.

    Proof. Suppose it is not true. Then there exists a sequence of connecting orbits, represented by

    solutions in temporal gauge ṽi := (vi, 0,Ψi) ∈ M̃bΘ, such that

    E(ṽi) > 0, limi→∞

    E(ṽi) = 0. (5.6)

    We first know that there is a compact subset KH ⊂ M such that for every i, the image vi(Θ) iscontained in KH . Then we must have

    limi→∞

    supΘ

    (|∂svi|+ |µ(vi)|) = 0. (5.7)

    Indeed, if the equality doesn’t hold, then we can find a subsequence which either bubbles off a

    nonconstant holomorphic sphere at some point z ∈ Θ (if the above limit is∞), or (after a sequenceof proper translation in s-direction) converges to a solution (with positive energy) on compact

    subsets (if the above limit is positive and finite). Either case contradicts the assumption. Therefore

    we conclude that for any � > 0, the image of vi lies in U� := µ−1(g∗� ) for i sufficiently large.

    Recall that we have projections πµ : U� → µ−1(0) and πµ : U� → M . Then for all large i andany s, πµ(vi(s, ·)) is C0-close to a periodic orbit of H in M . Since those orbits are discrete (inC0-topology, for example), we may fix one such orbit γ ∈ ZeroBH and choose a subsequence (stillindexed by i) such that

    lim supi→∞

    sup(s,t)∈Θ

    d (πµ(vi(s, t)), γ(t)) = 0. (5.8)

    Then, use a fixed Riemannian metric on M with its exponential map exp, we can write

    πµ(vi(s, t)) = expγ(t)ξi(s, t)

    where ξi ∈ Γ(S1, γ∗TM

    ). Let B�(γ

    ∗TM) be the �-disk of γ∗TM . Then expγ pulls back µ−1(0)→

    M to a G-bundle Q→ B�(γ∗TM), together with a bundle map γ̃ : Q→ µ−1(0). We can trivializeQ by some

    φ : Q→ G×B�(γ∗TM).

    Now we take a lift x̃ := (x, f) ∈ ZeroB̃H of γ. Then we can write

    φ(γ̃−1(x(t))

    )= (g0(t), γ(t)). (5.9)

    On the other hand, we write

    φ(γ̃−1πµ(vi(s, t))

    )= (gi(s, t), ξi(s, t)). (5.10)

    Take the gauge transformation g̃i(s, t) = gi(s, t)g0(t)−1. Then write

    ṽ′i := (v′i,Φ′i,Ψ′i) := g̃

    ∗i ṽi. (5.11)

  • GAUGED FLOER HOMOLOGY 31

    Then by the exponential convergence of vi as s→ ±∞, we see that ∂sgi(s, t) decays exponentiallyand hence Φ′i converges to zero as s→ ±∞. On the other hand, we see that

    φ(γ̃−1πµ(v

    ′i(s, t))

    )= (g0(t), ξ̃i(s, t)). (5.12)

    Therefore

    ṽ′i ∈ M̃(x̃, x̃). (5.13)

    But it is also easy to see that the homotopy class of ṽ′i is trivial. Because the energy of connecting

    orbits only depends on its homotopy class, we see that the energy of ṽ′i, and hence the energy of ṽi

    is actually equal to zero, which contradicts with the hypothesis. �

    5.4. Proof of the compactness theorem. It suffices to prove, without essential loss of gener-

    ality, that for any sequence [ũi] ∈ M ([x−], [x+]; J,H) represented by unbroken connecting orbits(ui,Φi,Ψi) ∈ M̃ (x−, x+), there exists a convergent subsequence. By the assumption that thereexists no nontrivial holomorphic sphere in M , we have

    supi,Θ|∂sui +XΦi(ui)| < +∞. (5.14)

    Then the limit (broken) connecting orbits can be constructed by induction and the energy quan-

    tization property (Proposition 5.5) guarantees that the induction stops at finite time. The details

    are standard and left to the reader.

    6. Floer homology

    In this section we use the moduli spacesM ([x−], [x+]; J,H) to define the vortex Floer homologygroup V HF∗ (M,µ;H). We also discuss further works and related problems in the last three

    subsections.

    By Corollary A.12, we can choose a generic S1-family of “admissible” almost complex structures

    J ∈ J̃ regH which is regular with respect to H. Such an object is a smooth t-dependent family ofalmost complex structures Jt, such that for each t, Jt is G-invariant, ω-compatible, and outside a

    neighborhood U of µ−1(0), Jt ≡ J; inside U , Jt preserves a distribution gCU . Being regular impliesthat for all pairs [x±] ∈ CritAH , the moduli space M ([x−], [x+]; J,H) is a smooth manifold with

    dimM ([x−], [x+]; J,H) = CZ([x−])− CZ([x+]). (6.1)

    Moreover, there is free R-action on M ([x−], [x+]; J,H) by time translation, whose orbit space isdenoted by M̂([x−], [x+]; J,H). Combining the compactness theorem, we have

    Proposition 6.1. If J ∈ J̃ regH , then

    CZ([x−])− CZ([x+]) ≤ 0 =⇒M ([x−], [x+]; J,H) = ∅; (6.2)

    CZ([x−])− CZ([x+]) = 1 =⇒ #M̂ ([x−], [x+]; J,H)

  • 32 GUANGBO XU

    6.1. The gluing map and coherent orientation. The boundary operator of the Floer chain

    complex is defined by the (signed) counting of M̂([x−], [x+]; J,H). If we want to define the Floerhomology over Z2, then we don’t need to orient the moduli space; otherwise, the orientation ofM̂([x−], [x+]; J,H) can be treated in the same way as the usual Hamiltonian Floer theory, sincethe augmented linearized operator Dũ (whose determinant is canonically isomorphic to the deter-

    minant of the actual linearization dF[ũ]), is of the same type of Fredholm operators considered inthe abstract setting of [10]. We first give the gluing construction and then discuss the coherent

    orientations of the moduli spaces.

    In this subsection we construct the gluing map for broken trajectories with only one breaking.

    The general case is similar. This construction is, in principle, the same as the standard construction

    in various types of Morse-Floer theory (see [32] [10]), with a gauge-theoretic flavor. The gauge

    symmetry makes the construction a bit more complex, since we always glue representatives, and

    we want to show that the gluing map is independent of the choice of the representatives.

    In this subsection, we fix the choice of the admissible family J ∈ J̃ regH and omit the dependenceof the moduli spaces on J and H.

    For any pair x± ∈ CritAH , we say that a solution ũ ∈ M̃ (x−, x+) is in r-temporal gauge, if itsrestrictions to [r,+∞)× S1 and (−∞, r]× S1 are in temporal gauge, for some r > 0. Now we fix anumber r = r0 and only consider solutions in r0-temporal gauge.

    Now we take three elements x, y, z ∈ CritÃH with

    CZ(x)− 1 = CZ(y) = CZ(z) + 1. (6.4)

    Assume y = (y, η) : S1 →M × g. We would like to construct, for a large R0 > 0, the gluing map

    glue : M̂ ([x], [y])× (R0,+∞)× M̂ ([y], [z])→ M̂ ([x], [z]) . (6.5)

    Now consider two trajectories [ũ±] = [u±,Φ±,Ψ±], [ũ−] ∈ M ([x], [y]), [ũ+] ∈ M ([y], [z]) withtheir representatives both in r0-temporal gauge and ũ± are asymptotic to y as s → ∓∞. Thenthere exists R1 > 0 such that

    ±s ≥ R1 =⇒ u∓(s, t) = expy(t) ξ∓(s, t). (6.6)

    Here Θ+R1 = [R1,+∞)× S1 and Θ−R1 = (−∞,−R1]× S

    1 and ξ± ∈W k,p(

    Θ∓R1 , y∗TM

    ).

    Next, we take a cut-off function ρ such that s ≥ 1 =⇒ ρ(s) = 1, s ≤ 0 =⇒ ρ(s) = 0. For eachR >> r0, denote ρR(s) = ρ(s−R). We construct the “connected sum”

    uR(s, t) =

    u−(s+R, t), s ≤ −R2 − 1

    expy(t)

    (ρR

    2(−s)ξ−(s+R, t) + ρR

    2(s)ξ+(s−R, t)

    )s ∈

    [−R2 − 1,

    R2 + 1

    ]u+(s−R, t), s ≥ R2 + 1

    (6.7)

    (ΦR,ΨR) (s, t) =

    (Φ−(s+R, t),Ψ−(s+R, t)) , s ≤ −R2 − 1(

    0, ρR2

    (−s)Ψ−(s+R, t) + ρR2

    (s)Ψ+(s−R, t)), s ∈

    [−R2 − 1,

    R2 + 1

    ](Φ+(s−R, t),Ψ+(s−R, t)) . s ≥ R2 + 1

    (6.8)

    Now it is easy to see that, if we change the choice of representatives ũ± which are also in r0-temporal

    gauge, the connected sum ũR := (uR,ΦR,ΨR) doesn’t change for s ∈ [−R2 − 1,R2 + 1] and hence

  • GAUGED FLOER HOMOLOGY 33

    we obtain a gauge equivalent connected sum. Moreover, if we change y by y′ which represent the

    same [y] ∈ CritAH , then we can obtain

    ũ′− ∈ M̃(x, y′

    ), ũ′+ ∈ M̃

    (y′, z

    )(6.9)

    which are also in r0-temporal gauge, and we obtain a connected sum ũ′R which is gauge equivalent

    to ũR.

    Now we consider the augmented linearized operator DR := DũR .

    Lemma 6.2. There exists c > 0 and R0 > 0 such that for every R ≥ R0 and η̃ ∈ Ẽ2,pũR ⊕W2,p (Θ, g),

    we have

    ‖D∗Rη̃‖W 1,p ≤ c ‖DRD∗Rη̃‖Lp . (6.10)

    Proof. Same as the proof of [32, Proposition 3.9] �

    Hence we can construct a right inverse

    QR := D∗R (DRD

    ∗R)−1 : Ẽ0,pũR ⊕ L

    p (Θ, g)→ TũRB̃1,p (6.11)

    with

    ‖QR‖ ≤ c. (6.12)

    Now we write QR := (QR,AR) with QR : Ẽ0,pũR → TũRB̃1,p (x, z). Then actually the image of QR

    lies in the kernel of d∗0 and therefore QR is a right inverse to dF̃ũR |kerd∗0 . Because our constructionis natural with respect to gauge transformations, we see that QR induces an injection

    QR : E0,p[ũR] → T[ũR]B1,p (6.13)

    which is a right inverse to the linearized operator dF[ũR] and which is bounded by c. By the implicitfunction theorem, we have

    Proposition 6.3. There exists R1 > 0, δ1 > 0 such that for each R ≥ R1, there exists a uniqueξ̃ ∈ ImQR = kerd∗0 ⊂ TũRB̃1,p,

    ∥∥∥ξ̃∥∥∥W 1,p

    ≤ δ1 such that

    F̃(

    expũR ξ̃)

    = 0,∥∥∥ξ̃∥∥∥

    W 1,p≤ 2c

    ∥∥∥F̃(ũR)∥∥∥Lp. (6.14)

    Therefore, the gluing map can be defined as

    glue ([ũ−], R, [ũ+]) =[expũR ξ̃

    ]∈ M̂ ([x], [z]; J,H) . (6.15)

    On the other hand, it is easy to see that the augmented linearized operators Dũ for all con-

    necting orbits ũ is of “class Σ” considered in [10]. Therefore, by the main theorem of [10], there

    exists a “coherent orientation” with respect to the gluing construction. Choosing such a coher-

    ent orientation, then to each zero-dimensional moduli space M̂([x], [y]; J,H), we can associate thecounting χJ([x], [y]) ∈ Z, where each trajectory [ũ] contributes to 1 (resp. -1) if the orientation of[ũ] coincides (resp. differ from) the “flow orientation” of the solution. Then we define

    δJ : V CFk (M,µ;H; ΛZ) → V CFk−1 (M,µ;H; ΛZ)

    [x] 7→∑

    [y]∈CritAH

    χJ([x], [y])[y](6.16)

  • 34 GUANGBO XU

    As in the usual Floer theory, we have

    Theorem 6.4. For each choice of the coherent orientation on the moduli spaces M([x], [y]; J,H),the operator δJ in (6.16) defines a morphism of ΛZ-modules satisfying δJ ◦ δJ = 0.

    This makes (V CF∗(M,µ;H; ΛZ), δJ) a chain complex of ΛZ-modules, to which will be generally

    referred as the vortex Floer chain complex. Therefore the vortex Floer homology group is defined

    as the ΛZ-module

    V HFk (M,µ; J,H; ΛZ) :=ker (δJ : V CFk (M,µ;H; ΛZ)→ V CFk−1 (M,µ;H; ΛZ))im (δJ : V CFk+1 (M,µ;H; ΛZ)→ V CFk (M,µ;H; ΛZ))

    (6.17)

    6.2. The continuation map. Now we prove that the vortex Floer homology group defined above is

    independent of the choice of admissible family of almost complex structures and the time-dependent

    Hamiltonians, and, if we use the moduli space of (1.8) instead of (1.7) to define the Floer homol-

    ogy, independent of the parameter λ > 0. So far the argument has been standard, by using the

    continuation principle.

    Let (Jα, Hα, λα) and(Jβ, Hβ, λβ

    )be two triples where λα, λβ > 0, Hα, Hβ are G-invariant

    Hamiltonians satisfy Hypothesis 2.1 and 2.3, and Jα ∈ J̃ regHα,λα , Jβ ∈ J̃ reg

    Hβ ,λβ(for notations refer to

    the appendix).

    We choose a cut-off function ρ : R → [0, 1] with s ≤ −1 =⇒ ρ(s) = 1 and s ≥ 1 =⇒ ρ(s) = 0.Then we define

    Hs,t = ρ(s)Hαt + (1− ρ(s))H

    βt , λs = ρ(s)λ

    α + (1− ρ(s))λβ. (6.18)

    We denote this family of Hamiltonians by H . Now, as in the appendix, we consider the space of

    families of admissible almost complex structures J̃(Jα, Jβ

    )consisting of smooth families of almost

    complex structures J = (Js,t)(s,t)∈Θ, such that for each l ≥ 1,∣∣∣e|s|Js,t − Jαt ∣∣∣Cl(Θ−×M)

  • GAUGED FLOER HOMOLOGY 35

    One thing to check in defining the continuation map is the energy bound of solutions, which

    implies the compactness of the moduli space. We define the energy to be

    E(ũ) = E(u,Φ,Ψ)

    =1

    2

    (|∂su+XΦ(u)|2L2 +

    ∣∣∂tu+XΨ(u)− YHs,t(u)∣∣2L2 + ∣∣λ−1s (∂sΨ− ∂tΦ + [Φ,Ψ])∣∣2L2 + |λsµ(u)|2L2)= |∂su+XΦ(u)|2L2 + |λsµ(u)|L2 (6.22)

    where, the last equality holds only for ũ a solution to (6.20). Then we have

    Proposition 6.5. For any solution ũ to (6.20) whose energy is finite and whose image in M has

    compact closure, we have

    E (ũ) = AHα([xα])−AHβ ([xβ])−∫

    Θ

    ∂Hs,t∂s

    (u)dsdt. (6.23)

    Proof. We can transform the solution ũ into temporal gauge. Then the energy density is

    |∂su|2 + |λsµ(u)|2 =ω(∂su, ∂tu+XΨ − YHs,t(u)

    )− µ(u) · ∂Ψ

    ∂s

    =ω(∂su, ∂tu)−∂

    ∂s(µ(u) ·Ψ) + ∂

    ∂s(Hs,t(u))−

    ∂Hs,t∂s

    (u).

    (6.24)

    Then integrating over Θ, we obtain (6.23). �

    Theorem 6.6. There exists a Baire subset J̃ regH ,λs(Jα, Jβ

    )⊂ J̃

    (Jα, Jβ

    ), such that for any J ∈

    J̃ regH ,λs(Jα, Jβ

    ), the moduli space N ([xα], [xβ]; J ,H , λs) is a smooth oriented manifold with

    dimN(

    [xα], [xβ]; J ,H , λs)

    = CZ([xα])− CZ([xβ]). (6.25)

    So in particular, when CZ(xα) = CZ(xβ), N is of zero dimension. The algebraic count of N givesan integer χ

    ([xα], [xβ]

    ). Then we define the continuation map

    contβα : V CF∗ (M,µ; Jα, Hα, λα; ΛZ) → V CF∗

    (M,µ; Jβ, Hβ, λβ; ΛZ

    )[xα] 7→

    ∑[xβ ]∈CritA

    χ(

    [xα], [xβ])

    [xβ]. (6.26)

    Now we have the similar results as in ordinary Hamiltonian Floer theory.

    Theorem 6.7. The map contβα is a chain map. The induced map on the vortex Floer homology

    groups is independent of the choice of the homotopy H , the family J of almost complex structures,

    the cut-off function ρ. In particular, contβα is a chain homotopy equivalence. If (Jγ , Hγ) is another

    admissible pair and λγ > 0, then in the level of homology

    contγβ ◦ contβα = cont

    γα.

    Proof. The proof is essentially based on the construction of various gluing maps and the compact-

    ness results about N([xα], [xβ]; J ,H , λs

    )when CZ([xα]) − CZ([xβ]) = 1. As in the gluing map

    constructed in proving the property δ2J = 0, we need to specify a gauge to construct the approxi-

    mate solutions. We can still use solutions in r-temp