arX

iv:1

111.

2636

v3 [

mat

h.R

T]

8 S

ep 2

019

COHOMOLOGICALLY INDUCED DISTINGUISHED

REPRESENTATIONS AND COHOMOLOGICAL TEST VECTORS

BINYONG SUN

Abstract. Let G be a real reductive group, and let χ be a character of areductive subgroup H of G. We construct χ-invariant linear functionals oncertain cohomologically induced representations of G, and show that these linearfunctionals do not vanish on the bottom layers. Applying this construction,we prove two Archimedean non-vanishing hypotheses, which are vital to thearithmetic study of special values of certain L-functions via modular symbols.

1. Introduction

1.1. Distinguished representations. Let G be a real reductive group, namely,it is a Lie group with the following properties:

• g is reductive;• G has only finitely many connected components;• there is a connected closed subgroup of G with finite center whose com-plexified Lie algebra equals [g, g].

Here and henceforth, we use the corresponding lower case Gothic letter to indicatethe complexified Lie algebra of a Lie group. In particular, g denotes the complex-ified Lie algebra of G. For applications to the theory of automorphic forms, weare interested in Casselman-Wallach representations of G. Recall that a (complex)representation of a real reductive group is said to be Casselman-Wallach if it isFrechet, smooth, of moderate growth, and its underlying Harish-Chandra moduleis admissible and finitely generated. The reader may consult [Cas], [Wa2, Chapter11] or [BK] for more details about Casselman-Wallach representations. To ease no-tation, we do not distinguish a representation with its underlying vector space, ora character of a Lie group with its corresponding one-dimensional representation.

Let H be a closed subgroup of G, and let χ : H → C× be a character. By aχ-distinguished representation of G, we mean a Casselman-Wallach representationπ of G, together with an H-equivariant continuous linear functional ϕ : π → χ.Distinguished representations are ubiquitous in representation theory and in thetheory of automorphic forms.

2000 Mathematics Subject Classification. 22E46, 22E41.Key words and phrases. real reductive group, distinguished representation, cohomological in-

duction, L-function, special value.1

2 B. SUN

Fix a Cartan involution θ of G. From now on we assume that H has only finitelymany connected components, and that θ(H) = H . Then H is also a real reductivegroup, and θ restricts to a Cartan involution of H . Write

K := Gθ (the fixed point group) and Hc := H ∩K,which are respectively maximal compact subgroups of G and H .

Recall that Casselman-Wallach globalizations establish an equivalence betweenthe category of finitely generated admissible (g, K)-modules and the category ofCasselman-Wallach representations of G. In particular, every Casselman-Wallachrepresentation π of G equals the Casselman-Wallach globalization of its underlying(g, K)-module π[K]. The restriction induces an injective linear map

(1) HomH(π, χ) → Homh,Hc(π[K], χ).

We say that the quadruple (G, θ,H, χ) has the automatic continuity propertyif the map (1) is surjective for all Casselman-Wallach representations π of G. Atleast when this is the case, one may study χ-distinguished representations in thepurely algebraic setting of (g, K)-modules. The reader is referred to [BK] for morediscussions on the automatic continuity property. It holds at least for symmetricsubgroups as in the following theorem.

Theorem 1.1. ([BaD, Theorem 1] and [BrD, Theorem 1]) If there is an involutiveautomorphism σ of G which commutes with θ such that H is an open subgroup ofGσ, then (G, θ,H, χ) has the automatic continuity property.

Remark. The proof of Theorem 1.1 by van den Ban-Delorme and Brylinski-Delormeis carried out for trivial χ, but the same proof works in general. The author thanksPatrick Delorme for confirming this.

Example 1.2. Let (G,H) := (GL2n(R), Sp2n(R)) (n ≥ 1). For every g ∈ G, let

(2) θ(g) := g−t (the inverse transpose)

and

σ(g) :=

[0 1n

−1n 0

]g−t

[0 −1n1n 0

](1n is the identity matrix of size n).

Then θ is a Cartan involution of G stabilizing H , and σ is an involutive automor-phism of G which commutes with θ such that Gσ = H .

The original motivation of this paper is to settle two non-vanishing hypotheseswhich are vital to the arithmetic study of special values of certain L-functions(Theorems 4.5 and 5.5 in this paper). For this purpose, we will also consider thefollowing two examples of (G,H, θ, σ).

Example 1.3. Let (G,H) := (GLn(K),GLn(R)) (n ≥ 1), where K is a topologicalfield which is topologically isomorphic to C. For every g ∈ G, let

θ(g) := g−t (the inverse of the conjugate transpose)

DISTINGUISHED REPRESENTATIONS 3

and

σ(g) := g (the complex conjugate).

Example 1.4. Let (G,H) := (GL2n(R),GLn(R) × GLn(R)) (n ≥ 1). For everyg ∈ G, let θ(g) be as in (2), and

σ(g) :=

[1n 00 −1n

]g

[1n 00 −1n

].

Like Example 1.2, in the above two examples, θ is a Cartan involution of Gstabilizing H , and σ is an involutive automorphism of G which commutes with θsuch that Gσ = H . Hence Theorem 1.1 applies to all of the above three examples.We remark that in all these three examples, (G,H) is a Gelfand pair, namely (see[AGS, AiS])

dimHomH(π,C) ≤ 1 (C stands for the trivial representation)

for every irreducible Casselman-Wallach representation π of G.

1.2. Cohomologically induced distinguished representations. The main themeof this paper is an algebraic construction of distinguished representations via co-homological induction.

To be precise, let q be a parabolic subalgebra of g which is θ-stable, namely,θ(q) = q. Here θ : g → g denotes the complexified differential of θ : G → G. Weuse “ ¯ ” to indicate the complex conjugation in various contexts. In particular,¯ : g → g denotes the complex conjugation with respect to the real form Lie(G)of g. Note that the parabolic subalgebras q and q are opposite to each other. Put

L := NG(q) = NG(q) (the normalizers),

and put Lc := L ∩ K. Then L is a θ-stable real reductive group, and Lc is amaximal compact subgroup of it.

Denote by n the nilpotent radical of q ∩ [g, g]. Then the parabolic subalgebrasq and q respectively have Levi decompositions

(3) q = l⊕ n and q = l⊕ n.

Put nc := n ∩ k.Write Πg,K

q,Lcfor the (dim nc)-th left derived functor of the functor

R(g, K)⊗R(q,Lc) ( · )

from the category of (q, Lc)-modules to the category of (g, K)-modules. Here “R ”indicates the Hecke algebra of a pair (see [KV, Chapter I, Section 5]). Let X be an(l, Lc)-module, to be viewed as a (q, Lc)-module via the trivial action of n. Then

we get the cohomologically induced (g, K)-module Πg,Kq,Lc

(X).

4 B. SUN

Note that L∩H is a θ-stable real reductive group in L, and Lc∩Hc = L∩H∩Kis a maximal compact subgroup of it. The adjoint representation of the compactgroup Lc ∩Hc on the one-dimensional space

∧top(hc/(lc ∩ hc)) (the top degree wedge product)

preserves a real form. Thus it corresponds to a quadratic character of Lc ∩ Hc.Consequently, it is also an (l ∩ h, Lc ∩ Hc)-module with the trivial (l ∩ h)-action.This (l∩h, Lc∩Hc)-module further corresponds to a quadratic character of L∩H ,which we denote by εL∩H. Using this character, we define a character on L∩H by

(4) χL∩H := εL∩H · χ|L∩H .Under the assumptions

(5) q+ h = g and q ∩ h = q ∩ h,

in Section 2, we will construct χ-invariant linear functionals on Πg,Kq,Lc

(X) fromχL∩H-invariant linear functionals on X , in other words, we will define a linear map

(6) Πg,Kq,Lc

: Homl∩h,Lc∩Hc(X,χL∩H) → Homh,Hc(Πg,Kq,Lc

(X), χ).

Example 1.5. Let G1 be a real reductive group with a Cartan involution θ1. Sup-pose G = G1×G1 and θ = θ1×θ1. Suppose H is the groupG1 diagonally embeddedin G. Let q1 be a θ1-stable parabolic subalgebra of g1. Then q := q1 ⊕ q1 satisfies(5).

Remark. By using the pair (G,H) of Example 1.5, our construction in (6) recoversthe construction of Shapovalov forms (cf. [KV, Section VI.4]) which are used toprove the unitarity of certain cohomologically induced representations.

Example 1.6. Suppose σ is as in Theorem 1.1 so that H is a symmetric subgroup.Let x be an element of the Lie algebra of K such that σ(x) = −x (here σ : g → g

denotes the complexified differential of σ : G→ G). Suppose q = qx is the sum ofthe eigenspaces with non-negative eigenvalues of the operator [x/

√−1, · ] : g → g.

Then q is a θ-stable parabolic subalgebra satisfying (5). In this case, σ(L) = L,and L ∩H is a symmetric subgroup of L.

Remark. Let the notation and the assumptions be as in Example 1.6. Flensted-Jensen [FJ] and Oshima-Matsuki [OM] classify the discrete series representations ofG on G/H , namely the irreducible subrepresentations of L2(G/H). In particular,they show that such a representation exists if and only if the symmetric spacesG/H and K/Hc have equal rank. Assume that G is connected. Schlichtkrull[Sch1] and Vogan [Vog2] prove that all these discrete series representations have

the form Πg,Kq,Lc

(X), where q = qx for an appropriate x as in Example 1.6, and X

is an appropriate character of L. The embedding Πg,Kq,Lc

(X) → L2(G/H) providedby the works of Flensted-Jensen and Oshima-Matsuki, followed by the evaluating

DISTINGUISHED REPRESENTATIONS 5

map at 1 ∈ G/H , yields an invariant linear functional on Πg,Kq,Lc

(X), namely anelement of

Homh,Hc(Πg,Kq,Lc

(X),C) (C stands for the trivial representation).

We expect that our construction (6) also produces this invariant linear functional.

Besides the symmetric subgroup case of the above two examples, there are someother interesting cases of q satisfying (5). See [Sun, Section 2.2] for an example ofthe Gross-Prasad case.

Remarks. (a) Assume that the (l, Lc)-module X is finitely generated and admis-sible, and ϕ ∈ Homl∩h,Lc∩Hc(X,χL∩H) continuously extends to the Casselman-Wallach globalization of X . It is natural to ask the following question: does thelinear functional Πg,K

q,Lc(ϕ) continuously extend to the Casselman-Wallach globaliza-

tion of Πg,Kq,Lc

(X)? By Theorem 1.1, the answer is yes in the symmetric subgroupcase. Due to the lack of a theory of cohomological inductions in the setting ofCasselman-Wallach representations, not much is known beyond this case.

(b) With the notation as in (a), assume that Πg,Kq,Lc

(ϕ) continuously extends

to the Casselman-Wallach globalization (Πg,Kq,Lc

(E))∞ of Πg,Kq,Lc

(E). Then we get aG-intertwining linear map

(Πg,Kq,Lc

(E))∞ → IndGHχ := {f ∈ C∞(G) | f(hg) = χ(h)f(g), h ∈ H, g ∈ G},

v 7→ (g 7→ (Πg,Kq,Lc

(ϕ))(g.v)).

For unitary χ, it is interesting to know in which cases the image of the above mapis contained in the space of square integrable sections. See [Sch1] for a relevantwork in the symmetric subgroup case.

If G is compact, then both H and L are compact, and the linear map (6) is alsowritten as

(7) Πg,Gq,L : HomL∩H(X,χL∩H) → HomH(Π

g,Gq,L(X), χ).

The following theorem will be proved in Section 3. It asserts that, when G iscompact and connected, our construction produces all the χ-invariant linear func-tionals.

Theorem 1.7. Suppose G is compact and connected. Let X be an L-module(namely an (l, L)-module). Assume (5) holds so that the linear map (7) is defined.Then the linear map (7) is surjective.

In many interesting cases, the two spaces in (7) are both one-dimensional. Whenthis is the case, the surjectivity of the linear map (7) implies that it is also injec-tive. In general, the following example shows that the linear map (7) may not beinjective.

6 B. SUN

Example 1.8. Suppose G = SU(2), H = SO(2) and χ = χk (k ∈ Z) is the characterof H given by [

cos t sin t− sin t cos t

]7→ cos(kt) +

√−1 sin(kt).

Suppose q is the Borel subalgebra of g consisting of the upper triangular matricesso that (5) is satisfied. Then

L =

{[a 00 a−1

]| a ∈ U(1)

}= U(1).

Suppose X = Xm is a one-dimensional representation of L of weight m ≥ 2. ThenΠg,G

q,L(Xm) is an irreducible representation of G of highest weight m− 2.Note that L∩H = {±1} and the character εL∩H is trivial. If k and m have the

same parity, then

dimHomL∩H(Xm, χk) = 1.

On the other hand,

dimHomH(Πg,Gq,L (Xm), χk) = 0

unless 2 − m ≤ k ≤ m − 2. Therefore, if k and m have the same parity, and|k| > m−2, then (7) is a linear map from a one-dimensional space to a zero space.Thus it is not injective.

1.3. Test vectors in the bottom layers. Now we return to the general casewhen Gmay or may not be compact. Given a χ-distinguished representation (π, ϕ)of G, it is an important (and often hard) problem to find an explicit vector v0 ∈ πsuch that ϕ(v0) 6= 0. Such a vector is called a test vector of the χ-distinguishedrepresentation. For arithmetic applications, we are particularly interested in thecase when π is an irreducible unitarizable representation with nonzero cohomology,and we hope to find a test vector in π which supports the cohomology.

Recall that all irreducible unitary representations with nonzero cohomology areobtained by cohomological induction. At least for these representations, the bot-tom layers coincide with the minimal K-types (in the sense of Vogan), and theyhave non-trivial contributions to the cohomologies (see [VZ] and [BW]). In thissense, we refer test vectors in the bottom layers as cohomological test vectors. InTheorem A.3 of Appendix A, we will show that the non-vanishing on the bottomlayers implies the non-vanishing of certain restriction maps of cohomology spaces.These restriction maps capture the Archimedean behaviors of modular symbols,and we call them modular symbols at infinity. The modular symbols are non-trivial and of arithmetic interest only when the corresponding modular symbolsat infinity are nonzero (see [AG, GR, AsS, GHL, KMS] for examples). In Sections4 and 5, we will give two examples of nonzero modular symbols at infinity, whichare respectively used in the arithmetic study of the Asai L-functions [GHL] andthe standard L-functions for GSpin(2n+ 1) [AG, GR].

DISTINGUISHED REPRESENTATIONS 7

Recall from [KV, Lemma 5.10] that if G is connected, then so is L. To be a littlemore general, we now assume that L is connected, but allow G to be disconnected.This includes the cases of (G,L) = (GL2n(R),GLn(C)) or (GL2n(R), (C

×)n) whichwe are considering in this paper. We use a superscript “ ◦ ” to indicate the identityconnected component of a Lie group. The connectedness of L implies that L◦

c =Lc ⊂ K◦.

Similar to (3), we have Levi decompositions

qc = lc ⊕ nc and qc = lc ⊕ nc,

where qc := q ∩ k is a parabolic subalgebra of k, and nc := n ∩ k is the nilpotentradical of qc ∩ [k, k]. Let E be an Lc-module, namely an (lc, Lc)-module. Similar

to the (g, K)-module Πg,Kq,Lc

(X), we have a cohomologically induced K◦-module

Πk,K◦

qc,Lc(E).

We say that an irreducible Lc-module is dominant (with respect to qc) if it isisomorphic to τ nc for an irreducible K◦-module τ . Here and as usual, a superscriptLie algebra indicates the vectors annihilated by the Lie algebra action. If E isirreducible, then the algebraic version of the Bott-Borel-Weil Theorem (cf. [KV,Corollary 4.160 and Proposition 4.173]) implies that

(8) Πk,K◦

qc,Lc(E) ∼=

{τ, if E ⊗ ∧dim ncnc ∼= τ nc is dominant;{0}, if E ⊗ ∧dim ncnc is not dominant.

Note that (5) implies that

(9) qc + hc = k and qc ∩ hc = qc ∩ hc.

Similar to (6), the equalities in (9) enable us to define a linear map

(10) Πk,K◦

qc,Lc: HomLc∩Hc(E, χL∩H) → HomK◦∩Hc(Π

k,K◦

qc,Lc(E), χ).

By Theorem 1.7, this map is surjective.Recall that X is an (l, Lc)-module. Fix a homomorphism φ ∈ HomLc(E,X).

Then we have the bottom layer map (see Section 2.4)

β(φ) ∈ HomK◦(Πk,K◦

qc,Lc(E),Πg,K

q,Lc(X)).

The following proposition asserts that our construction of invariant functionalsis compatible with the bottom layer map.

Proposition 1.9. (see Proposition 2.7) Suppose (5) is satisfied and L is connected.Let X be an (l, Lc)-module and let E be an Lc-module. Then the diagram

(11)

Πk,K◦

qc,Lc(E)

β(φ)−−−→ Πg,Kq,Lc

(X)yΠk,K◦

qc,Lc(ϕ◦φ)

yΠg,Kq,Lc

(ϕ)

χ χ

commutes, for all φ ∈ HomLc(E,X) and ϕ ∈ Homl∩h,Lc∩Hc(X,χL∩H).

8 B. SUN

As mentioned earlier, we hope to find test vectors in the bottom layers. Moreprecisely, at least in some interesting cases, we want to show that the χ-invariantlinear functional Πg,K

q,Lc(ϕ) does not vanish on the bottom layers. Proposition 1.9

reduces this problem to the case of compact connected groups. Consequently, wemay apply Theorem 1.7 to get the following result.

Theorem 1.10. Suppose (5) is satisfied and L is connected. Let X be an (l, Lc)-module, and let E be an irreducible Lc-submodule of X such that

dimHomLc∩Hc(E, χL∩H) = dimHomK◦∩Hc(Πk,K◦

qc,Lc(E), χ).

If there is a χL∩H-invariant linear functional on X which does not vanish on E,then there is a χ-invariant linear functional on Πg,K

q,Lc(X) which does not vanish on

Πk,K◦

qc,Lc(E).

Note that the assumptions of Theorem 1.10 imply that the K◦-module Πk,K◦

qc,Lc(E)

is nonzero, and hence irreducible. It is viewed as a K◦-submodule of Πg,Kq,Lc

(X) viathe bottom layer map which is injective (see [KV, Chapter V, Section 6]).

Example 1.11. Let (G,H, θ, σ) be as in Example 1.2 so that G = GL2n(R) andH = Sp2n(R). Combining [GOSS, Theorem A] and [AOS, Theorem 1.1], the set ofH-distinguished unitarizable irreducible Casselman-Wallach representations of Gis determined (see also [GSS]). Here an irreducible Casselman-Wallach representa-tion of G is said to be H-distinguished if there is a nonzero H-invariant continuouslinear functional on it. A major step in the determination is to show that if n iseven, then every Speh representation of G is H-distinguished. This is done in[GOSS, Proposition 4.0.2] by global method, and in [GSS, Theorem A] by analyticmethod. This also follows from Theorem 1.10 as explained in what follows.

For k ≥ 1, denote by Dk the unique (up to isomorphism) irreducible Cassleman-Wallach representation of GL2(R) in the relative discrete series which has the same

infinitesimal character as that of Sk−1(C2) ⊗ |det|− k−12 . Here C2 stands for the

standard representation of GL2(R), and “Sk−1” indicates the (k−1)-th symmetricpower. The Speh representation J2n,k of GL2n(R) is the unique irreducible quotientof

IndGP2,2,··· ,2

((Dk ⊗ |det|n−12 )⊗(Dk ⊗ |det|n−3

2 )⊗ · · · ⊗(Dk ⊗ |det| 1−n2 )).

Here and henceforth, P2,2,··· ,2 denotes the upper triangular parabolic subgroup oftype (2, 2, · · · , 2), “⊗” stands for the completed projective tensor product, and“Ind” stands for the normalized smooth induction. Let y be a skew-symmetricmatrix in gl2n(R) such that y2 = −12n. Define qy as in Example 1.6 and supposeq = qy. Define a unitary character

(12) κn : GLn(C) → C×, z 7→ det(z)

|det(z)| .

DISTINGUISHED REPRESENTATIONS 9

View y as a complex structure so that R2n is viewed as an n-dimensional complexvector space. Then L ∼= GLn(C) and we view κn as a character on L via thisisomorphism. It is known that the underlying (g, K)-module of J2n,k is isomorphic

to Πg,Kq,Lc

(κk+nn ) (see [Sp, Section 4]).

Now suppose n = 2m is even and

y =

0 1m 0 0−1m 0 0 00 0 0 −1m0 0 1m 0

.

Then σ(y) = −y and q satisfies (5). Note that

K◦ = SO(2n), Hc∼= U(n), Lc

∼= U(n), L∩H ∼= Sp2m(C), Lc∩Hc∼= Sp(m).

By (8), τ := Πk,K◦

qc,Lc(κk+n

n ) is an irreducible representation of SO(2n) of highestweight (k + 1, k + 1, · · · , k + 1). Cartan-Helgason Theorem (cf. [Hel, Chapter V,Theorem 4.1]) implies that

dimHomHc(τ,C) = 1.

Then it is clear from Theorem 1.10 and Theorem 1.1 that there is an H-invariantcontinuous linear functional on J2n,k which does not vanish on τ . We remark thatneither the global method of [GOSS] nor the analytic method of [GSS] proves thenon-vanishing of the invariant linear functional on the minimal K-type.

In Section 2, we will explain the general construction of cohomologically induceddistinguished representations. Section 3 is devoted to a proof of Theorem 1.7. InSection 4 and Section 5, we will give two arithmetically interesting applicationsof our construction. In Appendix A, a general non-vanishing result is proved formodular symbols at infinity, which contains Theorem 4.5 and Theorem 5.5 asspecial cases.

Acknowledgements. The author thanks Michael Harris for the suggestion to workon the non-vanishing hypothesis for Asai L-functions (Theorem 4.5), and thanksDihua Jiang for the suggestion to work on the non-vanishing hypothesis for L-functions for GSpin(2n + 1) (Theorem 5.5). He thanks Patrick Delorme for con-firming the automatic continuity theorem in the twisted case (Theorem 1.1), andthanks Wee Teck Gan for the suggestion to write down a general criterion (as inTheorem A.3) for the non-vanishing of modular symbols at infinity. He also thanksHongyu He, Fabian Januszewski and Chen-Bo Zhu for helpful discussions. Part ofthe work was done when the author participated in the “Analysis on Lie groups”program at Max Planck Institute for Mathematics, in 2011. The author thanks theorganizers for the invitation and thanks Max Planck Institute for Mathematics fortheir hospitality. He also thanks the referees for helpful suggestions which resultedin the improvement of the paper, in particular, for the suggestion to consider thepair (GL2n(R), Sp2n(R)) of Example 1.11. The work was supported in part by the

10 B. SUN

National Natural Science Foundation of China (No. 11525105, 11688101, 11621061and 11531008).

2. Cohomologically induced distinguished representations

2.1. Bernstein functors and Zuckerman functors. We begin with recallingsome basic facts concerning Bernstein functors and Zuckerman functors. Let (g, K)be a pair as in [KV, Section I.4], namely,

• g is a finite-dimensional Lie algebra over C;• K is a compact Lie group whose complexified Lie algebra k is identifiedwith a Lie subalgebra of g;

• there is given an action Ad of K on g by automorphisms which extendsthe adjoint representation of K;

• the differential of the action Ad equals the adjoint action of k on g.

Let Lc be a closed subgroup of K. Denote by B the Bernstein functor

R(g, K)⊗R(g,Lc) ( · )from the category of (g, Lc)-modules to the category of (g, K)-modules. Write Bj

for its j-th left derived functor (j ∈ Z). Likewise, denote by Z the Zuckermanfunctor

HomR(g,Lc)(R(g, K), ( · ))K-finite

from the category of (g, Lc)-modules to the category of (g, K)-modules, where“K-finite” indicates the space of K-finite vectors. Write Zj for its j-th rightderived functor (j ∈ Z).

As in [DV], in order to describe the functor Bj more explicitly, for each (g, Lc)-module V0, we introduce a linear action

(13) (k, Lc)× (g, K) y R(K)⊗ V0 (R(K) := R(k, K)),

as follows:

• the pair (k, Lc) acts by the tensor product of the right translation on R(K)and the restriction of the (g, Lc)-action on V0;

• the group K acts on R(K)⊗ V0 through the left translation on R(K);• the Lie algebra g acts on R(K)⊗ V0 such that

(14)

∫

K

f(k) d(X.(µ ⊗ v))(k) =

∫

K

f(k)(Adk−1X).v dµ(k),

for all X ∈ g, µ ∈ R(K), v ∈ V0, and f ∈ C[K].

Here C[K] denotes the space of all left K-finite (or equivalently, right K-finite)smooth functions on K. (Similar notation will be used for other compact Liegroups.) In the left hand side of (14), we view R(K)⊗ V0 as a space of V0-valuedmeasures on K.

DISTINGUISHED REPRESENTATIONS 11

Under these actions, R(K) ⊗ V0 becomes a (k, Lc)-module as well as a weak(g, K)-module (see [KV, Chapter I, Section 5] for the notion of weak (g, K)-modules). Furthermore, the (k, Lc)-action and the (g, K)-action commute witheach other, and

(15) Bj(V0) = Hj(k, Lc; R(K)⊗ V0), j ∈ Z,

as (g, K)-modules (cf. [KV, Section III.3]). In particular the homology spaceHj(k, Lc; R(K)⊗ V0) is not only a weak (g, K)-module, but also a (g, K)-module.The reader is referred to [KV, (2.126)] and [KV, (2.127)] for the explicit com-plexes which respectively compute the relative Lie algebra homology spaces andthe relative Lie algebra cohomology spaces.

Similarly, in order to describe the functor Zj more explicitly, for each (g, Lc)-module V , we introduce a linear action

(16) (k, Lc)× (g, K) y C[K]⊗ V,

as follows:

• the pair (k, Lc) acts by the tensor product of the left translation on C[K]and the restriction of the (g, Lc)-action on V ;

• the group K acts on C[K]⊗ V through the right translation on C[K];• the Lie algebra g acts by

(17) (X.f)(k) := (AdkX).f(k), k ∈ K, f ∈ C[K]⊗ V.

Here and as usual, C[K] ⊗ V is identified with a space of V -valued functions onK. Then similar to (15), we have that

(18) Zj(V ) = Hj(k, Lc;C[K]⊗ V ), j ∈ Z,

as (g, K)-modules.Recall the following Zuckerman Duality Theorem.

Theorem 2.1. (see [KV, Corollary 3.7]) For every (g, Lc)-module V0 and everyj ∈ Z, there is a canonical isomorphism

(19) Bj(V0) ∼= Zm−j(∧mk/lc ⊗ V0) (m := dim k/lc)

of (g, K)-modules. Here ∧mk/lc is viewed as a (g, Lc)-module such that g actstrivially, and Lc acts through the adjoint representation.

Proof. One checks that the linear isomorphism

Ij : ∧j(k/lc)⊗ R(K)⊗ V0 → HomC(∧m−j(k/lc),C[K]⊗ ∧mk/lc ⊗ V0),ω ⊗ fµK ⊗ v 7→ (ω′ 7→ f∨ ⊗ (ω ∧ ω′)⊗ v)

is Lc-equivariant and (g, K)-equivariant, and {(−1)j(j+1)

2 Ij}j∈Z restricts to a mor-phism of the chain complexes which compute (15) and (18). Here µK denotes theHaar measure on K such that K◦ has volume 1, and f∨ is the function on K suchthat f∨(x) = f(x−1) for all x ∈ K. Therefore Ij induces an isomorphism (19). See[KV, Chapter III] for more details. �

12 B. SUN

2.2. Some one-dimensional modules. Now we retain the notation of the In-troduction. Recall that we have the Lie groups

L

❍❍❍❍

❍❍❍❍

❍❍

L ∩H

rrrrrrrrrrr

❑❑❑❑

❑❑❑❑

❑❑Lc

✈✈✈✈✈✈✈✈✈

Lc ∩Hc

⊂

G

❈❈❈❈

❈❈❈❈

H

⑥⑥⑥⑥⑥⑥⑥⑥

❆❆❆❆

❆❆❆❆

K ,

⑤⑤⑤⑤⑤⑤⑤⑤

Hc

and the Lie algebras q = l⊕n and qc = lc⊕nc. We first allow L to be disconnected.Recall from (5) that

(20) q+ h = g and q ∩ h = q ∩ h.

Lemma 2.2. One has that

(21) g = q+ h = q+ h, q ∩ h = q ∩ h = l ∩ h,

and

(22) k = qc + hc = qc + hc, qc ∩ hc = qc ∩ hc = lc ∩ hc.

Proof. The equalities of (21) is an obvious consequence of the assumption (20).The equalities of (22) is implied by (21) since h, q, q and l are all θ-stable. �

Lemma 2.3. As representations of Lc ∩Hc,

(23) hc/(lc ∩ hc) ∼= nc ∼= nc ∼= k/(lc + hc),

and they are all self-dual.

Proof. By (22), we have that

(24) hc/(lc ∩ hc) = hc/(qc ∩ hc) ∼= k/qc ∼= nc ∼= qc/lc ∼= k/(lc + hc).

Similarly,

(25) hc/(lc ∩ hc) ∼= nc ∼= k/(lc + hc).

Note that nc and nc are dual to each other under the Killing form of k. Thereforethe lemma follows. �

Write S := dim nc for simplicity. One important consequence of our main as-sumption (20) is the following “degree match”:

dim(hc/(lc ∩ hc)) = S (this follows from (23)).

Recall from the Introduction that χ is a character of H . View ∧Shc/(lc ∩ hc)as an (h, Lc ∩Hc)-module such that h acts trivially and Lc ∩Hc acts through theadjoint representation. Define a one-dimensional (h, Lc ∩Hc)-module

ξ := (∧Shc/(lc ∩ hc))⊗ χ|(h,Lc∩Hc).

DISTINGUISHED REPRESENTATIONS 13

Note that the adjoint action of Lc on ∧2Sk/lc is trivial. We view ∧2Sk/lc asa trivial (g, Lc)-module, and likewise view ∧2S(k/lc)

∨ as a trivial (g, Lc)-module.Here and henceforth, a superscript “ ∨ ” indicates the dual module. Put

ξ0 := (∧2S(k/lc)∨)⊗ ξ.

Then the one-dimensional module (ξ0)|(l∩h,Lc∩Hc) corresponds to the character χL∩H

of L∩H (which is defined in the Introduction). We identify ξ0 with χL∩H in whatfollows.

2.3. The construction. Let X be an (l, Lc)-module, and let

ϕ ∈ Homl∩h,Lc∩Hc(X,χL∩H).

View X as a (q, Lc)-module such that n acts trivially on it. Put

V0 := U(g)⊗U(q) X (“U” indicates the universal enveloping algebra).

This is a (g, Lc)-module with g acts by the left multiplication, and Lc acts by thetensor product of its adjoint action on U(g) and its given action on X .

Lemma 2.4. There is a unique (h, Lc ∩Hc)-equivariant linear map

(26) ψ0 : V0 → ξ0

which extends ϕ.

Proof. By (21), we have that

V0 = U(g)⊗U(q) X = U(h)⊗U(l∩h) X

as an (h, Lc∩Hc)-module. Therefore the lemma is a form of Frobenious reciprocity.�

Define a (g, Lc)-moduleV := ∧2Sk/lc ⊗ V0.

The linear functional ψ0 of (26) induces an (h, Lc∩Hc)-equivariant linear functional

(27) ψ := 1∧2Sk/lc ⊗ ψ0 : V → ξ.

Similar to the action

(28) (k, Lc)× (g, K) y C[K]⊗ V

of (16), based on the (h, Lc ∩Hc)-action on ξ, we define an action

(29) (hc, Lc ∩Hc)× (h, Hc) y C[Hc]⊗ ξ.

Note that there is a component-wise inclusion

(hc, Lc ∩Hc)× (h, Hc) ⊂ (k, Lc)× (g, K),

and the map

(30) rK,Hc ⊗ ψ : C[K]⊗ V → C[Hc]⊗ ξ

is (hc, Lc ∩Hc)× (h, Hc)-equivariant, where rK,Hc denotes the restriction map.

14 B. SUN

Write ξ for the module

(31) (hc, Lc ∩Hc)× (h, Hc) y (∧Shc/(lc ∩ hc))|(hc,Lc∩Hc) ⊗ χ|(h,Hc).

It equals ξ as a vector space.

Lemma 2.5. The linear map

(32)C[Hc]⊗ ξ → ξ,

f 7→∫Hcχ(c)−1 f(c) dc

is (hc, Lc ∩ Hc) × (h, Hc)-equivariant, where dc is the Haar measure on Hc suchthat K◦∩Hc has volume 1, and as usual, C[Hc]⊗ξ is viewed as a space of ξ-valuedfunctions on Hc.

Proof. This is routine to check. �

Lemma 2.6. One has an identification

(33) HS(hc, Lc ∩Hc; ξ) = χ

of (h, Hc)-modules.

Proof. Poincare duality (cf. [KV, Corollary 3.6]) implies that

HS(hc, Lc ∩Hc;∧Shc/(lc ∩ hc)) = C.

Thus

HS(hc, Lc ∩Hc; ξ) = HS(hc, Lc ∩Hc;∧Shc/(lc ∩ hc))⊗ χ = C⊗ χ = χ.

�

Finally, we define the (h, Hc)-equivariant linear functional

(34) Πg,Kq,Lc

(ϕ) : Πg,Kq,Lc

(X) → χ

to be the composition of the following maps:

Πg,Kq,Lc

(X) = HS(k, Lc; R(K)⊗ V0)(35)

→ HS(k, Lc; C[K]⊗ V )

→ HS(hc, Lc ∩Hc; C[Hc]⊗ ξ)

→ HS(hc, Lc ∩Hc; ξ) = χ.

Here the first arrow is the Zuckerman duality isomorphism as in Theorem 2.1, thesecond arrow is the restriction of cohomology induced by the map (30), and thethird arrow is the linear map induced by the map (32).

It is clear that (34) yields a linear map

(36) Πg,Kq,Lc

: Homl∩h,Lc∩Hc(X,χL∩H) → Homh,Hc(Πg,Kq,Lc

(X), χ).

DISTINGUISHED REPRESENTATIONS 15

Remark. The construction of Πg,Kq,Lc

(ϕ) is functorial in the following sense: for allcommutative diagram

X1η−−−→ X2yϕ1

yϕ2

χL∩H χL∩H ,

where Xi is an (l, Lc)-module, ϕi ∈ Homl∩h,Lc∩Hc(Xi, χL∩H) (i = 1, 2), and η ∈Homl,Lc(X1, X2), the diagram

Πg,Kq,Lc

(X1)Πg,K

q,Lc(η)

−−−−−→ Πg,Kq,Lc

(X2)yΠg,Kq,Lc

(ϕ1)

yΠg,Kq,Lc

(ϕ2)

χ χ

commutes.

2.4. Bottom layers. Now assume that L is connected so that Lc = L◦c ⊂ K◦.

Recall from the Introduction that E is an Lc-module and φ ∈ HomLc(E,X). Thehomomorphism φ induces a (k, Lc)× (k, K◦)-equivariant linear map

R(K◦)⊗ (U(k)⊗U(qc) E) → R(K◦)⊗ (U(g)⊗U(q) X),µ⊗X ⊗ v 7→ µ⊗X ⊗ φ(v).

Taking the relative Lie algebra cohomologies, we get the bottom layer map

Πk,K◦

qc,Lc(E) = HS(k, Lc; R(K

◦)⊗ (U(k)⊗U(qc) E))

→ Πg,K◦

q,Lc(X) = HS(k, Lc; R(K

◦)⊗ (U(g)⊗U(q) X)).(37)

The extension by zero map

R(K◦) → R(K)

induces an injective linear map

Πg,K◦

q,Lc(X) = HS(k, Lc; R(K

◦)⊗ (U(g)⊗U(q) X))

→ Πg,Kq,Lc

(X) = HS(k, Lc; R(K)⊗ (U(g)⊗U(q) X)).(38)

Now the composition of (37) and (38) yields the following bottom layer map:

β(φ) : Πk,K◦

qc,Lc(E) → Πg,K

q,Lc(X).

Similar to (36) and using (22), we have a linear map

(39) Πk,K◦

qc,Lc: HomLc∩Hc(E, χL∩H) → HomK◦∩H(Π

k,K◦

qc,Lc(E), χ).

16 B. SUN

Proposition 2.7. Assume (20) holds and L is connected. Then the diagram

(40)

Πk,K◦

qc,Lc(E)

β(φ)−−−→ Πg,Kq,Lc

(X)yΠk,K◦

qc,Lc(ϕ◦φ)

yΠg,Kq,Lc

(ϕ)

χ χ

commutes for all φ ∈ HomLc(E,X), and ϕ ∈ Homl∩h,Lc∩Hc(X,χL∩H).

Proof. This is routine to check. We omit the details. �

3. A proof of Theorem 1.7

In this section, suppose G is connected and compact. Then K = G, Hc = H ,qc = q, nc = n and L = Lc = L◦

c .Recall that we assume

(41) q+ h = g and q ∩ h = q ∩ h.

Our goal is to prove the following Theorem (Theorem 1.7 of the Introduction).

Theorem 3.1. If G is connected and compact, then the linear map

(42) Πg,Gq,L : HomL∩H(X,χL∩H) → HomH(Π

g,Gq,L (X), χ)

is surjective for every L-module X.

3.1. The G-module Πg,Gq,L (X). Recall the (g, L)-module

V = ∧2Sg/l⊗ (U(g)⊗U(q) X),

and writeΠg,G

q,L (X) := HS(g, L; C[G]⊗ V ).

It is isomorphic to Πg,Gq,L(X) by the Zuckerman duality isomorphism. The last two

arrows of (35) yield a linear map

(43) Πg,Gq,L : HomL∩H(X,χL∩H) → HomH(Π

g,Gq,L (X), χ).

It is clear that the map (42) is surjective if and only if so is the map (43).Without loss of generality, assume that the L-module X is irreducible. If

Πg,Gq,L(X) is zero, then Theorem 3.1 is trivial. So we assume it is nonzero. Then by

(8), Πg,Gq,L (X) is isomorphic to an irreducible G-module τ such that

τ n ∼= X ⊗ ∧Sn.

View C[G] as a G × G-module such that the first factor acts through the lefttranslation, and the second factor acts through the right translation. View τ∨ ⊗ τas a G×G-submodule of C[G] via the linear embedding

τ∨ ⊗ τ → C[G],λ⊗ u 7→ (g 7→ 〈λ, g.u〉).

DISTINGUISHED REPRESENTATIONS 17

Since HS(g, L; C[G] ⊗ V ) ∼= τ is irreducible, Peter-Weyl’s Theorem implies thatwe have an identification

(44) HS(g, L; τ∨ ⊗ τ ⊗ V ) = HS(g, L; C[G]⊗ V ).

Lemma 3.2. There is an identification

(45) HomL(∧Sg/q, (τ∨)n ⊗ (∧2Sg/l)⊗X) = HS(g, L; τ∨ ⊗ V )

of one-dimensional vector spaces.

Proof. This is known to experts. We sketch a proof for the convenience of thereader. Fix an element x0 in the center of l such that all the eigenvalues of theoperator [x0, · ] : n → n are positive real numbers. By considering the action ofx0, we know that

HomL(∧jg/l, τ∨ ⊗ V ) =

{{0}, if j 6= S,HomL(∧Sg/q, (τ∨)n ⊗ (∧2Sg/l)⊗X), if j = S.

Hence the lemma follows.�

By (44) and (45), we have identifications

Πg,Gq,L (X) = HomL(∧Sg/q, (τ∨)n ⊗ (∧2Sg/l)⊗X)⊗ τ(46)

= HomL(∧Sg/q, (τ∨)n ⊗ (∧2Sg/l)⊗X ⊗ τ)

of G-modules. Here we let L act trivially on the G-module τ .

3.2. The linear functional Πg,Gq,L (ϕ). Recall that

ϕ ∈ HomL∩H(X,χL∩H),

whereχL∩H = ∧2S(g/l)∨ ⊗ ∧Sh/(l ∩ h)⊗ χ,

to be viewed as an (L∩H)-module. By tensoring with ∧2Sg/l, it obviously inducesan (L ∩H)-equivariant linear map

(47) ϕ′ : (∧2Sg/l)⊗X → ∧Sh/(l ∩ h)⊗ χ.

Define a linear map

ϕ : (τ∨)n ⊗ (∧2Sg/l)⊗X ⊗ τ → ∧Sh/(l ∩ h)⊗ χ,

λ⊗ ω ⊗ x⊗ u 7→ ϕ′(ω ⊗ x)∫Hχ−1(h) 〈λ, h.u〉 dh,

where dh is the Haar measure on H with total volume 1.Write

(48) χ = HomL∩H(∧Sh/(l ∩ h),∧Sh/(l ∩ h)⊗ χ).

Here we let L ∩H act trivially on the H-module χ. Write

ιh : ∧Sh/(l ∩ h) → ∧Sg/q

18 B. SUN

for the obvious linear map. This is a linear isomorphism by (41). The followinglemma is routine to check.

Lemma 3.3. Under the identifications (46) and (48), the linear map

Πg,Gq,L (ϕ) : Π

g,Gq,L (X) → χ

is identical to the following linear map:

HomL(∧Sg/q, (τ∨)n ⊗ (∧2Sg/l)⊗X ⊗ τ) → HomL∩H(∧Sh/(l ∩ h),∧Sh/(l ∩ h)⊗ χ),

f 7→ ϕ ◦ f ◦ ιh.Fix an isomorphism X ∼= τ n ⊗ ∧Sg/q. Identify ∧Sh/(l ∩ h) with ∧Sg/q via ιh.

Then the linear map ϕ′ of (47) obviously induces an (L ∩ H)-equivariant linearmap

ϕ′′ : (∧2Sg/l)⊗ τ n → χ.

It is easy to see that the latter map in Lemma 3.3 is identical to the followinglinear map:

((τ∨)n ⊗ (∧2Sg/l)⊗ τ n)L ⊗ τ → χ,

λ⊗ ω ⊗ y ⊗ u 7→ ϕ′′(ω ⊗ y)∫Hχ−1(h) 〈λ, h.u〉 dh.

Here and as usual, a superscript group indicates the fixed vectors of the groupaction.

Note that ∧2Sg/l is a trivial L-module. Thus in order to finish the proof ofTheorem 3.1, it suffices to prove the following proposition.

Proposition 3.4. The linear map

HomL∩H(τn, χ) → HomH(((τ

∨)n ⊗ τ n)L ⊗ τ, χ),

φ 7→(λ⊗ y ⊗ u 7→ φ(y)

∫Hχ−1(h) 〈λ, h.u〉 dh

)

is surjective.

3.3. A proof of Proposition 3.4. Write τ(χ) for the χ-isotypic subspace of τ ,and write pχ : τ → τ(χ) for the H-equivariant projection map. Then the linearmap in Proposition 3.4 equals the following one:

(49)HomL∩H(τ

n, χ) → HomH(((τ∨)n ⊗ τ n)L ⊗ τ, χ)

φ 7→ (λ⊗ y ⊗ u 7→ φ(y)〈λ, pχ(u)〉) .Take a basis (λ1, λ2, · · · , λr) of (τ∨)n and (y1, y2, · · · , yr) of τ n which are dual to

each other. Then ((τ∨)n ⊗ τ n)L is spanned by∑r

i=1 λi ⊗ ui. Using this generatorwe identify ((τ∨)n ⊗ τ n)L with C so that

((τ∨)n ⊗ τ n)L ⊗ τ = τ.

Then the linear map (49) equals the composition of the linear map

(50)HomL∩H(τ

n, χ) → χ⊗ (τ∨)n,

φ 7→ ∑ri=1 φ(yi)⊗ λi

DISTINGUISHED REPRESENTATIONS 19

and the linear map

(51)χ⊗ (τ∨)n → HomH(τ, χ),

a⊗ λ 7→ (u 7→ a〈λ, pχ(u)〉) .Thus Proposition 3.4 follows from the following Lemma 3.5 and Lemma 3.6.

Lemma 3.5. The image of the linear map (50) equals (χ⊗ (τ∨)n)L∩H .

Proof. Note that (50) is a restriction of the obvious linear isomorphism

HomC(τn, χ)

∼=−→ χ⊗ (τ∨)n.

This linear isomorphism is H-equivariant, and hence the lemma follows. �

Lemma 3.6. The restriction of the linear map (51) to (χ⊗(τ∨)n)L∩H is surjective.

Proof. By the (L ∩ H)-invariance of the linear map (51), it suffices to show thatthe map (51) itself is surjective.

Note thatHomH(τ, χ) = HomC(τ(χ), χ) = χ⊗ (τ(χ))∨,

and (51) is identical to the following linear map:

(52)χ⊗ (τ∨)n → χ⊗ (τ(χ))∨,

a⊗ λ 7→ a⊗ λ|τ(χ).Using the first equality of (41), we have that

U(g).(τ(χ) ∩ (n.τ)) = U(q).(U(h).(τ(χ) ∩ (n.τ)))

= U(q).(τ(χ) ∩ (n.τ))

⊂ n.τ ( τ.

Thus the irreducibility of τ implies that τ(χ) ∩ (n.τ) = {0}. Consequently, thenatural map

τ(χ) → τ/(n.τ)

is injective. Taking the transpose, we know that the map

(τ∨)n → (τ(χ))∨,λ 7→ λ|τ(χ)

is surjective. Thus the map (52) is also surjective and the lemma is proved.�

4. The first application

In this section and the next one, we retain the notation of the Introduction. Asapplications of our construction, we give two arithmetically interesting examplesof cohomological test vectors. For the first one, suppose (G,H, θ, σ) and K are asin Example 1.3 so that

G = GLn(K), H = GLn(R), K = U(n) and Hc = O(n).

Write ι1, ι2 : K → C for the two distinct isomorphisms.

20 B. SUN

4.1. Cohomological test vectors. Fix a sequence

(53) µ = (µ1 ≥ µ2 ≥ · · · ≥ µn; µn+1 ≥ µn+2 ≥ · · · ≥ µ2n) ∈ Z2n

such that

(54) µ1 + µ2n = µ2 + µ2n−1 = · · · = µn + µn+1 = 0.

Write κ : K× → C× for the unitary character whose square equals(ι1)|K×

(ι2)|K×. Define

the principal series representation

(55) πµ := IndGLn(K)Bn(K)

(κ2µ1+n−1 ⊗ κ2µ2+n−3 ⊗ · · · ⊗ κ2µn+1−n

),

where Bn(K) denotes the Borel subgroup of the upper-triangular matrices. This isan irreducible Casselman-Wallach representation of GLn(K) ([Vog1], see also [Bar,Theorem 1.2]).

Denote by Fµ the irreducible algebraic representation of GLn(C) × GLn(C) ofhighest weight µ. It is also viewed as an irreducible representation of GLn(K) byrestricting through the complexification map

(56) GLn(K) → GLn(C)×GLn(C), g 7→ (ι1(g), ι2(g)).

By Vogan-Zuckerman theory of cohomological representations [VZ], πµ is theunique (up to isomorphism) irreducible Casselman-Wallach representation of GLn(K)which is unitarizable and tempered, and whose total relative Lie algebra cohomol-ogy is nonzero (see [Clo, Section 3]):

(57) H∗(gln(C)× gln(C),U(n);F∨µ ⊗ πµ) 6= 0.

Here gln(C)× gln(C) is identified with g through the differential of (56).Recall that a result of Vogan [Vog1, Theorem 4.9] asserts that every irreducible

Casselman-Wallach representation of GLn(K) has a unique minimal U(n)-type,and it occurs with multiplicity one. Likewise, every irreducible Casselman-Wallachrepresentation of GLn(R) has a unique minimal O(n)-type, and it occurs withmultiplicity one.

Whenever a Lie group has exactly two connected components, we use sgn todenote the unique non-trivial quadratic character of it. We will prove the followingTheorem in the next subsection.

Theorem 4.1. The space HomGLn(R)(πµ, sgnn−1) is one-dimensional, and a nonzero

element of it does not vanish on the minimal U(n)-type of πµ.

Remarks. (a) Using the explicit description of πµ in (55), the analytic method ofmeromorphic continuations also produces a nonzero element of HomGLn(R)(πµ, sgn

n−1)(cf. [Osh, Theorem 6], [Ola, Theorem 5.1] and [Ban, Theorem 5.10]). But it seemshard to show the non-vanishing on the minimal U(n)-type by this analytic method.

(b) When n ≥ 2, some matrix coefficients of πµ are not integrable on SLn(R).Thus the method of [Sun, Lemma 4.5], which proves a similar result in the Gross-Prasad case by matrix coefficient integrals, does not apply to Theorem 4.1.

DISTINGUISHED REPRESENTATIONS 21

4.2. A proof of Theorem 4.1. Note that g = gln(C)× gln(C) and the complex-ified differential of θ : G→ G equals

θ : gln(C)× gln(C) → gln(C)× gln(C), (x, y) 7→ (−yt,−xt).Suppose

q = bn × btn ⊂ g = gln(C)× gln(C),

where bn denotes the Lie algebra of all upper-triangular matrices in gln(C). Thenq is a θ-stable parabolic subalgebra of g satisfying (5), and

(L, Lc, L ∩H, Lc ∩Hc) = ((K×)n, (U(1))n, (R×)n, {±1}n).Using the isomorphisms ι1, ι2 : K → C, we also have an identification

l = Cn × Cn.

Denote by λµ the unitary character of L whose complexified differential equals

(µ1+n−1, µ2+n−3, · · · , µn+1−n;µ2n+1−n, µ2n−1+3−n, · · · , µn+1+n−1).

Then by Vogan-Zuckerman theory [VZ], Πg,Kq,Lc

(λµ) is isomorphic to the (g, K)-

module ofK-finite vectors in πµ, and the irreducible representation τµ := Πk,Kqc,Lc

(λµ)of K occurs with multiplicity one in πµ (it is the unique minimal K-type of πµ).Identify K with C via ι1, then τµ has highest weight

(2µ1 + n− 1, 2µ2 + n− 3, · · · , 2µn + 1− n).

Lemma 4.2. One has that

dimHomHc(τµ, sgnn−1) = 1.

Proof. This is an instance of Cartan-Helgason Theorem (cf. [Hel, Chapter V, The-orem 4.1]). �

Lemma 4.3. There is an element of Homh,Hc(Πg,Kq,Lc

(λµ), sgnn−1) which does not

vanish on the minimal K-type τµ of Πg,Kq,Lc

(λµ).

Proof. Note that the one-dimensional representations

(λµ)|L∩H and χL∩H∼= sgnn−1|L∩H ⊗ ∧top(hc/(lc ∩ hc))

of L ∩H are both trivial. Thus Lemma 4.2 implies that

dimHomLc∩Hc(λµ, χL∩H) = dimHomHc(τµ, sgnn−1).

SinceHoml∩h,Lc∩Hc(λµ, χL∩H) 6= {0},

the lemma follows by Theorem 1.10. �

Lemma 4.4. For every irreducible Casselman-Wallach representation π of GLn(K),and every character χ0 of GLn(R), one has that

dimHomGLn(R)(π, χ0) ≤ 1.

22 B. SUN

Proof. When χ0 is trivial, the lemma is proved in [AGS, Theorem 8.2.5]. In general,since χ0 extends to a character of GLn(K), the lemma reduces to the case whenχ0 is trivial. �

Now Theorem 4.1 follows by combining Theorem 1.1, Lemma 4.3 and Lemma4.4.

4.3. The non-vanishing hypothesis for Asai L-functions. Write

GLn(K) = GL1n(K)× R×

+,

where GL1n(K) := {g ∈ GLn(K) | ι1(det g) · ι2(det g) = 1}, and the group R×

+

of positive real numbers is viewed as a subgroup of GLn(K) via the diagonalembedding. Following [GHL], writemG for the complexified Lie algebra of GL1

n(K).

Put tn := (n−1)(n+2)2

. Recall that [Clo, Lemma 3.14]

dimHtn(mG,U(n);F∨µ ⊗ πµ) = 1,

and

Hj(mG,U(n);F∨µ ⊗ πµ) = {0} for all j > tn.

Note that

dimHomGLn(R)(F∨µ ,C) = 1 and dimHtn(sln(C),O(n); sgnn−1) = 1.

Theorem 4.5. Let ϕ be a nonzero element of HomGLn(R)(πµ, sgnn−1), and let ψ be

a nonzero element of HomGLn(R)(F∨µ ,C). Then by restriction of cohomology, the

linear functional ψ ⊗ ϕ : F∨µ ⊗ πµ → sgnn−1 induces a nonzero linear map

Htn(mG,U(n);F∨µ ⊗ πµ) → Htn(sln(C),O(n); sgnn−1)

of one-dimensional vector spaces.

Theorem 4.5 is a representation theoretic reformulation of the non-vanishinghypothesis of Grobner-Harris-Lapid in the study of non-critical values of the AsaiL-function (see [GHL, Section 6.2]).

In view of Theorem 4.1, Theorem 4.5 follows by applying Theorem A.3 of Ap-pendix A to the group GL1

n(K).

5. The second application

For the second application, suppose (G,H, θ, σ) is as in Example 1.4 so that

G = GL2n(R), H = GLn(R)×GLn(R), K = O(2n) and Hc = O(n)×O(n).

DISTINGUISHED REPRESENTATIONS 23

5.1. Cohomological test vectors. Fix a sequence

ν = (ν1 ≥ ν2 ≥ · · · ≥ ν2n) ∈ Z2n (n ≥ 1)

such that

(58) ν1 + ν2n = ν2 + ν2n−1 = · · · = νn + νn+1 = w

for some w ∈ Z. Define

πν := |det|w2 ⊗ IndGP2,2,··· ,2

(Dν1−ν2n+2n−1⊗Dν2−ν2n−1+2n−3⊗ · · · ⊗Dνn−νn+1+1

),

where Dk (k ≥ 1) is a relative discrete series representation as in Example 1.11.Similar to (55), πν is an irreducible Casselman-Wallach representation of GL2n(R).

Denote by F ν the irreducible algebraic representation of GL2n(C) of highestweight ν. It is also viewed as an irreducible representation of GL2n(R) by re-striction. Similar to Section 4.1, πν is the unique (up to isomorphism) irreducibleCasselman-Wallach representation of GL2n(R) such that [Clo, Section 3]

• πν |SL±

2n(R)is unitarizable and tempered, and

• the total relative Lie algebra cohomology

(59) H∗(gl2n(C), SO(2n); (F ν)∨ ⊗ πν) 6= 0,

whereSL±

2n(R) := {g ∈ GL2n(R) | det(g) = ±1}.Suppose

(60) χ = χ1 ⊗ χ2 : GLn(R)×GLn(R) → C×

is a character such thatχ1 · χ2 = detw.

For each s ∈ C, let |det|s,−s denotes the character |det|s ⊗ |det|−s of GLn(R) ×GLn(R). The following Theorem will be proved in the next subsection.

Theorem 5.1. Up to scalar multiplication, there exists a unique nonzero elementϕ ∈ HomGLn(R)×GLn(R)(π

ν , χ) which extends to a holomorphic family in the follow-ing sense: there exists a map

ζ : πν × C → C

such that

• ζ( · , s) ∈ HomGLn(R)×GLn(R)(πν , χ · |det|s,−s), for all s ∈ C;

• ζ(v, · ) is an entire function, for all O(2n)-finite vector v ∈ πν;• ζ( · , 0) = ϕ.

Moreover, ϕ does not vanish on the minimal O(2n)-type of πν.

Similar to Theorem 4.1, a nonzero χ-invariant continuous linear functional onπν may also be constructed by the analytic method of meromorphic continuations.But the non-vanishing on the minimal O(2n)-type is not easy to show by thatmethod.

24 B. SUN

5.2. A proof of Theorem 5.1. Fix an embedding

(61) γ2n : (C×)2n → GL2n(C)

of algebraic groups which sends (a1, a2, · · · , a2n) to the matrix

a1+a2n2

0 · · · 0 0 · · · 0 a1−a2n2i

0 a2+a2n−1

2· · · 0 0 · · · a2−a2n−1

2i0

· · · · · · · · · · · · · · · · · · · · · · · ·0 0 · · · an+an+1

2an−an+1

2i· · · 0 0

0 0 · · · an+1−an2i

an+1+an2

· · · 0 0· · · · · · · · · · · · · · · · · · · · · · · ·0 a2n−1−a2

2i· · · 0 0 · · · a2n−1+a2

20

a2n−a12i

0 · · · 0 0 · · · a2n+a12

,

where i =√−1 ∈ C is the fixed square root of −1. View (C×)2n as a Cartan

subgroup of GL2n(C) via the embedding (61). Then the corresponding root systemis

(62) {±(ei − ej) | 1 ≤ i < j ≤ 2n} ⊂ Z2n.

Here e1, e2, · · · , e2n denote the standard basis of Z2n. Suppose q is the Borelsubalgebra of g which corresponds to the positive system

(63) {ei − ej | 1 ≤ i < j ≤ 2n} ⊂ Z2n

of (62). Then q is a θ-stable parabolic subalgebra of g satisfying (5), and

(L, Lc, L ∩H, Lc ∩Hc) = ((C×)n, (S1)n, (R×)n, {±1}n),where S1 denotes the group of complex numbers of modulus one. Here L is viewedas a subgroup of G via the embedding

(C×)n → GL2n(R),(a1, a2, · · · , an) 7→ γ2n(a1, a2, · · · , an, an, · · · , a2, a1).

Denote by λν the restriction to L of the character

(ν1 + 2n− 1, ν2 + 2n− 3, · · · , ν2n + 1− 2n)

of (C×)2n, through the embedding

L = (C×)n → (C×)2n,(a1, a2, · · · , an) 7→ (a1, a2, · · · , an, an, · · · , a2, a1).

Then by Vogan-Zuckerman theory [VZ], we know that Πg,Kq,Lc

(λν) is isomorphic tothe (g, K)-module of K-finite vectors in πν . Moreover, the irreducible K◦-module

τ ν := Πk,K◦

qc,Lc(λν) occurs with multiplicity one in πν (it is contained in the unique

minimal K-type of πν).

DISTINGUISHED REPRESENTATIONS 25

Note that Lc is a Cartan subgroup of K◦, and qc is a Borel subalgebra of k whichcorresponds to the positive system

{ei ± ej | 1 ≤ i < j ≤ n} ⊂ Zn = Hom(Lc,C×)

of the root system of K◦. The highest weight of τ ν is

(ν1 − ν2n + 2n, ν2 − ν2n−1 + 2(n− 1), · · · , νn − νn+1 + 2).

Note that H ∩K◦ equals

O(n)×{±1} O(n) (the fiber product over the determinant homomorphisms).

Thus it has exactly two connected components. Recall the integer w from (58).

Lemma 5.2. One has that

dimHomH∩K◦(τ ν , sgnw) = 1.

Proof. This is also an instance of Cartan-Helgason Theorem (cf. [Hel, Chapter V,Theorem 4.1]). �

Lemma 5.3. There is a map

(64) ζ0 : Πg,Kq,Lc

(λν)× C → C

with the following properties:

• ζ0( · , s) ∈ Homh,Hc(Πg,Kq,Lc

(λν), χ · |det|s,−s), for all s ∈ C;

• ζ0(v, · ) is a polynomial function, for all v ∈ Πg,Kq,Lc

(λν);

• ζ0( · , s) does not vanish on τ ν ⊂ Πg,Kq,Lc

(λν), for all s ∈ C.

Proof. Note that the (Lc∩Hc)-action on ∧Shc/(lc∩hc) is trivial, and the characterλν |L∩H and (χ · |det|s,−s)|L∩H (s ∈ C) are both equals to the following one:

(a1, a2, · · · , an) 7→ (a1a2 · · · · · an)w.Fix a nonzero element

ϕ ∈ HomL∩H(λν ,∧2S(k/lc)

∨ ⊗ ∧Shc/(lc ∩ hc)⊗ χ).

For all s ∈ C, let

ϕs ∈ HomL∩H(λν ,∧2S(k/lc)

∨ ⊗ ∧Shc/(lc ∩ hc)⊗ (χ · |det|s,−s))

be the element which is identical to ϕ when both χ and χ · |det|s,−s are identifiedwith C as vector spaces. Now we define a map

ζ0 : Πg,Kq,Lc

(λν)× C → C, (v, s) 7→ (Πg,Kq,Lc

(ϕs))(v).

Then the first two properties of the lemma hold by the construction of Πg,Kq,Lc

(ϕs).The third property holds by Proposition 1.9, Lemma 5.2, and Theorem 1.7. �

26 B. SUN

Identify πν with the Casselman-Wallach globalization of Πg,Kq,Lc

(λν). By the au-tomatic continuity theorem (Theorem 1.1), the map ζ0 of (64) extends to a map

ζ : πν × C → C

such that

• ζ( · , s) ∈ HomGLn(R)×GLn(R)(πν , χ · |det|s,−s), for all s ∈ C;

• ζ(v, · ) is a polynomial function, for all O(2n)-finite vectors v ∈ πν ;

• ζ( · , s) does not vanish on τ ν ⊂ Πg,Kq,Lc

(λν), for all s ∈ C.

Then ϕ := ζ( · , 0) satisfies the conditions of Theorem 5.1. Moreover, ϕ does notvanish on the minimal K-type of πν .

To prove the uniqueness of ϕ, recall the following multiplicity one result.

Lemma 5.4. ([CS, Theorem B]) Let π be an irreducible Casselman-Wallach repre-sentation of GL2n(R). Then for all but countably many characters χ′ of GLn(R)×GLn(R), the space HomGLn(R)×GLn(R)(π, χ

′) is at most one-dimensional.

Now let ζ ′ : πν × C → C be a map such that

• ζ ′( · , s) ∈ HomGLn(R)×GLn(R)(πν , χ · |det|s,−s), for all s ∈ C;

• ζ ′(v, · ) is an entire function, for all O(2n)-finite vectors v ∈ πν .

Pick a vector v0 ∈ τ ν ⊂ πν which does not vanish under a nonzero element ofHomH∩K◦(τ ν , sgnw). Then ζ(v0, ·) is a nowhere vanishing polynomial function,and is thus a nonzero constant. Put

γ(s) :=ζ ′(v0, s)

ζ(v0, s), s ∈ C.

Then Lemma 5.4 implies that

(65) ζ ′(·, s) = γ(s)ζ(·, s),

for all but countably many s ∈ C. Therefore for all v ∈ πν which is K-finite, thecontinuity of the both sides of (65) on the variable s ∈ C implies that

(66) ζ ′(v, s) = γ(s)ζ(v, s) for all s ∈ C.

Finally, the continuity of the both sides of the equality (66) on the variable v ∈ πν

implies that

(67) ζ ′(·, s) = γ(s)ζ(·, s) for all s ∈ C.

In particular, ζ ′(·, 0) is a scalar multiple of ϕ = ζ(·, 0). This proves the uniquenessof ϕ, and finishes the proof of Theorem 5.1.

DISTINGUISHED REPRESENTATIONS 27

5.3. The non-vanishing hypothesis for L-functions for GSpin(2n+ 1). Putt′n := n2 + n− 1. Then [Clo, Lemma 3.14]

Hν := Ht′n(sl2n(C), SO(2n); (F ν)∨ ⊗ πν) 6= 0,

and

Hj(sl2n(C), SO(2n); (F ν)∨ ⊗ πν) = 0 for all j > t′n.

The natural actions of the group O(2n) on sl2n(C), SO(2n) and (F ν)∨⊗πν induce arepresentation of O(2n)/SO(2n) = {±1} on Hν . It turns out that [Mah, Equation(3.2)]

(68) Hν ∼= C⊕ sgn (as representations of the group {±1}),where “C ” stands for the trivial representation.

Recall from (60) the character χ = χ1⊗χ2 with χ1 ·χ2 = detw. Now we assumethat

(69) (χi)|(GLn(R))◦ = (detwi)|(GLn(R))◦ , i = 1, 2,

for two integers w1, w2 such that

(70) w1 + w2 = w and νn ≥ wi ≥ νn+1 (i = 1, 2).

Then the space

Hχ := Ht′n(sl2n(C) ∩ (gln(C)× gln(C)),O(n)×{±1} O(n); det−w1,−w2 · χ)is one-dimensional, and naturally carries a representation of

(O(n)×O(n))/(O(n)×{±1} O(n)) = {±1}.Here det−w1,−w2 denotes the character det−w1⊗det−w2 of GLn(C)×GLn(C). Using(68), we conclude that

dimHom{±1}(Hν ,Hχ) = 1.

As an instance of H. Schlichtkrull’s generalization of Cartan-Helgason Theorem([Sch2, Theorem 7.2], see also [Kna, Theorem 2.1]), (70) implies that

dimHomGLn(C)×GLn(C)((Fν)∨, det−w1,−w2) = 1.

Theorem 5.5. Assume that (69) and (70) hold. Let ϕ be as in Theorem 5.1. Let ψbe a nonzero element of HomGLn(C)×GLn(C)((F

ν)∨, det−w1,−w2). Then by restrictionof cohomology, the linear functional ψ ⊗ ϕ : (F ν)∨ ⊗ πν → det−w1,−w2 · χ inducesa nonzero element of Hom{±1}(H

ν ,Hχ).

Theorem 5.5 is a representation theoretic reformulation of the non-vanishinghypothesis in the study of p-adic L-functions and critical values of L-functions forGSpin(2n + 1), using the Langlands lift to GL(2n) and Shalika models (see [AG,assumption (A2)], [GR, Section 6.6] and [AsS]).

28 B. SUN

Now we come to the proof of Theorem 5.5. Denote by πν◦ the Casselman-Wallach

globalization of Πg,K◦

q,Lc(λν). It is an irreducible representation of (GL2n(R))

◦ withminimal K◦-type τ ν . Moreover, we have a natural inclusion πν

◦ ⊂ πν , and

Hν◦ := Ht′n(sl2n(C), SO(2n); (F ν)∨ ⊗ πν

◦ )

is a one-dimensional subspace of the two-dimensional space

Hν := Ht′n(sl2n(C), SO(2n); (F ν)∨ ⊗ πν).

Denote by ϕ◦ the restriction of ϕ to πν◦ , which does not vanish on the minimal

K◦-type τ ν . Let ψ : (F ν)∨ → det−w1,−w2 be as in Theorem 5.5. By restriction ofcohomology, the linear functionals

ψ ⊗ ϕ◦ : (Fν)∨ ⊗ πν

◦ → det−w1,−w2 · χ and ψ ⊗ ϕ : (F ν)∨ ⊗ πν → det−w1,−w2 · χrespectively induce linear functionals

η◦ : Hν◦ → Hχ and η : Hν → Hχ .

Applying Theorem A.3 of Appendix A to the group SL2n(R), we know that η◦is nonzero. Therefore η is nonzero since it extends η◦. This finishes the proof ofTheorem 5.5.

Appendix A. Modular symbols at infinity

As mentioned in the Introduction, the Archimedean behaviors of modular sym-bols are captured by certain restriction maps of relative Lie algebra cohomologies.We call these restriction maps modular symbols at infinity. In this appendix, weshow (in Theorem A.3) that the non-vanishing on the bottom layers implies thenon-vanishing of certain modular symbols at infinity.

A.1. Unitary representations with nonzero cohomology. We first reviewsome basic facts concerning unitary representations with nonzero cohomology. LetG be a real reductive group (as in Section 1.1). For simplicity, assume it is con-nected and its center is compact. Fix a Cartan involution θ of G. Put K := Gθ,which is a maximal compact subgroup of G.

As usual, we use the corresponding lower case Gothic letter to indicate thecomplexified Lie algebra of a Lie group. Let F be an irreducible finite-dimensionalrepresentation of G, and let V be a unitarizable irreducible (g, K)-module suchthat the total relative Lie algebra cohomology

H∗(g, K;F∨ ⊗ V ) 6= 0.

Then by [VZ], there is a θ-stable parabolic subalgebra q of g with the followingproperties:

• the representation F n|L is one-dimensional and unitarizable;• V ∼= ΠS(F

n ⊗ ∧dim nn).

DISTINGUISHED REPRESENTATIONS 29

Here

n := the nilpotent radical of q ∩ [g, g],L := NG(q) = NG(q),S := dim(n ∩ k),

and ΠS denotes the S-th left derived functor of the functor

R(g, K)⊗R(q,Lc) ( · ) (Lc := L ∩K)

from the category of (q, Lc)-modules to the category of (g, K)-modules.Put

qc := q ∩ k, nc := n ∩ k,

and define two vector spaces

qn := q/qc, nn := n/nc.

Note that L and Lc are connected [KV, Lemma 5.10]. We introduce three irre-ducible representations τV , τF and τn of K such that

(71) (τV )nc ∼= F n|Lc ⊗ ∧dim nnnn, (τF )

nc ∼= F n|Lc, and (τn)nc ∼= ∧dim nnnn

as representations of Lc. Then τV occurs with multiplicity one in V (this is thebottom layer of the cohomological induction, and is the unique minimal K-typeof V , in the sense of Vogan); τF occurs with multiplicity one in F ; and τn occurswith multiplicity one in both ∧dim nng/k and ∧dim qng/k.

Lemma A.1. One has that

dimHomK(τn, τ∨F ⊗ τV ) = 1.

Proof. Note that τV is the Cartan product of τF and τn. (For details on Cartanproducts, see [Ea] for example.) Hence

dimHomK(τn, τ∨F ⊗ τV ) = dimHomK(τn ⊗ τF , τV ) = 1.

�

Theorem A.2. (a) For all j ∈ Z, Hj(g, K;F∨ ⊗ V ) = HomK(∧jg/k, F∨ ⊗ V ).(b) The space Hj(g, K;F∨ ⊗ V ) is zero unless dim nn ≤ j ≤ dim qn, and both

Hdim nn(g, K;F∨ ⊗ V ) and Hdim qn(g, K;F∨ ⊗ V ) are one-dimensional.

Proof. Part (a) is proved in [Wa1, Proposition 9.4.3], and part (b) is implied by[Wa1, Theorem 9.6.6].

�

Theorem A.2 and Lemma A.1 imply that we have identifications

(72) Hj(g, K;F∨ ⊗ V ) = HomK(τn, τ∨F ⊗ τV ), for j = dim nn or dim qn.

30 B. SUN

A.2. Non-vanishing of modular symbols at infinity. Let H be a θ-stableclosed subgroup of G with finitely many connected components. Put Hc := H∩K.Let χF and χV be two characters of H such that

(73) (χF · χV )|(h,Hc)∼= ∧dim h/hch/hc,

where ∧dim h/hch/hc carries the trivial representation of h and the adjoint represen-tation of Hc. Suppose we have two nonzero elements

λF ∈ HomH(F∨, χF ) and λV ∈ Homh,Hc(V, χV ).

By restriction of cohomology, the functional λF ⊗λV : F∨⊗V → χF ⊗χV inducesa linear map

(74) Hdim h/hc(g, K;F∨ ⊗ V ) → Hdim h/hc(h, Hc;χF ⊗ χV ) ∼= C.

The functional (74) reflects the Archimedean behaviors of various types of modularsymbols which are used in the arithmetic study of special values of L-functions. Wecall it a modular symbol at infinity. In the literature, authors concentrate on thecases when dim h/hc = dim nn or dim qn. See [AG, Har, GHL, KMS] for examples.The modular symbol is interesting only when the functional (74) is nonzero.

Recall the parabolic subalgebra q from Section A.1.

Theorem A.3. Assume that h + q = g and λV does not vanish on the K-subrepresentation τV of V . If either

(75) dim h/hc = dim nn and h ∩ q ⊂ k,

or

(76) dim h/hc = dim qn and h ∩ n ⊂ k,

then the linear functional (74) is nonzero.

We remark that the condition h ∩ q = h ∩ q implies that h ∩ n = 0 and henceh ∩ n ⊂ k. The condition (75) holds, for example, in the case of Rankin-Selbergconvolutions for GL(n)×GL(n− 1) (see [Sun]).

The rest of this appendix is devoted to a proof of Theorem A.3.

Lemma A.4. If (75) or (76) holds, then every nonzero element of the one-dimensional space HomK(∧dim h/hcg/k, τn) does not vanish on the one-dimensionalsubspace ∧dim h/hch/hc of ∧dim h/hcg/k.

Proof. We give a proof under the assumption that (75) holds, which is similar tothat of [Sun, Lemma 2.10]. The same proof works when (76) holds. Fix a K-invariant Hermitian inner product 〈 , 〉 on g/k. It induces a K-invariant Hermitianinner product 〈 , 〉∧ on ∧dim h/hcg/k.

View τn as aK-submodule of ∧dim h/hcg/k. It is generated by the one-dimensionalspace ∧dim h/hcn/nc. Note that every nonzero element of

HomK(∧dim h/hcg/k, τn)

DISTINGUISHED REPRESENTATIONS 31

is a scalar multiple of the orthogonal projection ∧dim h/hcg/k → τn. Therefore inorder the prove the lemma, it suffices to show that the one-dimensional spaces∧dim h/hch/hc and ∧dim h/hcn/nc are not perpendicular to each other under the form〈 , 〉∧. This is equivalent to saying that the pairing

〈 , 〉 : h/hc × n/nc → C

is non-degenerate. Note that

{x ∈ g/k | 〈x, n/nc〉 = 0} = q/qc,

and by (75), h/hc ∩ q/qc = {0}. This proves the lemma. �

Lemma A.5. Let u be a nonzero element of τn and v a nonzero element of τF .Then every nonzero element of HomK(τn ⊗ τF , τV ) does not vanish on u⊗ v.

Proof. The lemma holds since τV is the Cartan product of τn and τF (cf. [Ya,Section 2.1]). �

Lemma A.6. If h+ q = g, then

(77) dimHomHc(τV , χV ) ≤ 1.

Proof. Recall from (71) that dim τ ncV = 1. Note that h+ q = g implies hc + qc = k.Then we have that

(78) τV = U(k).τ ncV = U(hc).(U(qc).τncV ) = U(hc).τ

ncV

Therefore every element of HomHc(τV , χV ) is determined by its restriction to theone-dimensional space τ ncV . Hence (77) holds. �

Lemma A.7. Assume that h+q = g and λV does not vanish on τV . Then (χV )|Hc

occurs with multiplicity one in (τV )|Hc, and λV does not vanish on (χV )|Hc ⊂(τV )|Hc.

Proof. This is obviously implied by Lemma A.6. �

Lemma A.8. View τ∨F as a K-submodule of F∨. If h + q = g, then λF does notvanish on τ∨F .

Proof. Similar to (78), we have that

F∨ = U(h).(F∨)n.

Since τ∨F is generated by (F∨)n, the lemma follows. �

We are now ready to prove Theorem A.3. By (72), the map (74) is identifiedwith the linear map

(79) HomK(τn, τ∨F ⊗ τV ) → HomC(∧dim h/hch/hc, χF ⊗ χV )

which is induce by the linear functional

(λF )|τ∨F ⊗ (λV )|τV : τ∨F ⊗ τV → χF ⊗ χV

32 B. SUN

and the restriction to ∧dim h/hch/hc of the K-equivaraint projection map

pn : ∧dim h/hcg/k → τn.

The map (79) is identified with the obvious linear map

(80) HomK(τn ⊗ τF , τV ) → HomC(∧dim h/hch/hc ⊗ χ∨F , χV ).

The non-vanishing of the map (80) is equivalent to saying that the following com-position map is nonzero:

(81) ∧dim h/hch/hc ⊗ χ∨F → τn ⊗ τF → τV

(λV )|τV−→ χV .

Here the first arrow is the tensor product of (pn)|∧dimh/hch/hc and the transpose of(λF )|τ∨F , and the second arrow is a nonzero K-equivariant linear map. The firstarrow is nonzero by Lemma A.4 and Lemma A.8. By Lemma A.5, the compositionof the first two arrows of (81) is nonzero, and hence by (73), its image is a one-dimensional submodule of (τV )|Hc which is isomorphic to (χV )|Hc. Finally, byLemma A.7, the composition of (81) is nonzero. This finishes the proof of TheoremA.3.

References

[AGS] A. Aizenbud, D. Gourevitch and E. Sayag, Generalized Harish-Chandra descent, Gelfand

pairs, and an Archimedean analog of Jacquet-Rallis’s theorem, Duke Math. J. 149 (2009),509-567.

[AOS] A. Aizenbud, O. Offen and E. Sayag, Disjoint pairs for GL(n,R) and GL(n,C), C. R.Math. Acad. Sci. Paris 350 (2012), 9-11.

[AiS] A. Aizenbud and E. Sayag, Invariant distributions on non-distinguished nilpotent orbits

with application to the Gelfand property of (GL(2n,R), Sp(2n,R)), J. Lie Theory 22 (2012)137-153.

[AsS] M. Asgari and F. Shahidi, Generic transfer for general spin groups, Duke Math. J. 132(2006), 137-190.

[AG] A. Ash and D. Ginzburg, p-adic L-functions for GL(2n), Invent. Math. 116 (1994), 27-73.[Ban] E.P. van den Ban, The principal series for a reductive symmetric space. I. H-fixed distri-

bution vectors, Ann. Sci. ecole Norm. Sup. (4) 21 (1988), no. 3, 359-412.[BaD] E.P. van den Ban and P. Delorme, Quelques proprietes des representations spheriques

pour les espaces symetriques reductifs, J. Funct. Anal. 80(1988), 284-307.[Bar] E. Baruch, A proof of Kirillov’s conjecture, Ann. of Math. (2) 158 (2003), no. 1, 207-252.[BK] J. Bernstein and B. Krotz, Smooth Frechet globalizations of Harish-Chandra modules, Israel

Journal of Mathematics, 199 (2014), 45-111.[BW] A. Borel and N. Wallach, Continuous cohomology, discrete subgroups, and representations

of reductive groups, Second edition. Mathematical Surveys and Monographs, 67. AmericanMathematical Society, Providence, RI, 2000.

[BrD] J.-L. Brylinski and P. Delorme, Vecteurs distributions H-invariants pour les series princi-

pales generalisees d’espaces symetriques reductifs et prolongement meromorphe d’integrales

d’Eisenstein, Invent. Math. 109 (1992), no. 3, 619-664.[Cas] W. Casselman, Canonical extensions of Harish-Chandra moudles to representations of G,

Can. Jour. Math. 41, (1989), 385-438.

DISTINGUISHED REPRESENTATIONS 33

[CS] F. Chen and B. Sun, Uniqueness of twisted linear periods and twisted Shalika periods,arXiv:1703.06238

[Clo] L. Clozel, Motifs et formes automorphes: applications du principe de fonctorialite.(French)[Motives and automorphic forms: applications of the functoriality principle] Automorphicforms, Shimura varieties, and L-functions, Vol. I (Ann Arbor, MI, 1988), 77-159, Perspect.Math., 10, Academic Press, Boston, MA, 1990.

[DV] M. Duflo and M. Vergne, Sur les functeurs de Zuckerman, C. R. Acad. Sci. Paris 304(1987), 467-469.

[Ea] M. Eastwood, The Cartan product, Bull. Belg. Math. Soc. Simon Stevin, Volume 11, Number5 (2005), 641-651.

[FJ] M. Flensted-Jensen, Discrete series for semisimple symmetric spaces, Ann. of Math. (2)111 (1980), no. 2, 253-311.

[GOSS] D. Goureviteh, O. Offen, S. Sahi and E. Sayag, Existence of Klyachko models for

GL(n,R) and GL(n,C), J. Funct. Analysis 262 (2012), no. 8, 3465-3664.[GSS] D. Gourevitch, S. Sahi and E. Sayag, Invariant functionals on Speh representations, Trans-

formation groups 20 (2015), no. 4, 1023-1042.[GHL] H. Grobner, M. Harris and E. Lapid, Whittaker regulators and non-critical values of the

Asai L-function, Contemporary Mathematics, 664 (2016) 119-134.[GR] H. Grobner, A. Raghuram (with an appendix by W.T. Gan), On the arithmetic of Shalika

models and the critical values of L-functions for GL(2n), Amer. J. Math. 136 (2014), 675-728[Har] M. Harris, Occult period invariants and critical values of the degree four L-function of

GSp(4), in: Contributions to automorphic forms, geometry, and number theory, eds. H.Hida, D. Ramakrishnan, F. Shahidi, Johns Hopkins Univ. Press, (Baltimore, MD, 2004),331-354.

[Hel] S. Helgason, Groups and Geometric Analysis, Amer. Math. Soc., Providence, R.I. 2000.[KMS] D. Kazhdan, B. Mazur and C.-G. Schmidt, Relative modular symbols and Rankin-Selberg

convolutions, J. reine angew. Math. 519 (2000), 97-141.[Kna] A. Knapp, Branching theorems for compact symmetric spaces, Represent. Theory 5 (2001),

404-436.[KV] A. Knapp and D. Vogan, Cohomological induction and unitary Representations, 1995,

Princeton University Press, Princeton, NJ.[Mah] J. Mahnkopf, Cohomology of arithmetic groups, parabolic subgroups and the special values

of L-functions of GLn, J. Inst. Math. Jussieu., 4(4), 553-637 (2005).

[Ola] G. Olafsson, Fourier and Poisson transformation associated to a semisimple symmetric

space, Invent. Math. 90 (1987), no. 3, 605-629.[Osh] T. Oshima, Poisson transformations on affine symmetric spaces, Proc. Japan Acad. Ser.

A Math. Sci. 55 (1979), no. 9, 323-327.[OM] T. Oshima and T. Matsuki, A description of discrete series for semisimple symmetric

spaces, Group representations and systems of differential equations (Tokyo, 1982), 331-390,Adv. Stud. Pure Math., 4, North-Holland, Amsterdam, 1984.

[Sch1] H. Schlichtkrull, A series of unitary irreducible representations induced from a symmetric

subgroup of a semisimple Lie group, Invent. Math. 68 (1982), no. 3, 497-516.[Sch2] H. Schlichtkrull, One-dimensional K-types in finite dimensional representations of

semisimple Lie groups: A generalization of Helgason’s theorem, Math. Scand. 54 (1994),279-294.

[Sp] B. Speh, Unitary representations of GL(n,R)with nontrivial (g,K)-cohomology, Invent.Math. 71 (1983), 443-465.

[Sun] B. Sun, The nonvanishing hypothesis at infinity for Rankin-Selberg convolutions, J. Amer.Math. Soc. 30 (2017), 1-25.

34 B. SUN

[Vog1] D. Vogan, The unitary dual of GL(n) over an archimedian field, Invent. Math., 83 (1986),pp. 449-505.

[Vog2] D. Vogan, Irreducibility of discrete series representations for semisimple symmetric

spaces, Representations of Lie groups, Kyoto, Hiroshima, 1986, 191-221, Adv. Stud. PureMath., 14, Academic Press, Boston, MA, 1988.

[VZ] D. Vogan and G. Zuckerman, Unitary representations with nonzero cohomology, CompositioMath. 53 (1984)), 51-90.

[Wa1] N. Wallach, Real Reductive Groups I, Academic Press, San Diego, 1988.[Wa2] N. Wallach, Real Reductive Groups II, Academic Press, San Diego, 1992.[Ya] O. Yacobi, An analysis of the multiplicity spaces in branching of symplectic groups, Selecta

Math N.S., 16 (2010), 819-855.

HCMS, HLM, CEMS, Academy of Mathematics and Systems Science, Chinese

Academy of Sciences & School of Mathematical Sciences, University of Chinese

Academy of Sciences, Beijing 100190, China

E-mail address : sun@math.ac.cn