atmp016

c 2005 International Press Adv. Theor. Math. Phys. 9 (2005) 559 574 Open-string BRST cohomology for generalized complex branes Anton Kapustin, Yi Li California Institute of Technology, Pasadena, CA 91125, USA E-mail address: kapustin@theory.caltech.edu; yili@theory.caltech.edu Abstract It has been shown recently that the geometry of D-branes in general topologically twisted (2, 2) sigma-models can be described in the language of generalized complex (GC) structures. On general grounds, such D-branes (called GC branes) must form a category. We compute the BRST cohomology of open strings with both ends on the same GC brane. In mathematical terms, we determine spaces of endomorphisms in the category of GC branes. We find that the BRST cohomology can be expressed as the cohomology of a Lie algebroid canonically associated to any GC brane. In the special case of B-branes, this leads to an apparently new way to compute Ext groups of holomorphic line bundles supported on complex submanifolds: while the usual method leads to a spectral sequence converging to the Ext, our approach expresses the Ext group as the cohomology of a certain differential acting on the space of smooth sections of a graded vector bundle on the submanifold. In the case of coisotropic A-branes, our computation confirms a proposal of Orlov and one of the authors (A.K.). e-print archive: http://lanl.arxiv.org/abs/hep-th/0501071

560 ANTON KAPUSTIN AND YI LI 1 Introduction and summary More than a decade ago, Witten [10] explained how to manufacture 2d topological field theories from sigma-models whose target is a Kähler manifold. He showed that any sigma-model with a Kähler target space X admits a topologically twisted version called the A-model; if X is a Calabi Yau manifold, there is another topologically twisted theory, the B-model. Recently it was realized that A and B topological twists can be applied, under certain conditions, to more general sigma-models with (2, 2) supersymmetry. In these models, the target space is not Kähler, in general, and the H-flux is nonzero. We will call these more general TFTs the generalized A and B-models. It was shown in [5, 6] that the geometry of these TFTs is conveniently described in terms of generalized complex (GC) geometry [3, 4]. Namely, to any (2, 2) sigma-model, one can associate a pair of GC structures (see what follows) I and J on the target space, and the generalized A-model (resp. B-model) depends only on J (resp. I), at least if one neglects worldsheet instantons. Other papers discussing the relation between GC geometry and supersymmetric sigma-models include [1, 2, 9]. To any topologically twisted sigma-model one can associate a category of topological D-branes. These D-branes correspond to boundary conditions which are preserved by the BRST operator. D-branes associated to the generalized A- or B-model will be called GC branes. The geometry of such D-branes has been discussed in [5, 12]. We note that so far only GC branes with abelian gauge fields have been understood. By definition, the space of morphisms between a pair of topological D-branes E and E is the BRST cohomology of the space of open strings, with boundary conditions given by E and E. From the physical viewpoint, these are simply open-string states in the Ramond sector, which have zero energy. Open-string BRST cohomology for topological D-branes in A-and B-models has been intensively studied during the last decade both for physical and mathematical reasons. In this paper we begin a study of openstring cohomology for GC branes. Our main result is the description of this cohomology in geometric terms in the case E = E. That is, we compute endomorphisms in the category of GC branes. Specifically, we show that to any GC brane wrapped on a submanifold Y of a GC manifold X one can associate a Lie algebroid E Y whose cohomology computes the BRST cohomology. It turns out that this result has interesting implications even for the well-understood B-branes (i.e., topological D-branes of the ordinary B-model). Namely, it has been argued that the category of B-branes is equivalent to the bounded derived category of coherent sheaves on X. Even in the case of a holomorphic line bundle E

OPEN-STRING BRST COHOMOLOGY FOR GC BRANES 561 supported on a complex submanifold Y, the mathematical procedure for computing endomorphisms of the corresponding object in D b (X) is rather complicated and involves many arbitrary choices. The most explicit way to state the result is to say that there is a spectral sequence converging to the desired space of endomorphisms whose E 2 term is given by p,q H p (Λ q NY). (1.1) The differential d 2 can also be described completely explicitly [8] [T. Pantev, private communication]. It is the composition of the cohomology class β Y H 1 (TY NY ) corresponding to the exact sequence 0 TY TX Y NY 0 and the cohomology class [F ] H 1 (TY ) represented by the curvature of the line bundle E. The class [F ] is known as the Atiyah class of the holomorphic line bundle E; it is the obstruction to the existence of a holomorphic connection on E. The class β Y measures the extent to which TX Y fails to split holomorphically as TY NY. Their composition [F ] β Y is a class in H 2 (NY ). Back-of-the-envelope estimate of the open-string BRST cohomology gives E 2 as the physical result, but a more careful computation shows that the whole spectral sequence arises [8]. This serves as an important check that the category of B-branes is indeed equivalent to D b (X). Our result shows that one can dispense with the spectral sequence and write down an explicit graded vector bundle on Y and a differential Q Y on its space of sections, such that Q Y -cohomology computes the space of endomorphisms of the B-brane. Specifically, the graded bundle is isomorphic to p,q Ω 0,p Λ q NY 1,0 (1.2) (the grading being p + q), and the differential Q Y is mapped by this isomorphism to a deformation of the Dolbeault differential + δ(y,f). The correction term δ(y,f) has bidegree (2, 1) and depends both on the way Y sits in X and the curvature of the line bundle on Y. The correction term is by itself a differential, so one can write down a spectral sequence which converges to Q Y -cohomology and whose E 1 term is given by equation (1.1). We show that the differential d 1 is equal to [F ] β Y. This confirms that Lie algebroid cohomology computes Ext(E, E). The isomorphism of our graded bundle on Y with the graded bundle equation (1.2) is not canonical, and as a result the form δ(y,f) is not completely canonical either. However, the construction of the original graded bundle and the differential Q Y is completely canonical.

562 ANTON KAPUSTIN AND YI LI We note that in the case of B-branes of higher rank we do not have analogous results. The spectral sequence-computing endomorphisms still exists, but we do not know how to get it from a complex of vector bundles on Y. Hopefully, an extension of the computations in this paper will enable one to find such a complex. 1 2 Mathematical preliminaries Let X be a manifold and H be a closed 3-form on X. The twisted Dorfman bracket on smooth sections of TX TX is a bilinear operation defined as follows: (Z + ξ) (W + η) =[Z, W]+L Z η ι W dξ + ι Z ι W H, Z, W Γ(TX), ξ,η Γ(TX ). Its skew-symmetrization is called the twisted Courant bracket. A subbundle E of TX TX is called integrable if it is closed with respect to the twisted Dorfman bracket. Let q be the obvious symmetric bilinear form on TX TX : q(z + ξ,w + η) =Z(η)+W (ξ). If the subbundle E is isotropic with respect to q and integrable, then the twisted Dorfman bracket descends to a Lie bracket on sections of E. A Lie algebroid over X is a triple (E,[, ],a), where E is a real vector bundle over X, [, ] is a Lie bracket on smooth sections of E, and a is a bundle map a: E TX. These data must satisfy the following requirements: a([s 1,s 2 ])=[a(s 1 ),a(s 2 )] for any two smooth sections s 1,s 2 of E. [f s 1,s 2 ]=f [s 1,s 2 ] a(s 2 )(f) s 1 for any two smooth sections s 1,s 2 of E and any f C (X). The simplest Lie algebroid over X is TX itself, with a = id. All standard constructions using the Lie bracket on TX can be generalized to an arbitrary Lie algebroid over X. For example, on the sections of the exterior algebra bundle p Λ p E, (2.1) 1 We would like to emphasize that the results of [8] apply only to rank-1 bundles.

OPEN-STRING BRST COHOMOLOGY FOR GC BRANES 563 there is a degree-1 derivation d E which squares to zero. Its definition is a slight generalization of the definition of the usual exterior derivative: p (d E α)(s 0,...,s p )= ( 1) i a(s i )(α(s 0,...,ŝ i,...,s p )) i=0 + i<j ( 1) i+j α ([s i,s j ],s 0,...,ŝ i,...,ŝ j,...,s p ). (2.2) The cohomology of d E is called the Lie algebroid cohomology. An integrable isotropic subbundle E of TX TX can be made into a Lie algebroid by letting a: E TX to be the obvious projection to TX. A complex Lie algebroid is defined analogously, except a is a bundle map a: E TX C. If E is an integrable isotropic subbundle of the complexification of TX TX, then it has an obvious structure of a complex Lie algebroid. From now on we will drop the adjective complex ; since we will be dealing only with complex Lie algebroids in this paper, this cannot lead to confusion. Let X be a manifold and H be a closed 3-form on X. A twisted GC structure on (X, H) is an endomorphism I: TX TX TX TX, such that I 2 = 1, I preserves q, and the eigenbundle of I with eigenvalue i is integrable. We will denote the latter bundle E in the rest of the note. It is obviously isotropic, so we get a complex Lie algebroid for every GC structure I (we will drop twisted in the rest of the note, so GC will mean twisted GC ). The simplest examples of GC structures (with H = 0) are given by complex and symplectic structures. Given a complex structure I on X (regarded as an endomorphism of TX), we let, in an obvious notation, ( ) I 0 I = 0 I. Given a symplectic structure ω on X, welet ( ) 0 ω 1 I =. ω 0 One can easily check that these tensors define GC structures on X [4]. In this paper we will find the following equivalent definition of a Lie algebroid useful [11]. Given a vector bundle E we can construct a graded supermanifold E[1] by declaring the linear coordinates on the fibers of E to

564 ANTON KAPUSTIN AND YI LI be fermionic variables of degree 1. A Lie algebroid over X is a pair (E,Q), where E is a vector bundle on X and Q is a degree-1 vector field on the supermanifold E[1] satisfying Q 2 =0. To see the relation between the two definitions, let e a be a local basis of sections of E, let x i be local coordinates on X, and let θ a be fermionic linear coordinates on the fibers of E dual to e a. The most general vector field on E[1] of degree 1 has the form Q = a i αθ α i + c α βγ θβ θ γ θ α (2.3) for some locally defined functions a i α,c α βγ. We can use these functions to define a bundle map a: E TX and a bracket operation on sections of E by letting a(e α )=a i α i, [e β,e γ ]=c α βγ e α. The condition Q 2 = 0 is equivalent to the requirement that the triple (E,[, ], a) be a Lie algebroid. Note that in this alternative formulation, sections of the graded bundle equation (2.1) are regarded as functions on E[1], and the Lie algebroid differential d E is simply the derivative of a function along the vector field Q. 3 The Lie algebroid of a GC brane A brane of rank-1 is a submanifold Y together with a Hermitian line bundle E equipped with a unitary connection. Its curvature F = i 2 is a real closed 2-form on Y whose periods are integral multiples of 2π. In what follows, only the curvature of the connection will be important; for this reason we will regard as rank-one brane as a pair (Y,F). If H 0, then there is an additional constraint of Y : the restriction of H to Y must be exact. That is, while the B-field on X is not a globally well-defined 2-form, its restriction to Y is. The set of B-fields on X is acted upon by 1-form gauge transformations: B B + dλ, λ Ω 1 (X). Under this gauge transformation, the connection on E transforms as follows: iλ Y. The curvature of is not invariant under these transformations; the gaugeinvariant combination is F = B Y + F.

OPEN-STRING BRST COHOMOLOGY FOR GC BRANES 565 The generalized tangent bundle T Y F of a brane (Y,F) will be defined as the subbundle of ( TX TX ) Y defined by the following condition: Z + ξ TY F Z TY, ξ + ι Z F NY. Let (X, H) be a GC-manifold with a GC structure I. A GC brane on X is defined to be a brane (Y,F) such that its generalized tangent bundle T Y F is preserved by I. It was shown in [5] that rank-one topological branes of the generalized B-model are precisely GC branes. The definition of a GC brane simplifies somewhat when F = 0, because its generalized tangent bundle becomes the sum of the tangent and the conormal bundle of Y. We will call a GC brane with F = 0 a GC-submanifold. Let (Y,F) be a GC brane in (X, H, I). Let E Y be the i eigenbundle of the restriction of I to T Y F. E Y is a subbundle of the complexification of T Y F. It turns out there is a natural Lie algebroid structure on E Y. 2 The anchor map is the obvious projection to TY C. The Lie bracket is defined as follows. Given any two sections of E Y, we can regard them as sections of E Y (because E Y is a subbundle of E Y ). Extend them off Y, compute the twisted Dorfman bracket, and restrict back to Y. One can easily check that the result lies in E Y and does not depend on how we extend sections off Y. 4 Open-string BRST cohomology for GC branes In this section we show that the cohomology of the Lie algebroid E Y is isomorphic (classically, i.e., if one neglects instantons) to the BRST cohomology of the open-string space of states, where both ends of the open string are on the brane (Y,F). To begin with, let us reformulate the definition of the Lie algebroid E Y in terms of a nilpotent vector field on a graded manifold. Recall that E Y is a subbundle of E Y, and Y is a submanifold of X. Therefore the graded manifold E Y [1] is a graded submanifold of E[1]. If the vector field Q on E[1] happens to be tangent to E Y [1], then one can restrict it to the latter submanifold and get a nilpotent degree-1 vector field on E Y [1]. We claim that this is precisely what happens, and the vector field one gets in this way is Q Y. As apparent from the form of equation (2.3), it is 2 This Lie algebroid structure was independently found by Gualtieri [3].

566 ANTON KAPUSTIN AND YI LI sufficient to check this for functions of degree 0 and 1. Functions of degree 0 on E Y [1] and E[1] are ordinary functions on Y and X, respectively. Let f be a function on Y. We can extend it off Y in an arbitrary manner, compute its Q-derivative (which is a section s of E ), and restrict back to E Y [1], which means that we restrict s to Y and map it to a section of EY using the fact that EY is a quotient of E Y. The first part of our claim means here that the result does not depend on the way we extended f off Y. This is true simply because the anchor map a: E Y TX Y maps E Y E Y to TY. The second part of our statement (that Q Y is the restriction of Q) follows in this case from the fact that the anchor map for E Y is the restriction of the anchor map for E Y. Similarly, for degree-1 functions the claimed result follows from the fact that the Lie algebroid bracket for E Y is well defined. Now let us look at how the open-string BRST cohomology is to be computed. In the zero-mode approximation (which is sufficient for computing the BRST cohomology), open-string pre-observables are functions of both bosonic coordinates on Y and fermionic coordinates taking values in some vector bundle over Y. Fermionic coordinates can have R-charge 1 or 1. In order to compute the BRST cohomology, it is sufficient to consider fermionic coordinates with R-charge 1, since the BRST-variation of the ones with R-charge 1 contains spatial derivatives of bosonic coordinates. Let L be the vector bundle over Y where fermionic coordinates of charge 1 take values. Then the space of observables is the space of functions on the graded manifold L[1]. The generator of the BRST transformation is a degree-1 vector field on L[1] which squares to zero. What does the bundle L look like? For closed strings, the fermionic zero modes (ψ +,ψ ) take values in the bundle TX TX. To make contact with GC geometry, it is useful to work with their linear combinations which take values in TX TX [5]: ψ = 1 2 (ψ + + ψ ), ρ = 1 2 g (ψ + ψ ), where g is the Riemannian metric on X. Open-string boundary conditions put a linear constraint on the fermionic zero modes (ψ, ρ), which requires them to be in the fibers of the generalized tangent bundle of the brane (Y,F) [5]. Finally, the requirement that the R-charge of the fermions be +1 is equivalent to the requirement that the fermions take values in the subbundle E [6]. Since the intersection of E Y and the generalized tangent bundle is E Y, we conclude that L = E Y. It remains to show that the BRST operator acts on L[1] = E Y [1] like Q Y. But this is almost trivial: the open-string zero modes are simply a subset of the closed-string zero modes, and the BRST transformation for the former

OPEN-STRING BRST COHOMOLOGY FOR GC BRANES 567 is a specialization of the BRST transformation for the latter. (The fact that the BRST transformation leaves the subset of open-string zero modes invariant is equivalent to the BRST-invariance of the boundary conditions defining our D-brane.) We also know that the closed-string BRST operator coincides with Q. These two facts and the relation between Q and Q Y explained in the beginning of this section imply the desired result. 5 Examples 5.1 GC-submanifolds It is easy to verify that for GC-submanifolds (i.e., for F = 0) the application of the results of the previous section gives familiar results. For a GC structure I coming from a complex structure I on X (and B = 0), a GC-submanifold is simply a complex submanifold. For such a submanifold, E Y = TY 0,1 (NY ) 1,0. The Lie bracket is the obvious one: TY 0,1 has the standard Lie bracket, the conormal part is an abelian subalgebra, and TY 0,1 acts on sections of (NY ) 1,0 via the ordinary operator. Lie algebroid cohomology of E Y is therefore isomorphic to p,q H p ( Λ q NY 1,0). If I comes from a symplectic structure on X (and B = 0), then a GCsubmanifold is simply a Lagrangian submanifold, and E Y is isomorphic to TY C as a Lie algebroid. Hence Lie algebroid cohomology is isomorphic to the de Rham cohomology H (Y,C). 5.2 Rank-1 B-branes Now let B = 0 and let I come from a complex structure on X. Let (Y,F) be an arbitrary GC brane. This means that Y is a complex submanifold of X, and the curvature F of the connection is of type (1, 1) (i.e., the line bundle E is holomorphic). Let us compute the Lie algebroid E Y and show that it is a deformation of TY 0,1 (NY ) 1,0. Let (z α,u i )belocal holomorphic coordinates on X such that Y is locally given by the equations z α = 0. Their complex-conjugates will be denoted zᾱ, ūī. We want to choose a local basis of sections for E Y. The most obvious choice is e ī = Fīj du j, e α = dz α. ūī

568 ANTON KAPUSTIN AND YI LI It is easy to see that this is a local trivialization of E Y. Moreover, it is easy to check that all Lie brackets vanish (it is important here that df = 0). On the other hand, the obvious Lie algebroid EY 0 = TY0,1 (NY ) 1,0 has the following obvious local trivialization: f ī =, fα = dz α. ūī Obviously, all Lie brackets vanish as well. It seems at this stage that we have proved that the two Lie algebroids are isomorphic. However, this conclusion is premature, because the transition functions in the two cases are different. Namely, as one goes from chart to chart, the covectors du j mix up with dz α, and so e ī mix with e α ; on the other hand, f ī does not mix with f α. To compare the two Lie algebroids, it is convenient to choose a more complicated local trivialization for E Y, so that the transition functions are the same as for EY 0. This will prove that E Y and EY 0 are isomorphic as vector bundles. However, we will see that they are not isomorphic as Lie algebroids, in general, because some Lie brackets in the new basis will be nonvanishing. Suppose we have two overlapping charts. The holomorphic coordinates in the other chart will be denoted (y α,w i ). We have: = Ā jī ūī, w j duj = B j i dwi + Cαdy j α, where B = ( A 1) t is a square matrix whose entries are holomorphic functions of y (it is the gluing cocycle for TY ), Ā is the complex-conjugate of A, while C is a rectangular matrix whose entries are holomorphic functions of w, z. On the overlap of the two charts, consider the following holomorphic section of TY 1,0 (NY ) 1,0 : γ = Cα j u j dyα. γ is a Cech 1-cocycle with values in the coherent sheaf TY 1,0 (NY ) 1,0, which measures the failure of TX Y to split holomorphically as TY NY. Its class was denoted β Y in the Introduction. Using a partition of unity, we can write γ as a coboundary of a smooth 0-cocycle: γ = p j α u j dzα qα j w j dyα, where the matrices p j α and q j α are defined on the first and second chart, respectively, but are not holomorphic, in general.

OPEN-STRING BRST COHOMOLOGY FOR GC BRANES 569 Now consider a modified local trivialization: on the first chart we use ( Fīj du j p j αdz α), dz α, ūī while on the second chart we use ( Gīj dw j qαdy j α), dy α. wī Here G īj = B jī F jibj i is the matrix representing the 2-form F in the coordinate basis dw i,d wī. It is easy to check that the gluing cocycle between the modified local bases is exactly the same as for TY 0,1 (NY ) 1,0. Thus E Y is isomorphic to the latter as a vector bundle. Now let us compute the Lie brackets of the elements of the modified local basis of E Y. In terms of the old basis, the new one is ẽ ī = e ī + F īj p j αdz α, ẽ α = e α = dz α. It follows that the Lie brackets on the first chart are ( [ẽ ī, ẽ j] = F jk ī p k α F jp ) k īk α ẽ α, [ẽ α, ẽ β ]=0, [ẽ ī,e α ]=0, and similarly on the second chart (with p k α replaced with qα). k We observe that the commutator of ẽ ī and ẽ j differs from the commutator of f ī = ī and f j by a term δ(y,f)( ī, j), where δ(y,f) is a local section of Ω 0,2 (Y ) (NY ) 1,0 given in the first chart by ( δ(y,f)= F jk ī p k α F jp ) k īk α dz α d zī d z j. This section is actually globally well defined: this follows from the definition of p, q and the fact that the 1-cocycle γ is holomorphic. To see this more clearly, note that on the overlap of the two charts we have ī p k α u k dzα = ī qα k w k dyα. Here ī denotes either / ūī or / wī (they are related by the matrix Ā). Thus we have a global section d Y of Ω 0,1 (Y ) TY 1,0 (NY ) 1,0 whose local expression is d Y = pk α ūī u k dzα dūī. It is easy to see that d Y is -closed and therefore represents a class β Y H 1 (TY NY ). This is simply the Dolbeault representative of the

570 ANTON KAPUSTIN AND YI LI cohomology class whose Cech representative was denoted γ. The form δ(y,f) is obtained by taking the wedge product of d Y and F Ω 0,1 (TY ) 1,0 and contracting TY 1,0 with (TY ) 1,0. Since both F and d Y are -closed, so is δ(y,f). The Lie algebroid differential for E Y is now easily computed. Since E Y TY 0,1 (NY ) 1,0, it is a degree-1 differential Q Y acting on smooth sections of the graded bundle r,s Ω 0,r (Y ) Λ s NY 1,0, where the grading is given by r + s. One easily sees that if ζ is a section of this graded bundle, then Q Y (ζ) = ζ + δ(y,f) ζ. Here means contraction of NY and Λ s NY. We conclude that the Lie algebroids E Y and EY 0 are not isomorphic, in general: the former is a deformation of the latter. Note that the sheaf cohomology class represented by δ(y,f) is exactly the product of the class β Y H 1 (TY NY ) and a class in H 1 (TY ) represented by the (1, 1) form F. The latter class is the Atiyah class of the line bundle on the brane Y. Thus for Q Y -cohomology we get a spectral sequence whose first term (E 1 ) is simply the -cohomology: p,q H p (Λ q NY), and the first differential is the product of [d Y ]=β Y and the Atiyah class. This is precisely the E 2 term in the spectral sequence computing the Ext groups of the object of D b (X) corresponding to our brane Y [8] [T. Pantev, private communication]. (The object is the push-forward of the locally free sheaf E on Y to the ambient manifold X). This provides some evidence that the Lie algebroid cohomology computes the Ext groups. 5.3 Coisotropic A-branes Since the geometry of coisotropic A-branes is somewhat more complicated than that of B-branes, we start with a brief review of the data involved (see [7] for more details). A coisotropic A-brane is a triple (Y,,F) such that Y X is a coisotropic submanifold, and is a unitary connection on a line bundle on Y with curvature F. By definition, LY ker(ω Y ) forms an integrable distribution of constant rank, which is the codimension of Y. In addition, the curvature form F, regarded as a bundle map F : TY TY, must annihilate LY. So if we denote the quotient bundle TY/LY by FY, F descends to a section of 2 FY. Finally, the restriction of ω 1 F to

OPEN-STRING BRST COHOMOLOGY FOR GC BRANES 571 FY defines a transverse almost-complex structure on Y with respect to the foliation LY. (This transverse almost-complex structure is automatically integrable.) It follows from these conditions that the complex dimension of FY is even. Furthermore, both F and ω FY are of type (2, 0)+(0, 2) with respect to the transverse complex structure J = ω 1 F FY. The Lie algebroid associated with the brane (Y,F) ise Y = ker(i Y + i), where I Y is the restriction to Y of the GC structure associated to the symplectic structure ω on X: ( ) 0 ω 1 I =. ω 0 It is easy to see that topologically E Y is isomorphic to L C Y FY 1,0.However, we will show that E Y LY C FY 1,0 also as Lie algebroids. To this end, we will perform the same kind of calculation as in the B-brane case. Let us choose a local system of coordinates (x a,z i, zī,y µ )onx such that the submanifold Y is locally defined by x a =0,y µ parametrize the leaves of the foliation defined by LY, while the z s are holomorphic coordinates in the transverse directions. Note that the splitting of transverse coordinates into holomorphic and anti-holomorphic ones is done with respect to the complex structure J on FY. A local trivialization for E Y is given by e i = z i iω ijdz j iω ia dx a, e µ = y µ iω µadx a. To find the gluing cocycle in this basis, let us take another local system of coordinates (u, w, v), which overlaps with the old one. As equations x a =0 and u a = 0 define the same submanifold Y locally, one must have u = u(x) on the overlap. In addition, from ω(ly,ty ) = 0 one deduces that w = w(x, z), w = w(x, z). In other words, the Jacobian for the coordinate change takes the following upper triangular form 3 x/ u z/ u y/ u (x, z, y) (u, w, v) = 0 z/ w y/ w. 0 0 y/ v It immediately follows that the e µ s transform among themselves in a simple way: e µ = yν v µ e ν. 3 Here we write z to denote both z and z coordinates to simplify the notation. The same applies to w.

572 ANTON KAPUSTIN AND YI LI The transformation law for the e i s is slightly more complicated. In the new chart, we have e i = w i iω ijdw j iω iadu a. The form of the Jacobian implies that the components of ω transform according to ω ij = zk w i z l w j ω kl ω ia = zj x b w i u a ω jb + zj z k w i u a ω jk + yµ x b w i u a ω µb. Combining this with the transformation law for the coordinate basis, one can show that e i = zj w i e j + yµ w i e µ. This is the same gluing cocycle should we take ẽ i = z i, ẽ µ = y µ (5.1) as the obvious local trivialization of LY C FY 1,0. Therefore our choice of basis establishes an isomorphism between E Y and LY C FY 1,0 as vector bundles. One can further show that these basis sections of E Y commute under the Lie bracket derived from the Dorfman bracket on TX TX. For instance, we have [e i,e j ]= il i (ω jα dq α )+il j (ω iα dq α ) dι j (iω iα dq α ) = i( i ω jα j ω iα + α ω ij )dq α =0, (5.2) where q α denote all of x a,z i, zī,y µ. The last step follows directly from dω = 0. By the same token, we have [e µ,e ν ]=0,[e i,e µ ] = 0. Namely, our basis sections for E Y have the same (vanishing) Lie brackets among themselves, just as the basis sections equation (5.1) of LY C FY 1,0. This shows that E Y LY C FY 1,0 not only as vector bundles but also as Lie algebroids. Since Lie algebroid structures on E X are in one-to-one correspondence with degree-1 homological vector fields on E[1] (i.e., BRST operators in the jargon of TFT), we conclude that one can use the obvious Lie algebroid structure on LY C FY 1,0 to compute the open-string ground

OPEN-STRING BRST COHOMOLOGY FOR GC BRANES 573 states for a coisotropic A-brane. Namely, one can use the simplified BRST operator 4 Q Y = d LY + FY, where d LY is the de Rham differential in the leaf direction, and FY is the Dolbeault operator in the directions transverse to the foliation. This proves the claim by Orlov and one of the authors [7] that the open-string BRST cohomology for a coisotropic A-brane is isomorphic to the cohomology of the sheaf of functions locally constant along the leaves of the characteristic foliation of Y and holomorphic in the transverse directions. Acknowledgments A.K. would like to thank Alexey Bondal, Andrei Căldăraru, Tony Pantev, and Marco Gualtieri for helpful discussions. Y.L. would like to thank Yong- Geun Oh for an interesting discussion. We are also grateful to the organizers of the Workshop on Mirror Symmetry at the Perimeter Institute, Waterloo, for providing a stimulating atmosphere. This work was supported in part by the DOE grant DE-FG03-92-ER40701. References [1] L. Bergamin, Generalized complex geometry and the Poisson sigma model, arxiv:hep-th/0409283. [2] S. Chiantese, F. Gmeiner, and C. Jeschek, Mirror symmetry for topological sigma models with generalized Kaehler geometry, arxiv:hepth/0408169. [3] M. Gualtieri, Generalized complex geometry, D.Phil thesis, Oxford University, arxiv:math.dg/0401221. [4] N. Hitchin, Generalized Calabi Yau Manifolds, Q. J. Math. 54 (2003), 281; arxiv:math.dg/0209099. [5] A. Kapustin, Topological strings on noncommutative manifolds, Int. J. Geom. Meth. Mod. Phys. 1 (2004), 49; arxiv:hep-th/0310057. [6] A. Kapustin and Y. Li, Topological sigma-models with H-flux and twisted generalized complex manifolds, arxiv:hep-th/0407249. [7] A. Kapustin and D. Orlov, Remarks on A-branes, mirror symmetry, and the Fukaya category, J. Geom. Phys. 48 (2003), 84; arxiv:hepth/0109098. 4 We have changed to to conform with the usual conventions.

574 ANTON KAPUSTIN AND YI LI [8] S. Katz and E. Sharpe, D-branes, open string vertex operators, and Ext groups, Adv. Theor. Math. Phys. 6 (2003), 979 1030; arxiv:hepth/0208104. [9] U. Lindstrom, R. Minasian, A. Tomasiello, and M. Zabzine, Generalized complex manifolds and supersymmetry, arxiv:hep-th/0405085. [10] E. Witten, Mirror manifolds and topological field theory, in Essays on mirror manifolds, ed. S.T. Yau, International Press, Hong Kong, 1992, 120 158; arxiv:hep-th/9112056. [11] A. Yu. Vaintrob, Lie algebroids and homological vector fields, Uspekhi Mat. Nauk. 52 (1997), 161 162. [12] M. Zabzine, Geometry of D-branes for general N = (2,2) sigma models, arxiv:hep-th/0405240.