## Section 2 Abnormal polynomials

### Subsection 2.1 Abnormal curves and polynomials

In addition to the characterization as singular points of the endpoint map, abnormal curves can also be defined as characteristic curves of the canonical symplectic form on the cotangent bundle [14], see also Section 4.3.2 of [1]. In the setting of Carnot groups, right-trivialization of the cotangent bundle transfers this characterization from the cotangent bundle to the following description in terms of the Lie algebra, see Section 2.3 of [17].

###### Definition 2.1.

Let \(G\) be a Carnot group and \(\mathfrak{g}\) its Lie algebra. Let \(X\in\mathfrak{g}\) be a vector and \(\covector\in\mathfrak{g}^*\) a covector. The polynomial

is called an abnormal polynomial. Here \(\Ad{g}\in \mathrm{GL}(\mathfrak{g})\) is the adjoint map, i.e. the differential of the conjugation map \(G\to G\text{,}\) \(h\mapsto ghg^{-1}\text{.}\)

The abnormal polynomials were introduced with a different presentation in [15] and [16] under the name “extremal polynomial” for the purpose of characterizing abnormal curves. The presentation using the adjoint map was introduced in [17]. The characterization in Theorem 1.1 of [15] and Corollary 2.14 of [17] is the following.

###### Lemma 2.2.

A horizontal curve \(\gamma\) is abnormal if and only if there exists a nonzero \(\covector\in\mathfrak{g}^*\) such that \(\abnormalpolynomial{X}{\covector}\circ\gamma\equiv 0\) for all horizontal vectors \(X\in\lielayer{\mathfrak{g}}{1}\text{.}\)

An important notion for curves in Carnot groups is the horizontal lift. This allows transferring curves between different Carnot groups of equal rank.

###### Definition 2.3.

Let \(G\) and \(F\) be two Carnot groups and \(\mathfrak{g}\) and \(\mathfrak{f}\) their Lie algebras. A Lie group homomorphism \(\pi\colon F\to G\) is a Carnot group homomorphism if \(\pi_*(\lielayer{\mathfrak{f}}{1})\subset \lielayer{\mathfrak{g}}{1}\text{.}\)

###### Definition 2.4.

Let \(G\) and \(F\) be Carnot groups and let \(\pi\colon F\to G\) be a Carnot group homomorphism. Let \(\gamma\colon(a,b)\to G\) be a horizontal curve in \(G\text{.}\) A curve \(\tilde{\gamma}\colon (a,b)\to F\) is called a horizontal lift of the curve \(\gamma\text{,}\) if \(\tilde{\gamma}\) is horizontal in \(F\) and \(\pi\circ\tilde{\gamma}=\gamma\text{.}\)

An important fact is that the horizontal lifts of abnormal curves are themselves also abnormal curves, see Proposition 2.27 of [17].

The proof of the main result Theorem 1.1 will involve manipulating the abstract abnormal polynomials \(\abnormalpolynomial{X}{\covector}(g) = \covector(\Ad{g}X)\) in coordinates as concrete multivariate polynomials of a polynomial ring \(\polyring{x_1,\ldots,x_n}\text{.}\) Since the adjoint map \(\Ad{}\colon G\to \mathrm{GL}(\mathfrak{g})\) is a Lie group homomorphism, a convenient system of coordinates is given by exponential coordinates of the second kind. For a basis \(X_1,\ldots,X_n\) of the Lie algebra \(\mathfrak{g}\text{,}\) the exponential coordinates of the second kind are

###### Lemma 2.5.

In exponential coordinates of the second kind with respect to the basis \(X_1,\dots,X_n\text{,}\) the abnormal polynomial \(\abnormalpolynomial{X}{\covector}\) has the formula

###### Proof.

The claim follows directly by applying the two formulas

and

to expand the expression \(\abnormalpolynomial{X}{\covector}(x_1,\ldots,x_n) = \covector(\Ad{\exp(x_nX_n)\cdots\exp(x_1X_1)}X)\text{.}\)

### Subsection 2.2 Hall sets

The coordinate formula of Lemma 2.5 for the abnormal polynomials depends on the choice of the basis \(X_1,\ldots,X_n\) of the Lie algebra. In the case of free Lie algebras, possible bases are well understood in terms of Hall sets, see the original construction in [12]. The following definition is the one in Section 4.1 of [22], but with the trees structured as in Ch II, §2.10 of [7].

###### Definition 2.6.

Let \(\mathcal{A}\) be a finite set and let \(M(\mathcal{A})\) be the free magma on \(\mathcal{A}\text{,}\) i.e., the set of all binary trees with leaves in \(\mathcal{A}\text{.}\) For \(h_1,h_2\in M(\mathcal{A})\text{,}\) denote by \((h_1,h_2)\in M(\mathcal{A})\) the binary tree whose left subtree is \(h_1\) and right subtree is \(h_2\text{.}\)

A Hall set on \(\mathcal{A}\) is a totally ordered set \(\hallset\subset M(\mathcal{A})\) with the following properties:

- All elements \(a\in \mathcal{A}\) are in \(\hallset\text{.}\)
- Every tree \(h=(h_1,h_2)\in \hallset\) satisfies \(h_1,h_2\in\hallset\) and \(h_1\lt h_2\) and \(h_1\lt h\text{.}\)
- Every tree \((h_1,(h_{21},h_{22}))\in \hallset\) satisfies \(h_{21}\leq h_1\text{.}\)
- Every tree \(h\in M(\mathcal{A})\) with the properties of ii and iii is contained in \(\hallset\text{.}\)

The elements of \(\hallset\) are called Hall trees. The degree \(\deg(h)\) of a tree \(h\) is its number of leaves.

###### Remark 2.7.

The order constraint \(h_1\lt h\) in ii is sometimes imposed in the stronger form that \(\deg(h')\lt \deg(h)\) implies that \(h'\lt h\) for all \(h,h'\in\hallset\text{.}\) This stronger condition is used in much of the older literature, such as the original construction of [12] and in [7]. See the discussion in Section 4.5 of [22] for the history and reasons behind these different conditions.

Viewing the elements \(\mathcal{A}\) as the generators of a free Lie algebra, each Hall tree specifies some iterated Lie bracket of the generators and hence an element of the free Lie algebra. By Theorem 4.9 of [22], any Hall set on \(\mathcal{A}\) defines a basis of the free Lie algebra on \(\mathcal{A}\text{.}\)

Fixing a total order on the free magma \(M(\mathcal{A})\text{,}\) a Hall set can be defined by a recursively enumerating the trees satisfying the conditions of Definition 2.6, see Proposition 4.1 of [22]. The only requirement on the total order on the free magma is that \(h_1\lt h\) for any tree \(h=(h_1,h_2)\text{.}\) For the purposes of this article, the order will also need to be compatible with the degree. The following recursively defined order will be used in the rest of the paper.

###### Definition 2.8.

Let \((\mathcal{A},\lt)\) be a totally ordered set and let \(M(\mathcal{A})\) be the set of binary trees with leaves in \(\mathcal{A}\text{.}\) The deg-left-right order on \(M(\mathcal{A})\) is the total order extending \(\lt\) such that \(h\lt h'\) for \(h,h'\in M(\mathcal{A})\) if any of the following hold:

- \(\deg(h)\lt\deg(h')\text{,}\)
- \(\deg(h)=\deg(h')\) and \(h_1\lt h_1'\text{,}\) or
- \(\deg(h)=\deg(h')\) and \(h_1=h_1'\) and \(h_2\lt h_2'\text{,}\)

where

and \(h_1',h_2'\) are similarly the left and right subtrees for \(h'\text{.}\)

The deg-left-right Hall set for the free Lie algebra of rank \(r\) is the Hall set constructed with the deg-left-right order on \(M(\{1,2,\dots,r\})\text{,}\) where \(1\lt 2\lt\cdots \lt r\) is ordered in the standard way.

Each Hall tree \(h\in\hallset\subset M(\mathcal{A})\) has a corresponding Hall word \(w(h)\text{,}\) which is a word with letters in \(A\text{.}\) The Hall words are defined recursively for \(h=(h_1,h_2)\) by concatenations \(w(h) = w(h_1)w(h_2)\) starting from \(w(a)=a\) for each \(a\in \mathcal{A}\text{.}\) The Hall words are in one to one correspondence with Hall trees, see Corollary 4.5 of [22]. This correspondence defines the deg-left-right order on the deg-left-right Hall words.

###### Lemma 2.9.

Let \(\hallset\) be any Hall set such that the order is compatible with the degree, i.e., such that \(\deg(h)\lt \deg(h')\) implies \(h\lt h'\text{.}\) Then the Hall words of degree \(\geq 2\) are exactly the words that factor as

where

- \(i_1,\ldots,i_k\in \ZZ_+\) are arbitrary exponents, and
- \(w_{1}\lt w_{2}\lt \cdots \lt w_{k}\lt w\) are Hall words of lower degree.
- either \(\deg(w) = 1\) or there are Hall words \(w',w''\) such that \(w=w'w''\text{,}\) \(w'\lt w''\) and \(w_{1}\geq w'\text{.}\)

Moreover, the Hall tree for the Hall word \((w_{k})^{i_k}(w_{k-1})^{i_{k-1}}\cdots (w_{1})^{i_1}w\) describes the Lie bracket

###### Proof.

The claim that the Hall words factor as in the statement is straightforward. If \(h=(h_1,h_2)\) is a Hall tree of degree \(\geq 2\text{,}\) then by Definition 2.6 \(h_1\lt h_2\) and if \(h_2=(h_{21},h_{22})\text{,}\) then \(h_1\geq h_{21}\text{.}\) Then \(w(h)\) admits the factorization \(w(h) = w(h_1)w(h_2)\) and \(\ad{X_{w(h_1)}}X_{w(h_2)}\) is the corresponding Lie bracket.

For the converse claim, let \(h\) be the binary tree described by a Lie bracket \(\ad{X_{w_{k}}}^{i_k}\cdots \ad{X_{w_{1}}}^{i_1}X_{w}\) with the words \(w_1,\ldots,w_k\text{,}\) and \(w\) satisfying the conditions i–iii. Write the \(\ad{}\)-sequence without exponents as

where \(n=i_1+\cdots+i_k\text{,}\) \(Y_1=\ldots=Y_{i_1}=X_{w_{1}}\) and so on. For each \(i=1,\ldots,k\text{,}\) let \(h_i\) be the Hall tree corresponding to the Lie algebra element \(Y_i\) and let \(h_0\) be the Hall tree for the element \(X_{w}\text{,}\) so that \(h=(h_n,(h_{n-1},(\ldots,(h_1,h_0)\ldots)))\text{.}\) The claim that \(h\) is a Hall tree will follow by induction showing that every tree \(\bar{h}_i:=(h_i,(\ldots(h_1,h_0)\ldots))\) is a Hall tree. The case \(i=0\) holds by the choice of \(h_0=\bar{h}_0\text{.}\)

In the case \(i=1\text{,}\) assumption ii implies that \(h_1\lt h_0\text{.}\) If \(\deg(w)\gt 1\text{,}\) let \(h'\) and \(h''\) be the Hall trees corresponding to the Hall words \(w'\) and \(w''\text{,}\) so that \(h_0=(h',h'')\text{.}\) Then assumption iii implies that \(h_1\geq h'\text{.}\) By assumption the order is compatible with the degree, so \(h_1\lt (h_1,h_0)=\bar{h}_1\text{.}\) Hence the tree \(\bar{h}_1=(h_1,h_0)\) satisfies all the order constraints of Definition 2.6 and is a Hall tree.

Suppose the claim holds up to some \(i\geq 1\text{.}\) The tree \(\bar{h}_{i+1} = (h_{i+1},(h_i,\bar{h}_{i-1}))\) then consists of Hall trees. The degree first comparison guarantees that \(h_{i+1}\lt \bar{h}_{i}\lt \bar{h}_{i+1}\text{.}\) Combined with assumption ii, this guarantees that all of the order constraints \(h_{i+1}\lt (h_i,\bar{h}_{i-1})\text{,}\) \(h_{i+1}\lt \bar{h}_{i+1}\text{,}\) and \(h_i\leq h_{i+1}\) are satisfied, so \(\bar{h}_{i+1}\) is a Hall tree. By induction it follows that the tree \(h=\bar{h}_n\) is a Hall tree, proving the claim.

###### Example 2.10.

In rank 2, the deg-left-right Hall words up to degree 6 are

For the words of degree 6, the longest possible factorizations in the form of Lemma 2.9 are

### Subsection 2.3 Polynomials in free Lie algebras

In the proof of Theorem 1.1 it will be convenient to consider abnormal polynomials not as polynomials in a single Carnot group \(G\text{,}\) but as abstract multivariate polynomials that admit realizations as abnormal polynomials in several Carnot groups. Fixing a basis \(X_1,\ldots,X_n\) of the Lie algebra \(\mathfrak{g}\text{,}\) the coordinate formula of Lemma 2.5 realizes the identification of abnormal polynomials as multivariate polynomials \(\abnormalpolynomial{}{}\in \polyring{x_1,\ldots,x_n}\text{.}\) Fixing a Hall set \(\hallset=\{w_{1},w_{2},\ldots\}\) for the free Lie algebra \(\freelie{r}\) of rank \(r\) then allows viewing multivariate polynomials \(\abnormalpolynomial{}{}\in\polyring{x_{w_{1}},x_{w_{2}},\ldots}\) as polynomials in several Carnot groups of rank \(r\) at once, as will be described next.

###### Definition 2.11.

Let \(\hallset\) be a Hall set for the free Lie algebra \(\freelie{r}\) of rank \(r\text{.}\) Let \(G\) be a Carnot group of rank \(r\text{,}\) so its Lie algebra is the quotient \(\mathfrak{g}=\freelie{r}/I\) of the free Lie algebra \(\freelie{r}\) by some ideal \(I\subset\freelie{r}\text{.}\) The Carnot group \(G\) is said to be compatible with the Hall set \(\hallset\), if the ideal \(I\) has a basis consisting of elements of the Hall set.

###### Definition 2.12.

Let \(\hallset\) be a Hall set for a free Lie algebra \(\freelie{r}\) and let \(G\) be a Carnot group of rank \(r\) that is compatible with the Hall set \(\hallset\text{.}\) Let \(w_{1},\dots,w_{n}\) be the complementary Hall words to the ideal defining \(\mathfrak{g}\text{,}\) so that \(\mathfrak{g}=\operatorname{span}\{X_{w_{1}},\dots,X_{w_{n}}\}\text{.}\) The exponential coordinates of the second kind

are said to be adapted to the Hall set if \(w_{n}\gt\ldots\gt w_{1}\text{.}\)

###### Definition 2.13.

Let \(\hallset=\{w_{1},w_{2},\ldots\}\) be a Hall set for a free Lie algebra \(\freelie{r}\text{.}\) Let \(\polyring{\hallset}:=\polyring{x_{w_{1}},x_{w_{2}},\ldots}\) be the weighted polynomial ring with a countable number of generators, where the weight of the variable \(x_{w_{i}}\) is the degree of the Hall word \(w_{i}\text{.}\)

The above definitions allow identifying each polynomial in a Carnot group compatible with a Hall set \(\hallset\) with a unique polynomial in \(\polyring{\hallset}\text{.}\) Abnormal polynomials are not completely arbitrary polynomials however, so it will be relevant that the coordinate form of abnormal polynomials is preserved by lifting to higher step groups.

###### Lemma 2.14.

Let \(\mathfrak{f}\) be a stratified Lie algebra with a basis \(X_1,\ldots, X_n\text{.}\) Let \(\mathfrak{g}\) be a stratified Lie algebra defined as a quotient of \(\mathfrak{f}\) by an ideal \(I = \operatorname{span}\{X_{m+1},\ldots,X_n\}\text{.}\) Let \(F\) and \(G\) be Carnot groups with Lie algebras \(\mathfrak{f}\) and \(\mathfrak{g}\) respectively. Then for every vector \(X\in\mathfrak{g}\) and covector \(\covector\in \mathfrak{g}^*\) there exist lifts \(\tilde{X}\in\mathfrak{f}\) and \(\tilde{\covector}\in\mathfrak{f}^*\) such that the abnormal polynomials \(\abnormalpolynomial{X}{\covector}\) in \(G\) and \(\abnormalpolynomial{\tilde{X}}{\tilde{\covector}}\) in \(F\) have identical coordinate expressions in exponential coordinates of the second kind with respect to the bases \(X_1,\ldots,X_n\) and \(\pi(X_1),\ldots,\pi(X_m)\text{.}\)

###### Proof.

The lift of the covector is the pullback \(\tilde{\covector}:=\pi^*\covector\text{.}\) The vector \(X\) is lifted by setting the coefficients of the elements \(X_{m+1},\ldots,X_n\) to zero. That is, if \(X = \sum_{i=1}^{m}x_i\pi(X_i)\text{,}\) then the lift is \(\tilde{X} = \sum_{i=1}^{m}x_iX_i\text{.}\)

The projection \(\pi\colon\mathfrak{f}\to\mathfrak{g}\) is a Lie algebra homomorphism, so the definitions of the lifts imply that

Since \(\pi(X_{n})=\ldots=\pi(X_{m+1})=0\text{,}\) the formula of Lemma 2.5 implies that the abnormal polynomial \(\abnormalpolynomial{\tilde{X}}{\tilde{\covector}}\) in \(G\) only contains monomials with variables \(x_1,\ldots,x_m\text{.}\) Moreover by the same formula, the coefficient of each monomial containing only variables \(x_1,\ldots,x_m\) is exactly the coefficient of the same monomial in the abnormal polynomial \(\abnormalpolynomial{X}{\covector}\) in \(H\text{.}\)

In a Carnot group \(G\text{,}\) there is a natural action \(\mathfrak{g}\acts C^\infty(G)\) of the Lie algebra \(\mathfrak{g}\) on the space of smooth functions \(G\to\RR\text{,}\) defined by extending a vector \(X\in\mathfrak{g}\) to a left-invariant vector field \(\tilde{X}(g) = (L_{g})_*X\) and using the action of vector fields as derivations on the space of smooth functions. In coordinates this defines an action \(\mathfrak{g}\acts \polyring{x_1,\ldots,x_n}\) by derivations also on the space of polynomials.

###### Definition 2.15.

Let \(\hallset\) be a Hall set for a free Lie algebra \(\freelie{r}\text{.}\) Define an action \(\freelieaction\colon \freelie{r}\acts\polyring{\hallset}\) via derivations by extending all the actions \(\mathfrak{g}\acts \polyring{x_1,\ldots,x_n}\) as follows:

Let \(X\in\freelie{r}\) be a vector and \(P\in\polyring{\hallset}\) a polynomial. Let \(x_{w_{1}},\ldots,x_{w_{k}}\) be all the variables of \(P\) and let \(G\) be any Carnot group compatible with the Hall set \(\hallset\) such that none of the basis elements \(X_{w_{1}},\ldots,X_{w_{k}}\) are quotiented away. Fix exponential coordinates \(\RR^n\to G\) adapted to the Hall set \(\hallset\text{,}\) and let \(\iota\colon\polyring{x_1,\ldots,x_n}\into \polyring{\hallset}\) be the identification of polynomials in \(G\) in coordinates with polynomials in \(\polyring{\hallset}\text{.}\) The action \(\freelieaction\colon \freelie{r}\acts\polyring{\hallset}\) is defined by

where \(\pi\colon \freelie{r}\to\mathfrak{g}\) is the quotient projection, and \(\pi(X)\iota^{-1}(P)\) is defined by the action \(\mathfrak{g}\acts C^\infty(G)\text{.}\)

The well-posedness of Definition 2.15 critically relies on the assumption that all the coordinates on the Carnot groups involved are adapted to a single fixed Hall set. Without this assumption, the coordinate expressions of the actions \(\mathfrak{g}\acts\polyring{x_1,\ldots,x_n}\) are in general incompatible with each other.

An extremely useful feature of the abnormal polynomials is how their behavior under the action \(\mathfrak{g}\acts\polyring{x_1,\ldots,x_n}\) is linked with the structure of the Lie algebra \(\mathfrak{g}\text{.}\) The following result is a rephrasing of Theorem 1.1 of [16] and Proposition 2.22 of [17] using the linearity of the abnormal polynomials \(\abnormalpolynomial{Y}{\covector}\) in the vector \(Y\text{.}\)

###### Lemma 2.16.

Let \(G\) be a Carnot group and \(\mathfrak{g}\) its Lie algebra. Then for every covector \(\covector\in\mathfrak{g}^*\) and all vectors \(X,Y\in\mathfrak{g}\text{,}\) the \(X\)-derivative of the abnormal polynomial \(\abnormalpolynomial{Y}{\covector}\) is the abnormal polynomial \(X\abnormalpolynomial{Y}{\covector} = \abnormalpolynomial{[X,Y]}{\covector}\text{.}\)

In the proof of Theorem 1.1 it will be necessary to further simplify the monomial coefficients of abnormal polynomials given by the formula of Lemma 2.5. The following lemma is a criterion for when these coefficients are as simple as possible for the deg-left-right Hall set.

###### Lemma 2.17.

Let \(w\) be a deg-left-right Hall word with factorization \(w=w'w''\) and let \(x^I = x_{v_1}^{i_1}\ldots x_{v_\ell}^{i_\ell}\) be a monomial in \(\polyring{\hallset}\) such that \(v_1\lt\ldots\lt v_\ell\lt w\) and \(v_1\geq w'\text{.}\) Then \(vw:=(v_\ell)^{i_\ell}\cdots (v_1)^{i_1}w\) is a deg-left-right Hall word and in exponential coordinates adapted to the deg-left-right order the coefficient of the monomial \(x^I\) in the abnormal polynomial \(\abnormalpolynomial{w}{\covector}\) is \(\frac{1}{i_1!\cdots i_\ell!}\covector_{vw}\text{.}\)

###### Proof.

By the explicit formula of Lemma 2.5 the coefficient of the monomial \(x^I\) in the abnormal polynomial \(\abnormalpolynomial{w}{\covector}\) is

By Lemma 2.9, the assumption that \(v_1\lt\ldots\lt v_\ell\lt w\) and \(v_1\geq w'\) guarantees that \(vw\) is a deg-left-right Hall word. The second part of the lemma states the bracket \(\ad{X_{v_\ell}}^{i_\ell}\cdots \ad{X_{v_1}}^{i_1}X_{w}\) is exactly the Hall basis element \(X_{vw}\text{,}\) so

### Subsection 2.4 Poincaré series

The proof of Theorem 1.1 involves counting the number of monomials of a given degree depending on some subset of the variables \(x_{w}\) in \(\polyring{\hallset}\text{.}\) A convenient form to manipulate these quantities will be in terms of generating functions, i.e., in terms of Poincaré series.

###### Definition 2.18.

Let \(W = \bigoplus_{i=0}^\infty W_i\) be a graded vector space with all components \(W_i\) finite dimensional. The Poincaré series of the graded vector space \(W\) is the formal power series

The Poincaré series of a weighted polynomial ring has a rather simple expression as a rational function depending on the weights.

###### Lemma 2.19.

Let \(r\in\NN\) and \(d\in\NN\text{.}\) Let \(w_{1}\lt\dots\lt w_{n}\) be all the Hall words of degree up to \(d\) in the free Lie algebra \(\freelie{r}\) of rank \(r\text{,}\) and let \(d_{k}:=\deg(w_{k})\text{.}\) The Poincaré series for the weighted polynomial ring \(\polyring{x_{w_{1}},\ldots,x_{w_{n}}}\) with weights \(d_{1},\ldots,d_{n}\) is given by the rational function

###### Proof.

By Ch. V, §5.1, Proposition 1 of [6], the Poincaré series for the weighted polynomial ring is given by the first formula \(\poincare(t) = 1/\prod_{k=1}^{n} (1-t^{d_{k}})\text{.}\) Since there are exactly \(\dim\freelielayer{r}{k}\) Hall words of degree \(k\text{,}\) the second formula follows by grouping terms.