## Section2Abnormal polynomials

### Subsection2.1Abnormal curves and polynomials

In addition to the characterization as singular points of the endpoint map, abnormal curves can also be defined as characteristic curves of the canonical symplectic form on the cotangent bundle [14], see also Section 4.3.2 of [1]. In the setting of Carnot groups, right-trivialization of the cotangent bundle transfers this characterization from the cotangent bundle to the following description in terms of the Lie algebra, see Section 2.3 of [17].

###### Definition2.1.

Let $G$ be a Carnot group and $\mathfrak{g}$ its Lie algebra. Let $X\in\mathfrak{g}$ be a vector and $\covector\in\mathfrak{g}^*$ a covector. The polynomial

is called an abnormal polynomial. Here $\Ad{g}\in \mathrm{GL}(\mathfrak{g})$ is the adjoint map, i.e. the differential of the conjugation map $G\to G\text{,}$ $h\mapsto ghg^{-1}\text{.}$

The abnormal polynomials were introduced with a different presentation in [15] and [16] under the name “extremal polynomial” for the purpose of characterizing abnormal curves. The presentation using the adjoint map was introduced in [17]. The characterization in Theorem 1.1 of [15] and Corollary 2.14 of [17] is the following.

An important notion for curves in Carnot groups is the horizontal lift. This allows transferring curves between different Carnot groups of equal rank.

###### Definition2.3.

Let $G$ and $F$ be two Carnot groups and $\mathfrak{g}$ and $\mathfrak{f}$ their Lie algebras. A Lie group homomorphism $\pi\colon F\to G$ is a Carnot group homomorphism if $\pi_*(\lielayer{\mathfrak{f}}{1})\subset \lielayer{\mathfrak{g}}{1}\text{.}$

###### Definition2.4.

Let $G$ and $F$ be Carnot groups and let $\pi\colon F\to G$ be a Carnot group homomorphism. Let $\gamma\colon(a,b)\to G$ be a horizontal curve in $G\text{.}$ A curve $\tilde{\gamma}\colon (a,b)\to F$ is called a horizontal lift of the curve $\gamma\text{,}$ if $\tilde{\gamma}$ is horizontal in $F$ and $\pi\circ\tilde{\gamma}=\gamma\text{.}$

An important fact is that the horizontal lifts of abnormal curves are themselves also abnormal curves, see Proposition 2.27 of [17].

The proof of the main result Theorem 1.1 will involve manipulating the abstract abnormal polynomials $\abnormalpolynomial{X}{\covector}(g) = \covector(\Ad{g}X)$ in coordinates as concrete multivariate polynomials of a polynomial ring $\polyring{x_1,\ldots,x_n}\text{.}$ Since the adjoint map $\Ad{}\colon G\to \mathrm{GL}(\mathfrak{g})$ is a Lie group homomorphism, a convenient system of coordinates is given by exponential coordinates of the second kind. For a basis $X_1,\ldots,X_n$ of the Lie algebra $\mathfrak{g}\text{,}$ the exponential coordinates of the second kind are

\begin{equation*} \RR^n\to G,\quad (x_1,\ldots,x_n)\mapsto \exp(x_nX_n)\cdots\exp(x_1X_1)\text{.} \end{equation*}

The claim follows directly by applying the two formulas

and

to expand the expression $\abnormalpolynomial{X}{\covector}(x_1,\ldots,x_n) = \covector(\Ad{\exp(x_nX_n)\cdots\exp(x_1X_1)}X)\text{.}$

### Subsection2.2Hall sets

The coordinate formula of Lemma 2.5 for the abnormal polynomials depends on the choice of the basis $X_1,\ldots,X_n$ of the Lie algebra. In the case of free Lie algebras, possible bases are well understood in terms of Hall sets, see the original construction in [12]. The following definition is the one in Section 4.1 of [22], but with the trees structured as in Ch II, §2.10 of [7].

###### Definition2.6.

Let $\mathcal{A}$ be a finite set and let $M(\mathcal{A})$ be the free magma on $\mathcal{A}\text{,}$ i.e., the set of all binary trees with leaves in $\mathcal{A}\text{.}$ For $h_1,h_2\in M(\mathcal{A})\text{,}$ denote by $(h_1,h_2)\in M(\mathcal{A})$ the binary tree whose left subtree is $h_1$ and right subtree is $h_2\text{.}$

A Hall set on $\mathcal{A}$ is a totally ordered set $\hallset\subset M(\mathcal{A})$ with the following properties:

1. All elements $a\in \mathcal{A}$ are in $\hallset\text{.}$
2. Every tree $h=(h_1,h_2)\in \hallset$ satisfies $h_1,h_2\in\hallset$ and $h_1\lt h_2$ and $h_1\lt h\text{.}$
3. Every tree $(h_1,(h_{21},h_{22}))\in \hallset$ satisfies $h_{21}\leq h_1\text{.}$
4. Every tree $h\in M(\mathcal{A})$ with the properties of ii and iii is contained in $\hallset\text{.}$

The elements of $\hallset$ are called Hall trees. The degree $\deg(h)$ of a tree $h$ is its number of leaves.

###### Remark2.7.

The order constraint $h_1\lt h$ in ii is sometimes imposed in the stronger form that $\deg(h')\lt \deg(h)$ implies that $h'\lt h$ for all $h,h'\in\hallset\text{.}$ This stronger condition is used in much of the older literature, such as the original construction of [12] and in [7]. See the discussion in Section 4.5 of [22] for the history and reasons behind these different conditions.

Viewing the elements $\mathcal{A}$ as the generators of a free Lie algebra, each Hall tree specifies some iterated Lie bracket of the generators and hence an element of the free Lie algebra. By Theorem 4.9 of [22], any Hall set on $\mathcal{A}$ defines a basis of the free Lie algebra on $\mathcal{A}\text{.}$

Fixing a total order on the free magma $M(\mathcal{A})\text{,}$ a Hall set can be defined by a recursively enumerating the trees satisfying the conditions of Definition 2.6, see Proposition 4.1 of [22]. The only requirement on the total order on the free magma is that $h_1\lt h$ for any tree $h=(h_1,h_2)\text{.}$ For the purposes of this article, the order will also need to be compatible with the degree. The following recursively defined order will be used in the rest of the paper.

###### Definition2.8.

Let $(\mathcal{A},\lt)$ be a totally ordered set and let $M(\mathcal{A})$ be the set of binary trees with leaves in $\mathcal{A}\text{.}$ The deg-left-right order on $M(\mathcal{A})$ is the total order extending $\lt$ such that $h\lt h'$ for $h,h'\in M(\mathcal{A})$ if any of the following hold:

1. $\deg(h)\lt\deg(h')\text{,}$
2. $\deg(h)=\deg(h')$ and $h_1\lt h_1'\text{,}$ or
3. $\deg(h)=\deg(h')$ and $h_1=h_1'$ and $h_2\lt h_2'\text{,}$

where

\begin{equation*} h=\begin{cases}(h_1,h_2),\amp \text{if }\deg(h)\geq 2\\h_1,\amp \text{if }\deg(h)=1\end{cases} \end{equation*}

and $h_1',h_2'$ are similarly the left and right subtrees for $h'\text{.}$

The deg-left-right Hall set for the free Lie algebra of rank $r$ is the Hall set constructed with the deg-left-right order on $M(\{1,2,\dots,r\})\text{,}$ where $1\lt 2\lt\cdots \lt r$ is ordered in the standard way.

Each Hall tree $h\in\hallset\subset M(\mathcal{A})$ has a corresponding Hall word $w(h)\text{,}$ which is a word with letters in $A\text{.}$ The Hall words are defined recursively for $h=(h_1,h_2)$ by concatenations $w(h) = w(h_1)w(h_2)$ starting from $w(a)=a$ for each $a\in \mathcal{A}\text{.}$ The Hall words are in one to one correspondence with Hall trees, see Corollary 4.5 of [22]. This correspondence defines the deg-left-right order on the deg-left-right Hall words.

The claim that the Hall words factor as in the statement is straightforward. If $h=(h_1,h_2)$ is a Hall tree of degree $\geq 2\text{,}$ then by Definition 2.6 $h_1\lt h_2$ and if $h_2=(h_{21},h_{22})\text{,}$ then $h_1\geq h_{21}\text{.}$ Then $w(h)$ admits the factorization $w(h) = w(h_1)w(h_2)$ and $\ad{X_{w(h_1)}}X_{w(h_2)}$ is the corresponding Lie bracket.

For the converse claim, let $h$ be the binary tree described by a Lie bracket $\ad{X_{w_{k}}}^{i_k}\cdots \ad{X_{w_{1}}}^{i_1}X_{w}$ with the words $w_1,\ldots,w_k\text{,}$ and $w$ satisfying the conditions iiii. Write the $\ad{}$-sequence without exponents as

where $n=i_1+\cdots+i_k\text{,}$ $Y_1=\ldots=Y_{i_1}=X_{w_{1}}$ and so on. For each $i=1,\ldots,k\text{,}$ let $h_i$ be the Hall tree corresponding to the Lie algebra element $Y_i$ and let $h_0$ be the Hall tree for the element $X_{w}\text{,}$ so that $h=(h_n,(h_{n-1},(\ldots,(h_1,h_0)\ldots)))\text{.}$ The claim that $h$ is a Hall tree will follow by induction showing that every tree $\bar{h}_i:=(h_i,(\ldots(h_1,h_0)\ldots))$ is a Hall tree. The case $i=0$ holds by the choice of $h_0=\bar{h}_0\text{.}$

In the case $i=1\text{,}$ assumption ii implies that $h_1\lt h_0\text{.}$ If $\deg(w)\gt 1\text{,}$ let $h'$ and $h''$ be the Hall trees corresponding to the Hall words $w'$ and $w''\text{,}$ so that $h_0=(h',h'')\text{.}$ Then assumption iii implies that $h_1\geq h'\text{.}$ By assumption the order is compatible with the degree, so $h_1\lt (h_1,h_0)=\bar{h}_1\text{.}$ Hence the tree $\bar{h}_1=(h_1,h_0)$ satisfies all the order constraints of Definition 2.6 and is a Hall tree.

Suppose the claim holds up to some $i\geq 1\text{.}$ The tree $\bar{h}_{i+1} = (h_{i+1},(h_i,\bar{h}_{i-1}))$ then consists of Hall trees. The degree first comparison guarantees that $h_{i+1}\lt \bar{h}_{i}\lt \bar{h}_{i+1}\text{.}$ Combined with assumption ii, this guarantees that all of the order constraints $h_{i+1}\lt (h_i,\bar{h}_{i-1})\text{,}$ $h_{i+1}\lt \bar{h}_{i+1}\text{,}$ and $h_i\leq h_{i+1}$ are satisfied, so $\bar{h}_{i+1}$ is a Hall tree. By induction it follows that the tree $h=\bar{h}_n$ is a Hall tree, proving the claim.

In rank 2, the deg-left-right Hall words up to degree 6 are

\begin{align*} 1\amp\lt 2\lt 12\lt 112\lt 212\lt 1112\lt 2112\lt 2212\\ \amp\lt 11112\lt 21112\lt 22112\lt 22212\lt 12112\lt 12212\\ \amp\lt 111112\lt 211112\lt 221112\lt 222112\lt 222212\\ \amp\lt 121112\lt 122112\lt 122212\lt 112212\text{.} \end{align*}

For the words of degree 6, the longest possible factorizations in the form of Lemma 2.9 are

\begin{align*} 111112 \amp= (1)^5(2)\amp 222112 \amp= (2)^3(1)(12)\amp 122112 \amp= (12)(2)(112)\\ 211112 \amp= (2)(1)^3(12)\amp 222212 \amp= (2)^4(12)\amp 122212 \amp= (12)(2)(212)\\ 221112 \amp= (2)^2(1)^2(12)\amp 121112 \amp= (12)(1)(112)\amp 112212 \amp= (112)(212)\text{.} \end{align*}

### Subsection2.3Polynomials in free Lie algebras

In the proof of Theorem 1.1 it will be convenient to consider abnormal polynomials not as polynomials in a single Carnot group $G\text{,}$ but as abstract multivariate polynomials that admit realizations as abnormal polynomials in several Carnot groups. Fixing a basis $X_1,\ldots,X_n$ of the Lie algebra $\mathfrak{g}\text{,}$ the coordinate formula of Lemma 2.5 realizes the identification of abnormal polynomials as multivariate polynomials $\abnormalpolynomial{}{}\in \polyring{x_1,\ldots,x_n}\text{.}$ Fixing a Hall set $\hallset=\{w_{1},w_{2},\ldots\}$ for the free Lie algebra $\freelie{r}$ of rank $r$ then allows viewing multivariate polynomials $\abnormalpolynomial{}{}\in\polyring{x_{w_{1}},x_{w_{2}},\ldots}$ as polynomials in several Carnot groups of rank $r$ at once, as will be described next.

###### Definition2.11.

Let $\hallset$ be a Hall set for the free Lie algebra $\freelie{r}$ of rank $r\text{.}$ Let $G$ be a Carnot group of rank $r\text{,}$ so its Lie algebra is the quotient $\mathfrak{g}=\freelie{r}/I$ of the free Lie algebra $\freelie{r}$ by some ideal $I\subset\freelie{r}\text{.}$ The Carnot group $G$ is said to be compatible with the Hall set $\hallset$, if the ideal $I$ has a basis consisting of elements of the Hall set.

###### Definition2.12.

Let $\hallset$ be a Hall set for a free Lie algebra $\freelie{r}$ and let $G$ be a Carnot group of rank $r$ that is compatible with the Hall set $\hallset\text{.}$ Let $w_{1},\dots,w_{n}$ be the complementary Hall words to the ideal defining $\mathfrak{g}\text{,}$ so that $\mathfrak{g}=\operatorname{span}\{X_{w_{1}},\dots,X_{w_{n}}\}\text{.}$ The exponential coordinates of the second kind

\begin{equation*} \RR^n\to G,\quad (x_1,\dots,x_n) \mapsto \exp(x_nX_{w_{n}})\cdots\exp(x_1X_{w_{1}}) \end{equation*}

are said to be adapted to the Hall set if $w_{n}\gt\ldots\gt w_{1}\text{.}$

###### Definition2.13.

Let $\hallset=\{w_{1},w_{2},\ldots\}$ be a Hall set for a free Lie algebra $\freelie{r}\text{.}$ Let $\polyring{\hallset}:=\polyring{x_{w_{1}},x_{w_{2}},\ldots}$ be the weighted polynomial ring with a countable number of generators, where the weight of the variable $x_{w_{i}}$ is the degree of the Hall word $w_{i}\text{.}$

The above definitions allow identifying each polynomial in a Carnot group compatible with a Hall set $\hallset$ with a unique polynomial in $\polyring{\hallset}\text{.}$ Abnormal polynomials are not completely arbitrary polynomials however, so it will be relevant that the coordinate form of abnormal polynomials is preserved by lifting to higher step groups.

The lift of the covector is the pullback $\tilde{\covector}:=\pi^*\covector\text{.}$ The vector $X$ is lifted by setting the coefficients of the elements $X_{m+1},\ldots,X_n$ to zero. That is, if $X = \sum_{i=1}^{m}x_i\pi(X_i)\text{,}$ then the lift is $\tilde{X} = \sum_{i=1}^{m}x_iX_i\text{.}$

The projection $\pi\colon\mathfrak{f}\to\mathfrak{g}$ is a Lie algebra homomorphism, so the definitions of the lifts imply that

Since $\pi(X_{n})=\ldots=\pi(X_{m+1})=0\text{,}$ the formula of Lemma 2.5 implies that the abnormal polynomial $\abnormalpolynomial{\tilde{X}}{\tilde{\covector}}$ in $G$ only contains monomials with variables $x_1,\ldots,x_m\text{.}$ Moreover by the same formula, the coefficient of each monomial containing only variables $x_1,\ldots,x_m$ is exactly the coefficient of the same monomial in the abnormal polynomial $\abnormalpolynomial{X}{\covector}$ in $H\text{.}$

In a Carnot group $G\text{,}$ there is a natural action $\mathfrak{g}\acts C^\infty(G)$ of the Lie algebra $\mathfrak{g}$ on the space of smooth functions $G\to\RR\text{,}$ defined by extending a vector $X\in\mathfrak{g}$ to a left-invariant vector field $\tilde{X}(g) = (L_{g})_*X$ and using the action of vector fields as derivations on the space of smooth functions. In coordinates this defines an action $\mathfrak{g}\acts \polyring{x_1,\ldots,x_n}$ by derivations also on the space of polynomials.

###### Definition2.15.

Let $\hallset$ be a Hall set for a free Lie algebra $\freelie{r}\text{.}$ Define an action $\freelieaction\colon \freelie{r}\acts\polyring{\hallset}$ via derivations by extending all the actions $\mathfrak{g}\acts \polyring{x_1,\ldots,x_n}$ as follows:

Let $X\in\freelie{r}$ be a vector and $P\in\polyring{\hallset}$ a polynomial. Let $x_{w_{1}},\ldots,x_{w_{k}}$ be all the variables of $P$ and let $G$ be any Carnot group compatible with the Hall set $\hallset$ such that none of the basis elements $X_{w_{1}},\ldots,X_{w_{k}}$ are quotiented away. Fix exponential coordinates $\RR^n\to G$ adapted to the Hall set $\hallset\text{,}$ and let $\iota\colon\polyring{x_1,\ldots,x_n}\into \polyring{\hallset}$ be the identification of polynomials in $G$ in coordinates with polynomials in $\polyring{\hallset}\text{.}$ The action $\freelieaction\colon \freelie{r}\acts\polyring{\hallset}$ is defined by

\begin{equation*} \freelieaction\colon\freelie{r}\times\polyring{\hallset}\to\polyring{\hallset},\quad \freelieaction(X,P) := \iota\Big(\pi(X)\iota^{-1}(P)\Big)\text{,} \end{equation*}

where $\pi\colon \freelie{r}\to\mathfrak{g}$ is the quotient projection, and $\pi(X)\iota^{-1}(P)$ is defined by the action $\mathfrak{g}\acts C^\infty(G)\text{.}$

The well-posedness of Definition 2.15 critically relies on the assumption that all the coordinates on the Carnot groups involved are adapted to a single fixed Hall set. Without this assumption, the coordinate expressions of the actions $\mathfrak{g}\acts\polyring{x_1,\ldots,x_n}$ are in general incompatible with each other.

An extremely useful feature of the abnormal polynomials is how their behavior under the action $\mathfrak{g}\acts\polyring{x_1,\ldots,x_n}$ is linked with the structure of the Lie algebra $\mathfrak{g}\text{.}$ The following result is a rephrasing of Theorem 1.1 of [16] and Proposition 2.22 of [17] using the linearity of the abnormal polynomials $\abnormalpolynomial{Y}{\covector}$ in the vector $Y\text{.}$

In the proof of Theorem 1.1 it will be necessary to further simplify the monomial coefficients of abnormal polynomials given by the formula of Lemma 2.5. The following lemma is a criterion for when these coefficients are as simple as possible for the deg-left-right Hall set.

By the explicit formula of Lemma 2.5 the coefficient of the monomial $x^I$ in the abnormal polynomial $\abnormalpolynomial{w}{\covector}$ is

By Lemma 2.9, the assumption that $v_1\lt\ldots\lt v_\ell\lt w$ and $v_1\geq w'$ guarantees that $vw$ is a deg-left-right Hall word. The second part of the lemma states the bracket $\ad{X_{v_\ell}}^{i_\ell}\cdots \ad{X_{v_1}}^{i_1}X_{w}$ is exactly the Hall basis element $X_{vw}\text{,}$ so

### Subsection2.4Poincaré series

The proof of Theorem 1.1 involves counting the number of monomials of a given degree depending on some subset of the variables $x_{w}$ in $\polyring{\hallset}\text{.}$ A convenient form to manipulate these quantities will be in terms of generating functions, i.e., in terms of Poincaré series.

###### Definition2.18.

Let $W = \bigoplus_{i=0}^\infty W_i$ be a graded vector space with all components $W_i$ finite dimensional. The Poincaré series of the graded vector space $W$ is the formal power series

\begin{equation*} \poincare(t) = \sum_{k=0}^\infty (\dim W_k)t^k\text{.} \end{equation*}

The Poincaré series of a weighted polynomial ring has a rather simple expression as a rational function depending on the weights.

By Ch. V, §5.1, Proposition 1 of [6], the Poincaré series for the weighted polynomial ring is given by the first formula $\poincare(t) = 1/\prod_{k=1}^{n} (1-t^{d_{k}})\text{.}$ Since there are exactly $\dim\freelielayer{r}{k}$ Hall words of degree $k\text{,}$ the second formula follows by grouping terms.