On the groups of unitriangular automorphisms of relatively free groups

We describe the structure of the group Un of unitriangular automorphisms of the relatively free group Gn of finite rank n in an arbitrary variety C of groups. This enables us to introduce an effective concept of normal form for the elements and present Un by using generators and defining relations. The cases n = 1, 2 are obvious: U1 is trivial, and U2 is cyclic. For n ≥ 3 we prove the following: If Gn−1 is a nilpotent group then so is Un. If Gn−1 is a nilpotent-by-finite group then Un admits a faithful matrix representation. But if the variety C is different from the variety of all groups and Gn−1 is not nilpotent-by-finite then Un admits no faithful matrix representation over any field. Thus, we exhaustively classify linearity for the groups of unitriangular automorphisms of finite rank relatively free groups in proper varieties of groups, which complements the results of Olshanskii on the linearity of the full automorphism groups AutGn. Moreover, we introduce the concept of length of an automorphism of an arbitrary relatively free group Gn and estimate the length of the inverse automorphism in the case that it is unitriangular.