In mathematics, the **automorphism group** of an object *X* is the group consisting of automorphisms of *X* under composition of morphisms. For example, if *X* is a finite-dimensional vector space, then the automorphism group of *X* is the group of invertible linear transformations from *X* to itself (the general linear group of *X*). If instead *X* is a group, then its automorphism group is the group consisting of all group automorphisms of *X*.

Especially in geometric contexts, an automorphism group is also called a symmetry group. A subgroup of an automorphism group is sometimes called a **transformation group**.

Automorphism groups are studied in a general way in the field of category theory.

If *X* is a set with no additional structure, then any bijection from *X* to itself is an automorphism, and hence the automorphism group of *X* in this case is precisely the symmetric group of *X*. If the set *X* has additional structure, then it may be the case that not all bijections on the set preserve this structure, in which case the automorphism group will be a subgroup of the symmetric group on *X*. Some examples of this include the following:

- The automorphism group of a field extension is the group consisting of field automorphisms of
*L*that fix*K*. If the field extension is Galois, the automorphism group is called the Galois group of the field extension. - The automorphism group of the projective
*n*-space over a field*k*is the projective linear group^{[1]} - The automorphism group of a finite cyclic group of order
*n*is isomorphic to , the multiplicative group of integers modulo*n*, with the isomorphism given by .^{[2]}In particular, is an abelian group. - The automorphism group of a finite-dimensional real Lie algebra has the structure of a (real) Lie group (in fact, it is even a linear algebraic group: see below). If
*G*is a Lie group with Lie algebra , then the automorphism group of*G*has a structure of a Lie group induced from that on the automorphism group of .^{[3]}^{[4]}^{[a]}

If *G* is a group acting on a set *X*, the action amounts to a group homomorphism from *G* to the automorphism group of *X* and conversely. Indeed, each left *G*-action on a set *X* determines , and, conversely, each homomorphism defines an action by . This extends to the case when the set *X* has more structure than just a set. For example, if *X* is a vector space, then a group action of *G* on *X* is a *group representation* of the group *G*, representing *G* as a group of linear transformations (automorphisms) of *X*; these representations are the main object of study in the field of representation theory.

Here are some other facts about automorphism groups:

- Let be two finite sets of the same cardinality and the set of all bijections . Then , which is a symmetric group (see above), acts on from the left freely and transitively; that is to say, is a torsor for (cf. #In category theory).
- Let
*P*be a finitely generated projective module over a ring*R*. Then there is an embedding , unique up to inner automorphisms.^{[5]}

Automorphism groups appear very naturally in category theory.

If *X* is an object in a category, then the automorphism group of *X* is the group consisting of all the invertible morphisms from *X* to itself. It is the unit group of the endomorphism monoid of *X*. (For some examples, see PROP.)

If are objects in some category, then the set of all is a left -torsor. In practical terms, this says that a different choice of a base point of differs unambiguously by an element of , or that each choice of a base point is precisely a choice of a trivialization of the torsor.

If and are objects in categories and , and if is a functor mapping to , then induces a group homomorphism , as it maps invertible morphisms to invertible morphisms.

In particular, if *G* is a group viewed as a category with a single object * or, more generally, if *G* is a groupoid, then each functor , *C* a category, is called an action or a representation of *G* on the object , or the objects . Those objects are then said to be -objects (as they are acted by ); cf. -object. If is a module category like the category of finite-dimensional vector spaces, then -objects are also called -modules.

Let be a finite-dimensional vector space over a field *k* that is equipped with some algebraic structure (that is, *M* is a finite-dimensional algebra over *k*). It can be, for example, an associative algebra or a Lie algebra.

Now, consider *k*-linear maps that preserve the algebraic structure: they form a vector subspace of . The unit group of is the automorphism group . When a basis on *M* is chosen, is the space of square matrices and is the zero set of some polynomial equations, and the invertibility is again described by polynomials. Hence, is a linear algebraic group over *k*.

Now base extensions applied to the above discussion determines a functor:^{[6]} namely, for each commutative ring *R* over *k*, consider the *R*-linear maps preserving the algebraic structure: denote it by . Then the unit group of the matrix ring over *R* is the automorphism group and is a group functor: a functor from the category of commutative rings over *k* to the category of groups. Even better, it is represented by a scheme (since the automorphism groups are defined by polynomials): this scheme is called the **automorphism group scheme** and is denoted by .

In general, however, an automorphism group functor may not be represented by a scheme.