Find all affine transformations that map the hyperbola…
Find all affine transformations that map the hyperbola $\dfrac{x^2}{a^2}-\dfrac{y^2}{b^2}=1$ onto itself.
Explanation of Method
All affine maps that map a canonical hyperbola onto itself.
Step 1: Reduction
The centre is fixed, so the map is linear. It also permutes the asymptotes. With $X=\tfrac xa$, $Y=\tfrac yb$ the curve is $X^2-Y^2=1$, and the map must preserve the form $X^2-Y^2$. These are the (pseudo-orthogonal) matrices
\[ \pm\begin{pmatrix}\cosh t&\sinh t\\ \sinh t&\cosh t\end{pmatrix},\qquad \pm\begin{pmatrix}\cosh t&-\sinh t\\ \sinh t&-\cosh t\end{pmatrix},\qquad t\in\mathbb R. \]
Indeed, $(X\cosh t+Y\sinh t)^2-(X\sinh t+Y\cosh t)^2=X^2-Y^2$.
Step 2: Back to $x,y$
\[\text{I: }\ x=\pm\left(x'\cosh t+\tfrac ab\,y'\sinh t\right),\quad y=\pm\left(\tfrac ba\,x'\sinh t+y'\cosh t\right)\]
\[\text{II: }\ x=\pm\left(x'\cosh t-\tfrac ab\,y'\sinh t\right),\quad y=\pm\left(\tfrac ba\,x'\sinh t-y'\cosh t\right)\]
Here the same sign is taken in both lines. For $t=0$ we get the identity, the central symmetry $(x,y)\mapsto(-x,-y)$, and the reflections in the $x$-axis and in the $y$-axis.
Family I with sign $+$ maps $P(a\cosh s,b\sinh s)$ to $(a\cosh(s+t),b\sinh(s+t))$. It moves points along their branch and keeps each asymptote fixed as a line. The sign $-$ interchanges the branches.
Result
\[\boxed{\,\text{I: }\ x=\pm\left(x'\cosh t+\tfrac ab\,y'\sinh t\right),\quad y=\pm\left(\tfrac ba\,x'\sinh t+y'\cosh t\right)\,}\]
\[\boxed{\,\text{II: }\ x=\pm\left(x'\cosh t-\tfrac ab\,y'\sinh t\right),\quad y=\pm\left(\tfrac ba\,x'\sinh t-y'\cosh t\right)\,}\]
Remark
Correction: the families given in the source (with $a\cosh t$, $b\sinh t$ as coefficients) do not reduce to the identity at $t=0$: they give $x=a x'$, which does not preserve the hyperbola unless $a=1$. The correct coefficients are $\cosh t$, $\sinh t$ with the factors $\tfrac ab$ and $\tfrac ba$, exactly as for the ellipse in Problem 157.