By using a suitable substitution, bring the equation…
By using a suitable substitution, bring the equation $y'=(ax+by+c)^2$ to a form where the variables can be separated, and then integrate it.
Explanation of Method
Since the right side depends only on the linear combination $ax+by+c$, substitute $v=ax+by+c$ to reduce the equation to the separable Riccati-type form $v'=a+bv^2$, then integrate — the antiderivative is an arctangent when $ab>0$ and a logarithm when $ab<0$.
Step 1: Substitute $v=ax+by+c$
Differentiating $v=ax+by+c$ gives $v'=a+by'$, so $y'=\dfrac{v'-a}{b}$ (for $b\neq0$). Substituting into $y'=(ax+by+c)^2=v^2$:
\[\frac{v'-a}{b}=v^2 \quad\Longrightarrow\quad v'=a+bv^2.\]
Step 2: Separate variables
\[\frac{dv}{a+bv^2}=dx.\]
Step 3: Integrate — case $ab>0$
Factoring out $a$ and substituting $u=\sqrt{b/a}\,v$ gives the standard arctangent integral:
\[\frac{1}{\sqrt{ab}}\arctan\!\left(v\sqrt{\tfrac{b}{a}}\right)=x+C \quad\Longrightarrow\quad v=\sqrt{\tfrac{a}{b}}\,\tan\!\big(\sqrt{ab}\,(x+C)\big).\]
Step 4: Integrate — case $ab<0$
Writing $b=-\beta$ with $\beta>0$, the integral becomes logarithmic:
\[\frac{1}{2\sqrt{a\beta}}\ln\left|\frac{1+v\sqrt{\beta/a}}{1-v\sqrt{\beta/a}}\right|=x+C.\]
Step 5: Back-substitute
Replacing $v=ax+by+c$ and solving for $y$ in the case $ab>0$:
\[\boxed{\,y=\frac1b\left[\sqrt{\tfrac{a}{b}}\,\tan\!\big(\sqrt{ab}\,(x+C)\big)-ax-c\right],\qquad ab>0.}\]
For $ab<0$ the implicit form is
\[\boxed{\,\ln\left|\dfrac{1+(ax+by+c)\sqrt{-b/a}}{1-(ax+by+c)\sqrt{-b/a}}\right|=2\sqrt{-ab}\,(x+C),\qquad ab<0.}\]
(If $b=0$, the equation is already separable as $y'=(ax+c)^2$.)
Result
Final answer:
\[\boxed{\,v'=a+bv^2,\ \ v=ax+by+c\ \Longrightarrow\ y=\dfrac1b\!\left[\sqrt{\tfrac{a}{b}}\tan\big(\sqrt{ab}(x+C)\big)-ax-c\right]\ (ab>0),\ \text{or the log form } (ab<0).}\]