Project Euler Problem 914
For a given integer R consider all primitive Pythagorean triangles that can fit inside, without touching, a circle with
Solution
Answer: 414213562371805310
Using Euclid’s parametrization, every primitive Pythagorean triple is
$$a=m^2-n^2,\quad b=2mn,\quad c=m^2+n^2$$
with $m>n$, $\gcd(m,n)=1$, and opposite parity.
For a right triangle, the inradius is
$$r=\frac{a+b-c}{2}=n(m-n).$$
Let
$$x=m-n,\qquad y=n.$$
Then
$$r=xy,$$
and the hypotenuse becomes
$$c=(x+y)^2+y^2=x^2+2xy+2y^2.$$
A triangle fits strictly inside a circle of radius $R$ iff its circumradius satisfies
$$\frac c2<R \quad\Longleftrightarrow\quad x^2+2xy+2y^2<2R.$$
So we must maximize
$$xy$$
subject to
$$x^2+2xy+2y^2<2R,$$
with $x$ odd and $\gcd(x,y)=1$.
The continuous optimum occurs at
$$x=\sqrt2,y,$$
giving
$$y_0=\sqrt{\frac{R}{2+\sqrt2}}.$$
For $R=10^{18}$,
$$y_0 \approx 541196100.146.$$
Searching only a small neighborhood around this optimum (the objective falls quadratically away from it) gives the best admissible pair
$$x=765366155,\qquad y=541196602.$$
Check:
$$x^2+2xy+2y^2 = 1999999999995787453 < 2\times10^{18},$$
and
$$r=xy = 765366155\cdot541196602 = 414213562371805310.$$
Therefore,
Answer: 414213562371805310