The sum of the squares of the roots of $x^2-(p-2)x-(p+1)=0$ is smallest when $p$ equals:

The roots satisfy α + β = $p-2$ and αβ = $-(p+1)$, so α² + β² = $(p-2{)}^2+2(p+1)=p^2-2p+6=(p-1{)}^2+5$. A square is never negative, so the least value 5 occurs at $p=1$. (Setting $p-2=0$ gives 2 and setting $p+1=0$ gives −1 — both fix a coefficient rather than minimise the sum.) At $p=1$ the equation is $x^2+x-2=0$ with roots 1 and −2, whose squares add to 5.