The Beta Function

Split the integral at $t = 1/2$.

On $(0, 1/2]$ we have $\frac12 \leq 1-t < 1$, so $(1-t)^{q-1}$ is trapped between the positive constants $c_1 = \min(1, 2^{1-q})$ and $c_2 = \max(1, 2^{1-q})$.

Hence $c_1 t^{p-1} \leq t^{p-1}(1-t)^{q-1} \leq c_2 t^{p-1}$, and the comparison test reduces the convergence of $\int_0^{1/2} t^{p-1}(1-t)^{q-1}\,dt$ to that of $\int_0^{1/2} t^{p-1}\,dt$.

$\int_\delta^{1/2} t^{p-1}\,dt = \frac{(1/2)^p - \delta^p}{p}$ for $p \neq 0$, which converges to $\frac{2^{-p}}{p}$ as $\delta \to 0^+$ when $p > 0$.

For $p = 0$ it is $-\ln(2\delta) \to \infty$, and for $p < 0$ it is $\frac{\delta^p}{|p|} \to \infty$.

So $\int_0^{1/2} t^{p-1}\,dt$ converges exactly when $p > 0$.

The end $t = 1$ is symmetric. With $s = 1-t$, $\int_{1/2}^{1-\varepsilon} t^{p-1}(1-t)^{q-1}\,dt = \int_{\varepsilon}^{1/2} (1-s)^{p-1} s^{q-1}\,ds$ (a legitimate $C^1$ substitution on a compact interval), so the same argument with $p$ and $q$ swapped shows $\int_{1/2}^1 t^{p-1}(1-t)^{q-1}\,dt$ converges exactly when $q > 0$.

On the middle piece $[\delta, 1-\delta]$ the integrand is continuous, so the integral is always finite. Hence the improper integral $B(p,q)$ converges exactly when $p > 0$ and $q > 0$ — the divergences at the two ends cannot cancel, since the integrand is positive.

Problem 1 gives convergence for $p, q > 0$, so we may substitute $u = 1-t$ ($du = -dt$) in the truncated integral $\int_\delta^{1-\varepsilon} t^{p-1}(1-t)^{q-1}\,dt$ (a $C^1$ substitution on a compact interval).

The interval flips: $t = \delta$ gives $u = 1-\delta$ and $t = 1-\varepsilon$ gives $u = \varepsilon$, so $\int_\delta^{1-\varepsilon} t^{p-1}(1-t)^{q-1}\,dt = \int_{1-\delta}^{\varepsilon} (1-u)^{p-1}u^{q-1}(-du) = \int_{\varepsilon}^{1-\delta} u^{q-1}(1-u)^{p-1}\,du$.

Sending $\delta, \varepsilon \to 0^+$, convergence from problem 1 turns the two sides into $B(p,q)$ and $B(q,p)$, so $B(p,q) = B(q,p)$. Flipping the curve $t^{p-1}(1-t)^{q-1}$ left-right leaves the area underneath unchanged.

Do not apply parts directly to the improper integral.

On the truncated interval $[\delta, 1-\varepsilon]$ ($0 < \delta < 1-\varepsilon < 1$) the integrand is smooth, so apply parts to the truncated integral of $B(p+1,q) = \int_0^1 t^p(1-t)^{q-1}\,dt$ with $u = t^p$, $dv = (1-t)^{q-1}\,dt$.

Then $du = p t^{p-1}\,dt$ and $v = -\frac{(1-t)^q}{q}$.

The truncation is needed because for $0 < p < 1$ or $0 < q < 1$ the derivative is singular at the endpoints.

On the truncated interval the boundary term is $\left[-\frac{t^p(1-t)^q}{q}\right]_{\delta}^{1-\varepsilon} = -\frac{(1-\varepsilon)^p \varepsilon^q}{q} + \frac{\delta^p (1-\delta)^q}{q}$.

Since $p, q > 0$, $t^p (1-t)^q \to 0$ at both ends: $\delta^p \to 0$ as $\delta \to 0^+$ and $\varepsilon^q \to 0$ as $\varepsilon \to 0^+$, so the boundary term tends to $0$.

Sending $\delta, \varepsilon \to 0^+$ and using convergence from problem 1 gives $B(p+1,q) = \frac{p}{q}\int_0^1 t^{p-1}(1-t)^q\,dt = \frac{p}{q}B(p,q+1)$.

Meanwhile, multiplying the integrand of $B(p,q) = \int_0^1 t^{p-1}(1-t)^{q-1}\,dt$ by $1 = t + (1-t)$ gives $\int_0^1 t^p(1-t)^{q-1}\,dt + \int_0^1 t^{p-1}(1-t)^q\,dt$, that is, $B(p+1,q) + B(p,q+1)$.

From the second step, $B(p,q+1) = \frac{q}{p}B(p+1,q)$; substituting gives $B(p,q) = B(p+1,q) + \frac{q}{p}B(p+1,q) = \frac{p+q}{p}B(p+1,q)$. Rearranging yields $B(p+1,q) = \frac{p}{p+q}B(p,q)$.

A note (an idea-level sketch, not a proof): the bridge to the gamma function, $B(p,q) = \frac{\Gamma(p)\Gamma(q)}{\Gamma(p+q)}$, follows the standard sketch — write $\Gamma(p)\Gamma(q)$ as a double integral over $(0,\infty)^2$; the polar substitution $t = r^2\sin^2\theta$, $s = r^2\cos^2\theta$ then separates the radial part into $\Gamma(p+q)$ and the angular part into $B(p,q)$. The full justification, including Fubini's theorem on the unbounded domain, goes beyond this scene, so we take the relation as the bridge, not as a proved theorem here.