First extend the hypothesis to all polynomials. If p(x)=∑j=0mcjxj, linearity of the integral gives ∫01p(x)f(x)dx=∑j=0mcj∫01xjf(x)dx=0. So ∫01pf=0 for every polynomial p.
By the Weierstrass approximation theorem, there are polynomials (pk) with pk→f uniformly on [0,1]: ∥pk−f∥∞→0. Put M=max[0,1]∣f∣ (a continuous function on a compact interval attains its maximum). Then
∫01pkf−∫01f2=∫01(pk−f)f≤∥pk−f∥∞⋅M⟶0.
Since each ∫01pkf=0, passing to the limit gives ∫01f(x)2dx=0. As f2 is continuous and nonnegative, f2≡0: if f(c)2=η>0 at some c, continuity would force f2≥η/2 on an interval, making the integral positive. Hence f≡0.