We consider the onset of chaos in a pendulum driven by a Brownian noise environment. The behavior of the Lyapunov exponents of the system as a function of the strength of the coupling to the thermal environment is investigated for two models of thermal coupling (given by the Langevin equation and by random stop-start motion). For sufficiently strong coupling to the environment, the motion is nonchaotic and almost all trajectories are stable but, for each initial velocity and realization of the Brownian noise, an exceptional set of unstable trajectories exists, in analogy to the existence of basin boundaries in deterministically driven systems. The initial points of the unstable trajectories form a fractal set, the dimension of which is zero for coupling to the environment greater than a (temperature-independent) critical value, grows as the environmental coupling is decreased, and equals one at the transition to chaotic behavior.