© Author(s) 2019. Hysteresis behavior widely exists in the transmission process of harmonic drives. Eliminating the hysteresis effect is highly desired in the high-precision mechanical transmission, which results in challenges in the control design. This paper aims to improve the tracking accuracy of the motor-harmonic drive serial system. Firstly, a modified Bouc-Wen model based on uniform smooth approximating function is applied to describe the hysteresis behavior of the harmonic drive. By using coordinate transformation and accurate state feedback linearization, we then obtain the mathematical model of the serial system of the motor-harmonic drive. Finally, the reference trajectory is tracked by a compound optimal controller that is based on a linear quadratic regulator. Simulation results show that compared with the disturbance observer-based control (DOBC) using a linear observer, the new compound optimal controller in this paper presents a smoother control signal with the elimination of large amount of high-frequency oscillations. Furthermore, the relative error in the steady state tracking tends to approach to zero and no cyclic fluctuations appears. With the employing of optimal control, the output of the harmonic drive can trace more complex trajectory.