Periodic and period-doubling vibrations in a gear transmission system subjected to forced excitations with multipiecewise linear functions are investigated in this work by using the incremental harmonic balance (IHB) method. The nonlinear ordinary differential equations that govern vibration of the gear transmission system are formulated by employing the Newton's second law. Analytical results reveal abundant interesting phenomena, including jumps, bifurcations, softening-spring behaviors, and primary, super-harmonic, and subharmonic resonances, which have not been exhibited in existing investigations on nonlinear vibrations of gear transmission systems. Nonlinear phenomena and resonances of the gear transmission system are revealed by considering different numbers of degrees-of-freedom of the system. The bifurcations containing saddle-node, period-doubling, and Hopf bifurcations are observed in frequency response curves. The period-doubling phenomena are characterized by phase-plane diagrams and Fourier spectra. Further, analytical results obtained by the IHB method match very well with those from numerical integration.