Iterative algorithms are widely applied in reliability analysis and design optimization. Nevertheless, phenomena of failed convergence, such as periodic oscillation, bifurcation, and chaos, are oftentimes observed in iterative procedures of solving some nonlinear problems. In the present paper, the essential causes of numerical instabilities including periodic oscillation and chaos of iterative solutions are revealed by the eigenvalue-based stability analysis of iterative schemes. To understand and control these instabilities, the stability transformation method (STM), which is capable of tackling numerical instabilities of iterative algorithms in reliability analysis and design optimization, is proposed. Finally, several benchmark examples of convergence control of PMA (performance measure approach) for probabilistic analysis and the SORA (sequential optimization and reliability assessment) for reliability-based design optimization (RBDO) are presented. The observations from the benchmark examples indicate that the STM is a promising approach to achieve convergence control for iterative algorithms in reliability analysis and design optimization.