Ultrametricity is a fundamental mathematical concept that describes spaces with the metric satisfying the strong triangle inequality and the induced topology of hierarchical architectures. Due to the violation of the Archimedes' axiom of measurement in such a metric, such spaces are also called non-Archimedean. Examples can be found in taxonomy, where phylogenetic trees are ultrametric, geography for measuring landscape complexity, and physics, where complex systems have intrinsically an ultrametric structure. The Noble Prize Giorgio Parisi demonstrated this within the theory of spin glasses, where the overlap between spins exhibits ultrametricity, with the mathematical solution given by the full replica symmetry breaking. In the talk, we discuss the last two decades progress in application of ultrametric models in physics of complex systems. Particular attention will be paid to the model of ultrametric diffusion with a sink, which allowed us to describe the process of CO- rebinding kinetics in Myoglobin; and the ultrametric properties of periodic orbits in a Hamiltonian system with fully chaotic dynamics.