This work concerns the analysis of electromagnetic dispersive media modelled by generalized Lorentz models. More precisely, this paper is the second of two articles dedicated to the long time behaviour of solutions of Maxwell's equations in dissipative Lorentz media, via the long time decay rate of the electromagnetic energy for the corresponding Cauchy problem. In opposition to the frequency dependent Lyapunov functions approach used in [Cassier, Joly, Rosas Martínez, Z. Angew. Math. Phys. 74 (2023), 115], we develop a method based on the spectral analysis of the underlying non-self-adjoint operator of the model. Although more involved, this approach is closer to physics, as it uses the dispersion relation of the model, and has the advantage to provide more precise and more optimal results, leading to distinguish the notion of weak and strong dissipation.