On the complexity of inverting integer and polynomial matrices