let no-one put asunder."
There's marriage, and then there's sacramental marriage. They're not the same.
The Church doesn't say that annulled marriages were not legally valid marriages.
It just says they didn't constitute a sacrament, or a marriage in the true Christian sense of the term. "These two folks may have been joined together--but it wasn't God who did the joining. Hence, this joining can properly be put asunder."
But if God did the joining, then no, no-one should put that asunder. To do so would violate God's will. The tricky part is determining whether the joining was God's will or not. Let's say it is, but one party insists upon pursuing an adulterous relationship--should the Church just sanction that?
On the other hand, let's say that the adulterer was so psychologically immature that on his/her wedding day, s/he was not genuinely capable of appreciating the sacramental or true nature of marriage, as it is understood by the Church. Well, in the latter case, s/he would have been incapable of being a minister of the sacrament. In Catholic theology, the ministers of the sacrament of marriage are the couple getting married, not the priest. If their consent is defective in some way, then they cannot validly confer the sacrement.
This link gives information on the grounds for nullity according to Canon Law.