There are varying false premises behind the rigorism --
-- Pleasure is evil (false) ->Puritanism/Manicheism/rigorism
-- ONLY the primary end of marriage (or ONLY the primary and secondary ends of marriage) are licit for relations (pleasure is not one of these). This is different from the first one because it doesn't say pleasure is evil, just that it's not allowed when it comes to the activities ordered toward procreation
BOTH THESE ARE FALSE
Now, it's certainly easy for us, laboring under Original Sin, to have varying degrees of disorder, where the pleasure takes on more of a role than it should/would in our unfallen state, and one might argue that this disorder constitutes anywhere from a mere imperfection to venial sin, but there is not mortal sin unless the primary/secondary ends are excluded deliberately. Even if one doesn't intellectually consider them at all during the relations, as long as one has a persistent VIRTUAL intention, there's no grave sin.