Just because you had stronger erections with more rest doesn't necessarily mean that it's better for gains. But yea, if it's getting in the way of sex than I sure as hell would cut down too
Another thing, you can't help but think even if the fatigue just temporarily decreases the erection power, Penis Enlargement'ing to that state over a long period of time might have some permanent effects.