PVC shouldn't be a problem touching your skin. Probably more people have latex allergies than PVC. I think the issues might be with the edge of the PVC being harder than latex and irritating your skin, or with the ability of it to hold on to the other parts.
Yeah, PVC pipe would be extremely difficult to get over the ends of the rods...plus its really stiff, so I don't see how that would be a snug ring type situation on your package...