I have the same issue on the right side. It's difficult to find any solutions. Most say that it's harmless so don't worry about it. The thing that bothers me about this, other than the look of it when it slips up there, is certain sexual positions. When she's on top and I'm fully inside, that hip rocking motion that they seem to like forces it to pop inside followed by pressure on it from their weight. If I don't fix it, it'll pop out forcefully also. Very uncomfortable and natural reaction forces me to "jump" even if I know it's about to happen. And then they wonder what happened.
I looked on a couple ball stretching sites and even bought a leather 1" stretcher. The hardest part is getting it on without pinching. It probably takes me a full 5 minutes due to one ball hanging right and the other being so high. I have trouble using with any regularity because it hurts after a bit also. I've considered trying one of the weights also but not sure if it would be any easier. If one of y'all try it, let me know.