I need help understanding

While growing up I never really drank milk, at all, I just didn’t like it. But lately I have been trying to and I drank milk for a week everyday, but then my back started to hurt and that was the only change I’ve done for my back to hurt, so to test this theory I stopped drinking milk and then my back eventually stopped hurting. After forgetting about it after a while, I have started to drink milk again, and low and behold, my back started to hurt again. Is this a sort of medical condition or something? I don’t have a clue, all I know is that i can’t be lactose intolerant because I eat dairy products in other foods (Mac n cheese, cheese, hamburger helper, etc.) Btw, I did go to the doctor to get a physical done to see what possibly might be the issue and they said that it could be an issue with where my vertebrae(back bone) is connected to my ilium (hip bone) and the pressure it’s been under, but as said, nothing has changed over the year but me drinking the milk and I suddenly got better after stopping.