I think I like my exes brother

My ex boyfriend’s brothers moved in next door to my apartment complex and I used to go to school with one of them and we were good friends. I’ll call him J. J and I were in the same friend group and hung out every day and I thought he was cute before I went out with his brother and people from our friends group used to tell me that he liked me but nothing every happened between us but afterwards we didn’t really speak too much especially because he had moved schools. Now like five years after breaking up with his brother J lives next door and we recently started talking and hanging out a little bit we have gone for car rides a few times at night when he gets home from work and we text each other couple times a day not too many, nothing has happened between us but Idk if his brother (my ex) knows how close we have gotten because I feel like maybe he will not be okay with us hanging out. He might think that we like each other or that I want revenge or something since he started going out with someone that used to be my best friend but that is not the case. I love hanging out with J, we laugh and talk about anything and we have known each other for the longest time but we have never tried to be physical. When he moved in I found out he used to work with my dad so they know each other already and my mom gets along with him well too. I think we both see each other as just friends for now but for some reason I want to try to be more than that and at least start getting physical with him when we hang out but I’m not sure how to start. Whenever we are talking we do keep our distance from each other but sometimes we do get close or he sits too close beside me. He’s really funny and our energy just match and we’re both Scorpios I’m not sure if that’s as good thing tho. I don’t know if he doesn’t want to make a move because obviously I was with his brother and there’s bro code. But I want to try to see if there can be a connection between us and I’m not sure how to initiate it. I get that it may look bad that I was with someone and now I want to try with his brother but idk I just have a different thing going on with J that I never felt with my ex and it’s better and not negative. How can I get close to J without making the situation uncomfortable or ruining our friendship?!! :/