Disconection from game, on xbox, shortly after joining friends custom games.

Everytime I start up a custom game in survival, (and I would assume creative too) on Xbox One S with my buddy, he looses connection in less than 10 minutes on my server (custom game), and visa versa on his server(custom game). I see hundreds of other people are having the same problem on xbox. My buddy also plays on PC, and he said this has NEVER happened to him, and he has over 800 hours with his other friend on steam. We both run roughly 40-230mps. So internet is not a problem here. I payed for the deluxe edition of the game too. I was very excited to play, but I am now getting discouraged and frustrated due to lagging out of any game i join within 1-10 minutes, and sometimes it doesn't even connect at all. This problem makes tge game unplayable with friends. I love the game so far eith the little time I have gotten to play with it. But due too this game breaking bug, I may have to refund the game until this bug is fixed on xbox for me, and the many other people swarming to reddit to try and find answers. If you would like any other details, I will absolutely provide! I appreciate your time. So here is a rundown for troubleshooting... The problem is not my internet or his (we are fine trying to play any other games together). This is happening in custom games with default survival settings specifically on the, 'Home System' map. When I host, he laggs out. When he hosts, I lag out. Online mode was set to friends only. There are no mods bc we're on console. Other people are running into the same issue, on YouTube and google. My friend said this only happens on xbox. And lastly, there seems to be no correlation or cause, 'in game' that would seem to triger this connection error (its not to many objects or a processing problem). Thank you for your time, I hope yall can fix this, then I could play this amazing game again!