Cannot travel to respite since I placed three portals near my cairn/base

So, ever since I build three portals (two worlds & one estate card) I cannot travel to my respite anymore if they are all open. It seems like after 5-10 minutes I can (usually) travel to respite again, but in the meantime I am locked out.

When I click travel to respite on the map, the game crashes and when I travel to respite from the main menu the game gives me the following error: "Player already in session"

Note: I do not know that building the three portals is causing this issue, all I know is the issue started happening when I built them and does not seem to happen if I do not open more than one portal in my respite realm. Also when I can finally travel back to my realm, only the portal to the realm I was in is still open.