It would be nice to be able to dismiss a follower to stay are your Respite. This way you do not always have to have a follower with you when you don't want one, but can get a favored follower back again when you need them.
Players could explore on their own if they want to, or go to private vaults with others without having to also have a crowd of NPCs. Without risking permanently losing a favorite follower.