When a remote location connection is slow, it can take some time to refresh a remote directory. While refreshing, one can freely browse another directory in other panel. But when refreshing directory is finished, related panel is forcibly focused.
This patch prevents that.