Always unload model before a new load

Since the count is known during the load we can also preserve some space
to improve performance a bit.