We only show the "Remove this applet" option when the panel controller is opened to prevent the user from accidentally messing up his setup.
However, the "Remove this panel" option, albeit in a submenu, was always there and is even more destructive.