There should be a "disable" button, like in the theme because currently, the user/groups configuration override the one in production, on a live set up.
Also, you should on this screen add a warning:
"Use this feature for development phase. You should make sure you disabled it when delivering the plugin to production, otherwise you risk either to break your authentication system, or to delete all the created users and groups and replace them by the one defined here if you let the "Always" policy.
Note: in the future we will make it possible to enable/disable users and groups separately.