.
B, Middle Mouse Button.
Alternatively:
This removes what is in the 'box' from the selection.
Thank you !!