Taken from https://github.com/coonrad/Debian-Xfce4-Minimal-Install, which is a tool that will do most of this for you, if you want

1. Do normal install, but do not install xfce4, or any desktop
2. Following does a base install, using slim for the login

   1. apt install xfce4 libxfce4ui-utils thunar xfce4-appfinder xfce4-panel \
      xfce4-session xfce4-settings xfce4-terminal xfconf xfdesktop4 \
      xfwm4 slim

3. You may install whatever else you want/need from this point on. See the original article for more details.