Always keep files together in the installation directory.

To remove the software, use the uninstall option, available via for instance the Windows start menu. Please remove manually eventual remaining files like *.ini and *.his files and then remove eventually the empty directory.