List Manager and Email Experience Manager serve two different functions. Therefore, yes you can use List Manager without EXM.
List Manager's responsibility is simple: to create lists of Contacts. It does so, either by Contact Lists (manual lists) or Segmented Lists (rule-based lists) of Contacts.
List Manager does not care about Email Experience Manager and is not aware of its existence.
On the other hand, Email Experience Manager's sole responsibility is to create, merge, and distribute email to contacts. It uses lists from List Manager to maintain Contact Lists so that Email Experience Manager does not need to perform that responsibility.
EXM heavily relies upon List Manager because you can't send email without knowing who to send to.
In Sitecore versions prior to Sitecore 9, List Manager is natively part of the Sitecore platform as a function within the xDB marketing tools. However, because EXM was the only module that really made significant use of the List Manager module, it was heavily associated with Email Experience Manager. List Manager has significant issues in Sitecore versions prior to 8.2.6, and many hotfixes are available from Sitecore Support if you are working with versions prior.
In Sitecore Experience Platform v9+, List Manager is significantly improved and many of the issues in prior versions of Sitecore have been resolved. Additionally, Email Experience Manager is also now part of the platform natively (like List Manager) but is not required in order to utilize List Manager.