This is a long known issue with Solr, where if a field is stored / has analyzers / is tokenized, it returns incorrect results. This is vaguely mentioned on the documentation here.
It was blogged about years ago by Rich Seal here who said that you can add a
copyField with parameters
indexed="true" stored="false" and then sort by that field.
This method works well, however it is somewhat of a pain to modify the managed schema manually for this, and then there's always the likelihood that someone will click "Populate managed schema" in Sitecore and overwrite it.
So I have updated the SmartSolrSchema module to include a sortable date computed field that you can use to quickly add any date fields that you need to sort by, which works nicely with the normal "Populate managed schema" in Sitecore.