In your scenario, yes you will need to update your Microsoft.IdentityModel.Tokens.Saml and Microsoft.IdentityModel.Xml assemblies to match your version 5.2.4
This of course means you need to ensure Sitecore.IdentityServer.Host.deps.json is synced to reflect your updated assembly versions.
Please test and verify this works. If not, there is probably more ...
You can access IdToken in SecurityTokenValidated or OnRedirectToIdentityProvider event within your ConfigureServices OWIN pipeline.
You may need to persist this IdToken in say Redis to access it elsewhere in your business logic.
We have examples of how this has been done, such as in this post and this one
Sample ConfigureServices snippet, original code on ...
You may want to run the identity server from a command prompt, so you can see what the errors are.
I had to update versions of several assemblies, and update the Sitecore.IdentityServer.Host.deps.json file.
Integrating the SusyemSys.Saml2 into the identity server has some challenges. I ran into these same issues, but eventually got around them with some pain. Of course, there is a chance I some of the changes I made below could use some improvement, especially with respect to the updated dependencies.
You do need to include updated versions of Microsoft....
In my case after removing the custom configuration of Publish:end event resolved the issue.
<handler type="Sitecore.Publishing.HtmlCacheClearer, Sitecore.Kernel" method="ClearCache">