In Experience Profile, I can see what Pattern Card a Contact currently has. How would I review the history of that specific Contact to see the Patterns they have triggered?
You cannot unfortunately.
Pattern Cards are calculated when requested so "on the fly".
You could however intercept pipeline which is responsible for calculating it and after calculation you can save results on custom facet on contact.
This will however not help as it will only show matches when requested not all historical changes.
Maybe you can intercept end visit pipeline and trigger calculating pipeline there and save results on custom facet on contact. This should match your requirements.
I cannot quickly find which pipeline is responsible for calculation. Will dig deeper or maybe somebody will comment this answer meanwhile...