This can happen if you have made your SXA Tenant unpublishable. There seems to be some logic in some components that doesn't resolve the Context (master/web) correctly and needs to have the site published.
Simply make your SXA Tenant publishable (remove publishing restrictions) and publish to resolve this issue.
I think I figured it out! Sitecore is merging messages with the same text, so making the message unique to the validation failure fixes the error. Originally I was setting the error message text as follows in the parameters section of the validator item:
Instead of doing this, I am now setting the message within the custom validator itself as follows: