We can set document-css to false to turn off all the embedded style, assuming we aren't relying on some parts of it. Otherwise, yeah, setting it to the CSS variables would be the next thing to try.
Alternatively, we could make our own modified version of pandoc's built-in styles.html, but that is probably overkill if the other approaches work