Enable Compiler Warnings, Set Werror on CI

This makes our build a lot more stricter in terms of things allowed. During going through all the warnings and fixing them I indeed found a few that could very well be (have been) bugs or lead to odd behavior.

Of course this now adds lots of static_casts in different places, but mostly because the underlying classes were written quite carelessly. In some places I already refactored, in others we can do that over time.