But another “problem” is that you don’t know if the compiled program you use is actually based on the open source code or if the developer merged it with some shady code no one knows about.

Actually, there is a Debian project working on exactly that problem, called reproducible builds

https://wiki.debian.org/ReproducibleBuilds