You can mitigate both problems vastly with design - especially (all kinds of) noise levels, which were an afterthought with Concord (which was still used 27 years).

The sonic booms alone don’t need to be so huge & take do much energy.

Still don’t think we need such (or that many) planes tho - but if we (for some reason) do & there is negligible difference to subsonic ones, why not.