The drawing, as intricate as it may be, may not be quite to scale.
Moving the caliper down and angling the adaptor slightly to compensate for rotation as you say, would solve the problem.
I'd take a picture of this with the caliper on the forks where it would need to be mounted to make up the difference, but I cannot be arsed
But as you can see, nothing about this idea is absurd in the slightest.