Stopping Brexit would leave Britain fractured: finance minister