My vote for 540i, it’s a lighter car (than X5), will feel nimble without the need for gas gussling v8.
5 series interior build quality, comfort and choice or materials in just a different world, same at X5. I’d hesitate to go for m340i just for the engine. (I had a 335i before, have had the pleasure to drive two 5s, both with the in-line 6 turbos). Multi contour seats and luxury seating is not offered on 3 series, switching between x5 and your second car on a day to day basis will feel seamless with a 5 series.
If money isn’t an issue go M550i, it’s not necessary in my opinion since you pretty much get all bells and whistles in 540i except the engine (which is more efficient with the 48v mild hybrid introduction with LCI).
Also, getting a 3 series would likely rule that out for taking two toddlers in the future, seating may be fine, the trunk space is limited, resulting in sole reliance on your X5. I personally like more options when I have two expensive vehicles.
Hear you on driving sedans vs suvs, test drove an x3, not a fan at all, it’s like sitting in a bus. (Though I hear the Macan drives like a car!