low light - What would be a better lens, 40 or 50mm prime, for walk around?

I have a Canon EOS Rebel T5i/700D body, an EF-S 18-55 wide angle lens, a 70-300mm telephoto lens, and the kit lens that came with the body.

I am still building my lens collection and am looking for something of a walk-around lens, I'd also like it to double as something fairly fast for low light. Sharpness is key for this one. Because of this my first thought was a prime lens, generally build quality is very important to me so the 50mm f/1.8 "nifty 50" is out. I also prefer EF glass, so I don't have to reacquire my lenses again when/if I decide to go full sized.

All that said, I have it down to two lenses - the EF 50mm f1.4 and the EF 40mm f2.8. What lens would you suggest in my case?

I like the light weight and small size of the 40mm but the 50mm should about mimic an 85mm portrait lens on my APS-C camera.