You think Elixir is not floral? Smells VERY floral to me. Smells a lot like a white peony, although I don't think it lists that note specifically. The neroli on top of the fragrance is floral.
By the way, I LOVE Elixir. I think it's the best of the regular Penhaligon's line.
Interestingly, when I wear it my wife says it smells feminine. So I told her she should try wearing it, since it's "unisex." She put it on one day. She tells me on her it smells "masculine." Bizarre, right? On a man, it smells feminine, and on a woman, it smells masculine. That's her story and she's sticking with it. Maybe that's one of the problem with the whole concept of "unisex." Ultimately, I told her that I'm going to wear it now and then, whether she likes it or not, because I like it too much to let it go to waste.
I like Elixir, but I don't think of it as floral, despite the floral notes.
The masculine/feminine thing is odd. Basenotes lists it as masculine, but for a while The Perfumed Court described it as the "best women's Penhaligon's" scent. I e-mailed Penhaligon's and they said it was a men's fragrance. Now The Perfumed Court describes it as unisex.
So I'm with kingfisher - wear it if you like it.