I drive in sport mode all the time! Really though, if I know I can make the trip in EV only I'll do so as it is my preferred driving mode. However, when I know I'll be kicking on the ICE for one reason or another I just put it into Sport mode. Why, you might ask?
Well, 'hybrid' mode still kills the battery pretty quickly and is OK in some circumstances. It lacks regenerative braking, instead opting for "coast" mode when you take your foot off the accelerator. Sure, I can push the shifter over to 'manual' or tap the downshift button on the wheel to get some regen, but it's not much. Plus, after downshifting you either need to 1. stop, or 2. upshift until the car goes back into automatic mode. So for me it's not a good experience. Where I live hybrid mode doesn't make sense to use for me.
'Hold' mode is also just OK. I dislike the coast and regen options just as I did with hybrid, but you of course lose the benefit of using the battery. I'm only ever in 'hold' mode when the battery is completely flat, or if I'm on the highway and not using charge mode. I should say it's the default mode I use when driving on the highway since it is the most fuel efficient for long drives.
'Charge' mode is also something I only use when driving long distances where I know I'll want to have a full battery (city center for example), but won't have the opportunity to charge from the grid. Even here regen when coasting is poor.
Sport mode is in effect 'hold' mode, but with a healthy boost from the battery. So we get decent acceleration in essentially all conditions, from pulling away from a stoplight to passing in traffic. There's a nice fat range of torque available, and while it's not as fast as an S4, it feels closer to a GTI than a Prius. The off pedal regen is better than any other mode, and I find will keep the battery charged even better than standard hold mode - all while getting me better mileage along the way. The engine still shuts off when coasting, stopped, or when only using very small throttle inputs. If I want to slow down a little bit, instead of needing to touch the brakes, I can just lift off the throttle a little. When driving on a highway through the city where speeds go from 60mph - 0mph back to 60mph frequently, this is a great way to start/stop again with minimal brake usage.
Anyway, it of course depends on where you live and the roads you drive on, so if you're mostly on the highway or mostly crawling around the city it's not very useful. But if you find yourself in-between I think you should give it a shot, you might be surprised.