If you’re talking about tomatoes, the difference is the context, and it isn’t a choice between colloquial vs scientific taxonomy, but between culinary/nutritional vs botany/taxonomy (and). You can talk about either in a colloquial context or a formal context, though generally there isn’t much reason to talk about botany in a colloquial setting.

From a nutritional perspective, mushrooms are generally considered vegetables, too.

afaik vegetable is a purely colloquial term anyway.

I thought you were wrong but I looked it up and I appear to have been mistaken. It makes “tomatoes are fruits, not vegetables” sound nonsensical, as it implies that “vegetable” is a different taxonomical option, when really it’s just a word for objects with a particular collection of traits that are relevant in a different context. What we should he saying is “While tomatoes are not fruit in the food pyramid, taxonomically, they are.” Doesn’t really roll off the tongue, though. Maybe “Tomatoes are vegetables AND fruits!” would solve that?