I think it's because veal is fed with lots of rich, fatty foods - like cream and milch - before it's killed to make the meat tender, whereas lamm is just sent off to the slaughterhouse as soon as it's old enough.
It doesn't bother me too much - veal is delicious!! :)
