Before getting into the details, I need to make one thing clear. Nested transactions are a lie. They do not exist in SQL Server.
This is part 1 of a three-part series by Gail Shaw. Read the whole thing. Also read Paul Randal:
Nested transactions do not actually behave the way the syntax would have you believe. I have no idea why they were coded this way in SQL Server – all I can think of is someone from the dim and distant past is continually thumbing their nose at the SQL Server community and going “ha – fooled you!!”.
Nested transactions are somebody’s attempt at trolling. They succeeded.