So I've been wondering for a while now how one would reject a whole triangle if obscured by another. I've thought about using dot product to test if one triangle is behind another, is the direction from triangle A to triangle B going in the same direction as the camera view?
This would require the pipeline to have knowledge of the camera view, so this isn't the option.
I thought about getting the normal of the triangle using the cross product method and then using dot product the resulting vector with the direction from A to B. This might tell the pipeline if a triangle is behind another, but not if it is obscured.
Something just occurred to me a couple of days ago, and once you mentioned the Geometry shader returning 0 triangles, it drove the idea home for me. I've been messing around with barycentric coordinates for a few weeks now and what they do is determine if a given point is inside a triangle or not. So comparing the depth of the two triangle's vertices and calculating the barycentric coordinates for each vertex in the back triangle and determining if all three are inside the front triangle, you can reject the obscured triangle.
Now the issue is the Geometry shader only knowing about the triangle that is passed in, so again, there is a problem. I'm guessing this type of polygon rejection isn't done here either. I'm going to guess it's somewhere after the vertex shader and before the geometry shader.
All that being said, I actually don't think DX or OGL does this for you. I got this idea from AMD and the primitive culling they added to their Vega ( can't remember if it is also on their RX480,RX470,etc...Polaris ) GPUs.
3D Fundamentals  Geometry shader

 Posts: 4171
 Joined: February 28th, 2013, 3:23 am
 Location: Oklahoma, United States
3D Fundamentals  Geometry shader
If you think paging some data from disk into RAM is slow, try paging it into a simian cerebrum over a pair of optical nerves.  gameprogrammingpatterns.com