The problem is that too many people look at this stuff in a bubble.
You've taken two games out of 16 for each team. That's two games out of 48 total.
You have to look at the entire body of work for each. Patriots were good all year long. Giants were above average in a tough division. We were awful for a good part of the season and then possible average for the rest. Super Bowl contenders don't play as poorly as we have all season. The Patriots might play poorly here and there but overall they know how to play and win as a collective unit.
We do not. And that's why we're not as close to a Super Bowl as people might think.