No, not the points in the actual game.
I dont think it really goes against you, if you beat someone who had won no other games, maybe because they signed up and never came back to the site, but the other player had beat someone who had won all their other games, then they would get a higher S-B than you. That seems fair to me.
And if you both had the same S-B then you would both go through, presumably!
Of course it counts all the games you have won, each player you beat, their games would decide your S-B.
A bit like when deciding your BKR the other players rating is the deciding factor.
When deciding the S-B, the other players games are taken into account.