Czuch Chuckers: That is correct.

If you are 3-4 down and double, your opponent should refuse the double even if only slightly behind, since that is a chance to even up the game again (rather than continue the current game at a disadvantage).

Hence it is important to double before the game gets going.