This paper proposes a learning method for uncertain two-player games. We focus on the under-explored yet important problem of learning game payoffs by observing actions. We present a fully differentiable module capable of learning small zero-sum games purely from observing the actions of individual players. We demonstrate the effectiveness of the learning method on several security game tasks. The proposed method also leads to potential applications in the domains of reinforcement and deep learning.