Predicting the gender of users in social media has aroused great interests in recent years. Almost all existing studies rely on the the content features extracted from the main texts like tweets or reviews. It is sometimes difficult to extract content information since many users do not write any posts at all. In this paper, we present a novel framework which uses only the users' ids and their social contexts for gender prediction. The key idea is to represent users in the embedding connection space. A user often has the social context of family members, schoolmates, colleagues, and friends. This is similar to a word and its contexts in documents, which motivates our study. However, when modifying the word embedding technique for user embedding, there are two major challenges. First, unlike the syntax in language, no rule is responsible for the composition of the social contexts. Second, new users were not seen when learning the representations and thus they do not have embedding vectors. Two strategies circular ordering and incremental updating are proposed to solve these problems. We evaluate our methodology on two real data sets. Experimental results demonstrate that our proposed approach is significantly better than the traditional graph representation and the state-of-the-art graph embedding baselines. It also outperforms the content based approaches by a large margin.