Humans can be identified through the way they walk, according to medical studies, each person has an almost unique way of walking. Over the past decade, numerous gait recognition approach has been presented by researchers through which human can be identified, but gait recognition can be affected by a variety of wearable objects such as clothes, carry bags, shoes and others. The silhouette-image with invariant features can be obtained if the wearer objects are removed from it. The Generator Advisory Network (GAN) model is used to remove covariate objects from the silhouette images. The main challenge of this approach to make input-output pairs of frames to train GAN model, for this silhouette image has to mapped with another silhouette image with same pose. Since pose analysis has been done in Pose Energy Image (PEI) approach so PEI is used to label the frames with a specific key pose. After labelling silhouette frames, pair them with the recognition of same pose and train the GAN model. The trained GAN model used to generate a new gait invariant feature which is named as Covariate Factors Omitted Silhouette Image (CFOSI). For cross-validation of the proposed approach, three publically available datasets are used, where proposed approach found 5.50, 4.71 and 17.88 percentage more accurate results with respect to CASIA B, CASIA C and TUM-GAID in case of Gait Energy Image (GEI). We can conclude that after the omission of covariate objects from silhouette images our approach performance is more accurate than others.