Key Generation Using Generalized Pell's Equation in Public Key Cryptography Based on the Prime Fake Modulus Principle to Image Encryption and Its Security Analysis

RSA is one among the most popular public key cryptographic algorithm for security systems. It is explored in the results that RSA is prone to factorization problem, since it is sharing common modulus and public key exponent. In this paper the concept of fake modulus and generalized Pell's equation is used for enhancing the security of RSA. Using generalized Pell's equation it is explored that public key exponent depends on several parameters, hence obtaining private key parameter itself is a big challenge. Fake modulus concept eliminates the distribution of common modulus, by replacing it with a prime integer, which will reduce the problem of factorization. It also emphasizes the algebraic cryptanalysis methods by exploring Fermat's factorization, Wiener's attack, and Trial and division attacks.