Although the view that mystical experiences are ineffable is present in many mystics in both East and West, mystical philosophies are rare in the West, where "scientific," common sense, or ordinary language philosophies dominate. One exception is Wittgenstein's Tractatus which holds that there are ineffable mystical things about which one "must be silent". Indeed, Wittgenstein, throughout his career admired India's Tagore, who held mystical views. However, many scholars agree with Nieli, who argues that Wittgenstein, in his "Private Language Argument" replaces his earlier Tractatus-mysticism with the view that all genuine language and experience is public. The paper argues that Nieli is incorrect. Mysticism is still present in Wittgenstein's Philosophical Investigations but in a more subtle form than that of the Tractatus. Specifically, the mysticism of Tractatus is what has been called an "autobiographical" mysticism while that of Wittgenstein's Philosophical Investigations is what has been called a "radiant" mysticism. Given Wittgenstein's longstanding admiration for Tagore, the persistence of mysticism into Wittgenstein's Philosophical Investigations should be no surprise. The paper argues that rather than denying the mystical, Wittgenstein's private language argument actually spell out the place for the ineffable more precisely than had been done in the Tractatus, thereby helping to explain Wittgenstein's longstanding admiration for Tagore.