In this paper, we introduced a concept of cascaded phase-truncated Fourier transforms (CPTFTs), which is a nonlinear iterative operator. Meanwhile, an optical/digital hybrid system was presented to implement the operator of CPTFTs. Thereafter, by adopting CPTFTs as main processing unit, we have done some research on optical one-way cryptosystem: an optical Hash function (O-Hash) and a keyed optical Hash function (K-O-Hash) were put forward respectively, which can be applied to message integrity check and authentication. Both of the O-Hash and K-O-Hash can transform an arbitrary-length pre-encoded message (without or with a secret key) into a fixed-length output by means of taking a two-step one-way encryption process. Finally, a set of numerical experiments were carried out to evaluate the performance of avalanche effect and collision resistance of the O-Hash and K-O-Hash constructed with our approach.