We present new results of point-contact Andreev-reflection ( PCAR) spectroscopy in single-phase Mg1-xAlxB2 single crystals with x up to 0.32. Fitting the conductance curves of our point contacts with the two-band Blonder-Tinkham-Klapwijk model allowed us to extract the gap amplitudes Delta(sigma) and Delta(pi). The gap values agree rather well with other PCAR results for Al-doped crystals and polycrystals up to x=0.2 reported in the literature, and extend them to higher Al contents. In the low-doping regime, however, we observed an increase in the small gap Delta(pi) on increasing x (or decreasing the local critical temperature of the junctions, T-c(A)) which is not as clearly found for other samples. On further decreasing T-c(A) below 30 K, both the gaps decrease and, up to the highest doping level x = 0.32 and down to T-c(A) = 12 K, no gap merging is observed. A detailed analysis of the data within the two-band Eliashberg theory shows that this gap trend can be explained as being mainly due to the band filling and to an increase in the interband scattering which is necessary to account for the increase in Delta(pi) at low Al contents ( x < 0.1). We suggest interpreting the following decrease of Delta(pi) for T-c(A) < 30 K as being governed by the onset of the inhomogeneity and disorder in the Al distribution that partly mask the intrinsic effects of doping and are not taken into account in standard theoretical approaches.