A rigorous theoretical investigation has been made of electromagnetic modes and associated filamentational instabilities in a nonuniform, partially ionized, self-gravitating, dusty magnetoplasma, taking into account the effects of the neutral fluid dynamics, inhomogeneities in the external magnetic field and the plasma density, the drag forces, the ionization and recombination, etc. It is found that there exist two new types of filamentational instabilities; one type is due to the combined effects of collisions of dust particles or ions with stationary neutrals and the self-gravitational force, and the other type is due to the combined effects of the dusty plasma and the neutral fluid dynamics. Furthermore, the effects of the external magnetic field and its inhomogeneity, the neutral fluid temperature, collisions of dust particles with neutrals, and the ionization play the stabilizing role, whereas the effects of the neutral fluid mass density, the plasma number density inhomogeneity, collisions of ions with neutrals, and the recombination play the destabilizing role. The newly found instabilities may be responsible for the formation of stars in galaxies via fragmentation or filamentation of interstellar dense molecular clouds. (C) 2000 American Institute of Physics. [S1070-664X(00)03009-3].