Mixed micelles containing Nonidet P40 (NP-40) (829 mu M or 4.8 mM), phosphatidylserine (PS) (14.5 or 8 mol %), and 1,2-diacylglycerol (DG) (0.5 or 1 mol %) when preincubated with protein kinase C (PKC) assay mixture containing cationic substrate and CaCl2 (400 mu M) formed aggregates in a time-, temperature-, and substrate concentration-dependent manner with a t(1/2) similar to 3-12 min (22 degrees C). Concomitant with the formation of these aggregates there was a substantial loss of substrate phosphorylation catalyzed by the Ca2+-dependent PKC alpha, beta, and gamma but not the Ca2+-independent PKC delta and E. All cationic PKC substrates tested, neurogranin peptide analog(29-47), neurogranin, and histone III-S, formed aggregates with PS/DG/NP-40/Ca2+ mixed micelles in a time-dependent fashion. The poly(cationic-anionic) PKC substrate protamine sulfate also forms aggregates with the mixed micelles in the presence of Ca2+, but without affecting the substrate phosphorylation by the kinase. Under similar conditions, but at 4 degrees C, neither aggregation nor loss of cationic substrate phosphorylation was observed. Another nonionic detergent, octyl glucoside, behaved similarly to NP-40. Phosphatidylinositol (PI) and phosphatidylglycerol like PS, were effective in forming aggregates with NP-40/cationic polypeptide/DG/Ca2+ as monitored by light scattering, yet without affecting substrate phosphorylation. Phosphorylation of cationic substrates by M-kinase, derived from trypsinized PKC beta, was also greatly diminished by the aggregation. In contrast, [H-3]phorbol 12,13-dibutyrate binding to PKC beta was unaffected. Formation of the aggregates that were selectively utilized by the Ca2+-independent PKCs was dependent on the ratio of cationic substrate to the number of mixed micelles. Lipid vesicles containing PS and DG or a phospholipid composition mimicking that of the cell membrane, phosphatidylcholine/phosphatidylethanolamine/sphingomyelin/PI/PS (247, 1 27, 120, 47, and 40 mu g/mL), 80 mu g/ml DG, and Ca2+, did not form such aggregates in the absence of nonionic detergent. These results indicate that the aggregation of cationic polypeptide with nonionic detergent/PS/DG/Ca2+ renders the substrate phosphorylation sites inaccessible to the Ca2+-dependent subgroup of PKCs. The current findings suggest caution in the use of cationic polypeptides that form aggregates with phospholipid/nonionic detergent/Ca2+ mixed micelles in the assay of Ca2+-dependent PKCs.