Crowdsourcing privacy policy analysis: Potential, challenges and best practices

Privacy policies are supposed to provide transparency about a service's data practices and help consumers make informed choices about which services to entrust with their personal information. In practice, those privacy policies are typically long and complex documents that are largely ignored by consumers. Even for regulators and data protection authorities privacy policies are difficult to assess at scale. Crowdsourcing offers the potential to scale the analysis of privacy policies with microtasks, for instance by assessing how specific data practices are addressed in privacy policies or extracting information about data practices of interest, which can then facilitate further analysis or be provided to users in more effective notice formats. Crowdsourcing the analysis of complex privacy policy documents to non-expert crowd workers poses particular challenges. We discuss best practices, lessons learned and research challenges for crowdsourcing privacy policy analysis.