The Need for New Processes, Methodologies and Tools to Support Big Data Teams and Improve Big Data Project Effectiveness

As data continues to be produced in massive amounts, with increasing volume, velocity and variety, big data projects are growing in frequency and importance. However, the growth in the use of big data has outstripped the knowledge of how to support teams that need to do big data projects. In fact, while much has been written in terms of the use of algorithms that can help generate insightful analysis, much less has been written about methodologies, tools and frameworks that could enable teams to more effectively and efficiently "do" big data projects. Hence, this paper discusses the key research questions relating methodologies, tools and frameworks to improve big data team effectiveness as well as the potential goals for a big data process methodology. Finally, the paper also discusses related domains, such as software development, operations research and business intelligence, since these fields might provide insight into how to define a big data process methodology.