Big Data Project Management

As with any project, this scope increase needs to be considered at the very beginning. Of course, it is the project manager’s job to make all this happen.

Big data product projects have challenges in finding the right resources. Usually it requires both software development expertise on a big data platform and data scientists familiar with big data, including building models. According to Gartner, “in spite of having a huge demand for people with Big Data skills, al- most two third of the IT jobs will not be filled due to lack of talent”. Our experience confirmed Gartner’s findings. In one project we looked for a Hadoop developer for three months.  More than a dozen vendors provided over 100 resumes; however, only five were available for onsite inter- view and none of them met our needs. We moved a resource from another project to solve this resource issue. Considering that Hadoop was created in 2005 and became popular in the last 5 years, which makes it an “old” big data technology it surprised us that we weren’t able to find appropriate resources. For us, the best approach was to team the ex- pert contractors with our employees so that we could on- board the skills we needed. Time to train will need to be taken into account in the project schedule.

Big data product projects are usually new initiatives for companies.  Business decision makers typically tie their career to the success of these projects. There is an eager, almost ravenous desire, to birth the product quickly; however, due to complex scope and tools and employee skill sets and training, delivery time will be affected. The challenge for project managers is to manage this time expectation with reality.  This involves identifying interim milestones in order to show solid progress and deliver a useful product, balanced with providing the time for the team to learn new skill sets and get into a rhythm.  We’ve found that Agile methodologies work best in this situation. We have been using both VersionOne and JIRA Agile as project management tools, finding pros and cons in both tools.  We found that VersionOne provided great reports for line management while JIRA Agile was very customizable.  Regardless  of the toolset,  we needed to address the requirements for  corporate taxation and Sarbanes-Oxley (or  SOX) reporting which added to the tasks hat teams needed to achieve, which in  turn added to both product complexity and  project delivery time.  As always, project managers need to factor in all the elements that affect the timeline as it is being built.

Lastly, the quality of the product takes on additional meaning for these projects.  Big data projects typically need to consider multiple QA processes, including but not limited to the quality of the software code, the quality of the data, and the quality of the modeling. These processes add to the complexity of the team, process, and schedule coordination.

Big data projects pose many new challenges. However, these key project management elements still are key. An experienced project manager must be able to learn new skills quickly in order to efficiently manage scope, resources, and time. Becoming aware of these challenges and allocating time to learn new skills are important steps for project managers to successfully manage big data projects.

Acxiom is an enterprise data, analytics and software-as-a-service Company that uniquely fuse trust, experience and scale to fuel data-driven results. For over 40 years, Acxiom has been an innovator in harnessing the most important sources and uses of data to strengthen connections between people, businesses and their partners. Utilizing a channel and media neutral approach, we leverage cutting-edge, data-oriented products and services to maximize customer value. Every week, Acxiom powers more than a trillion trans- actions that enable better living for people and better results for our 7,000+ global clients.