Home

Big Data Project Management

Ray ZhanSu, Director-Vendor and Project Management, Acxiom and Tom Shrum, Expert Project Manager, Acxiom

Ray ZhanSu, Director-Vendor and Project Management, Acxiom

As more and more companies are developing big data products, or implementing bigdata projects, project managers are presented with new challenges. Since both new technologies and supply chains are involved, we’re really asking “how can old dogs learn new tricks?” In this paper, we want to share our experiences managing big data projects.

There are several types of big data projects. The common types are software development projects, involving either big data platform or big data analytics, and big data hardware infrastructure projects. We’ll limit our discussion to software development projects.

To frame the challenges encountered with big data product development, we should go back to the basics of project management. The key elements of project management are delivering quality results within the limitations of scope, resource, and time.

Big data product projects have a bigger scope. Besides the standard software development deliverables, there are both, data and analytic modeling pieces to consider. There are additional business groups to involve and separate visions for management to consolidate. Also, there are more tools and platforms to use, which usually involves new tool evaluation, new platform evaluation, and vendor contract negotiation.

Check Out: Top PLM Software Companies

“Utilizing a channel and media neutral approach, we leverage cuttingedge, dataoriented products and services to maximize customer value”

When we kicked off a new product two years ago, we worked with a consulting vendor to evaluate the data platform to use. We spent two months developing a Proof of Concept prototype followed by contractor negotiation. What makes the big data product project different is that, the emerging technologies such as Hadoop and service providers such as Amazon Web Services (or AWS) and Rackspace are involved, which make it both easier and harder to move forward. It is easier because these companies make it very easy to use their services, enabling new environments in a matter of minutes. It’s harder be- cause contracting the companies typically don’t have processes in place to address new supply chain business models that involve immediate payment via credit card.

As with any project, this scope increase needs to be considered at the very beginning. Of course, it is the project manager’s job to make all this happen.

Big data product projects have challenges in finding the right resources. Usually it requires both software development expertise on a big data platform and data scientists familiar with big data, including building models. According to Gartner, “in spite of having a huge demand for people with Big Data skills, al- most two third of the IT jobs will not be filled due to lack of talent”. Our experience confirmed Gartner’s findings. In one project we looked for a Hadoop developer for three months. More than a dozen vendors provided over 100 resumes; however, only five were available for onsite inter- view and none of them met our needs. We moved a resource from another project to solve this resource issue. Considering that Hadoop was created in 2005 and became popular in the last 5 years, which makes it an “old” big data technology it surprised us that we weren’t able to find appropriate resources. For us, the best approach was to team the ex- pert contractors with our employees so that we could on- board the skills we needed. Time to train will need to be taken into account in the project schedule.

Big data product projects are usually new initiatives for companies. Business decision makers typically tie their career to the success of these projects. There is an eager, almost ravenous desire, to birth the product quickly; however, due to complex scope and tools and employee skill sets and training, delivery time will be affected. The challenge for project managers is to manage this time expectation with reality. This involves identifying interim milestones in order to show solid progress and deliver a useful product, balanced with providing the time for the team to learn new skill sets and get into a rhythm. We’ve found that Agile methodologies work best in this situation. We have been using both VersionOne and JIRA Agile as project management tools, finding pros and cons in both tools. We found that VersionOne provided great reports for line management while JIRA Agile was very customizable. Regardless of the toolset, we needed to address the requirements for corporate taxation and Sarbanes-Oxley (or SOX) reporting which added to the tasks hat teams needed to achieve, which in turn added to both product complexity and project delivery time. As always, project managers need to factor in all the elements that affect the timeline as it is being built.

Lastly, the quality of the product takes on additional meaning for these projects. Big data projects typically need to consider multiple QA processes, including but not limited to the quality of the software code, the quality of the data, and the quality of the modeling. These processes add to the complexity of the team, process, and schedule coordination.

Big data projects pose many new challenges. However, these key project management elements still are key. An experienced project manager must be able to learn new skills quickly in order to efficiently manage scope, resources, and time. Becoming aware of these challenges and allocating time to learn new skills are important steps for project managers to successfully manage big data projects.

Acxiom is an enterprise data, analytics and software-as-a-service Company that uniquely fuse trust, experience and scale to fuel data-driven results. For over 40 years, Acxiom has been an innovator in harnessing the most important sources and uses of data to strengthen connections between people, businesses and their partners. Utilizing a channel and media neutral approach, we leverage cutting-edge, data-oriented products and services to maximize customer value. Every week, Acxiom powers more than a trillion trans- actions that enable better living for people and better results for our 7,000+ global clients.