The rise of Information Technology had led to capture of massive amounts of data about consumers, their behaviors, markets, competitors, etc and data collection, storage, analysis is further fuelled by the falling prices of networking equipments, databases, storage devices, development of advanced analytical software and rapid rise of cloud computing and social media. Banking, Financial Services, Insurance, Retail, Manufacturing, Telecom, Automobile, etc industries have been collecting data about customers, markets, competitors, etc since many years and this data collection is more digitized with the help of sensors, transaction software, mobile devices, social networks, etc. The explosion of the data in the past few years have led to coining of the term "Big Data" which refers to data sets with size ranging from terabytes to petabytes and are often very difficult to handle and requires advanced storage, analytical and hardware technologies for collection, storage and real time processing so that data can be effectively utilized in day to day operations and organizational business strategy development. Big data requires significant amount of investments in terms of technology infrastructure, advanced software tools, recruit highly talented and skilled professionals and changes to the enterprise IT policy and strategy.
Big data is critical for businesses and organizations as it can create tremendous value in terms of accurate and usable information for better decision making and formulate strategies, improve the performance in terms of productivity & profitability, understand consumers, their behaviors, needs, markets and develop products and services accordingly. To survive in the volatile and highly competitive business environment, businesses ranging from small to large see capturing and analyzing big data as a key differentiator in the market and are investing in tools and technologies that help in capture, storage and analyze in real time which will improve the decision making of the management and ultimately lead to the organizational growth and profitability. Employees in these organizations too can improve their performance and productivity with the help of big data. Despite having such a huge potential big data is seeing slower adoption as the tools, technologies and devices are still being developed and there are some issues like privacy, security of data, intellectual property issues, investment of significant amount of resources be it monetary and technology infrastructures, etc. Business Organizations are looking ways for making sense of the huge amounts of data that they are collecting and storing and are looking for tools and technologies that will help them do so and are willing to invest.
IDC forecasted the market for big data technology and services will reach US$16.9 billion by 2015 from US$3.2 billion in 2010 i.e. CAGR of 40%. According to IDC big data projects are those projects that involve two or more data formats, involve collection of more than 100 terabytes of data, smaller data sets that grow by more than 60% a year and involve high speed real time data streaming and analysis. Businesses are focusing more on the value in terms of improving operational efficiencies and innovation that the big data is going to generate for them and there has also been a rise in the appliance sales, cloud computing and outsourcing deals involving big data. Major IT vendors and many startups are developing new products and technologies and there has been significant M&A in this space where large IT vendors are acquiring the smaller players and startups and integrating them into their core offerings such as database solutions, analytics, etc. IDC report also highlighted the shortage of data technology and analytics experts which will hinder the aggressive adoption of businesses. Big data software and services, infrastructure technologies and storage will see strong growth till 2015.
Deloitte forecasted that by end of 2012 90% of Fortune 500 companies will start on big data initiatives and market penetration will increase leading to US$1-1.5 billion in revenues. The industry is still at its infancy and existing traditional databases and tools are meeting the client needs. Adoption of big data technologies is also not equal across the industrial verticals and most of the adoption is seen in Internet companies, BFSI, retail, etc. The industry also needs to focus on developing talented big data professionals in near future and businesses have to include big data in their enterprise IT and business strategy. According to Wikibon Research, big data is a US$5 billion market early 2012 and is expected to reach US$50 billion by 2017 which is CAGR of 58%. IBM is the market leader followed by Intel, HP and Oracle. Big data pure player category (players who get more than 50% revenues from big data related) includes players like Vertica, Aster Data, Splunk, Greenplum, etc. Pure players are at the fore front of innovation and larger IT players are acquiring these players and integrating into their product offerings which also highlights the fact there is going to be lot of consolidation in this market. Wikibon also believes big data is the new definitive source of competitive advantage across all the industrial verticals. For the big data technology and services to achieve the projected growth in future, major as well as smaller players have to develop innovative products and service offerings that will generate significant business value for the clients.
Despite the positive picture being highlighted by major research firms and major IT vendors there are certain issues that are of concern like securely storing the large data sets and protecting the privacy of the customers, regulatory policies, large amounts of investments in terms of monetary, technology and human resources, and integrating the big data technology and services into the organizational IT infrastructure and using it in the decision making. Data crunching and collection is being done by the businesses for the past many years and there are database technologies and tools that collect, store and analyze data at a comparatively lower levels exist. But the amount of data had exploded in the recent past due to mobility, cloud computing, commoditization of IT and social media that led to the big data technology and services. Businesses are looking for more Return on investment from the big data initiatives and use cases have to further evolve to convince them for increased adoptions. The amount of data, types of data and complexity of data and tools, technologies have forced the businesses to either take external help or outsource. Utility of the analysis of big data and its use in the decision making is also a major area of concern as the businesses may not be in a position to utilize it. Since big data requires significant investments management need to be convinced in regards to the business value in terms of profitability, customer satisfaction, etc that the big data is going to provide and its role as a key differentiator from the competition.
Vendors, clients and other stakeholders need to invest in developing the talented professionals who are trained in the big data technologies as this is one of the major areas of concern for the business adoption. The adoption has to be increased in the existing industry verticals and the new verticals too have to increase their adoption. Also the macro and micro economic environment should also be conducive so that clients can invest in such technologies and the current volatile economic environment is a major concern. The rise of cloud computing, social media, artificial intelligence, machine learning and natural language processing, etc will definitely drive the adoption of the big data technologies. There has been a lot of startup funding and larger IT vendors too are investing in the development of big data technologies and with significant M&A occurring in this industry the consolidation will be more advantageous for the customers. Finally high end analytics based on the advanced statistical techniques, data mining, etc provide the relevant intelligence for the effective decision making by the management and this should lead to organizational growth and profitability. Analytics software should make sense of the big data that is highly complex, involves multiple formats and the output intelligence has to be reliable for the businesses in the decision making. Industry specific solutions and cost effective tools technologies based on cloud computing is the requirement of many clients. Outsourcing Vendors are also playing a key role in pushing big data technologies and services as they are looking at these offerings as non linear revenues which will generate high revenues and margins but they have to recruit talented big data professionals and train them. Overall big data is going to see increased adoption in 2012 and is expected to achieve the forecasted targets.
Wikibon’s forecast for the Big Data market
Wikibon’s Top Big Data pure-play vendors
For Total 2011 Big Data Revenue by Vendor – Please visit http://wikibon.org/wiki/v/Big_Data_Market_Size_and_Vendor_Revenues