The big data conundrum
Jinfo Blog

By Tim Buckley Owen

Abstract

Chief information officers' latest headache may be the purchase of big data solutions, with major companies competing in this market. However, it will be the expertise of information professionals that will ultimately prove the success or otherwise of the venture.

Item

Big data solutions continue to jostle for position in large numbers. It may be the chief information officer’s (CIO) ultimate responsibility to put their reputation on the line with a purchase decision, but information managers may be able to help by taking on the role of expert user.

CIOs face the prospect of big data increasing by up to 80% a year over the next three years, said a recent survey of 5,000 companies from Virgin Media Business – and bandwidth is the issue that concerns more CIOs than anything else. Organisations have to be able to expand their networks as and when they need to “without being penalised for the privilege”, as Virgin put it.

Unsurprisingly, Virgin believes it has the solution. But it is not alone, and currently doesn’t feature in Gartner’s top half dozen business intelligence, analytics and performance management software leaders.

Nor does Connotate, despite describing itself as “the leading provider of solutions that help organisations monitor and collect data and content from the web”. However it has just secured a further $7 million in funding from a group of venture capitalists led by Castile Ventures, and it’s recently been on the acquisition trail, buying the analytics firm Fetch Technologies (LiveWire comment here).

IBM, on the other hand, is on Gartner’s list – and it’s just unveiled its DB2 10 and InfoSphere Warehouse 10 software. During testing, clients performed data warehouse queries up to 10 times faster and freed up storage space by up to 90%, it claims – and it also throws out a challenge to another market leader, Oracle, about data migration costs.

SAP, SAS, Institute and Microsoft are the other leaders (by 2011 revenue estimates) that Gartner’s figures show – but it has also identified more than 100 innovative vendors “jostling for position”. So perhaps information industry sage David Worlock, chairman of the Business Information Industry Association, can provide some clarity.

Suppliers can’t “productise” information services for ever, Worlock said in a recent blog. The biggest customers are quite clever enough to customise them for themselves.

So suppliers could just license everything and leave the customers to get on with it – but that would deprive the vendors of vital customer feedback. At the other extreme, they could customise everything for their major clients – but that would ignore the huge market of smaller customers.

Those customers require packaged solutions, but also represent a big opportunity in terms of at least scaled down customisation. But that might alarm the Oracles, IBMs and SAPs, who are quite prepared to buy on a large scale to prevent their own perceived territory from being undermined.

Big challenges for CIOs, then. But it’s information managers who may have to live with the consequences.

« Blog