In 2008, software engineer Vidur Dhanda was consulting on a risk management project on Wall Street when he noticed a fundamental problem: Even the country’s biggest lenders, in the midst of the financial crisis, did not have good data on the health of America’s small and midsize businesses.

His effort to remedy the problem resulted in the Business Default Index, which his Longmeadow (Mass.) startup, WAIN Street, launched last fall. The monthly index is based on the credit performance of nearly 18 million U.S. businesses. “I saw an opportunity. Could we get more data?” recalls Dhanda, who plans to market his index to banks that have long had problems getting lending data. A couple of them are piloting the product now, he says.

Humongous data sets are integral to businesses from Google (GOOG) and Amazon (AMZN) to quant traders on Wall Street. Now startups and small companies are latching onto big data, a buzzword that’s become shorthand in the business world for ways to use voluminous information to gain insights into customers, employees, or products.

Rod Smith, vice president of emerging technologies at IBM (IBM), says small businesses started asking about big data about 18 months ago, prompted in many cases by their larger corporate vendors and partners. But interest really exploded last fall after details emerged about how big data played a role in President Obama’s campaign.

“I was inundated with calls, not about politics, but about that impressive campaign. It was a great way for people to see how you could use [big data] very broadly,” says Smith. In the past two years, IBM has committed $5 billion in financing to help 8,500 small and midsize companies harness technologies like big data analytics, Smith says.

Many small companies want to improve their marketing by pinpointing the best sales prospects through social media. Peter Bordes, founder and chief executive officer of 10-employee, New York City-based Internet Media Labs, says small businesses can analyze what their customers are saying online “to engage in much more meaningful conversations with potential or existing customers, because that big data allows them to understand [those customers] better.”

In tests of his upcoming product, OneQube, Bordes says users have gotten click-through rates of 80 percent to 98 percent on highly targeted marketing campaigns, numbers that are unheard of with traditional direct mail or e-mail campaigns. “You’re giving [prospects] communications that are so relevant they’re reacting to them,” he says. The program mines comments and conversations on social networks like Twitter to identify the most relevant prospects, Bordes says.

Clients testing OneQube haven’t gotten any pushback about being too intrusive. But small companies interested in using big data for marketing should keep privacy concerns paramount, IBM’s Smith says. “You want to build customer relationships, but not have customers feel like you’re looking over their shoulder. [Small and midsize businesses] don’t want to appear icky, but the line is gray, so we’re working with our partners on best practices,” he says. “Sort of like Miss Manners for big data.”

Other companies, like Chicago-based review site G2Crowd, avoid the privacy issue by getting users to opt in: People who post business software reviews must validate their identities by signing in with their LinkedIn (LNKD) accounts, in effect giving permission to have their profile data mined.

G2Crowd provides basic access to reviews for free, which helps small companies without big research budgets identify the best products for them. Then it sells premium research, which incorporates reviewers’ professional backgrounds and social media scores, to corporate IT buyers, says CEO Godard Abel, who founded the 10-employee company last May. Those paying customers will benefit from G2Crowd’s algorithm that sifts data on reviewers’ experience and influence and then uses them to rate various IT products on their effectiveness, popularity, and momentum.

There are some surprising uses of big data as well. Take Catalyst IT Services, an onshore IT outsourcing company founded in 2001 by Michael Rosenbaum, a onetime Harvard Fellow in Law and Economics. He built technology and devised algorithms that would collect and sort large amounts of job applicant data that he believed would predict employee success by matching individuals’ traits and talents with clients’ company cultures. He felt his model could do a better job than sorting through résumés, which largely reflect applicants’ socioeconomic backgrounds and are often poor predictors of success.

When he started, Rosenbaum’s data-crunching approach felt like “swimming upstream,” he says, but in the past few years technology has allowed the company to store and parse far more data at a lower cost. Catalyst’s business has “accelerated enormously,” he says. The Baltimore company, with nearly 200 employees, expects to add another 100 this year.

Data lead to more successful pairings between clients and teams from Catalyst, Rosenbaum says. “Using large amounts of data allows you to suck subjectivity out of a system and make better decisions as a result,” he says. “That has huge implications not only for technology but for all kinds of businesses.”

Source:http://www.businessweek.com/articles/2013-04-03/how-small-businesses-use-big-data