“Big data” is a big catchphrase, and like it or not, your business will be affected by it, even if you don’t directly process big data.
How big is big?
• In 2010, Walmart was collecting data on more than a million transactions per hour, which at the time of the study was 167 times as much data as is held in the Library of Congress.
• A telescope observatory being constructed in Chile will collect in five days the same amount of data the Sloan Digital Sky Survey collected from 2000 to 2010.
• Data doubles in volume every two years.
• Two days’ worth of data at the world’s current data production from all sources is equal to the amount of information created from all conversations in history.
In some areas, big data has earned its keep, allowing shipping companies to maintain fleets more effectively by using unorthodox maintenance schedules, and allowing the medical sector to develop treatments that are more tailored to patient needs.
Many companies are hiring chief data officers, and “data scientist” is one of the hottest job categories today. Here are some of the main concepts your IT team should understand to help your company be ready when the big data juggernaut rolls in.
“I have a trillion more of these waiting on the truck. Should I leave all of them here?”
Breaking Down Silos and Redefining Data Ownership
Breaking down the “silos” into which data generated by the various departments is housed is becoming more important in the big data era. When organizations either don’t allow or never thought of making the boundaries between data sets more permeable, they can miss out on the comprehensive view that all that data could be providing. But think about it: with all the silos that hold data in a typical company, just getting the various data sets to talk to one another is major undertaking. Your IT team may be facing this task sooner than you think.
Protecting Privacy and Keeping It Legal
Big data brings with it big privacy and security demands. Even if your organization doesn’t fall under the rules imposed by PCI-DSS, HIPAA, and the like, legal concerns over big data will raise their heads sooner or later. The privacy and security issues that go along with big data will keep your IT team busy for the foreseeable future. Analyzing data requires it to be copied, so you have to be certain that you have the necessary ownership or licensee rights for the data your organization uses. Your organization will also have to address issues like who is responsible for inaccurate data, who “owns” the data you generate, and how to cope should data fall into the wrong hands.
Trying to Predict Potential Problems
If you haven’t upgraded your crystal ball from Windows XP, you’re going to have a hard time.
With big data, you not only need to stay abreast of current laws, you have to be able to anticipate potential problems. For example, can data be used to discriminate illegally? Anonymous data sets are routinely mined to dig up health and other private information. And while leaving analysis to algorithms may make it seem like discrimination can’t occur, in fact, big data analysis could result in “data redlining” – a more precise form of discrimination.
A Cambridge University study found that very sensitive information including use of addictive substances, sexual orientation, and various personality traits can be predicted with surprising accuracy just based on what things people Like on Facebook. Transparent, understandable big data usage policies and data governance teams can help prevent problems.
Letting Data Overrule Emotion
When big data is behind the wheel of the decision-making truck, you’re going to run into situations where analytics say one thing and conventional wisdom says another. Remember the 2012 election and Nate Silver’s big data triumph that let him accurately predict the outcomes in all 50 states? Many pundits hated him because by extracting the human element from the analysis, Silver shone a bright, unflattering light on the fact that TV pundits are about as reliable as a coin toss. Some skeptics may have to be won over to the idea of big data-driven decisions.
Big data is flowing into just about every business sector, and your IT team has to be prepared. Your organization can’t afford to have an IT service desk that’s hampered by unreliable or inadequate tools. Samanage makes leading IT service management software that has the most powerful service desk and IT asset management tools baked right in. Since it’s cloud-hosted, it scales easily, and powerful new features are added automatically as they are developed. With the best IT service management system, your organization will have a more solid foundation to support the masses of data you can expect in future years.