Thursday, May 30, 2013

What do you do when every object is Internet connected?

Actually, there is a spoon.

            Welcome to the future where everything collects data, not just your car, your phone and your Fuel band, but your toilet.  Yeah, your toilet knows you're slightly dehydrated and you don't eat enough fiber, and it already called your nurse practitioner.  Check your messages.

            But what do we do when everything is connected? 

            So we're collecting all this data, and storage keeps getting cheaper.  Heck, the biowizards at Harvard stored 700 terabytes on a single gram of DNA, so stop worrying about storage.[1]  The problem is you.

            You can't push a pumpkin through a garden hose.

            What?

            The three-pound engine at the top of your neck is a bottleneck.  Everything around you is going to be spewing data, and you will have no idea what to do with it because you don't know how to process it all.

Hurricanes and hedge funds. 

            We analyze complex systems every day with the fastest computers and the best brains, but we don't understand them.  So we collect more data and pile it deeper, but we can't process it all.  We have more data, not better choices. 

            In "How Hurricane Forecasting Got So Good," Sarah Fecht wrote, "Hurricane forecasting begins with lots and lots of data. More data than weather modelers know what to do with, really."  Yet when a hurricane is just two days from landfall, our predictions miss by an average of 100 miles.  During Superstorm Sandy, we outdid ourselves and called it within 50 miles.  Wow.[2]

            Was our data on Sandy incomplete? No, our analysis was.

Buy.  Sell.  Jump.

            Complex systems.  The stock market is complex.  Lotsa numbers, but computers are good at numbers and we have plenty of computers.  Yet every investment group pays Stock Analysts to figure out what all that data means and distill it down into something they can understand and act on.

            Hedge funds deal in billions of dollars.  There are at least 30 different recognized hedge fund strategies, but many of them boil down to considering a single factor.  All that data, and some hedge fund manager is looking at interest rates.  Only interest rates. 

            In Running Money, hedge funder Andy Kessler wrote, "Because information is distributed in milliseconds, there is no time advantage anymore.  You have to be ahead of news."[3]  Someone has to sit down and analyze all this stuff, because the guy across the street has access to all the same data you do.  So why do some hedge funds make money and others don't? Better analysis.

The Plans for the Death Star

            We already collect more data than we know how to process.  When all the objects in your life start chatting with you and each other, you'll be drowning in data.  It's time to plan ahead.

            The same computers that collect and store this stuff are not built to analyze it.  There are numbers and letters and charts, but which ones are important and what do they tell us?

            We need to build ourselves some new tools that can help us analyze this flood.  Then we need to ask these tools the right questions so they can give us good advice.  Because data is useless if we can't put it to work.

Sources:

[1] "Harvard cracks DNA storage, crams 700 terabytes of data into a single gram" by Sebastian Anthony for ExtremeTech. http://www.extremetech.com/extreme/134672-harvard-cracks-dna-storage-crams-700-terabytes-of-data-into-a-single-gram
[2] "How Hurricane Forecasting Got So Good" by Sarah Fecht for Popular Mechanics.  http://www.popularmechanics.com/science/environment/natural-disasters/how-hurricane-forecasting-got-so-good-14544602
[3] Running Money by Andy Kessler.  HarperCollins Publishers Inc.  www.andykessler.com.  www.harpercollins.com.

1 comment:

  1. The Chimp lives!
    The timbre of this article leads me to believe the chimp is overloaded. It is very true that information is bombarding us constantly. According to researchers at the University of California at Berkeley's School of Information, mankind has produced more information in the last 30 years than in all the previous 5,000 years combined. (‘How Can You Deal With Information Overload?’ article by Becky Sweat)

    “Produce more information” is not knowledge. I sometimes joke that Google and Wikipedia has made geniuses out of the likes of me when in fact the opposite is true. I can quickly research and recite simple facts without ever having to learn them.
    So the Chimp suggests we develop tools to process the data. Another alternative is to simply filter the data up front. Filtering data simply means ignoring data. There are so many interesting things in the world and so many avenues to get specialized data. Sometimes we just have to close a few of these avenues and accept that we will ignore some interesting and useful data.

    ReplyDelete