Modern business is largely about data - gathering it, sorting it, consuming it, drawing inferences from it. Even if you manufacture something, or provide a service, you need data to understand where you are in the market. Well there's a dirty little secret that no one talks about. ...The way you encode and store your data determines what questions you can ask...AND what types of questions you'll even THINK to ask. If you can't think of the right questions how are you going to lead your field? Let's see how this has played out in the past.
Europe – 1436 A.D. – Gutenberg Printing Press leads to the Industrial Revolution
When I was student long ago in a place far away, a Professor explained to me that the Gutenberg Printing Press was the 'linchpin' technology that enabled the Industrial Revolution by allowing mass production and distribution of books which in turn allowed the synergistic sharing of techniques and secrets between Trades.
Japan - 1000 A.D. – Silk Screen Printing fails to lead to an Industrial Revolution. Why?
Reflecting upon this professor’s assertion years later, I realized that Japan had the ability to mass produce documents via silk screening almost 500 years before Gutenberg’s invention. So why didn’t the Industrial Revolution start there?
A Different Linchpin Technology
In reality, the key factor that enabled the Industrial Revolution was Europe’s ability to share information between Trades. Ship Builders could study the aspects of metal working that would allow them to build better ships. Trades could exchange information. This was possible because Europe stored their information in books written with a 26 character alphabet. This is short enough to allow the books to be stored in a library in alphabetical order. I.e. the information was index-able. A ship builder could go right to the section of the library that detailed metal working.
By contrast, In Japan the Kanji alphabet is 3000 – 50,000 long. This representation of the data doesn’t allow for the CONCEPT of sorting by alphabetical order - Imagine looking up a phone number for someone in a large city before the age of computers? This meant that Japan stayed with a Feudal Master/Apprentice knowledge transference system, with its closely guarded silos of information, long after mass printing technology was available.
Their Metal Workers could make fantastic swords. Their Wood Workers were equally adept. What they couldn’t do was share information across Trades. They needed an index into their knowledge base and they didn’t have one. And worse, their information encoding system (Kanji) didn’t allow them to consider that indexing might be possible.
A similar issue arose within mathematics
The Roman number encoding system (X, III, IV, etc.) works well enough. But they didn’t have a symbol to represent zero. No zero means the concept of Double Entry Accounting can’t be invented. It took the introduction of zero from the scientists/mathematicians of the Muslim world to allow the Italians to invent this form of accounting and ultimately own the first banking empire of the world.
So what does this mean?
When you store data, you need to pay attention to the way that you encode the information. If you choose a method that works for your immediate needs but which is limited in some way (can’t be indexed, etc.). You won’t be able to ask the questions you need to ask of it. Even worse, you may not realize that you can’t ask the questions you need to ask.
Let’s look at a database example: I have a 50 petabyte data store. It’s structured but unindexed. This means that pretty much any search for information requires a table scan (i.e. look at each row of information one by one). I have a second database of the same size with similar information but this one is indexed. The indexes speeds up searches by a factor of 100,000 for a particularly complex query
This query that takes 30 minutes to run under the indexed system takes about 5 years to complete under the non-indexed system. Clearly, I’m not going to ask the non-indexed system this type of question!
Granted this is a contrived example…or is it?
If you have data in silos and you haven’t spent the time and money to cross map (cross index) the data between those silos, well then you can’t ask the questions of your data that you need to in order to transcend your competition. You can't innovate at the speed you need to. Your operations from Business Development to Legal Compliance are all compromised. This is reality at many big businesses today.
The Race is on!
The first companies to index their data assets will be at a significant competitive advantage. The first companies that figure out better ways to encode their data (AI?) will be at a competitive advantage. The others ....won't. The lead dog has the best view.
Originally published on LinkedIn.
