Category: Data science

Digging Data in the New Wild West

APMALeave a comment
hello world data power

“We do well to remember that there’s no such thing as a free lunch. (…). Data and algorithms don’t just have the power to predict our shopping habits. They also have the power to rob someone of their freedom”

[Hello World: Being Human in the Age of Algorithm, Hannah Fry]

There are many FREE apps for tracking your routine such as walking, jogging, eating, book reading, shopping, or studying. Thanks to these productive apps, we can check our daily routine and change our routine for better performance. By the way, how do these free apps make money? There is no free lunch in the world. They make their profit from the data you recorded. In the age of Big Data, data is the new gold and many companies are digging such gold in our daily routines now. We might say we live in the new Wild West.

Someone might think that Data is just Data. That is true but the AI model can spot important (hidden) patterns from massive data effectively. They can dig gold in the mine by efficient tools. Moreover, they make precise categories for people’s behaviors, leading to an accurate prediction (classification) for new customers. Hence, AI models are becoming more sophisticated as increasing the number of data they collect. Amazon and other online retailers provide irresistible deals and coupons every day. Netflix and other streaming services recommend the best movies we will like so we cannot help clicking the next movie. In these days, we cannot blame a shopaholic. because (internet) shopping addiction is not caused by a lack of self-control but caused by a sophisticated AI model. That is why I purchase more books on Amazon today (Don’t blame me!).

Who Is our Future AI and What Is our Role?

APMALeave a comment
hello world algorithm

“This tendency of ours to view things in black and white – seeing algorithms as either omnipotent masters or a useless pile of junk – present quite a problem in our high-tech age.”

[Hello World: Being Human in the Age of Algorithm, Hannah Fry]

In the famous marvel movie, “Avengers: Age of Ultron”, Two different AIs appeared. The first one is JARVIS (Just A Rather Very Intelligent System) that helps the Iron man – Good AI. Another one is Ultron, the AI supervillain, to destroy the world like “Skynet” in Terminator – Bad AI. Who (or what) will be in our future world? It is hard to answer this question. A lot of books written by AI experts are divided into two forecasts; AI utopia and AI dystopia. However, all the books speak with one voice that the future of AI depends on our actions. Hence, we don’t need to forecast our future in black and white. The (real) future of AI, I believe, will be in between and will be adjustable by us.

Artificial Intelligence is a system consisting of mathematical algorithms to take action that maximizes the probability of success for the given task. It is just (complicated but) a set of algorithms, not a supernatural power. That is, there is still room for understanding it and making it good. First, we should reaffirm fundamental mathematics inside of the AI algorithm as many as we can and eliminate hidden mathematical errors (or computer bugs). Second, we feed them to unbiased and correct data so that AI makes an impartial model to decide their actions. Third, we need to set clear and socially approved objectives for AI models. The first two actions are relatively practicable but the last part requires a social consensus to make a good AI model. For example, the United Nations platform, AI for Good, has tried to offer a route for sustainable development goals. So, please think about the future of AI and about your roles for making a good AIs.

Who Does Make It a Rule? Human? or Machine?

APMALeave a comment
hello world machine learning

“Rule-based algorithms have instructions written by humans, they’re easy to comprehend. (…) Machine-learning algorithms, by contrast, have recently proved to be remarkably good at tackling problems where writing a list of instructions won’t work.”

[Hello World: Being Human in the Age of Algorithm, Hannah Fry]

Nowadays we often hear the word “algorithm” on the news and social networks. By the way, what is the algorithm? The “algorithm” is a (mathematical) recipe to accomplish a certain task. So, your grandma’s recipe for chicken soup is, in some ways, an established algorithm. But when we say about the algorithm recently, it usually refers to a computer algorithm, a series of computer languages to solve a certain problem. There are two different types of algorithms: (1) a rule-based algorithm that follows the prescribed details by humans and (2) a machine-learning algorithm that makes its own rule by machine (computer) itself.

Who does make it a rule for a new task in the future? Humans can make a crystal clear algorithm so that anybody can check the inherent bias or errors of the new rule. Machines, on the other hand, can make a high-performance algorithm without any prior knowledge and deep understanding of the new system. In the age of AI, the power of machine-learning algorithms is no way negligible and the use of this power in various fields is inevitable. However, “Great power comes great responsibility”. So, we, as humans, repeatedly scrutinize such black-box algorithms and prevent misuse of algorithms. We should always know that the final decision should come from humans because machines have no responsibility for their decision. Also, a human should provide some important rules to machine-learning algorithms such as consideration for others, tolerance, and sacrifice, which may lead to creating not only better performance algorithms but also impartial algorithms.

More Approximations, More Problems in Your Life

APMALeave a comment
humble pi approximation

“When the time value is small, the error is also small. But the problem with a percentage error is that, as the value gets bigger, the error grows with it.”

[Humble PI: A Comedy of Maths Errors, Matt Parker]

Rounding a number is a quite simple method to approximate some complicated numbers, leading to easy calculation and good memory. However, such approximation makes a percentage error which makes a big problem when the value is big. Moreover, error grows bigger when multiplying this approximated number many times. To reduce a percentage error for mass production, in the late 20th century, many companies have focused on quality management such as Six Sigma. Specifically, companies have made continuous efforts to reduce a (percentage) error, leading to successful and predictable process results. In data science, to obtain high-fidelity prediction with big numbers, they have kept a significant digit as many as they can.

What about individuals? we often think that we made a 100% effort for my task but we actually made a 95% effort and rounded off. This approximation will make a problem in the future. What? you can say that the 95% effort is high enough to say 100%. Is it true? Let’s take an example, you usually complete a task with a 95% effort (here, say, a success rate) and you have 20 tasks now. Then, the probability that you complete all 20 tasks successfully is 35.8%. However, if you make a 99% effort for each task, then the probability will be 81.8% (and 98% with a 99.9% effort). This example shows that a small different percentage makes a big difference, and we agree that we should complete more than 20 tasks to achieve success in your life. That’s why we should do our best (close to a 100% effort) every time to reach your goal in life.

Please Give Math More Time to Pick up the Pieces

APMALeave a comment
humble pi

“We make things beyond what we understand, and we always have done. … When theory lags behind application, there will always be mathematical surprises lying in wait.”

[Humble PI: A Comedy of Maths Errors, Matt Parker]

In human history (specifically engineering and technology), many successful achievements have shown their effectiveness without proven scientific theories or rigorous mathematics. For example, first flying to the moon in 1969 was achieved with little knowledge of rocket science, aerodynamics, and astrophysics. Sometimes, we called such achievements “the greatest challenge for humans.” However, the word “challenge” here implies that we do not know a mechanism or theory well. I don’t want to underestimate such a challenge but we have experienced that applications without coherent theories may lead to a catastrophic disaster.

Then, in data science, when is the right time to adopt a new model? Should we wait until we understand all theories and mechanisms and prove them all by mathematics? It may be too late. So we should decide the right time by ourselves but we always keep in mind the negative effect when the introduced model fails. So, we try to quantify uncertainties of the model (or our decision) and estimate a probable disaster as insurance companies do. It is much robust thinking rather than efficient thinking. It may lead to slow progress and more cost but it can give us a second chance to correct the model when the model fails. So, if you don’t have rigorous mathematical support, please think uncertainty and make your model robust. Moreover, when you decide something without evidence in your life, please make your decision robust, too.

What Number Is a Really Big Number?

APMALeave a comment
humble pi

“As humans, we are not good at judging the size of large numbers. And even when we know one is bigger than another, we don’t appreciate the size of the difference.”

[Humble Pi: A Comedy of Maths Errors, Matt Parker]

In the Stone Age, a hundred might be a sufficient number to count a herd of deer for hunting or to count gathered nuts. In the early (and mid) 20th century, a million is enough to call the rich people ‘Millionaire’ but now it is too small to count Mark Zuckerberg’s net worth (a million is still BIG money for me by-the-way). In the age of Big Data, what number is a really big number? In the 1980s, Bill Gates, the pioneer to usher in the computer age, said: “for computer memory, 640K ought to be enough for anybody.” Nobody can predict the big number correctly and this is human nature. 

However, we need to estimate a certain big number for a data-driven model, for our business, or for our blogs. After unveiling a smartphone, data acquisition speed is now super fast, leading to the age of AI and Big Data. Nowadays, when we make a model, we consider its own capacity to deal with tremendous data (beyond a trillion). The recent introduction of the Internet of Things (IoT) and the autonomous vehicle will generate countless data every second. Then, we need to keep thinking about the big number again and again. That is why I am preparing for the first event for the Billionth visitor to my blog. Do you think this number is still small? It depends on your action. please visit my blog more! 

Everything is Connected but Not Correlated

APMALeave a comment
How not to be wrong

“Correlation is not transitive. … The non-transitivity of correlation is somehow obvious and mysterious at the same time.”

[How not to be wrong, Jordan Ellenberg]

In Hollywood, the Bacon Number of an actress/actor represents the closest connectivity to the actor, Keven Bacon through movies. Surprisingly, we observed that almost all the actresses/actors can be connected to Keven Bacon within six steps, called this: “Six Degrees of Separation” or “Small World.” This concept originally stems from “Erdős Number” in mathematics and science research, representing a collaborative distance to the mathematician, Paul Erdős. (My Erdős number is 4 by-the-way). What a small world and we feel that everybody is connected!

Sometimes, we confuse a correlation with a connection (or relation). A correlation is not transitive. Even though A and B are strongly correlated and B and C are also correlated, nobody can guarantee that A and C are correlated. However, we often think that there should be a correlation between A and C because we get used to syllogistic reasoning. Moreover, when we mixed up with causality, correlation, and relation, it’s a disaster. So, please do not make any transitivity for mutually correlated data. Also, we keep in mind that uncorrelated data can have a relationship with each other. We, you and I, are connected in the small world but we may not (or may) be correlated with each other.

Make Your Problem Harder!

APMALeave a comment
How not to be wrong

“Instead, we turn to the other strategy, which is the one Birbier used: make the problem harder. That doesn’t sound promising. But when it works, it works like a charm.”

[How not to be wrong, Jordan Ellenberg]

When your friend was struggling with a difficult problem, we often said: “Don’t make it complex, just start with a simple problem”. This is because we have experienced that this simplification provides some clues for solving the difficult problem. This is what mathematicians actually do every day. When proving some statements, they start from the simplest case and expand it to the target problem. However, sometimes, making the problem harder suggests a simple alternative way to solve your real problems effectively.

Many data scientists have focused only on reducing the number of features to make a data-driven model simper. However, this approach does not always give the simplest model. The projection onto the low-dimension (fewer features) may make the data structure more complicated, leading to a failure of spotting the hidden pattern. Hence, sometimes, they need to increase features to make a model simpler (because of more data, more simple). This alternative thinking (adding more features) embodies the trade-off between a simpler model with many features and a complicated model with few features.

Can We Predict our Future in Chaos?

APMALeave a comment

“For human action we have no such model and may never have one. That makes the prediction problem massively harder.”

[How not to be wrong, Jordan Ellenberg]

In the weather, the very tiny scale of energy at a certain location can change the global outcome dramatically – we called this chaos. Edward Lorenz discovered this and wrote: “if the theory were correct, one flap of a sea gull’s wing would be enough to alter the course of the weather forever”. Even though we have an accurate mathematical model (or a data-driven model) for the weather forecast with tons of measured data, we can make only a short-range prediction.

Our behaviors in society are much more chaotic than the weather, leading to a failure of prediction of future outcomes. Moreover, we have no mathematical model to describe our behaviors effectively. Hence, it is really hard (or impossible) to find “right” causation from the massive data. In this chaotic system, we should keep in mind the followings: (1) don’t make any causation from your success (rather, say, just “lucky”); (2) don’t follow others’ successes (a tiny different condition makes a totally different outcome); (3) don’t prejudge the situation using “common sense” (no one can predict the outcome).

Improbable Things Happen All the Time

APMALeave a comment

“The universe is big, and if you’re sufficiently attuned to amazingly improbable occurrences, you’ll find them. Improbable things happen a lot.”

[How not to be wrong, Jordan Ellenberg]

You have a card deck and draw five cards from this. Surprisingly, five cards you drawn are spade A, 2, 3, 4, and 5. (Congrats! you made a straight flush). Then, you might think that this is a new card deck so it is not shuffled yet because drawing these five cards in a row might be improbable (or much lower probable). However, improbable things happen all the time. Please go to Las Vegas and check this!

When analyzing some results, we need to get used to a BIG number in our fields. Our field of interest is pretty big and you can see many improbable occurrences (we can see winners of the lottery every week). Hence, we should be careful not to make any causality from a chance occurrence. In data science, even though the data-driven model finds some patterns from Big Data, we should examine that this pattern can be made by randomness or not. (It may be improbable that millions of people read this post and like it but improbable things happen all the time!!)