Category: Math/Stat

Searching for Average Man

APMALeave a comment
humble pi average

“How many of the 4,063 people in the survey could wear such an approximately average uniform? The answer is zero.”

[Humble PI: A Comedy of Maths Errors, Matt Parker]

An average is the simplest statistic to measure the characteristics of data; just add up all the values of data and divide it by the number of data. This simple calculation has been widely used to set a standard for assessment such as average earning, average height, or average life expectancy. Statistically, the average is vulnerable to an outlier. Moreover, the average does not give us information about the distribution of data. That’s why statistical analysis usually provides the average and its standard deviation for sample data. Still, these two values are not enough to represent data due to a large variability (e.g. using any values of average and of standard deviation, we can draw a dinosaur silhouette). Hence, you should choose the right statistic for a clear understanding of the data.

Then, can we say the average is the best number for representing our group or society? The answer is NO. The average man is not a real person and also not a standard for us. In the book “The End of Average“, the author, Todd Rose, showed the failure of our education based on the statistical average. An average is just a number calculated without any context. So, we don’t need to set the average as our standard for success or failure. The annual performance rate cannot measure your success rate in your life. The academic GPA cannot give you the grade of your life. Please don’t follow a phantom of the average. Keep pace with your own plan in your life. After being an outlier in success, please tell us your amazing story, not just give a number.

Probably, We Are Not Independent

APMALeave a comment
humble pi joint probabitlity

“Getting our head around probabilities is very hard for humans. But in high-stakes cases like this, we have to get it right.”

[Humble PI: A Comedy of Maths Errors, Matt Parker]

Since a human is born to be a prediction machine, we naturally estimate the probability of everything. When amazing things are happening in a row, we think it is improbable. This is because we usually estimate the probability that each thing happens and multiply all these probabilities in our head, leading to the tiny probability (close to zero); this is a simple rule to calculate a joint probability. However, to use this rule, there is a hidden assumption that all the events are independent (the probability that one event occurs does not affect the probability that another event occurs). But all the events in our life are not always independent. Some events are correlated, have a cause-and-effect relationship, or occur in the same environment. So, amazing things may happen in a row with a high probability.

If you want to predict some extreme results using such a calculation of joint probability, you consider the independence of each event first. Only the right calculation of the probability is a guardian against a catastrophic result. For example, on the street, you met ten persons whose height is over 6′ 4” in a row. Then, you think that a strange thing happens today. But if you met the same persons in front of a basketball court, then you feel that this situation is somewhat reasonable. The same persons but the different places can change your probability (sometimes we consider a conditional probability). Hence, a careful calculation of a joint probability changes looks-like improbable things to probable things (or vise versa). Even though you calculate the joint probability and the result is very small, you should keep in mind that improbable things happen all the time.

More Approximations, More Problems in Your Life

APMALeave a comment
humble pi approximation

“When the time value is small, the error is also small. But the problem with a percentage error is that, as the value gets bigger, the error grows with it.”

[Humble PI: A Comedy of Maths Errors, Matt Parker]

Rounding a number is a quite simple method to approximate some complicated numbers, leading to easy calculation and good memory. However, such approximation makes a percentage error which makes a big problem when the value is big. Moreover, error grows bigger when multiplying this approximated number many times. To reduce a percentage error for mass production, in the late 20th century, many companies have focused on quality management such as Six Sigma. Specifically, companies have made continuous efforts to reduce a (percentage) error, leading to successful and predictable process results. In data science, to obtain high-fidelity prediction with big numbers, they have kept a significant digit as many as they can.

What about individuals? we often think that we made a 100% effort for my task but we actually made a 95% effort and rounded off. This approximation will make a problem in the future. What? you can say that the 95% effort is high enough to say 100%. Is it true? Let’s take an example, you usually complete a task with a 95% effort (here, say, a success rate) and you have 20 tasks now. Then, the probability that you complete all 20 tasks successfully is 35.8%. However, if you make a 99% effort for each task, then the probability will be 81.8% (and 98% with a 99.9% effort). This example shows that a small different percentage makes a big difference, and we agree that we should complete more than 20 tasks to achieve success in your life. That’s why we should do our best (close to a 100% effort) every time to reach your goal in life.

I Don’t Count on You When You Count Numbers

APMALeave a comment
Humble PI

“The only downside is that you break the link between the number you are using to keep track of your counting with the number of things you are counting.”

[Humble PI: A Comedy of Maths Errors, Matt Parker]

Since Adam was a boy, counting has long been recognized as the most important skill for humans to survive in the world. So, I believe that you (even if you are a toddler) can count numbers well. Let’s check it out. how many numbers you can count on your fingers? The answer is eleven (not ten). This is because you can also count zero with all folded fingers (the more correct answer is 1024, please search for “finger binary” on google). Next, how many natural numbers from 10 to 99? The answer is 90 (not 89). Hooray, you got the right answers, I count on you!

We are more getting into trouble when counting large numbers in efficient ways. For example, when we count the total number of events for calculating probability, we use some math skills such as permutation and combination. However, these are too tricky to use simply. So, when you make a decision based on probability (e.g. Bayesian approach), miscounting the number of events results in a totally different probability, leading to a wrong decision. Please don’t count on yourself when you count numbers (specifically, counting sheep to sleep or counting cards to win the blackjack).

Please Give Math More Time to Pick up the Pieces

APMALeave a comment
humble pi

“We make things beyond what we understand, and we always have done. … When theory lags behind application, there will always be mathematical surprises lying in wait.”

[Humble PI: A Comedy of Maths Errors, Matt Parker]

In human history (specifically engineering and technology), many successful achievements have shown their effectiveness without proven scientific theories or rigorous mathematics. For example, first flying to the moon in 1969 was achieved with little knowledge of rocket science, aerodynamics, and astrophysics. Sometimes, we called such achievements “the greatest challenge for humans.” However, the word “challenge” here implies that we do not know a mechanism or theory well. I don’t want to underestimate such a challenge but we have experienced that applications without coherent theories may lead to a catastrophic disaster.

Then, in data science, when is the right time to adopt a new model? Should we wait until we understand all theories and mechanisms and prove them all by mathematics? It may be too late. So we should decide the right time by ourselves but we always keep in mind the negative effect when the introduced model fails. So, we try to quantify uncertainties of the model (or our decision) and estimate a probable disaster as insurance companies do. It is much robust thinking rather than efficient thinking. It may lead to slow progress and more cost but it can give us a second chance to correct the model when the model fails. So, if you don’t have rigorous mathematical support, please think uncertainty and make your model robust. Moreover, when you decide something without evidence in your life, please make your decision robust, too.

What Number Is a Really Big Number?

APMALeave a comment
humble pi

“As humans, we are not good at judging the size of large numbers. And even when we know one is bigger than another, we don’t appreciate the size of the difference.”

[Humble Pi: A Comedy of Maths Errors, Matt Parker]

In the Stone Age, a hundred might be a sufficient number to count a herd of deer for hunting or to count gathered nuts. In the early (and mid) 20th century, a million is enough to call the rich people ‘Millionaire’ but now it is too small to count Mark Zuckerberg’s net worth (a million is still BIG money for me by-the-way). In the age of Big Data, what number is a really big number? In the 1980s, Bill Gates, the pioneer to usher in the computer age, said: “for computer memory, 640K ought to be enough for anybody.” Nobody can predict the big number correctly and this is human nature. 

However, we need to estimate a certain big number for a data-driven model, for our business, or for our blogs. After unveiling a smartphone, data acquisition speed is now super fast, leading to the age of AI and Big Data. Nowadays, when we make a model, we consider its own capacity to deal with tremendous data (beyond a trillion). The recent introduction of the Internet of Things (IoT) and the autonomous vehicle will generate countless data every second. Then, we need to keep thinking about the big number again and again. That is why I am preparing for the first event for the Billionth visitor to my blog. Do you think this number is still small? It depends on your action. please visit my blog more! 

When You Meet a Mathematical Genius

APMA2 Comments
how not to be wrong

“Athletes don’t quit their sport just because one of their teammates outshines them. And yet I see promising young mathematicians quit every year, even though they love mathematics, because someone in their range of vision was ahead of them.”

[How not to be wrong, Jordan Ellenberg]

I know this is a little bit off the topic (and the style) of this blog but I would like to write this post for kids/students who want to be a future mathematician. In my life, I have met several mathematical geniuses equipped with complete mathematical skill sets, intuition, reasoning, and creativity. Many people may think that geniuses are not willing to work hard but all the geniuses I met before put their whole energy into solving some mathematical problems always. Hence, when I had met them, I had felt that there is NO chance to defeat such kinds of geniuses and had felt depressed every single day. Many prodigious kids/students give up chasing their dream to become a great mathematician like this way.

However, doing mathematics is not a race and competition to choose the only one winner. It is more like team sports. For example, in Football, the Quarterback looks like the one and only hero to win the game but it is not true. There are many unsung heroes to try to get a score and win the game. Likewise, the development of mathematics is not the exclusive property of the math geniuses. I don’t want to underestimate the role of math geniuses; they always give us a new point of view about mathematical thinking. But rather, I would like to redound to many roles of other mathematicians such as building rigorous mathematical formulation from brilliant ideas and applying this mathematical concept to various problems in the real world. Hence, “Genius” may represent not a person but a team (or generation).

So, here is my humble advice when you meet mathematical geniuses in your life:
(1) Do not compare yourself to them (everybody has an important role in developing mathematics).
(2) Learn everything from them as much as you can.
(3) Put your whole energy into developing, extending, and applying their brilliant idea.
(4) Do not give up.
(5) Be grateful for being contemporaneous with the great geniuses.

Everything is Connected but Not Correlated

APMALeave a comment
How not to be wrong

“Correlation is not transitive. … The non-transitivity of correlation is somehow obvious and mysterious at the same time.”

[How not to be wrong, Jordan Ellenberg]

In Hollywood, the Bacon Number of an actress/actor represents the closest connectivity to the actor, Keven Bacon through movies. Surprisingly, we observed that almost all the actresses/actors can be connected to Keven Bacon within six steps, called this: “Six Degrees of Separation” or “Small World.” This concept originally stems from “Erdős Number” in mathematics and science research, representing a collaborative distance to the mathematician, Paul Erdős. (My Erdős number is 4 by-the-way). What a small world and we feel that everybody is connected!

Sometimes, we confuse a correlation with a connection (or relation). A correlation is not transitive. Even though A and B are strongly correlated and B and C are also correlated, nobody can guarantee that A and C are correlated. However, we often think that there should be a correlation between A and C because we get used to syllogistic reasoning. Moreover, when we mixed up with causality, correlation, and relation, it’s a disaster. So, please do not make any transitivity for mutually correlated data. Also, we keep in mind that uncorrelated data can have a relationship with each other. We, you and I, are connected in the small world but we may not (or may) be correlated with each other.

The Triumph of Mediocrity: Do not Stumble on Your Success

APMALeave a comment
Triumph of Mediocrity

“That’s what causes regression to the mean: not a mysterious mediocrity-loving force, but the simple working of heredity intermingled with chance.”

[How not to be wrong, Jordan Ellenberg]

At the beginning of the month, I check the number of visitors and views on my blog and say: “What? Too many people come in! Then, my blog is ON PACE to break my monthly record!!” I am really excited about this shock rise. At the end of the month, my eyes widen in surprise because the average number of people visited, no new record (Sigh). This shows “The Triumph of Mediocrity.” Some data intertwined with deterministic factors and uncertainties show a tendency to regress to the mean.

This simple mathematical observation gives a lesson about how to live. There is no (deterministic) equation of success. Even if it exists, it has too many uncertainties so we cannot solve this equation. When you achieved something that you want, this success does not only stem from your skills, abilities, intelligence, and effort. Rather, uncertainties (many people call this “luck”) may drive your way to success. Just when you think that you find the equation of the success, your next try may fail and you will be back to the mean – we call this “Sophomore Slump.” So please be humble. please do not stumble on your success. Also, if you did your best but failed, please try one more, the triumph of mediocrity may take you to the success.

Make Your Problem Harder!

APMALeave a comment
How not to be wrong

“Instead, we turn to the other strategy, which is the one Birbier used: make the problem harder. That doesn’t sound promising. But when it works, it works like a charm.”

[How not to be wrong, Jordan Ellenberg]

When your friend was struggling with a difficult problem, we often said: “Don’t make it complex, just start with a simple problem”. This is because we have experienced that this simplification provides some clues for solving the difficult problem. This is what mathematicians actually do every day. When proving some statements, they start from the simplest case and expand it to the target problem. However, sometimes, making the problem harder suggests a simple alternative way to solve your real problems effectively.

Many data scientists have focused only on reducing the number of features to make a data-driven model simper. However, this approach does not always give the simplest model. The projection onto the low-dimension (fewer features) may make the data structure more complicated, leading to a failure of spotting the hidden pattern. Hence, sometimes, they need to increase features to make a model simpler (because of more data, more simple). This alternative thinking (adding more features) embodies the trade-off between a simpler model with many features and a complicated model with few features.