Last year, an aunt was diagnosed with extremely low bone density. She had been complaining of back pain and weakness, and a few tests later, her orthopedic confirmed that bone density was the problem. She was put on a course of medication, and then was given by shots. A year later, she got her bone…
Exponential need not mean explosive
Earlier on this blog I’ve written about the misuse of the term “exponential” when it is used to describe explosive increase in a particular number. My suspicion is that this misuse of the word “exponential” comes from Computer Science and complexity theory – where the hardest problems to crack are those which require time/space that…
Standard deviation is over
I first learnt about the concept of Standard Deviation sometime in 1999, when we were being taught introductory statistics in class 12. It was classified under the topic of “measures of dispersion”, and after having learnt the concepts of “mean deviation from median” (and learning that “mean deviation from mean” is identically zero) and “mean…
Calibration and test sets
When you’re doing any statistical analysis, the standard thing to do is to divide your data into “calibration” and “test” data sets. You build the model on the “calibration” data set, and then test it on the “test” data set. The purpose of this slightly complicated procedure is so that you don’t “overfit” your model.…
Analytics and complexity
I recently learnt that a number of people think that the more the number of variables you use in your model, the better your model is! What has surprised me is that I’ve met a lot of people who think so, and recommendations for simple models haven’t been taken too kindly. The conversation usually goes…
Black Box Models
A few years ago, Felix Salmon wrote this article in Wired called “The Formula That Killed Wall Street“. It was about a formula called “Gaussian Copula”, which was a formula for estimating the joint probability of a set of events happening, if you knew the individual probabilities. It was a mathematical breakthrough. Unfortunately, it fell…
Switching Off
Since last night I’ve been terribly sick. I slept fitfully, if at all, all of last night, and I’ve been totally out of action all day today. It’s nothing particularly serious – just a bad attack of the common cold, and I expect it to take its normal course. Yet, through the day, as I’ve…
Long mails
As you might have noticed from my blog posts over the years, I like writing long essays. By long, I mean blog post long. Somewhere of the length of 800-1000 words. I can’t write longer than that, because of which my attempts to write a book have come to nought. Now, thanks to regular blogging…
On finding the right signal
It is not necessary that every problem yields a “signal”. It is well possible that sometimes you try to solve a problem using data and you are simply unable to find any signal. This does not mean that you have failed in your quest – the fact that you have found the absence of a…
Missed opportunities in cross-selling
Talk to any analytics or “business intelligence” provider – be it a large commoditized outsourcing firm or a rather niche consultant – and one thing they all claim to advise their clients on is strategies for “cross sell”. However, my personal experience suggests that implementation of cross-sell strategies among retailers I encounter is extremely poor.…
Networking eatings
Given that I’m a freelancer and do several things to earn my money, and that there is no consistency in my income flow, I need to do a lot of “networking”. Essentially, this is about generally catching up with someone over an informal chat, discussing what we do, and exploring if there were any synergies…
Nate Silver Interview At HBR Blogs
HBR Blogs has interviewed Nate Silver on analytics, building a career in analytics and how organizations should manage analytics. I agree with his views on pretty much everything. Some money quotes: HBR: Say an organization brings in a bunch of ‘stat heads’ to use your terminology. Do you silo them in their own department that serves…
Hedgehogs and foxes: Or, a day in the life of a quant
I must state at the outset that this post is inspired by the second chapter of Nate Silver’s book The Signal and the Noise. In that chapter, which is about election forecasting, Silver draws upon the old Russian parable of the hedgehog and the fox. According to that story, the fox knows several tricks while the…
Why standard deviation is not a good measure of volatility
Most finance textbooks, at least the ones that are popular in Business Schools, use standard deviation as a measure of volatility of a stock price. In this post, we will examine why it is not a great idea. To put it in one line, the use of standard deviation loses information on the ordering of…
Anscombe’s Quartet and Analytics
Many “analytics professionals” or “quants” I know or have worked with have no hesitation in diving straight into a statistical model when they are faced with a problem, rather than trying to understand the data. However, that is not the way I work. Whenever I set out solving a new problem, I start with spending…
Exponential increase
“Increasing exponentially” is a common phrase used by analysts and commentators to describe a certain kind of growth. However, more often than not, this phrase is misused and any fast growth is termed as “exponential”. If f(x) is a function of x, f(x) is said to be exponential if and only if it can be…
Would you want a free membership card?
Last weekend I was at Cafe Coffee Day on MG Road, waiting to meet a prospective client, when one of the store staff walked up to me with a card. “This is a free loyalty card, Sir”, he said, going on to tell me that if I were to buy three coffees using the card…
Should you have an analytics team?
In an earlier post, I had talked about the importance of business people knowing numbers and numbers people knowing business, and had put in a small advertisement for my consulting services by mentioning that I know both business and numbers and work at their cusp. In this post, I take that further and analyze if it…
Perfect and imperfect interviews
The movies give this impression of what is a “perfect” job interview. This usually happens during the stage of the movie when the protagonist has started turning things around, things are looking up for him/her. The “perfect” job interview barely lasts minutes, and the protagonist returns with the offer letter. As good as it can…
Should you have an analytics team?
In an earlier post a couple of weeks back, I had talked about the importance of business people knowing numbers and numbers people knowing business, and had put in a small advertisement for my consulting services by mentioning that I know both business and numbers and work at their cusp. In this post, I take…
Duckworth Lewis and Sprinting a Marathon
How would you like it if you were running a marathon and someone were to set you targets for every 100 meters? “Run the first 100m in 25 seconds. The second in 24 seconds” and so on? It is very likely that you would hate the idea. You would argue that the idea of the…