Showing posts with label Soapbox. Show all posts
Showing posts with label Soapbox. Show all posts
MacBook Air battery replacement
After four years of daily use our MacBook Air informed us that it needed a battery replacement. That's kind of nice to know, in particular as it still feels speedy and otherwise just works.
A new battery isn't that expensive and according to iFixit it appeared to be quite easy to replace it. I needn't to worry, it was actually super simple, given appropriate tools:
Here are a few pictures of the surgery:
A new battery isn't that expensive and according to iFixit it appeared to be quite easy to replace it. I needn't to worry, it was actually super simple, given appropriate tools:
- Remove 10 screws from bottom case
- Open case
- Disconnect battery
- Remove 5 screws from battery
- Swap battery
- Reassemble everything back together
- Job done.
Here are a few pictures of the surgery:
 |
| Lower case with screws removed |
 |
| Old battery pack |
 |
| New battery pack |
It is the small data that matters the most
Everyone is talking about Big Data1, but it is the small data that is holding everything together. The small slowly changing reference tables are the linchpins. Unfortunately, too often politics gets in the way as those small tables, maintained by humans, don't get the attention they deserve; or in other words their owners, if they exists - many of these little tables are orphans, make changes without understanding the potential consequences on downstream systems.
Here is a little story as it happened over the weekend.
A friend of mine flew on Saturday night from Vienna to Yerevan. The plane was scheduled to leave Austria at 10:15 PM and to arrive at 4:35 AM in Armenia. However, last weekend was the weekend when the summer time or daylight saving time (DST) ended in many European countries, including Austria, but not in Armenia.
Armenia used DST in 1981-1995 and 1997-2011. This may look a little odd, but Armenia is bar far not the only region that has changed its policy over the years. Russia decided in 2011 not to switch to winter time and keep DST all year round, only to switch to permanent winter time last weekend again.
Back to my friend, she arrives early Sunday morning at the airport in Yerivan, but her friend who promised to pick her up is not there. After a little while she takes a cab to her friends house, making it just in time before he was about to leave for the airport. What happened? Well, her friend relied on his mobile phone for his wake-up call. However, his mobile phone wasn't aware of the fact that Armenia didn't have DST anymore and set his clock back by one hour, allowing him more sleep. His phone might either be a little old or getting its signal from Russia, I don't know. Yet, this little story illustrates nicely how reliant we are nowadays on our interconnected devices having access to the correct reference data.
Imagine being in an intensive care unit relying on medial devices that have to work correctly together on time signals, when they are sourced from different companies, countries and running on different systems. Try to stay healthy, particular in late March and October, would be my advise.
So, how do systems/computers actually know which timezone to use?
Most computers will use the tz reference code and database maintained by a group of volunteers until 2011 and now by IANA. Still, how do they get informed that a region or state decided to change the rules? By keeping their ears on the ground!
The following extract from the Asia file of the tz database 2014b release is quite telling:
Does this all sounds familiar to you and the challenges in your own organisation? Well, I gave a talk on Small & Big Data recently with a colleague of mine, should you be interested to find out more about this topic.
1. or Tiny Data in Rasmus' case↩
Here is a little story as it happened over the weekend.
A friend of mine flew on Saturday night from Vienna to Yerevan. The plane was scheduled to leave Austria at 10:15 PM and to arrive at 4:35 AM in Armenia. However, last weekend was the weekend when the summer time or daylight saving time (DST) ended in many European countries, including Austria, but not in Armenia.
Armenia used DST in 1981-1995 and 1997-2011. This may look a little odd, but Armenia is bar far not the only region that has changed its policy over the years. Russia decided in 2011 not to switch to winter time and keep DST all year round, only to switch to permanent winter time last weekend again.
Back to my friend, she arrives early Sunday morning at the airport in Yerivan, but her friend who promised to pick her up is not there. After a little while she takes a cab to her friends house, making it just in time before he was about to leave for the airport. What happened? Well, her friend relied on his mobile phone for his wake-up call. However, his mobile phone wasn't aware of the fact that Armenia didn't have DST anymore and set his clock back by one hour, allowing him more sleep. His phone might either be a little old or getting its signal from Russia, I don't know. Yet, this little story illustrates nicely how reliant we are nowadays on our interconnected devices having access to the correct reference data.
Imagine being in an intensive care unit relying on medial devices that have to work correctly together on time signals, when they are sourced from different companies, countries and running on different systems. Try to stay healthy, particular in late March and October, would be my advise.
So, how do systems/computers actually know which timezone to use?
Most computers will use the tz reference code and database maintained by a group of volunteers until 2011 and now by IANA. Still, how do they get informed that a region or state decided to change the rules? By keeping their ears on the ground!
The following extract from the Asia file of the tz database 2014b release is quite telling:
# Armenia
# From Paul Eggert (2006-03-22):
# Shanks & Pottenger have Yerevan switching to 3:00 (with Russian DST)
# in spring 1991, then to 4:00 with no DST in fall 1995, then
# readopting Russian DST in 1997. Go with Shanks & Pottenger, even
# when they disagree with others. Edgar Der-Danieliantz
# reported (1996-05-04) that Yerevan probably wouldn't use DST
# in 1996, though it did use DST in 1995. IATA SSIM (1991/1998) reports
# that Armenia switched from 3:00 to 4:00 in 1998 and observed DST
# after 1991, but started switching at 3:00s in 1998.
# From Arthur David Olson (2011-06-15):
# While Russia abandoned DST in 2011, Armenia may choose to
# follow Russia's "old" rules.
# From Alexander Krivenyshev (2012-02-10):
# According to News Armenia, on Feb 9, 2012,
# http://newsarmenia.ru/society/20120209/42609695.html
#
# The Armenia National Assembly adopted final reading of Amendments to the
# Law "On procedure of calculation time on the territory of the Republic of
# Armenia" according to which Armenia [is] abolishing Daylight Saving Time.
# or
# (brief)
# http://www.worldtimezone.com/dst_news/dst_news_armenia03.html
# Zone NAME GMTOFF RULES FORMAT [UNTIL]
Zone Asia/Yerevan 2:58:00 - LMT 1924 May 2
3:00 - YERT 1957 Mar # Yerevan Time
4:00 RussiaAsia YER%sT 1991 Mar 31 2:00s
3:00 1:00 YERST 1991 Sep 23 # independence
3:00 RussiaAsia AM%sT 1995 Sep 24 2:00s
4:00 - AMT 1997
4:00 RussiaAsia AM%sT 2012 Mar 25 2:00s
4:00 - AMT
Does this all sounds familiar to you and the challenges in your own organisation? Well, I gave a talk on Small & Big Data recently with a colleague of mine, should you be interested to find out more about this topic.
1. or Tiny Data in Rasmus' case↩
Approximating the impact of inflation
The other day someone mentioned to me a rule of thumb that he was using to estimate the number of years \(n\) it would take for inflation to destroy half of the purchasing power of today's money:
\[ n = \frac{70}{p}\]
Here \(p\) is the inflation in percent, e.g. if the inflation rate is \(2\%\) then today's money would buy only half of today's goods and services in 35 years. You can also think of a saving account with an interest rate of \(2\%\) that would double your money in 35 years.
It is not difficult to derive this formula. The starting point is:
\[
2K = K (1 + \frac{p}{100})^n
\]
This is equivalent to:
\[
2 = (1 + \frac{p}{100})^n
\]
Taking the log gives:
\[
\log(2) = n \log(1 + \frac{p}{100})
\]
The first term of the Taylor series approximation of \(\log(1+x)\) for small \(x\) is \(x\). Hence for small \(p\) I can set:
\[
\log(2) \doteq n \, \frac{p}{100}
\]
Next I have to estimate the value for \(\log(2)\). Writing it as an integral leads to:
\[
\log(2) = \int_1^2 \frac{1}{x} \,dx
\]
Using Simpson's rule I can approximate the integral with:
\[
\int_1^2 \frac{1}{x} \,dx \doteq \frac{2-1}{6} (1+4\frac{2}{1+2}+\frac{1}{2} )
= \frac{25}{36} \doteq 0.7
\]
Thus,
\[
n \doteq \frac{70}{p}
\]
Plotting the two formulas against each other reveals that the approximation works pretty well, even for inflation rates up to 10%.
\[ n = \frac{70}{p}\]
Here \(p\) is the inflation in percent, e.g. if the inflation rate is \(2\%\) then today's money would buy only half of today's goods and services in 35 years. You can also think of a saving account with an interest rate of \(2\%\) that would double your money in 35 years.
It is not difficult to derive this formula. The starting point is:
\[
2K = K (1 + \frac{p}{100})^n
\]
This is equivalent to:
\[
2 = (1 + \frac{p}{100})^n
\]
Taking the log gives:
\[
\log(2) = n \log(1 + \frac{p}{100})
\]
The first term of the Taylor series approximation of \(\log(1+x)\) for small \(x\) is \(x\). Hence for small \(p\) I can set:
\[
\log(2) \doteq n \, \frac{p}{100}
\]
Next I have to estimate the value for \(\log(2)\). Writing it as an integral leads to:
\[
\log(2) = \int_1^2 \frac{1}{x} \,dx
\]
Using Simpson's rule I can approximate the integral with:
\[
\int_1^2 \frac{1}{x} \,dx \doteq \frac{2-1}{6} (1+4\frac{2}{1+2}+\frac{1}{2} )
= \frac{25}{36} \doteq 0.7
\]
Thus,
\[
n \doteq \frac{70}{p}
\]
Plotting the two formulas against each other reveals that the approximation works pretty well, even for inflation rates up to 10%.
R Code
Here is the R code to reproduce the plot.curve(70/x, from=1, to=10,
xlab="Inflation rate p%",
ylab="Number of years for purchaing power to half",
main="Impact of inflation on purchasing power",
col="blue",
type="p", pch=16, cex=0.5)
curve(log(2)/(log(1+x/100)),
from=1, to=10, add=TRUE,
col="red")
legend("topright",
legend=c("70/p","log(2)/log(1+p/100)"),
bty="n",
col=c("blue", "red"),
pch=c(16,16), pt.cex=c(1,1))The Wiener takes it all? A review of the 2014 Eurovision results
Saturday's Eurovision Song Contest (ESC) from Copenhagen was hilarious as usual with acts from all over Europe and some more or less sensible gimmicks: a circular piano, a giant hamster wheel, a sea-saw, or indeed a beard and fancy dress.
The results of the ESC were only a little different to what the bookmakers in the UK had predicted before the event started. Sweden was seen as the favourite, followed by Austria, Netherlands, Armenia and the UK.
In the end Conchita Wurst from Austria won with 290 points in front of the acts from the Netherlands with 238, Sweden with 218 and Armenia with 174 points. The UK ended on the 17th rank with only 40 points. Perhaps the reason the English bookies had put the UK in fifth place reflected the bias of their clients towards their home country.
The points were given as a combination of a public tele vote and a jury in each country. But how big was the resemblance between the votes of the juries and the public? Is there much variance between countries?
The detailed voting data are available from the Eurovision site. Out of the 37 countries that participated in the voting, two countries, San Marino and Albania, had no tele voting and one country, Georgia, had no jury. In the remaining 34 countries Austria and the Netherlands made into the top 2 of the jury and public voting results if they were treated independently. Sweden made it into the top 5 of both as well, but the other countries differed.
So, how consistent was the voting of the jury and public for the top 5 in each country?
Well, in only 11 countries out of 34, the jury agreed with the public on more than 2 of the top 5 songs. In 25 countries the jury and public agreed on at least two candidates.
In some cases the differences between public and jury were so wide, that although a candidate was voted into the top 5 of the tele rankings the act wouldn't get any points. The top public winner in Belgium (Armenia), Ireland (Poland), Montenegro (Russia) and the UK (Poland) didn't get any points at all.
Still, getting the top favourites right seems much easier then any of the followers. Or in other words, it is really difficult to produce a hit, a song/act on which many can agree. But when you hear one, it is much easier to identify it as such, something you really like and believe others would like as well.
R Code
Eurovision
,
googleVis
,
R
,
Soapbox
How many more R-bloggers posts can I expect?
I noticed that the monthly number of posts on R-bloggers stopped increasing over the last year. Indeed, the last couple of months saw a decline in posts compared to the previous year. Thus, has most been said and written about R already?
Who knows? Well, I took a stab at looking into the future. However, I can tell you already that I am not convinced by my predictions. But maybe someone else will be inspired to take this work forward.
First, I have to get the data - that's easy, I can scrape the monthly post counts from the R-bloggers homepage.
Looking at the incremental and cumulative plots, and believing that eventually the number of R posts will decrease, I thought that a logistic growth function would provide a nice fit to the data and also give an asymptotic view of the total number of posts on R-bloggers.
Although the fit, see below, looks reasonable at first glance, I don't believe it provides a sensible prediction of the future. The model would forecast only another 1,269 post by the end of 2016 with not much more to expect after that. Indeed the asymptotic total number of posts K is only 14,396. I don't believe this can be right, not even as a proxy, when the current count of monthly posts is well above 100.
I played around with data and the logistic growth function a little further, using annual instead of monthly data, changing the time horizon and fixing K, yet without much success.
Eventually I recalled a talk by Rob Hyndman's about his forecast package. After all, I have a time series here. So, applying the
Well, I presented two models here: One predicts a rapid decline in monthly posts on R-bloggers, while the other forecasts an increase. Neither feels right to me. Of course time will tell, but have you got any ideas or views?
Who knows? Well, I took a stab at looking into the future. However, I can tell you already that I am not convinced by my predictions. But maybe someone else will be inspired to take this work forward.
First, I have to get the data - that's easy, I can scrape the monthly post counts from the R-bloggers homepage.
Looking at the incremental and cumulative plots, and believing that eventually the number of R posts will decrease, I thought that a logistic growth function would provide a nice fit to the data and also give an asymptotic view of the total number of posts on R-bloggers.
Although the fit, see below, looks reasonable at first glance, I don't believe it provides a sensible prediction of the future. The model would forecast only another 1,269 post by the end of 2016 with not much more to expect after that. Indeed the asymptotic total number of posts K is only 14,396. I don't believe this can be right, not even as a proxy, when the current count of monthly posts is well above 100.
I played around with data and the logistic growth function a little further, using annual instead of monthly data, changing the time horizon and fixing K, yet without much success.
Eventually I recalled a talk by Rob Hyndman's about his forecast package. After all, I have a time series here. So, applying the
forecast function to the incremental data provides a somewhat more realistic prediction of 2,695 posts for the next 12 months, but with an increasing trend in monthly posts for 2014, which I find hard to believe given the observations over the last year.
Well, I presented two models here: One predicts a rapid decline in monthly posts on R-bloggers, while the other forecasts an increase. Neither feels right to me. Of course time will tell, but have you got any ideas or views?
Session Info
R version 3.0.2 (2013-09-25)
Platform: x86_64-apple-darwin10.8.0 (64-bit)
locale:
[1] en_GB.UTF-8/en_GB.UTF-8/en_GB.UTF-8/C/en_GB.UTF-8/en_GB.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] forecast_4.8 xts_0.9-7 zoo_1.7-10 XML_3.95-0.2
loaded via a namespace (and not attached):
[1] colorspace_1.2-4 fracdiff_1.4-2
[3] grid_3.0.2 lattice_0.20-23
[5] nnet_7.3-7 parallel_3.0.2
[7] quadprog_1.5-5 Rcpp_0.10.6
[9] RcppArmadillo_0.3.920.1 tools_3.0.2
[11] tseries_0.10-32
forecast
,
logistic growth
,
R
,
R-bloggers.com
,
Soapbox
Whale charts - Visualising customer profitability
The Christmas and New Year's break is over, yet there is still time to return unwanted presents. Return to Santa was the title of an article in the Economist that highlighted the impact on online retailers, as return rates can be alarmingly high.
The article quotes a study by Christian Schulze of the Frankfurt School of Finance and Management, which analyses the return habits of customers who bought at least five items over a five year period from a large European online retailer. Although only a few figures are cited I will attempt to create a little model that replicates the customer behaviour and visualises the impact on overall profitability.
The study found that 5% of customers sent back more than 80% of the items they had bought; and that 1% of customers sent back at least 90% of their purchases. Or in other words 95% of customers send back less than 80% and 99% of customers send back less than 90%. To model this behaviour an S-shape curve seems appropriate, such as the logistic curve, as no-one can return more than they bought or less than nothing. With location and scale parameters m and s the logistic function can be fitted to the data, see the R code below.
The return rates do look quite high. However, if the products were shoes rather than books then I find them believable.
Additionally the article cites studies that suggest handling each returned item costs online sellers between $6 and $18, not to mention losses from items that are returned in unsaleable condition. Furthermore, without the cost of returns, online retailer's profits would be almost 50% higher.
Thus, to spin my toy model further, I assume 100 customers with revenues following an exponential distribution (λ=1/250), the cost ratio of sold goods to be lognormal (μ=-0.1, σ=0.1) and the cost of returns to follow a normal distribution with mean of $12 and standard deviation of $6.
In my simulation I could have made a profit of $1,979 instead of $1,441. Clearly the customers who return many items cause a real dent to my bottom line.
This situation is best visualised in what is often called a Whale Chart. Here I plot the cumulative profit against customers, with the most profitable customer on the left and the least profitable customer on the right. This chart shows me how much profit the first x number of customers generated. Often this graphs looks like a whale coming out of the water - hence its name.
In my little toy simulation I note that the first 20 most profitable customers would have generated more profit than the revenue of all customers. Indeed, profitability could have been 37% higher if it wasn't for loss making customers.
So, what shall I do? Manage my customers, know who I should reward and keep and whose loss wouldn't hurt at all. More customers are not the answer. I need more customers who return less.
The article quotes a study by Christian Schulze of the Frankfurt School of Finance and Management, which analyses the return habits of customers who bought at least five items over a five year period from a large European online retailer. Although only a few figures are cited I will attempt to create a little model that replicates the customer behaviour and visualises the impact on overall profitability.
The study found that 5% of customers sent back more than 80% of the items they had bought; and that 1% of customers sent back at least 90% of their purchases. Or in other words 95% of customers send back less than 80% and 99% of customers send back less than 90%. To model this behaviour an S-shape curve seems appropriate, such as the logistic curve, as no-one can return more than they bought or less than nothing. With location and scale parameters m and s the logistic function can be fitted to the data, see the R code below.
The return rates do look quite high. However, if the products were shoes rather than books then I find them believable.
Additionally the article cites studies that suggest handling each returned item costs online sellers between $6 and $18, not to mention losses from items that are returned in unsaleable condition. Furthermore, without the cost of returns, online retailer's profits would be almost 50% higher.
Thus, to spin my toy model further, I assume 100 customers with revenues following an exponential distribution (λ=1/250), the cost ratio of sold goods to be lognormal (μ=-0.1, σ=0.1) and the cost of returns to follow a normal distribution with mean of $12 and standard deviation of $6.
In my simulation I could have made a profit of $1,979 instead of $1,441. Clearly the customers who return many items cause a real dent to my bottom line.
This situation is best visualised in what is often called a Whale Chart. Here I plot the cumulative profit against customers, with the most profitable customer on the left and the least profitable customer on the right. This chart shows me how much profit the first x number of customers generated. Often this graphs looks like a whale coming out of the water - hence its name.
In my little toy simulation I note that the first 20 most profitable customers would have generated more profit than the revenue of all customers. Indeed, profitability could have been 37% higher if it wasn't for loss making customers.
So, what shall I do? Manage my customers, know who I should reward and keep and whose loss wouldn't hurt at all. More customers are not the answer. I need more customers who return less.
R Code
Economist
,
R
,
Soapbox
,
Whale chart
Why models need a certain culture to flourish
About half a year ago Ian Branagan, Chief Risk Officer of Renaissance Re - a Bermudian reinsurance company with a focus on property catastrophe insurance, gave a talk about the usage of models in risk management and how they evolved over the last twenty years. Ian's presentation, titled with the famous quote of George E.P. Box: "All models are wrong, but some are useful", was part of the lunch time lecture series of talks at Lloyd's, organised by the Insurance Institute of London.
I re-discovered the talk online over the weekend and found it most enlightening again.
So, what makes models useful? And here I mean models that estimate extreme outcomes / percentiles. Three factors are critical, according to Ian, to embed models successfully in risk management and decision making processes.
Where in the past senior management may have relied on advisors' expert judgement to guide them in their decision makings, they have to use models in a similar way now as well. I suppose, in the same way as it takes time and effort to build effective relationships with people, it is true for models as well. And equally, decisions should never rely purely on either other people's opinion or indeed model output. As Ian put it, outsourcing all modelling/thinking, and with that the decision making to vendors of models, such as catastrophe modelling companies or rating agencies, who both aim to provide probabilities for extreme events (catastrophes and companies failures) may be sufficient to tick a risk management box, but can ultimately put the company at risk, if model assumptions and limitations are not well understood.
Perhaps we are at the dawn of another enlightenment? Recall Kant's first sentence of his essay What is enlightenment?: "Enlightenment is man's emergence from his self-incurred immaturity." Indeed, it doesn't matter if we use experts' opinions or the output of models, relying blindly on them is dangerous and foolish. Don't stop thinking for yourself. Be critical! Remember, all models are wrong, but some are useful.
I re-discovered the talk online over the weekend and found it most enlightening again.
So, what makes models useful? And here I mean models that estimate extreme outcomes / percentiles. Three factors are critical, according to Ian, to embed models successfully in risk management and decision making processes.
- Need - A clear defined need for the model.
- Capabilities - The skills and resources to build and maintain the model.
- Culture - An organisational culture that embraces, understands and challenges the model.
Where in the past senior management may have relied on advisors' expert judgement to guide them in their decision makings, they have to use models in a similar way now as well. I suppose, in the same way as it takes time and effort to build effective relationships with people, it is true for models as well. And equally, decisions should never rely purely on either other people's opinion or indeed model output. As Ian put it, outsourcing all modelling/thinking, and with that the decision making to vendors of models, such as catastrophe modelling companies or rating agencies, who both aim to provide probabilities for extreme events (catastrophes and companies failures) may be sufficient to tick a risk management box, but can ultimately put the company at risk, if model assumptions and limitations are not well understood.
Perhaps we are at the dawn of another enlightenment? Recall Kant's first sentence of his essay What is enlightenment?: "Enlightenment is man's emergence from his self-incurred immaturity." Indeed, it doesn't matter if we use experts' opinions or the output of models, relying blindly on them is dangerous and foolish. Don't stop thinking for yourself. Be critical! Remember, all models are wrong, but some are useful.
Installing a SSD drive into a mid-2007 iMac
I have a mid-2007 iMac with a 2.4 GHz Core2Duo processor and despite the fact that it is already six years old, it still does a good job. However, compared to a friend's recent MacBook Air with a solid state disk (SSD) it felt sluggish when opening programmes and loading larger documents.
So, I thought it would be worthwhile to replace the old spinning hard disk drive with an SSD, instead of buying a new computer. I still like the display of the iMac. Hence, I got myself a Samsung 840 SATA drive, as it came with a USB cable and bracket hard drive holder.
It actually wasn't that difficult to replace the hard drive in my iMac. Of course I can give no guarantee that this works for you as well. Here are the steps I took:
Here are a few pictures of the surgery:
It actually wasn't that difficult to replace the hard drive in my iMac. Of course I can give no guarantee that this works for you as well. Here are the steps I took:
- Connected the new SSD via USB to my iMac
- Cloned old internal hard drive to new SSD, using Carbon Copy Cloner
- Tested new SSD extensively. Booted the Mac from the new SSD connected externally via USB.
- Followed iFixit guide and replaced hard drive
- Crossed my fingers and switched my iMac on again.
Here are a few pictures of the surgery:
 |
| iMac with glass panel and bezel removed |
 |
| Display and hard drive removed |
 |
| New SSD and old HDD |
 |
| Screen shot of the new system preferences |
iMac
,
replace hard drive
,
Soapbox
,
SSD
Don't be misguided by the beauty of mathematics, if the data tells you otherwise
I was trained as a mathematician and it was only last year, when I attended the Royal Statistical Society conference and met many statisticians that I understood how different the two groups are.
In mathematics you often start with some axioms, things you assume to be true, and these axioms are then the basis from which new theory is derived. In statistics or more general in science you start with a theory, or better a hypothesis and try to disprove it. And if you can't disprove it, you accept it until you have other evidence. Or to phrase it like Karl R. Popper: you can only be proven wrong.
Now, why do I mention this? I have met many mathematicians who talk about the beauty of mathematics and I agree, a mathematical concept, theorem or proof can indeed be beautiful. However, when you work in applied mathematics and particular when you use mathematics to build models, there is a danger that you stick to the beautiful idea and ignore reality. Remember the financial crisis?
For example, it might be handy to assume that your data follow a normal distribution, e.g. to make the calculations easier. However, if the data tells you otherwise then be bold and ruthless and change your model. As strange as it might sound, it is has to be your aim to prove a model doesn't work in order to use it successfully.
Remember Pythagoras? He believed in beautiful integers and the realisation that the square root of two was not a fraction of two integers caused a big crisis.
I would argue that we need mathematics to do statistics and statistics to do science. The developments over the last 350 years really demonstrate the success the scientific method. Of course some ideas had to go: the earth can no longer be regarded as the centre our solar system - instead it appears more like a little pale blue dot.
Diggle and Chetwynd, from Lancaster University, published a nice little book that gives a good introduction into statistics and of the scientific method. Two quotes of the book stuck in my mind (pages 1&2):
A scientific theory cannot be proved in the rigours sense of a mathematical theorem. But it can be falsified, meaning that we can conceive of an experimental or observational study that would show the theory to be false.
...
The American physicist Richard Feynman memorable said that 'theory' was just a fancy name for a guess. If observation is inconsistent with theory then the theory, however elegant, has to go. Nature cannot be fooled.
In mathematics you often start with some axioms, things you assume to be true, and these axioms are then the basis from which new theory is derived. In statistics or more general in science you start with a theory, or better a hypothesis and try to disprove it. And if you can't disprove it, you accept it until you have other evidence. Or to phrase it like Karl R. Popper: you can only be proven wrong.
Now, why do I mention this? I have met many mathematicians who talk about the beauty of mathematics and I agree, a mathematical concept, theorem or proof can indeed be beautiful. However, when you work in applied mathematics and particular when you use mathematics to build models, there is a danger that you stick to the beautiful idea and ignore reality. Remember the financial crisis?
For example, it might be handy to assume that your data follow a normal distribution, e.g. to make the calculations easier. However, if the data tells you otherwise then be bold and ruthless and change your model. As strange as it might sound, it is has to be your aim to prove a model doesn't work in order to use it successfully.
Remember Pythagoras? He believed in beautiful integers and the realisation that the square root of two was not a fraction of two integers caused a big crisis.
I would argue that we need mathematics to do statistics and statistics to do science. The developments over the last 350 years really demonstrate the success the scientific method. Of course some ideas had to go: the earth can no longer be regarded as the centre our solar system - instead it appears more like a little pale blue dot.
Diggle and Chetwynd, from Lancaster University, published a nice little book that gives a good introduction into statistics and of the scientific method. Two quotes of the book stuck in my mind (pages 1&2):
A scientific theory cannot be proved in the rigours sense of a mathematical theorem. But it can be falsified, meaning that we can conceive of an experimental or observational study that would show the theory to be false.
...
The American physicist Richard Feynman memorable said that 'theory' was just a fancy name for a guess. If observation is inconsistent with theory then the theory, however elegant, has to go. Nature cannot be fooled.
falsifiability
,
mathematics
,
models
,
science
,
Soapbox
,
statistics
Test Driven Analysis?
At the last LondonR meeting Francine Bennett from Mastodon C shared some of her experience and findings from an analysis of a large prescriptions data set of the UK's national health service (NHS). However, it was her last slide, which I found the most thought provoking. It asked for the definition of the following term:
Indeed, how do I go about test driven analysis? How do I know that I haven't made a mistake, when I start an analysis of a new data set? Well, I don't. But I try to mitigate risks. Similar to TDD, I consider which outputs I should expect from my analysis. Those outputs form the test scenarios of my analysis. Basically I try to write down everything I know, before I start working with the data, e.g.
Test-driven analysis?Francine explained that test driven development (TDD) is a concept often used in software development for quality assurance and she wondered if a similar approach could be also used for data analysis. Unfortunately the audience couldn't provide her with the answer, but many expressed that they face similar challenges. So do I.
Indeed, how do I go about test driven analysis? How do I know that I haven't made a mistake, when I start an analysis of a new data set? Well, I don't. But I try to mitigate risks. Similar to TDD, I consider which outputs I should expect from my analysis. Those outputs form the test scenarios of my analysis. Basically I try to write down everything I know, before I start working with the data, e.g.
- any other data sets or reports I can use for cross referencing,
- any back-of-the-envelope analysis I can carry out to provide ballpark answers,
- any relativities and ratios which should hold true,
- any known boundaries and thresholds,
- test scenarios for my code with small well known data, for which I know the outcome,
- names of experts, who could sense check and peer review my output.
Analysis
,
LondonR
,
R
,
Soapbox
,
Test Driven Analysis
Next Kölner R User Meeting: 12 April 2013
Quick reminder: The next Cologne R user group meeting is scheduled for this Friday, 12 April 2013. We will discuss cluster analysis and shiny. Further details and the agenda are available on our KölnRUG Meetup site. Please sign up if you would like to come along. Notes from the last Cologne R user group meeting are available here.
Thanks also to Revolution Analytics, who sponsors the Cologne R user group as part of their vector programme.
Kölner R Users
,
KölnR
,
News
,
R
,
Soapbox
Top 10 tips to get started with R
- Be motivated. R has a steep learning curve. Find a problem you can't solve otherwise. E.g. plotting multivariate data, a statistical analysis for which an R function exists already.
- Download and install R. Get to know the R console. Learn how to install additional packages, how to access the history, how to use auto completion and open the help system. Review the R Installation and Administration manual and check out the free books section on CRAN.
- Get familiar with the R help files. They can appear cryptic at the start, but there is a structure to them. Read and re-read a couple of help files again and again. Look out for the input and output sections, execute the examples, run the demos, e.g.
demo(graphics). Subscribe to R-help and read questions and answers, check out stackoverflow, follow blogs. Search with Rseek.org.
- Learn how to get your data into R. The easiest way is usually via a CSV-file (CSV=comma separated values), using
read.csv. Look into XLConnect, if you have to deal with spreadsheet files. Move on to write queries against data bases, e.g. using RODBC. Skim through the R Data Import/Export manual.
- Try to understand the different data types in R and how to modify them. What are the differences between a matrix and a data frame? What is a factor? What is a list? Think about the different use cases. Review the Introduction to R manual.
- Do charts! Lots of charts. They are rewarding and keep you motivated. Be inspired by the R Graph Gallery. Check out the following packages: lattice, plotrix, ggplot2, deducer, googleVis.
- Learn how you can modify and reshape data in R and apply functions on subsets using
by, apply, lapply, ave, reshape, sweep, with, within, etc. Set aside a weekend to think about these functions. - Write your R code into files instead of typing it all into the R console. Use an integrated development environment (IDE), e.g. ESS Emacs, RStudio, StatET Eclipse.
- Understand the concept of functions. Write a function, which gives "Hello World" back. Modify it, so it has an input argument NAME and it prints "Hello NAME". Review the code of existing R functions. Copy from existing code.
- Document your code! Start your code by explaining what you want to achieve and only code that much, then write down the next step in plain English and code again. How will you know that your code does what you want it to do? Testing can help. Think about your code style and how you will be versioning your files.
Bonus tip
Next Kölner R User Meeting: 6 February 2013
Quick reminder: The next Cologne R user group meeting is scheduled for tomorrow, 6 February 2013. All details and the agenda are available on the KölnRUG Meetup site. Please sign up if you would like to come along. Notes from the last Cologne R user group meeting are available here.
Thanks also to Revolution Analytics, who are sponsoring the Cologne R user group as part of their vector programme.
Kölner R Users
,
KölnR
,
News
,
R
,
Soapbox
Follow the ants to richness
A friend of mine told me the secret of making money at the stock market. "It's easy", he said.
All I would have to do is to buy a big jar of ants. Then I should observe the ants movement on my kitchen table, while following the stock market.
I shall keep the ants which walk in line with the stock market and remove those who don't. Eventually I would have one ant left that walked all the way in line with the stock market.
Bingo! This is the one I have to keep feeding well and observe, as it clearly can predict the movements of the stock market.
For more complex problems I recommend to use animals with bigger brains.
All I would have to do is to buy a big jar of ants. Then I should observe the ants movement on my kitchen table, while following the stock market.
I shall keep the ants which walk in line with the stock market and remove those who don't. Eventually I would have one ant left that walked all the way in line with the stock market.
Bingo! This is the one I have to keep feeding well and observe, as it clearly can predict the movements of the stock market.
For more complex problems I recommend to use animals with bigger brains.
ants
,
R
,
random walk
,
Soapbox
,
stock market
Comparing regions: maps, cartograms and tree maps
Last week I attended a seminar where a talk was given about the economic opportunities in the SAAAME (South-America, Asia, Africa and Middle East) regions. Of course a map was shown with those regions highlighted. The map was not that disimilar to the one below.
Greenland appears to be of the same size as Brazil or Australia and Africa seems to be only about four times as big as Greenland. Of course this is not true. Africa is about 14 times the size of Greenland, while Brazil and Australia are about four times the size of Greenland, with Brazil slightly larger than Australia and nine times the population. Thus, talking about regional opportunities without a comparable scale can give a misleading picture.
Of course you can use a different projection and XKCD helps you to find one which fits your personality. On the other hand, Michael Gastner and Mark Newman developed cartograms, which can reshape the world based on data [1]. Doing this in R is a bit tricky. Duncan Temple Lang provides the Rcartogram package on Omegahat based on Mark Newman's code and Mark Ward has some examples using the package on his 2009 fall course page to get you started.
A simple example of a cartogram is given as part of the maps package. It shows the US population by state:
library(RColorBrewer)
library(rworldmap)
data(countryExData)
par(mai=c(0,0,0.2,0),xaxs="i",yaxs="i")
mapByRegion( countryExData,
nameDataColumn="GDP_capita.MRYA",
joinCode="ISO3", nameJoinColumn="ISO3V10",
regionType="Stern", mapTitle=" ", addLegend=FALSE,
FUN="mean", colourPalette=brewer.pal(6, "Blues"))Greenland appears to be of the same size as Brazil or Australia and Africa seems to be only about four times as big as Greenland. Of course this is not true. Africa is about 14 times the size of Greenland, while Brazil and Australia are about four times the size of Greenland, with Brazil slightly larger than Australia and nine times the population. Thus, talking about regional opportunities without a comparable scale can give a misleading picture.
Of course you can use a different projection and XKCD helps you to find one which fits your personality. On the other hand, Michael Gastner and Mark Newman developed cartograms, which can reshape the world based on data [1]. Doing this in R is a bit tricky. Duncan Temple Lang provides the Rcartogram package on Omegahat based on Mark Newman's code and Mark Ward has some examples using the package on his 2009 fall course page to get you started.
A simple example of a cartogram is given as part of the maps package. It shows the US population by state:
Time for an old classic game: Moon-buggy
I discovered an old classic game of mine again: Moon-buggy by Jochen Voss, based on the even older Moon Patrol, which celebrates its 30th birthday his year.
game
,
moon-buggy
,
Soapbox
From guts to data driven decision making
 |
| Source: Wikipedia, License: CC0 |
The whole sketch is hilarious and is often regarded as a fine observation of miss-communication.
Yet, I think it really points out two different approaches in decision making: You can trust your guts or use data/measurements to support your decision.
Next Kölner R User Meeting: 5 October 2012
The next Cologne R user group meeting is scheduled for 5 October 2012. All details and the agenda are available on the KölnRUG Meetup site. Please sign up if you would like to come along. Notes from the last Cologne R user group meeting are available here.
Thanks also to Revolution Analytics, who are sponsoring the Cologne R user group as part of their vector programme.
Kölner R Users
,
KölnR
,
News
,
R
,
Soapbox
Connecting data to the real world - The next sexy job?
At last week's Royal Statistical Society (RSS) conference Hal Varian, Chief Economist at Google, gave a panel talk about 'Statistics at Google'. Could he get a better audience than the RSS?
Hal talked about his career in academia and at Google. He reminded us of the days when Google was still a small start up with no real idea about how they could actually generate revenue. At that time Eric Schmidt asked him to 'take a look' at advertising because 'it might make us a little money'. Thus, Hal got involved in Google's ad auctions.
 |
| Hal Varian at the Royal Statistical Society conference 2012 |
Another projects Hal talked about was predicting the present. Predicting the present, or 'nowcasting', is about finding correlations between events. The idea is to forecast economic behaviour, which in return can help to answer when to run certain ads. He gave the example of comparing the search requests for 'vodka' (peaking Saturdays) with 'hangover' (peaking Sundays) using Google Insight.
Hal Varian
,
robotics
,
RSS
,
Sebastian Thrun
,
Soapbox
,
statistics
Are career motivations changing?
The German news magazine Der Spiegel published a series of articles [1, 2] around career developments. The stories suggest that career aspirations of young professionals today are somewhat different to those of previous generations in Germany.
Apparently money and people management responsibility are less desirable for new starters compared to being able to participate in interesting projects and to maintain a healthy work life balance. Hierarchies are seen as a mean to an end, and should be more flexible, depending on requirements and skills sets. Similar to how they evolve in online communities and projects.
career
,
motivation
,
Simon Sinek
,
Soapbox
,
The RSA
