mages' blog

Last week I was invited to give an introduction to googleVis at Lancaster University. This time I decided to use the R package slidify for my talk. Slidify, like knitr, is built on Markdown and makes it very easy to create beautiful HTML5 presentations.

Introduction to googleVis

Separating content from layout is always a good idea. Markup languages such as TeX/LaTeX or HTML are built on this principle. Ramnath Vaidyanathan has done a fantastic job with slidify, as it is very straightforward to create presentations with R. There are a couple of advantages compared to traditional presentation software packages:

• RMarkdown helps me to focus on the content
• Integration of R code is build in
• HTML5 allows me to embed interactive content, such as
  
  • Videos
  • googleVis and other interactive charts
  • shiny apps (more on this next week)

In the past I have used knitr in combination with pandoc to generate a slidy presentation. However, with slidfiy I can do all this in R directly. And better, Ramnath provides me with a choice of different layout frameworks and syntax highlighting options. Finally to top it all, publishing the slides on Github was only one more R statement: publish('mages', 'Introduction_to_googleVis').

I will give a half-day tutorial on googleVis with Diego de Castillo at useR2013! in Albacete on 9 July 2013. I hope to see some of you there.

I was trained as a mathematician and it was only last year, when I attended the Royal Statistical Society conference and met many statisticians that I understood how different the two groups are.

In mathematics you often start with some axioms, things you assume to be true, and these axioms are then the basis from which new theory is derived. In statistics or more general in science you start with a theory, or better a hypothesis and try to disprove it. And if you can't disprove it, you accept it until you have other evidence. Or to phrase it like Karl R. Popper: you can only be proven wrong.

Now, why do I mention this? I have met many mathematicians who talk about the beauty of mathematics and I agree, a mathematical concept, theorem or proof can indeed be beautiful. However, when you work in applied mathematics and particular when you use mathematics to build models, there is a danger that you stick to the beautiful idea and ignore reality. Remember the financial crisis?

For example, it might be handy to assume that your data follow a normal distribution, e.g. to make the calculations easier. However, if the data tells you otherwise then be bold and ruthless and change your model. As strange as it might sound, it is has to be your aim to prove a model doesn't work in order to use it successfully.

Remember Pythagoras? He believed in beautiful integers and the realisation that the square root of two was not a fraction of two integers caused a big crisis.

I would argue that we need mathematics to do statistics and statistics to do science. The developments over the last 350 years really demonstrate the success the scientific method. Of course some ideas had to go: the earth can no longer be regarded as the centre our solar system - instead it appears more like a little pale blue dot.

Diggle and Chetwynd, from Lancaster University, published a nice little book that gives a good introduction into statistics and of the scientific method. Two quotes of the book stuck in my mind (pages 1&2):


A scientific theory cannot be proved in the rigours sense of a mathematical theorem. But it can be falsified, meaning that we can conceive of an experimental or observational study that would show the theory to be false.
...
The American physicist Richard Feynman memorable said that 'theory' was just a fancy name for a guess. If observation is inconsistent with theory then the theory, however elegant, has to go. Nature cannot be fooled.

Over the last year I worked with two colleagues of mine on the subject of inflation and claims inflation in particular. I didn't expect it to be such a challenging topic, but we ended up with more questions than answers. The key question and biggest challenge is to define what inflation, or indeed claims inflation actually is and how to measure it. We published a summary of our thoughts and findings in this month's issue of The Actuary.

Last year's discussion about the differences between the retail price index (RPI) and consumer price index (CPI) in the UK only exemplified the challenge. The economist Tim Harford illustrated the differences between the RPI and CPI with a simple example of price changes for a shirt and blouse in his Radio 4 programme More or Less. The radio podcast is still available from the BBC. Start listening after about 18 minutes into the show.

I am delighted to announce that the programme and abstracts for the first R in Insurance conference at Cass Business School in London, 15 July 2013, have been published.

The conference committee received strong abstracts from academia and the industry, covering:

• Pricing
• Reserving
• Data mining
• Capital modelling
• Automate reporting
• Catastrophe modelling
• High-performance computing
• Software development management

Register by the end of May to get the early bird booking fee.

We gratefully acknowledge the sponsorship of Mango Solutions and CYBAEA, without whom the event wouldn't be possible.

Programme and Abstracts

Register by the end of May to get the early bird booking fee.

Often I like to reduce the alpha value (level of transparency) of colours to identify patterns of over-plotting when displaying lots of data points with R. So, here is a tiny function that allows me to add an alpha value to a given vector of colours, e.g. a RColorBrewer palette, using col2rgb and rgb, which has an argument for alpha, in combination with the wonderful apply and sapply functions.


The example below illustrates how this function can be used with colours provided in different formats, thanks to the col2rgb function.

Our 5th Cologne R user group meeting was the best attended meeting so far, with 20 members finding their way to the Institute of Sociology for two talks by Diego de Castillo on shiny and Stephan Holtmeier on cluster analysis, followed by beer and schnitzel at the Lux, a gastropub nearby.

Shiny

Diego gave an overview of the design principles behind shiny, which provides a powerful API to build web apps in pure R. His explanation of the reactive programming model was particularly helpful to understand how shiny works under the hood and why it is so responsive. His live demonstrations of shiny even included shiny server, which he had running in a virtual machine. Diego's slides are available via our Meetup site.

Diego de Castillo: Introduction to shiny

You can hear more from Diego and me at the UseR!2013 conference in Albacete, where we will give a googleVis tutorial. We will touch on googleVis on shiny as well. A dedicated shiny tutorial will be given in the afternoon by Josh and Winston from RStudio.

Cluster analysis

Stephan Holtmeier, who is a psychologist by background, presented an introduction to cluster analysis with R, motivated by his work in analysing survey data. As a toy example he used a 360° feedback survey of a group of managers within a big company. In his example he wanted to understand the profile of those managers better. Stephan illustrated how a cluster analysis can help to identify groups of managers with similar strengths, e.g. for communication, leadership and/or performance. Depending on how he measured the distance between managers he could look for people who have similar levels of competency or a similar profile (correlation). Stephan also touched on the differences between hierarchical and centroid based cluster analysis, such as k-means. You can find Stephan's slides (in German) also on our Meetup site.

Stephan Holtmeier: Cluster Analysis with R

For more information on cluster analysis functions in R see also the cluster task view on CRAN. If you would like to get an overview of how psychologists look at data, then check out William Revelle's vignette of the psych package. Finally, if you are interested in how a k-means cluster analysis can be used for image manipulation, see an earlier post of mine.

Next Kölner R meeting, 19 July 2013

The next meeting has been scheduled for 19 July. Günter Faes will present his experiences using the XLConnect package as an interface between R and Excel. Dietmar Janetzko agreed to present how he used R and Twitter to predict exchange rate movements. Of course, the evening will close with a few Kölsch in a nearby beer-garden.

Please get in touch if you would like to present and share your experience, or indeed if you have a request for a topic you would like to hear more about. For more details see also our Meetup page.

Thanks again to Bernd Weiß for hosting the event and Revolution Analytics for their sponsorship.

At the last LondonR meeting Francine Bennett from Mastodon C shared some of her experience and findings from an analysis of a large prescriptions data set of the UK's national health service (NHS). However, it was her last slide, which I found the most thought provoking. It asked for the definition of the following term:

Test-driven analysis?

Francine explained that test driven development (TDD) is a concept often used in software development for quality assurance and she wondered if a similar approach could be also used for data analysis. Unfortunately the audience couldn't provide her with the answer, but many expressed that they face similar challenges. So do I.


Indeed, how do I go about test driven analysis? How do I know that I haven't made a mistake, when I start an analysis of a new data set? Well, I don't. But I try to mitigate risks. Similar to TDD, I consider which outputs I should expect from my analysis. Those outputs form the test scenarios of my analysis. Basically I try to write down everything I know, before I start working with the data, e.g.

• any other data sets or reports I can use for cross referencing,
• any back-of-the-envelope analysis I can carry out to provide ballpark answers,
• any relativities and ratios which should hold true,
• any known boundaries and thresholds,
• test scenarios for my code with small well known data, for which I know the outcome,
• names of experts, who could sense check and peer review my output.

But most importantly: I try to think long and hard which questions I want to answer, following the advice of John Tukey: Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise.