Leveraging data science and machine learning at scale, JustGiving is providing personalised experiences and identifying the causes that people actually care about. Richard Freeman, PhD, lead data and machine learning engineer, talks through the tools and processes underpinning the tech-for-good company’s platform, and why you should think twice before relying on third-party services
JustGiving has a long history of embracing data science. One of the earliest projects began in 2012 and funded by NESTA. We successfully built a recommendation engine that could actually identify people that are likely to fundraise.
In 2013, I joined to lead and deliver machine learning (ML) into production, in what was called the PANDA platform. This was challenging and rare – at a time when most ML was done offline and not embedded directly in a high-profile consumer product with 26 million users.
Back in 2013, PANDA was leveraging Apache MapReduce jobs and building model trainers’ pipelines and low latency model scoring APIs. By 2014, we had managed to change JustGiving products to make them ML-driven through recommendations, predictions and suggestions to help users raise more for good causes. Back then there was no Apache Spark, TensorFlow library, Docker container, or serverless computing which was only introduced into the platform later.
In parallel, around 2014, we noticed that our data scientists spent a lot of time preparing data, that our queries were growing in complexity, and that we were ingesting big data sets like web analytics data.
In response, I led the delivery of our in-house data science platform I called RAVEN in AWS, centred around massively parallel processing data warehouse Amazon Redshift.
RAVEN allowed us to join transactional data with non-transactional data (giving insight into user journeys), run experiments but also prepare the data for machine learning training and scoring at scale in PANDA.
The ML deployment objectives are to streamline the training, running of experiments and deployment of models into JustGiving products such as the fundraising page, feed and email campaigns.