Building a Future-Ready Data Platform:

The challenge 

IWSR wanted to evolve this forecast, applying a data science approach and combining other data sources to enhance it. This was to meet growing customer demand for improved forecasts, including forecasts that were able to go further into the future. Due to the forecasts currently being built manually, IWSR engaged a third-party to train an Artificial Intelligence (AI) model to produce forecasts but needed help to integrate this with their existing data and infrastructure, so that it could be used to automate the full forecasting process and data provisioning. 

To be able to continue innovating leveraging AI, IWSR would need to consolidate their data into a single platform in the long run.

What we delivered

As a trusted Databricks partner, Codurance has expertise in designing and delivering cutting-edge data platform solutions. The team strategically recommended Databricks, not just to address IWSR’s immediate challenge but to provide a scalable foundation for future data science and data engineering growth. 

Over a four month period, Codurance worked closely with senior stakeholders within IWSR including the CTO, Head of Data Engineering and Head of Research in an iterative manner to build out a data platform which integrated the AI model with IWSR’s data, and then published the AI model outputs via APIs for other systems to consume. As part of this the Codurance team set up and configured an instance of Databricks, which was hosted on AWS. The team also built a Microsoft Excel add-in tool to retrieve the AI model results set from an API exposed on Databricks to allow researchers to view and manipulate the data in an Excel spreadsheet. 

Outcomes

The data platform Codurance put in place allowed IWSR to productionise their forecasting ML model allowing them to fully realise its value. IWSR now has a standard process to interface with it which is used across the business. As a result, IWSR now has a more efficient data forecasting process than ever before. 

Critically, the data platform and processes Codurance put in place mean that data changes can now be fully versioned and audited, allowing for better governance of their data which in turn leads to increased data accuracy. 

The new improved forecasting process has also meant that individual researchers can now benefit from each other's data, something not previously possible.    

The data platform and an increased level of automation around data curation, transformation and provision has increased IWSR’s ability to scale into new markets globally. Leveraging Databricks, IWSR can continue its AI journey enabling the building of additional services providing additional revenue streams.