Paddy Power Betfair: Ripping up the sports betting rule book through digitisation

Headquartered in Dublin, Ireland, bookmaker Paddy Power Betfair (PPB) has come a long way in the short time since its formation through the merger of rivals Paddy Power and Betfair in 2015.  Both organisations came from humble origins taking a journey that has completely digitised the customer experience, opened new markets and developed new tools to drive innovation to further attract and retain its growing customer base.

Technology has been critical to the growth of PPB, as the global online betting market has risen from $20bn in 2009 to over $40bn in 2016. PPB has put an emphasis on offering the best features in responsive applications coupled with powerful and keen pricing models that offer greater value to customers. Digitisation has also enabled the business to offer the right products for customers at optimal times, harnessing sophisticated analytics whilst developing an exceptional digital infrastructure along the way. 

“Accelerating innovation around customer application development has become a focus. We work to constantly shorten the ideation-to-production cycle using agile, continuous delivery and infrastructure automation. In parallel, improving pricing for customers requires running increasingly complex mathematical models on progressively more sophisticated infrastructure. We have witnessed significant investment and advancement in these areas in recent years,” explains Technology Director, Cathal Sheridan.

“Infrastructure is at the heart of this. For example, within a pricing model a Monte Carlo analysis can consume vast arrays of hardware with huge fluctuating capacity demands in low latency environments. Similarly, developing analytics engines to understand customer preferences requires significant processing power and large data stores.”

The PPB Infrastructure team has set about achieving a No/Low-Ops environment for its software delivery teams by removing itself as a dependency in the code-to-deployment process. To do this the team created its new infrastructure stack, i2 (Infrastructure v2), encompassing a private software defined datacenter (SDDC) and a fully automated continuous integration/deployment (CI/CD) pipeline. This has given software delivery teams the ability to independently deploy their applications using the i2 pipeline to acquire compute, storage, and network resources, independent of PPB’s infrastructure team.

“The pipeline had to be flexible, not only offering rolling updates and blue-green deployments, but also powerful A/B testing features to allow delivery teams to trial the effectiveness of new customer features,” says Sheridan.

“Creating the pipeline technology has been a joint project between infrastructure and delivery teams, resulting in the ability to make thousands of deployments weekly.

“Both Paddy Power and Betfair had very strong technology assets to draw on. Betfair had put a greater focus on owning the software that encapsulates differentiating product IP, crucial to creating market-winning customer experiences. While Paddy Power had a particular strength in infrastructure resilience and scaling.

“Interestingly at the time of the merger we discovered that both teams had highly aligned views on the future of CI/CD and the importance of delivery pipelines for iterative product development and rapid delivery,” he continues.

“Right from the start we had common ground making it relatively straightforward to choose the strongest design thinking from each organisation and create the optimum private cloud and CD pipeline configuration.”

From there, the full design of i2 was established and PPB set about creating a global platform capable of serving multiple products to millions of customers across its many brands and geographies. This became a fundamental integration work stream within the newly merged entity.

By rapidly decommissioning its ‘heritage’ stacks in favour of its new i2 architecture, PPB has on-boarded almost all of its service catalogue to the new infrastructure to provide the benefits of the ‘software defined everything’ environment as early as possible.

“This was about enabling innovation by expediting the product and feature delivery process. We knew this would also have a positive impact on our ability to resolve service issues through immutable software and virtualized hardware. On top of that, the new platform brought us to higher levels of resilience,” notes Sheridan.

Building resilience

Whilst the company continues to make serious inroads into the public cloud, using it extensively for its corporate systems, the retail technology estate, and a growing portion of its online customer workloads, PPB continues to invest in the i2 private cloud for strategic and regulatory reasons.

The i2 private cloud harnesses the open-source OpenStack platform to better serve its developer community. All i2 infrastructure elements are now virtualised guaranteeing complete autonomy in the provision of infrastructure for delivery teams.

“We chose OpenStack to create self-service infrastructure provisioning. We leveraged its well-designed infrastructure abstraction layer to create the i2 CD pipeline and our developers directly translated this into faster release cycles,” explains Sheridan.

“At the same time, OpenStack gave the Infrastructure team real choice around vendor selection for both technology and services based on its open source credentials. This allowed us to join a thriving community building a great SDDC platform backed by serious industry contributors.

“We were able to cherry-pick from a wide selection of vendors to build multiple service level options into the stack. For example, we reviewed a range of storage platforms and included a number of offerings with the majority of the developers choosing NetApp for applications with high I/O workloads,” he continues.

“OpenStack has proven to be both stable and performant and allowed us to push on with other elements of our integration work without having to revisit these fundamental requirements.”

By partnering with international technology company Computacenter, PPB was able to move ahead quickly with its technology transformation leveraging the Computacenter expertise in Infrastructure design and implementation.

“Our challenge was to build a new stack and migrate more than 5 million customers to it from an incumbent architecture built up over a decade – all within two years,” observes Sheridan.

“PPB is a FTSE 100 company with a great reputation for technology, there was really no room for error. Having a partner like Computacenter definitely lightens the load.

“Choosing the right partners is important. In addition to bringing Computacenter onboard, we also made an early call to build i2 on the Red Hat OpenStack distribution. Red Hat gave us access to a level of expertise and support that translated into delivering a more robust infrastructure in a shorter timeframe.

“The cross-vendor support model we have created around OpenStack has allowed us to rapidly problem-solve and determine root cause as we worked through transformation issues. We achieved this without any material service interruption to our business.

DevOps to NoOps

PPB was an early convert to DevOps and had integrated infrastructure engineers within delivery teams to reduce handovers and dependencies between engineering functions. This enabled it to embrace the Component Ownership model which flourished to deliver a more reliable overall technology estate. It is now setting its sights firmly on a Low/No-Ops model, whereby all infrastructure is available at the highest possible level of abstraction. Consequently development teams are now spending less time on infrastructure scripting and more on innovating and solving for customers.

“From a customer outcome perspective this is a far more productive use of our developers’ time, especially as we can now automate much of how the infrastructure handles resilience,” says Sheridan.

“We faced a collective technology challenge while moving away from DevOps teams having direct access to hardware, to instead using the i2 pipeline as the gateway to the infrastructure. This involved huge collaboration across the entire technology team but the advantages of standardised access to infrastructure was clear to all.

Migrating its software onto the new i2 stack therefore involved significant testing, verification and collaboration. This is, of course, on top of complexities resulting from the unveiling of the annual sporting calendar and the subsequent change-vigilance periods this imposes on the business.

“We had to paint the aircraft while we were still in the air. We had to flex, scale, and provide resilience within a platform we were actively building while simultaneously delivering capacity to meet fluctuating customer demand” depicts Sheridan.

“There were also significant technical challenges along the way as the target stack was made up of a component/vendor configuration that was new at the time. We set aside considerable resources to thoroughly stress test the infrastructure at key points in the build. We relied on the fact that the OpenStack community had done a great job defining the roles that compliant components must adhere to. We benefitted directly from this but also contributed back the community”.

Customer focused

From its high level corporate mission to its individual brand goals and developed features, PPB adopts a customer-first approach.

As all objectives are linked to KPIs based on measurable customer benefits, a huge aligning effect has been created across the entire organisation and driven a common language across disciplines. This has made communicating ideas faster and driven further innovation.

Much of this innovation work concerns the value customers place on application speed and ease of use, as well as launching products that fit neatly with customer lifestyles. PPB is also working hard to further marry its online and retail offerings to guarantee an exceptional and consistent experience across all touch points. Providing the widest possible market choice and a continuous stream of new features has been fully underpinned by the use of data analytics, automation, and modern infrastructure.

Its product performance in terms of customer journey efficiency, page load times, transaction times etc., are all actively monitored. The company not only collects quantitative and qualitative feedback from customer groups, but also runs regular user experience lab sessions with customers to find product enhancements around usability, speed and personalisation.

“Many of these activities require ever more compute cycles that rely on the ability of i2 to scale-out seamlessly based on (a) OpenStack’s capacity to easily deliver more virtual infrastructure to our teams and (b) the manner in which the i2 network architecture (leaf-spine) allows us to deploy new and immediately-addressable physical hardware.

OpenStack’s flexibility is foundational to i2 and PPB has maximised its use of this by working with vendors that embrace the elasticity of the environment. For example, the i2 network is based on Nuage Networks’ SDN technology. This allows developers create virtual networks and scale them in an on-demand manner. This type of flexibility was simply unavailable just a few years ago.

Ongoing growth

Long-term, from a high level digital perspective, PPB will continue to look at its overall operating model to further strengthen its ties between its commercial, product, and technology teams.

“We never stop trying to find better ways to run our business. We rely on our capacity to evaluate and consume the latest generation of technology to differentiate and solve issues, including the scale challenge we expect as our customer base moves through another major growth phase,” reflects Sheridan.

At a corporate level, the business is also set to further expand its reach in the US sports betting market. Its recent long-term agreements with Meadowlands Racetrack in New Jersey and Tioga Downs in New York will see it offer a full suite of cutting-edge products, compete with US competitors and provide a world-class customer experience.

“Technology continues to play a significant role in our growth story and we constantly deploy new tools, such as data streaming, public cloud, AI and machine learning to drive this. From an infrastructure perspective, we have also put a real focus on Kubernetes and containers as a hosting platform and will likely see much of our customer-facing workloads move on to this technology through 2018/19 using the i2 pipeline capabilities” adds Sheridan.

By investing in building multi-skilled teams and continuous learning, PPB has fully embraced constant change to remain a key player by empowering its people. The company remains a formidable competitor in a thriving industry. 

“We work in a large complex organisation but one that, importantly, has held tight to its entrepreneurial roots where ideas come from anywhere and your personal impact on the bottom line is always visible,” concludes Sheridan.

“Our teams solve new problems every day. The pace we operate means we can see the results of our efforts very quickly. That’s what makes it a really great place to work.”