The DOAT Project: How We’re Revolutionising Device Onboarding at Vodafone

Our Vodafone Network teams across AMAP and EU countries are responsible for the security posture and management of over 27M devices (routers, firewalls, load balancers, proxies, switches, and VPN-concentrators).

Assuring the compliance of our network devices against a set baseline had until recently been a mishmash of supplier-led configuration management systems (CMS), mega-sized spreadsheets and sample auditing. The result was a costly and time-intensive process for new device onboarding, resource-intensive accounting which was prone to human error, and a limited assurance picture. So, alongside an expert team, I was appointed delivery lead / solution expert to create Vodafone’s DOAT (Device Operational Assurance Tool), which went on to win awards at CyberCon and more recently at the Secure World Awards.

Here’s a snapshot of how DOAT came together, how we overcame its inevitable challenges, and the impact it’s already had on Vodafone worldwide.

A quick overview of the tool

DOAT is an in-house developed central reporting tool that assures the CSB4 compliance status for network devices across our estate. It addresses the challenges of business growth and maturity that has outgrown all existing tooling capability.

Currently it’s being used to report on the compliance of 98 different network device types in scope for CSB4 (85% coverage achieved in a year) and has over 60,000 network devices onboarded so far.

Whilst we’ve used other solutions, none have yielded as much success for us as DOAT; it has been instrumental in allowing our markets to achieve the required network hardening Cyber Security Baseline. By giving them device level data on their compliance status DOAT allows targeted remediation activities.

Getting things going

Aside from me, our team consisted of two Python developers (a discovery graduate and a contractor), a senior network engineer, a part-time BI data analyst, a part-time Secure by Design Architect and a product manager, all operating under a lean, flat organisational structure.

Adopting some of the best practices of Scaled Agile Framework (SAFe), we developed on GitHub Enterprise using Python and YAML, and distributed new releases via the Azure DevOps CI/CD pipeline – compliance results are uploaded to a multifaceted central BI dashboard.

The biggest challenge to getting DOAT off the ground was to win over the Heads of Cyber (HoCs) and their local network teams. We had to convince them that this was a tool that they wanted to use and not something being imposed on them; that it was better and more expedient than anything available commercially off the shelf (CoTS); and that its meticulous compliance checking capability was better than any sampling audits.

To get this buy-in and win their trust, we ran several roadshows with all our stakeholders, addressing their concerns, ensuring all their requirements were captured and creating effective communications via a newly formed stakeholder community group.

Building DOAT, from A to B

To begin we made a Critical Path Analysis of the previous approaches to compliance assurance, which highlighted that probing production devices directly added unnecessary complexity, effort, lead time and cost to onboarding new devices. A previous CMS system had taken seven years to onboard less than 500 devices.

Our idea was that sourcing network configuration data from off-line backup servers would negate the need to access production devices, therefore simplifying and expediting new network device onboarding and compliance assurance reporting. In May 2021 we completed a limited scope proof of concept (PoC) that validated the feasibility of this approach.

In August we ran a three-month pilot phase with Albania, Portugal and Cyber that successfully tested DOAT under load and helped to shape its future development and delivery. In September we mobilised a small team (three core members) to kick off full development and the roll-out of DOAT to our global network teams that were responsible for devices in scope of CSB4.0 (60 network teams across 27 countries).

Get it done, together!

Given the diverse network device types and brands that are implemented onto our network, we couldn’t have succeeded without the invaluable knowledge and collaboration of our DOAT community and global network teams. Their support was crucial as technically there are several different ways to configure network devices that will achieve the correct compliance outcomes; the network teams all helped provide the necessary input to ensure that DOAT tolerated all their customisations and did not to return false positives.

There has now been a significant shift away from reporting CSB4 compliance via manually created (often old and inaccurate) mega-sized spreadsheets or via costly supplier-led tooling (eg IBM CMS). Instead, everybody is moving towards using DOAT as reflected on our near real time dashboard. Remedial hardening activities triggered by DOAT results are improving our security posture for our enterprise customers and our local markets week on week. I couldn’t be happier with what we’ve achieved here, and that Vodafone spirit of “Get it done together” has truly been exemplified in this delivery!

Want to become part of Vodafone innovation? Check our careers page for opportunities to join the team!