A “Dev-Net-Ops” Framework for Digital Telco’s (DSP’s)
From vendor defined tools to Telco Applications CI/CD Playbooks
As the Business adoption to DevOps accelerates at Stupendous pace . In a recent working with Gartner we conclude that by 2022 expected 75% of such project will fail to deliver business value as we put DevOps idea to practice. It means for Comm-SPs really we need to put some realistic framework on ground to ensure we adopt DevOps in steps using realistic use cases that deliver real value to business. This paper is all about this how author thinks the industry should address this problem. This white paper describes key practices and how they can benefit the telco world, which has conventionally relied on separate development and operations to meet its stringent service quality requirements in a heavily regulated market.
I think once the Telco applications will be based on Cloud Native and micros services it will be easy to use many of repository and tools from IT but what about situation today. From where we shall start with technology as heart of transformation what is ground step-0 , I think it still lies on vision and approach to not go after fancy marketing stuff in a frenzy and define a clear use case , Lets say to introduce DevOps in Telco can we select a simple open source Application and cloud it with a tool chain to manage it through a Digital market place from inception to delivery across the life cycles .
For Comm-SPs what will be the right definition of DevOps for Network or Dev-Net-Ops , the answers can be many as most Telco’s want to introduce a pure IT concept to Telco’s , however the fact is that the Development pipe line of Telco’s today is totally controlled by Vendors not visible on Community Hubs or commercial ready repositories .So to introduce Dev-Net-Ops in Network Telco’s should prepare a customized framework for DevOps can help them automate the Networks , enable agile service delivery and most importantly is open enough to introduce nimble players in the Networks .
Then there is a commercial aspect of the same, when I started to build my career in IT a decade ago always we are trying to build a solution where product /licenses can be procured from vendors while the service is managed in house. However Telco’s operate in a more vendor dominated mode since decades and it is due to very reason the Service cost is eating huge part of their CAPEX. A systematic approach in to DevOps and scoping it in right direction will help solve the issues
Current NFV/SDN pain issues related to DevOps use cases
Starting NFV/SDN with high expectations with Multi-vendor solutions, half a decade on the difficult road and still find difficult to operationalize the network. The list of Comm-SP pain points for transformation can be huge and adding Operational complexity without automation and analytics using DevOps can make this issue further complex, however for the purpose of this paper I will like to demarcate framework list can be big but can be summarized along following three points.
(1) End to End system integration is still not a pure decoupled offering from vendors, which ultimately means in a Multi-vendor project an Operator thinks Service will be led by integrator but ultimately, they find them in the mid-way requiring service procured from all players and dealing with them all the way. We believe a focus on Test environment and Validation defined by Net-Dev-Ops can help solve both technical and commercial issues.
(2) VNF On boarding is still a silo scenario , normally for 1st time VNF certification and onboarding we can digest more cost/time but subsequent VNF’s of such type/scale must be a clone and should be accessible at Zero service cost , after this is what we want to achieve a Network offering as a code and programmable network . a Dev-Net-Ops can enable us for this.
(3) TaaS or Testing service need to be based on open source tools and automated in CI/CD, in other words Telco should be able to customize tool chain and use variety of tools from different vendors. A correct Dev-Net-Ops will help us solve this.
(4) On Top of everything DevOps with relying to breaking the code and aligned with automation and frequent builds seems contradicting with Telco world strict requirement for highly reliable and performance accelerated service.
From Automation to DevOps Pipe Line
We all agree the Telco Networks are still not automated unlike their IT and Cloud Counterparts and in fact require lot of manual tweaking whether it is for integration, deployment, testing and worst all for troubleshooting. So, one might be asking if Automation can fix those issues?
“Twitter user @ tweeted this perspective on code reviews: “10 lines of code = 10 issues. 500 lines of code = looks fine.” There simply are not enough people with enough time for manual processes to adequately oversee application delivery”
*Courtesy of Ramyram GitHub
The answer to such type of question is not definitive, because Telco industry Development pipe is still controlled and tested by Vendors. So for us as a Telco it makes a lot of sense if we address those issues of Automation that help us to use set of tools that enable agile integration, fast testing and nevertheless help solve issues in an automated manner. So what should be the characteristics of such a Tool Chain?
- Simple to use
- Simple to integrate with both Cloud and Legacy Network
- Support a programmable interfaces that are Secure
- Support native coherence with IT automation tools JIRA /JFROG/JENKINS/GITHUB
- Put Orchestration as first way to achieve automation not focus much on Below layers
Why I think Automation will not solve issues for Telco Cloud Journey is due to two main reasons once is because most application are still Fat and secondly too much reliance on vendors specially on Dev part simply means complete automation is not possible and secondly if we achieve fake automation define by vendor it will simply not deliver desired results
I think at least of now CI /CD /CT that enable smart FCAPS is key to introduce in Telco Networks, this should be a first step and that as we will see reply strongly on developing, using many of the tools in Operator Test environments that enable them to smoothly roll out the Next Era services. . A Culture of Shared Responsibility, as defined in the Scaled Agile Framework (SAFe) supports the aim of DevOps to tear down silos and build shared tool-sets, vocabularies and communication structures focused on one goal: delivering value rapidly and safely.
Bringing Synergy with IT for Telco Net-Dev-Ops
“DevOps automation can be a complex and arduous task as toolchains emerge from the vast number of tools used to deliver applications in faster more agile ways. I&O leaders require a top down view to designing and implementing successful toolchains. Tool chains are often built from stand-alone disconnected tools that are available to I&O leaders in their organization” Gartner
*Courtesy of colleague Mateo Červar
With Mature DevOps in IT and NFV/SDN bringing a lot of Open source tools the value chain requires an organic growth means it can reuse existing tools and leverage it with new Tools. Secondly it supports Mapping of many tools let’s say from open source to package it in one tools in simple to use interface like Nokia Matrix. In other words the DevOps tool chain must be Orchestrated as well .
Why I not speak about the Culture
It’s a toil of exercise to know how a big Telco want to evolve to DevOps and the only sad part is too much reliance on Culture and Processes without solving the Technology Architecture is like pushing water from top of a tank that is broken form bottom, the more water you puts in the more will be passed out. So, the question of technology framework and tool chain has to be solved first.
Infact as we try to apply the traditional DevOps concepts In Comm-SPs , the main reasons can be building a DevOps environment that can solve Operational issues in a way that do not require a major change in existing Operational environments of the Operators. Then the rigidness of Resources in Telco’s is far more than those existing in IT . For example the H/W resource in IT is cheap making it possible to roll out PoD in short notice and to apply pure IaaS model.
Can a Comm-SP adopt the same ? The current Infrastructure selection triggered by application itself and the high cost of NFVI makes it more difficult to build an open DevOps solution. Obviously these are not areas strictly depending on DevOps but rather deciding how Dev-Net-Ops will finally roll out in the Networks. I think these points also define value chain of Orchestration vs. pure automation. In most cases pure Ansible is not something that can automate an NFV environment than to use an Open Orchestration solution.
Cost of Net-Dev-Ops
In Legacy world and still in NFV transition the question arises cost impacts for DevOps . For this first point is that once the tool chain is ready and working still, we need to optimize it continuously. So, a Comm-SP should carefully select the use case to make DevOps scale to business requirements and ensure we not invest in something we cannot quantify to deliver business results
This is a BOQ Cost item which obviously lies outside the DevOps umbrella but so vital for Comm-SP’s that simply DevOps can not seem deliver value if it is not addressed. I personally feel developing a DevOps Tool chain specially the CI/CD tool chain is simply what is Telco teams require. In our latest user story with Delivery teams we understand from new feature or new product development all items on line across pipeline with clear ownership and progress is vital, After all this is what we all have seen from the GitHub about open software’s. Today industry stands in two extremes Redmine and Jenkins that simply is control and process with no tool chain and vendors tool chain like Huawei NICS or Nokia Matrix which simply is not open control and process and in worst case not easily to optimize across iterations. This is an important area STC have been participating and influencing partners and industry and I hope building a framework will speed Net-Dev-Ops adoption across industry.
Role of vendors in Telco Dev-Net-Ops
Comm-SP’s as of today comprise of networks with many vendors and such situation will become more prevalent as procurement will follow principles of open source supply chain. Hence a need to build a framework and followed by reference architecture that is agreed among all stakeholders is key to success. This commitment to continuous planning, integration, testing and deployment will deliver rapid innovation through collaboration. Continuous delivery in a multi vendor environment requires the automated integration and testing of all service components to ensure high reliability of service. Complexity grows exponentially when the number of service components and their individual version increases. Synchronizing delivery from multiple vendors is essential. Portions of the operator’s infrastructure must also be open to allow testing and verification of new solutions. New channels for the automated collection of customized operational feedback on demand will also allow development teams to improve their components based on production data. This chapter of paper is based on author discussion with Nokia sales marketing team
The most important for DevOps successful transformation is selecting the right use case and what can deliver best results than automate the testing , its advantages are many for example it will allow a Comm-SP not worry about Release management or whatever changes in the infrastructure .CI and API based based Tool chains will definitely help solve the issues .DT is one of the operator who has really focused on this case and certainly it helped early adoption of DevOps .
*Ericsson public document titled NFV DevOps Life cycle automation PoC
Dev-Net-OPS to offer Security as a Service
Web scale architectures and early adapters already see advantage of offering pipeline as a service. Disney offer DevOps as a service for developers but for Telco’s NFV so much restrained by security will benefit more if we have a DevOps use case of security as a service.
Why I think so because in NFV/SDN with so many integration points and thousands of API calls in a complex multi-vendor solution means ensuring and monitoring a costly initiative for a company. In our recent study we found cost of deploying and managing security solutions for Telco DC’s is almost same as of deploying a solution and I sincerely believe DevOps on this front with Red-Blue teams will solve this issue
DevOps for Comm-SP’s Open Source solutions and analytics
There are two more areas I want to address in this thought Paper, the first is a question to industry can we really rely on closed or Fat vendors to offer us OSS solutions? I will certainly not support it because they simply not good at that and not know how to control, manager and support this. I think all of those vendors want to package certain OSS versions from community under their tool that is fixed and a new release will require lot of changes, simply stating it means a big commercial and financial hit for a company expecting new release every two weeks. Can we use tools with JFROG, SALT STACK like frameworks solve above issue?
The second main question is how to deliver automation cases that is based on data analysis like cross layer cross relation, data mining and ML . Simply stating both of above cases must be solved as DevOps use case and if we adopt to vendor solutions, we see great risk of lock in. A detailed study of these models will certainly help Dev-Net-Ops help achieve real early wins for a comm-SP who is trying to evolve to target 2020+ digital architecture.
I sincerely hope through this thought paper we have shared our vision of what will be the DevOps for Telco DSP , all comments are welcome to let us understand the Telco issues together to refine and redefine Dev-Net-Ops 2.0 . I hope such a thought paper will follow a collaborative engagement , shared responsibility and standards alignment if ever such transformation will create real value chain for the industry as a whole .
- Juniper Automation with JUNOS
- Gartner How to Navigate Your DevOps Journey
- GitHub (https://github.com/capitalone/Hygieia)
- For a deeper dive into application release automation features and functionality, see the Forrester report “Vendor DevOps automation”
- Landscape: Application Release Automation Tools”
- Nokia DevOps for the Telco World
- Ericsson DevOps in Verizon a public document