Architecting Scalable and Production ready Orchestration frameworks for a Digital Business

Part-I

<Telco , Applications and Enterprise domain>

“To achieve great things, two things are needed: a plan and not quite enough time.”   

Leonard Bernstein

Subsequent to my talk followed by nice demo of ETSI OSM by  Telefonica’s Francisco Javier Ramón Telefonica at the U.S Network transformation congress organized by  Layer 1,2,3 Network Transformation Congress one question always nodded me

 “when exactly Industry promises around software control and Orchestration will be Production ready with clear use cases and a distinct framework “

Almost after a six month gap I am here at the ONS Open Networking Summit Europe  and still I am thinking the same one.

 

All of us will agree what appear simple on paper in Practice a daunting task. This never dilutes the importance of front runners and practice after all with out delving on this journey also we can not solve the issues .

Therefore, as an active community member in the ETSI NFV NOC ,  OSM and Linux foundation EUAG I think is important to explain my own point of view where we stand and to apply a focused approach to solve it .

Therefore, the purpose of this paper is to summarize key challenging faced by Telco and IT industry on this domain and thereby defining an architecture based on best of the breed SDO and community frameworks that will ultimately drive a carrier RFP and vendor roadmaps to define the missing Lego of the Future Network.

The part1 of this paper is to enlist the Situation today that really makes it difficult to Automate and orchestrate the networks using a standard modeling framework. If you are reading it make sure you also read Part-2 the Ultimate story to be published in Nov 2019 .

Orchestration market as we know today is full of fragmentation and we are still finding the optimum way to address it . However, our years of working in this domain has made clear that for Orchestration to be successful it must solve both the abstraction view (hide service from underlying network) and automation view (a way to model network as a software program) . Obviously the two are related as automation is deeply stitched on Abstraction and modeling . Low abstraction means difficult to model and difficult to automate

I think our approach of selecting right Orchestration vendor has not been fair where we either tilted towards network vendors or to the BSS/OSS vendors with out setting right criteria hoping vendors will set it correctly. This is why as I recommend to CSP’s  to select an automation and orchestration product must consider following five characteristics as a baseline .

1. NFVO must be modular in nature and not a full product
2. Simple to design (day 0 , Web based , Drag and Drop )
3. Should support truly open source with strong community support (e.g instead of ETSI MANO promote ONAP/OSM)
4. Intelligent automation (Predictive analysis, Day1 /Day2 automation)
5. Operational efficiency (Only domain truly exist now VNF Day1 , FCAPS , RCA , Monitoring)

Further these characteristics are fully endorsed by how industry views it with TMForum projects like NaaS , MEF LSO and ETSI SO frameworks. However, having said this still we find the full abstraction from business layer to underlying layer is not achieved till date  by E2E SO  or upper BSS layers making the full cycle still a manual and vendor defined way . The situation as existing in TMForum NaaS program and suggestion can be depicted below

NFVO Challenges

Having speak about these two points it is also important how we have deployed our projects anyways. Based on projects experience we believe major setback to Orchestration is coming from bottom up approach where all efforts have been primarily VNF driven with less focus on software control , e.g CSP finds NFVO from vendor X will only deliver value if VIM ,VNF from Vendor Y is selected , issues of standards specially unified meta modeling for VNF are not followed well by all vendors .

So to devise solution first list down issues we need solve .

• Network service is different from VNF /Interfaces and Template  standardization [Get Rid of vendor defined VNFDs]
• How to separate NFVI FCAPS solutions from NFVO ?
• Is it possible to use NFVO components / From different vendors to make one NFVO
• How NFVO controls MEC and other / D-Cloud architectures
• One NFVO fits to support MV VIM , MV VNF scnerio , Telco , Enterprise etc  and may be PNF/VNF scnerio?
• Service Orchestration like offer VoLTE / Service from portal one click?

A part from above there are some challenges coming from industry itself which can be summed as follows

• ONAP vs OSM
• Rely on vendor or community as most successful companies in IT Cloud /Automation were those who build their own solutions by the power of community
• Can we promote a production ready G-VNFM and remove S-VNFM totally?
• Can Telco and IT workloads can be automated in the same manner (Cloudify vs Rift approach)

 

NFVO Scaling and Performance Issues

It seems clear what started off as an industry idea and university R&D project is a commercial reality now and it means the solution must deliver a comprehensive TCO and savings if we have to deploy and use it . This is why a company CXO want to visualize Orchestration across the whole network from the Core Cloud to the Edge.

However although all EUAG’s have analyzed the Orchestration solution and know that real value chain exists at the Edge but still are finding the best way to  do it at scale .  There seems to be some clear challenges.

The real issues of Orchestration around scaling and orchestration can be defined as follows .

• ETSI MANO was good in central DC but a more open and developer API based orchestration means it will have limitations to scale the Edge
• ONAP seems modular, scalar , programmable but same time its complex. If you are not a developer type of company its really hard to keep pace with it
• OSM is good for Operator specially its IM model , VNF/PNF support but will it be really carrier grade at the Edge with too much dependency on adaptors
• Finally, should be deploy a virtualization at the Edge based on VM style (Akraino/StarlingX) or container style (Akraino/Airship).

 

NFVO and DevOps 

Telco Needs DevOps and that is why the Orchestration and automation business is taking a huge momentum, let us understand phases in which we can evolve .

1. Phase1: When We believe we have reached a point for Virtual Network automation specially Day1
2. Phase2: Next we want focus on Day0 design and Day2 specially VNF configuration automation for 3^rd Party VNF’s
3. Phase3: Next main idea if build complete CI/CD Pipe line through all artifacts and tools form many vendors including Open source ones
4. Phase4 the pure Orchestra: This vision is important to support Play store vision for Orchestration

Knowing this vision is as vital as knowing the road to your destination. For a long journey we all know what matters most is we travel a handsome distance but making sure we know our target .

Defining Business use cases of automation

“ETSI NFV TST006 is still not mature , primarily DevOps in CSP will address Digital service management like ZSM through portal ,Auto Slice management , E2E Closed Loop control , service automation equals (===) Telco DevOps”

To ensure automation and orchestration business takes momentum it is vital to link it with a business use cases , however DevOps is still a fancy buzz word but with ETSI Release 3 (TST 006) we are coming out to define a more clear interpretation for CSP’s however it I still a Early draft and mainly only talks about process and responsibility split between developer (Vendor) and provider (CSP) . It obviously is important because in Telco we work with so many vendors  and their responsibilities and management is very important .

However still we want follow direction to introduce DevOps based on  a use cases to deliver maximum value for CSP with key use case which are shown below . Below use cases are a good start to ensure the value of automation is indeed delivered to business

Based on experience one can see there are lot of issues but definitely possible solutions and directions to address them .

I will wrap this paper here and will come back soon with Part2 – The Orchestration solution . This is based on our recent work with communities and to come to a conclusion that industry success will be more relied on coalescence and support for hybrid solutions than to take only one direction to solve issue.

This will be the epilogue of the Part2 of this paper and I sincerely hope it will add value to what you are doing irrespective of Telco or IT business .

References:

https://www.onug.net/blog/network-orchestration-and-visibility-for-managed-sd-wan/

ETSI docs

TMForum.org

Cloudify.io

Rift.io

 

 

 

 

Advertisement

Reshaping the Telco Industry of future through Service Innovation and Simplification

Sheikh is Huawei Middle East Senior Solution and Delivery Architect covering domains of NFV, SDN ,Cloud,Internet of Things (IoT) and DevOps based innovation ,specially interested in those disruptive technology driving the industry transformation

Reshaping the Telco Industry of future through Service Innovation and Simplification

“Change your head (Mind) or wait for its removal”

According to the World’s Economic Forum report 2016-17 the investment in the global Telecom industry has underpinned an immense shift in the information, cash flows and societal development mostly adding value the global economy and improving lives of masses

In fact in next 5-10years the real value that society can reap due to Digitization and information will depend on the Telecom industry provide the underlying infrastructure and improve on customer experience side but the fact is that the Telecom industry profit has declined from 58% in 2010 to 44% in 2016 end (As per survey) and it seems clear the Health of this industry is a question in itself.

Infact this issue in telecom industry is like a iceberg theory and many coupled issues so CEO need to solve these underlying issues instead of just barraging old themes of aggressive products and rollout to improve profits .In fact such an approach can only bleed the industry woes to the bottom line . Traditionally IT and Telecom industries considered separate but the fact is that the silent nimble players like Skype, WhatsAPP, IMO , Wechat has silently taken huge profits while using Telco infrastructures to reach customers merely through digitization of services

Below are some key actions to focus, not only ideas some on ground fact that will put the Telco’s back in command

Viz.

1.    Re-Architecting the Future Networks

Network synergy, Energy efficiency, Infrastructure sharing, high capacity OTN networks, Single RAN, Virtualization, NFV, Cloud based. These are all the terms you will listen from time to time but theme is simple make architecture simple and modular

It will enable you to not only offer a competitive price but also make new services innovate easily

2.    Operations simplification

This is a very important target for industry. I remember the ETSI Phase2 specification specially considered the integration between NFV and SDN to solve enterprise needs for use cases that extend beyond geographies of a typical data center.

Similarly the internet style of operations where main skill is based on programming and Meta modeling of issues instead of troubleshooting skills by written guides is very important.

In Huawei we have developed a customized process for the Telco’s based on DevOps but meeting more specific needs from carriers like high quality testing , fault demarcation and continuous delivery making best product based on Pilot and Production environment synergy and concept of remote delivery . We call this process IntOps.

The key point is that Telco’s can not merely copy IT companies default DevOps, Chef and Ansible playbooks to use them directly.

I think this is the main theme of CTIO to think how to achieve this goal.

3.    Orchestration

What is our definition of this metric is how the experience will be delivered to customer through Omni channel. The standardization of Os-Ma interface will lead us to a world where customer can not only purchase but can drive the service creation through online experience.

The Service Logic and Open Graphs will be intelligent enough to create services across many infrastructures

4.    IoT

The internet of everything is a game changer, and due to the reason each IoT use case is customized in terms of QOS and service modeling makes Operators prime drivers to speed up and monetize this opportunity for their transformation.

5.    Security ,Cyber crimes and personal data integrity 

As data volume increases beyond imagination and networks become converged the threats for safe communication is immensely threaten by the Security guarantees and the intrusion detection.

·        Security policy interaction with security devices

·        Technology to defense APT attack

·        Utilizing Industry Best Practice (CIS Standard) to enhance OS

·        Security abilities more than the original OpenStack

·        Security isolation of Virtual Resource for VNF

·        The integrity protection based on TPM

·        Introspection technology for business information encryption

·        Secured deployment, upgrading and PRIST

·        Actively participate in ETSI NFV Security Specifications

6.    Organization Re-structuring

Who will solve this issue when the decision maker itself fears his redundancy? Infact this point is complex with Telecom aging workforce. It seems difficult to tell a person all the skills he acquired in two decades are not relevant now.

How to let people work in cross function Scrum teams, how to replace each manger by leader or problem solver.

To solve idea is simple, nothing can be achieved overnight. Let us promote guys who accept the change and upgrade. Set targets and make policies that discourage lazy team’s .Focus on software skills more. Python /Cheff /OpenStack /JSON

7.    Application or Infrastructure make clear decisions

Compared to IT industries, still the Telecom industry reliance on infrastructures is huge .By this I means the Telco services are not Infrastructure independent. While on Enterprise Architecture for IT the whole pie lies on the Application itself.

NFV micro service is all the stuff targeting Telco separate from infrastructure but fact is that still many services rely on his part like SR-IOV, Smart NICS, FPGA and accelerators

The strategy set on this level will have huge impact on the future of Operators. The industry leaders like AT&T , Verizon , NTT ,DT all have make clear plans to be the Software defined operator so I think actions should follow the same .

8.    Replace troubleshooting by service automation

This is a crazy idea. The SON and Software defined networks can scale and heal automatically. But the programming logic and map need to build by operations team.

It require more specialized skills on both service modeling and coding and obviously many operations jobs will be depleted .

9.    Co-deployments and converged clouds

Telco in recent years have already deployed Cloud based applications like Cloud Core, Cloud Edge, VAS Cloud but the fact is that customer CTIO must view Cloud as a shared pool which is independent of any application. The year must see ways how to join the clouds.

Till now the Global resource sharing among tenants is key point that is not standardized and difficult to customize based on each tenant. I think the community is not doing business driven approach to solve such issues. This is also a reason why commercial companies look for productions ready Distros like Huawei Fusion, Mirantis Ocala MOS, and Redhat OSP etc

10.                       Process mapping 

This is last and most important part, as from decades Telco’s have followed a very standardized process and structures to run the organization. with the digitization many of these things is going to change like Team structures , process , procedures , upgrade the eTOM processes to the ITIL based processes , Define SLA’s across IT and Telecom domains and incorporate orchestration and key enabler

I have this idea to make the processes the Scrum leader must be appointed and below team member is essential to map the processes. It is good idea to match the processes with PMI ® ACP XP (Extreme programming matrix)

Based on across industry analysis it seems evident that with huge challenges ahead the time is now to adapt .On operational level its more about transforming the ways of working and on strategic level is to deliver complete digital experience ln addition we need to solve the cross industry Silows and specially legal and regulatory concerns

According to the World Economic Forum report there are still other factors like micro branding, atomization, legal and lastly digitization of user experience side but since the scope is too wide and I am not expert on such topics, so I am leaving those for other industry experts.

I have a sincere hope that these 10 points are very important for CXO to drive the Digitization of services across industries. Future we will not listen Cloud , NFV , SDN as different terms , also industries like banks , Telco’s , enterprise not separate Isle’s but joint realities driven by the common goal of digitization

Reference:

1>ETSI standards

2>ITIL expert boot camp

3>World Economic forum report Released in January 2017

https://www.linkedin.com/pulse/reshaping-telco-industry-future-through-service-saad-sheikh

 

 

 

 

Adoption challenges in Open source and opportunities for niche System Integrator

In 2019 the IT and Telecom industry has really come to synergy point through the power of open Source solutions. I still remember days in September 2012 when we are firstly looking the Open Source solutions .It was really difficult that time to even think the Open Source can integrate the so called real Isle Industries of IT and Telecom. However the fact is that big Companies already doing production ready work for consolidation of IT and Telecom Datacenters.

What is most exciting for me is that I started 2019 with a CXO workshop asking the way forward to include the Banking Application on the same Data Center. Obviously at this time the idea is very vague because the Banking as we all know requires highly secure and reliable. Many of the ideas of traditional IT and Telecom cannot be directly applied there .However the message is clear the future IT services will cover whole Ecosystem and infact each industry is Just like Application appliance .

The Cloud Based infrastructure is an idea which is so important to develop in multi tenant environment because each vertical will come with its own set of requirements. It is clear that there is huge market demand for this and actually no vendor can eat all this market though many will try to do this.

This type of Cross industry and Multi tenant solutions will strongly depend on the power of Open Source solutions to couple the solutions form many vendors as a module. The concept which is known in industry as a micro service .However the integration of all these Open Source is very complex especially when it comes to adaption and matching CIO requirements. It is because although the solution can be Open but customer requirements can be customized. It is because no customer want to live in a house build on the wish of someone .The Personalization of each customer solution is very important.

Obviously all customer wish to transform to an agile infrastructure and reduce OPEX but really what must be there first steps like how to select the Components from the Open Source Solution? What must be there Cloud, VIM, MANO, VNF, Repository etc. This type of question many CXO already put to vendors and the answer of each vendor depend on its business model like what is there Ecosystem without delving to details whether it can satisfy customer real needs

I think this strategy is damaging the NFV adoption on a wider scale, because vendors want to push a solution while the best strategy for open Source solution must be a Pull solution which matches customer needs. Without this it is difficult to increase the real demand of NFV and Open Source Projects

After selecting the software the next daunting task must be how to make it functional.  The product view of this scenario is to develop all applications as a VNF Appliance through image file. It means any application can be available and only usage is licenses. However the story is not so simple. Infact it is in stark contrast to the Idea of App Store as prevailing in IT industry.

The part of such endeavors must be how to efficiently use these images to develop a service, something customer really need, way to auto test and auto deploy the service and obviously how to apply Puppet, Ansible and Chef Play books schemes to achieve it. I mean to say just the Appliance and their abstraction is not enough how to model the service and make it interwork is key to success for this realization

This type of open Solution ecosystem will support the wide range of adoption across the industry; maybe we will see the NFV going live in both Private and Public Clouds and the application will be as freely distributed as the Windows of today

However with more and more open solutions the dilemma to integrate them will become more and more important question by all CIO’s , not only this we will come to a situation where customer will question the reliability and performance of open source solutions in commercial deployments ? Can SI can use all open Source tools and offer something a single Pane glass operations modeled around Open API’s and programming abstraction .

This is the real value of System integrator, the Hercules the IT and Telco industry waiting for years.

Can 2019 be the year of that SI’s making an impact? The story is open and the best team will make it happen. System integration team has to listen to customers listening to the industry, partners and customer alike to solve the issues of Ecosystem.

Collaboration is the key and sharing of experiences to solve industry issues is critical to be the System integrator of choice for the industry

https://www.linkedin.com/pulse/nascent-open-source-solutions-success-looking-si-saad-sheikh-be-pe