Tata Communications (TCL) is a leading global Tier-1 SP, offering IZO SD-WAN for managed branch customers across 130 countries in 700 cities. TCL team evaluated multiple network service orchestration solutions including Cisco NSO, Glue Networks and Anuta NCX for their multi-vendor infrastructure consisting of Cisco, Riverbed, Versa and Juniper. They chose Anuta NCX because of its flexibility, multi-vendor coverage, scalability, development toolkit as well as customer service.
Listen to Peter Juffernholz, AVP of Virtual Network Services, talking about his experience of introducing DevOps into his organization and the challenges he faced which included organization alignment, training of current staff and developing service models.
Using Anuta NCX, TCL reduced CPE provisioning time from 60 minutes to 5 minutes.
00:00 Welcome to packet Pushers, the data networking podcast that repeatedly, predictably and reliably arrives in your podcatcher every week.
The business value of infrastructure automation remains a tough, topic to discuss. It’s just a difficult thing to talk about when there is just so many complicated issues that you really need to take so many corners. But the business value of infrastructure automation is the hardest part. And the first stages of that automation process of turning the business around or taking it where you want to go is different for every company and for every team of network engineers. There are just so many areas that can be improved and it’s just hard to pick out which one to go with first.
In this sponsored show, we are talking directly with a customer from Anuta Networks – Tata Communications. And they are going to be talking about their Anuta NCX experience and how Tata Communication as a global service provider business in more than 130 countries have been using Anuta NCX to orchestrate MPLS Network and SD-WAN services that it offers as part of its portfolio.
Our guest today is Peter Juffernholz, Peter please introduce yourself to the Packet Pushers audience and tell us what you do at Tata.
01:03 Peter: Yeah, Hi my name is Peter Juffernholz and I’m with Tata Communications for now about ten years. I’m looking presently after their virtual network service portfolio. So here we are trying to bring to markets not just SD-WAN, but also other virtualized services around NFV implementation etc. And in the past, I’ve looked after the VPN environment as well.
Wow, managing VPN’s have been pretty hard for a number of years. So you certainly doing that manually must have been a bit of…you remember the battle times I imagine.
Peter: Yes, yes all of those.
01:41 And I imagine that automation and orchestration for you is really a way to change the way this works. Like when I have worked with carriers in the past and they’ve wanted to deploy VPN’s that’s taken them months to get everything set up. And that seems to be a situation which is ripe for a change.
Peter: Yes it is, and I think part of that is to have the tools of the trade to do that zero touch provisioning with embedding existing services and running on top of it through SD-WAN technologies for example, so that you can onboard customers quicker and faster and do migrations on a different pace. You also eliminate errors and faults along the way and simplify the way you can deploy services once the initial design is done and you have created for example templates, then it can be easily applied.
02:33 Well, yeah, once you’ve got the design and you repeatedly execute on the design week after week is how I would see it.
Peter: Yes, although every customer is unique, so they are all different and we will always have to go through that process with them. But a lot of the large network rollouts tend to last a long time. We are looking at rollouts that go 12 or 18 months and it’s not unusual for global MPLS Networks.
Intro to IZO SD-WAN
03:04 So is that some context for a conversation Peter? We are talking with you about Tata communications they have an IZO SD-WAN service. Can you tell us a little bit more about that and how it ties into Anuta?
Peter: Yeah, certainly. So IZO SD-WAN is part of our cloud enablement efforts under the name IZO we have grouped together some of our private cloud services as well as our network services, particularly our Hybrid WAN offerings and cloud enablement private connect into public cloud services as well. And the SD-WAN service is the latest member of that and what we have done is, we have a two-pronged approach so to speak.
We have one approach called SELECT that is a vendor based architecture where we onboarded versa networks and then we have an in-house built capability around standard Cisco ISR IOS and that we automated through Anuta to have capabilities around ZDP’s or zero deployments as well as dynamic path selection and congestion management that allows us to address customer requirements on their journey to cloud in hybrid networking.
Scope of Anuta NCX Deployment:
04:19 So what I’m hearing there is that you are using Anuta then to orchestrate the setup of IOS devices using DM-VPN and all of the technologies that we know and love today. Not one of these new fancy shmancy SD-WAN technologies that are all different. This, as you talk about the prime product, that’s just; the routers that we know and love and you’ve orchestrated those using Anuta?
Peter: Yes, well whether we love them is a different question. That’s what we are about, yes. We are orchestrating a standard ISRs we started with G2 series and we are doing 4K’s and CSR’s etc. So it’s really standard fare.
05:04 But one other things about the Anuta product is the multi-vendor nature of it. So they have in previous shows, say back in show 301 we talked to Anuta about their product. And they were keen to emphasize their ability to scale across many, many different vendors. Is that a key factor for you too?
Peter: Yeah, certainly, it is. So the Cisco story is only the part of it. So in most of the deployments that we are doing, we have more than just a router. We have firewalls, we have WAN optimization services. We have multiple deployments scenarios, virtual and physical. So in order to do this, we certainly rely on the flexibility of the NCX platform.
05:47 So when you are rolling out the service to customers, every time you stood up the service, you had to go out and touch a Cisco router, you had to touch a firewall, you had to touch WAN optimization to setup the service individually?
Peter: Yeah, correct. And then also depending on what underlay you are connected to, there might be other different systems involved. They come with their own IP address management etc. So this is all something that we can sidestep by templatizing and automating that deployment.
06:18 And so you are actually offering proxy services and firewalls as part of your IZO product?
Peter: Yes, so we have integrated in-house managed security services. So we have managed branch deployments so get traditional physical box on the customer premises We have virtual solutions, we have inbuilt UTM services, cloud-based UTM services and we also bundled with Zscaler.
06:48 Wow, that means you are actually spanning, not only a wide range of Cisco devices, you are also managing a wide range of security vendors all under a single tool.
Peter: Yes, it is understood that the configurations are managed so partner integration is yet, not part of us directly interfacing API’s with Zscaler or anything like that.
Zero Touch Deployment:
07:08 One of the challenges; I mean I’m just thinking how difficult it would be to deploy routers out on site and get them to installed and configured and tested and validated. What sort of timeframe are you looking at to deploy this? If you are deploying 130 countries and thousands of customers, you really need to have this much slicker than the way we’ve done it in the past. How long does it take to get a new edge device down on the ground?
Peter: So from a physical perspective, the delivery of the device isn’t really changing right but what changes is the speed with which we can configure on-boarded if we can run on top of an existing deployment then we can have a device onsite probably within two weeks and turn it up depending on what; you know if there is internet service already available. Then we can connect using that service and we can setup also some LTE based services in the meanwhile until last miles are provisioned and connected.
Time Saved is Money Earned
08:14 So how long was taking you before Anuta came along to get a service up and running?
Peter: So from a timeline perspective the times that we are saving are multiple right So number 1 we are reducing the time that it takes to provision the router itself on the network once it is even installed. We can compress that time from 60 minutes somewhere, about 45 to 60 minutes down to about 5 minutes now. Because all we do is push a template. So the pre-configuration work is also much faster. It takes the trained engineers very little time to pick the templates and customize them to customer requirement. That takes about 10 minute time or so. And then it can be deployed. Now the physical device as I said hasn’t really changed. What changed is the way we onboard them and that we can also provide zero-touch configuration for all those. So we can ship a device on premises that connects to internet at the IP address and connects to the controller.
09:26 You have effectively built a cloud platform so a lot of the other SD-WAN vendors out there have built cloud platforms and they have custom devices and they wake up and they connect to the system over the internet and then zero-touch. But you have done that with existing devices that most customers would have already have today using your Anuta platform.
Peter: Yea, we can do that.
09:47 I think that fascinates me in a lot of ways because there are companies who need to build their own solution. And the challenge to me has always been that so many different CLI commands that you would take to build a VPN and really what you are saying is you’ve shrunk the entire configuration process down, just to a template which really just need customization for what? The IP address of the customer or the naming convention or something like that?
Peter: Yeah, for example, yes. So it depends on what features the customers are getting. For now it’s all a new configurations only. We haven’t really touched the onboarding of existing customers that we have already deployed or had deployed before we worked with Anuta that are migrating from an existing configuration for example, into IZO prime offering. And for that, that’s going to be available in July as far as I recall. It will definitely help us; here again to do the migration work after careful planning etc. to roll that out and push the configuration instead of device by device with an engineer doing scripting etc. Now we push it out to 100 or whatever, 200 devices at the time.
11:02 So that makes things like configuration audits and reconciling asset list and things like that pretty straightforward too. And you are actually using Anuta to do your asset management?
Peter: No, we don’t use them for asset management but it does auto discovery. So Anuta platform is discovering the devices in the Network and takes the configurations that it finds and that makes it a lot easier for us to then push new configurations for the devices. So we still have external IPAM and other services and inventory management right?
11:34 Right, but those solutions are often integrated with other parts of your Network though?
11:40 I’ll say you know as Tata you have dozens of services and many business units, there might be a system for inventory management that actually spans the entire company I can imagine.
Peter: Yes, correct.
11:50 So you actually want to use that inventory system because the rest of the company wants to use the same thing. So you don’t necessarily want. So but in terms of doing things like role-based access control or IOS updates, that’s all managed quickly, like to your level of what you need?
Peter: Yea, again the Anuta platform is providing us with the capability to create audit trails and to create the right authentication levels for that. So we are able to maintain the integrity of the service and to document the integrity of our changes throughout the process.
Why TCL Chose Anuta over Competition?
12:31 So did you consider; before you got to Anuta, I am assuming you considered a couple of other different products. I mean there are quite a few out there in this space these days. Which were the ones that you looked at?
Peter: we looked closely really after an initial survey, we looked very closely at three different vendors. One was Tail-f now Cisco NSO right, as well as Glue. Glueware was another contender for that. However, because of the timelines that we were looking at and the speed to have all the components with role based changes and so forth ready that was only really possible with Anuta.
13:17 So you really felt that Anuta was ahead of these other options? What about writing your own? I mean Tata Communications has access to very large self-development team and building your own must have been attractive.
Peter: Well, to a degree, so that’s maybe overstating it a little bit. We are still a carrier, right. We are not notoriously good at building software. We have done that on the data center side so there is a set of guys that have actually built their own cloud orchestration platform in-house that bridges private and public cloud. But on the Network side, we didn’t feel that we had either the time or the talent to do that, right. So the Anuta platform provided a nice framework for us to onboard that rather quickly. Because it’s more than just having few lines of code. You have to have a working UI. And the stability of deployments that had been done and the platform is around for a while. So the Kinks are out of the box.
14:33 I did forget really, but we have been doing; Anuta has been coming on the Packet Pushers now for like 3 or 4 years. They have been around for longer than that. That must have been a consideration.
Peter: Yes, sure. So a proven track record of capabilities was important to us. Anuta has done a lot of development in they have really had some really smart guys that know inside out what to do with Network automation. And that was really one of the key elements. They were able to convince us and demonstrate to us that they can deliver in the timelines that we gave them.
15:15 Right, so let’s talk a little bit about how NCX works. Like the whole NCX platform itself is very much based around the Yang models and we know that a lot of our current vendors today sort of have a love-hate relationship with Yang and their implementations of Yang. How has your experience been around a model-driven device mapping or model-driven device models and how you get down to the Network? Is that working or how is that playing out in the market today?
Peter: We didn’t see any problems with that Yang model for the task that we have taken on so far and that we see to come up in the near future for our services, right. So I don’t really have much of an opinion on Yang versus other approaches. I got to say it worked really to the satisfaction of what we have deployed. And all the way up to the integration with our self-service portal. So I have really no issues on the Yang modelling.
16:23 How much experience does Tata Communications have with Yang before you began your engagement with Anuta. Is there something you were familiar with or you just skilled upon a little bit?
Peter: No we had to skill upon that, yeah, certainly.
16:34 Was that hard or was that challenging or was it just a thing?
Peter: Well, it’s always hard to get engineers out to take on new trainings while they are having a day job. But yeah, I think it wasn’t so much the skill-set. It was really taking them out of the job and getting them a crash course and then implementing it internally as well and understanding what it means for. But we put the KPI’s around it because it is part of our future as a Networking company to have automation and virtualization throughout the organization.
17:20 And it seems like Yang is the path forward anyway. Like the generalized support from the IETF and from the…; some vendors are embracing Yang wholeheartedly and some not so much. Like our beloved security vendors, for example, seemed to have forgotten what Yang is all about.
17:37 But then again security companies live in their own little bubble so maybe that’s normal.
So I’m going to say that you are saying that you are confident that the networking engineers, the operators you are working with. They have the capability of learning Yang. It was just a question of how do we fit this into their schedule while they are still doing their day jobs and keeping things running. Also going out and learning this modelling concept and then bringing it into the workplace in getting it running.
Peter: Yeah, correct. So I mean the concepts are known but obviously knowing something intellectually and applying it in templates and so forth, you know, that took a couple of weeks to get them ready. But then working alongside with the Anuta people was very good as well. Because I think that was the proof of the pudding to ultimately put those templates into play. And so now the team is now doing their own Yang models as well. So it’s no longer relying on Anuta for every change and modelling element.
18:36 And the good part about the models is, today you can use it to CLI, but tomorrow you can move it to protocols like NETCONF, GRPC or whatever emerges as a different way to configure the device.
Peter: Yeah, correct.
18:50 You don’t have to rewrite your entire program to take advantage of the change from CLI to GRPC or something. It’s just a case of building a southbound API.
Talk about scale. What sort of scale are you sort of intending to get to with this and where are you down that path now?
Peter: Well, I think the original scale that we looked at was several hundred thousand devices and instances. So that was the scaling in general that we looked at. The initial deployment that we have done is supporting about ten thousand devices for the time being. And so far we just began rolling it out a couple of months back. And so far we have done some 250 devices with about 300 network configurations.
19:36 But this is also multi-tenancy? You’ve done 250 devices with more to come but also across multiple tenants. It’s not just one big Network here.
Peter: Yeah, that’s multiple customers. It’s now three customers that are onboarded on the platform and the self-management and administration we are doing through our portal. So the customer portal has functionality built in to make certain changes – allow our customers to adjust the configurations right. For now it is very limited we only allow them to change QOS policies and create or change applications into different application classes, especially customer identified or self-managed applications.
20:28 And you mentioned earlier this CPE configuration time. You said it went from about 60 minutes per device down to 5 minutes per device. So you’ve actually measured that out in the field of these deployments?
Peter: Yea, we did. I mean after the initial ones, once we have the templates the team was ready to go and run. It was a significant improvement. We averaged it out and the timelines are just phenomenal. I mean it’s so good to not have engineers do manual work that can be automated and that can really make a difference. As I said, instead of configuring one device after another, you just apply the template once and push it out and we have so far detected zero error rate. We haven’t had really any instances where the configuration failed. If it failed for whatever reasons, it was because the device was busy and push is repeated and ultimately go through.
21:37 And not because of the copy paste failed. The human on the other end got something wrong.
Peter: Yea, correct, no copy and paste anymore.
21:47 So much could go wrong with the copy and paste and we forget that really.
21:51 Somebody puts too many characters in the buffer and paste it down and boom when you place it in the wrong order.
Future Plans with Anuta NCX:
So you’ve got the system built out, you’ve` gone through first phase. What’s the next phases now that you’ve got the green field situation working, where do you go?
Peter: So the next phase as I highlighted already is to go to have brownfield deployments as well. So migrate customers from other services and onboard them into the SD-WAN service. So that means more reading of existing configurations and pushing a modified template back onto the device. That’s one of the next steps and we have also in the road map to onboard some application optimization services alongside to the Cisco devices that are going to be for Riverbed in the first round. And then we also look at more building feedback loops and reactions that will allow us to do a bit of PBR.
22:54 Okay, so really sophisticated stuffs. You are talking about building feedback loops on the Network so you can start monitoring the state end to end from those devices and start making adaptive configuration changes in an automated way.
Peter: Yes, so because today we can do this based on the CPE measurements. And already we do application offloading. We do dynamic congestion management with adaptive shaping. But that’s all living on the devices, right. This doesn’t use the control plane It doesn’t use the flow information that we have yet. And that will be the next step to see how we can integrate the actual netflow measurements to trigger configuration changes.
23:45 This is always the stuff that I always want to configure but we spend so much time just getting the basics done that we never get to play with the advance stuff.
Peter: Yea, I hope it doesn’t happen to us this time.
24:00 Well, I think this idea; I mean people misunderstand the concept of models. The idea behind the model is that you say, this is what the device should be able to do. And you have this software layer which then turns that into a Yang model. And then you have a thing below the Yang model which says, how do turn this Yang model into a configuration for this device? So that might be straight over the NETCONF protocol into the device or it might be through a CLI interface. Or it might be through some other method of configuring the device. But the point is that the model doesn’t change from device to device. It’s consistent across the entire network. You just have some way of adapting it to each device that you are going to use. And I think that’s the real transition here. I imagine that you are on the cusp of saying we can always bring on any device as long as it maps into the model.
Peter: Yes, yes, technically yes. The only issue is you still need to troubleshoot that as well, right. So you need to have these processes around it as well. But yeah, in general it is true. I mean as long as you have the right model and the right adapter to the device, you can basically automate any of the configuration tasks that are required. And that’s one of the reasons why flexible Anuta platform is really helpful because it can do that for so many different vendor platform and devices and models around it.
25:21 So there is the technical aspects of putting an orchestration system out and the sort of operational and human aspects of it. Did you have to do this cross silos in your organization? Did you have to go among different groups and get sort of a unified buy-in on this? What was the inside process?
Peter: To a degree, yeah. So because we have multiple different product lines obviously that are operating on there we have to underlaying product lines, we have to CPE management. We have my team and then we have the security services and then I think the cloud connectors and other product team there. So to cross all of them we had to make sure that we have at least common understanding of what we are trying to achieve and what we are buying into. Because there is also no reason for us to stop with SD-WAN and not move this forward to any and all of our Cisco managed CPE’s. So there is nothing stopping us from that. It is just that the urgent need was around the SD-WAN service because that’s where we have more complex CLI work done. I think somewhere our engineer said that it was, I don’t know, some 400 CLI’s that they had to do across some different vendor platforms. A lot of platforms involved. So that’s no way to do this and we have already in the deployments that we had done before, we had the automation ready. We already discovered that we have a lot of manual labouring there and that causes problems. Because mistakes are made and repeat configurations have taken precious time away from moving towards the delivery target, right`. So that was a very, very key driver. But I think the uniformity of such an automation project and the benefits that it can reap is definitely understood by other teams.
Working with Anuta Team:
27:34 I just get the sense that, you don’t seem to be bothered. You just seem to say this was all pretty straightforward. It wasn’t enormously difficult.
Peter: So, internally it was. It was more difficult internally than actually working with the Anuta guys. So I got to say they’ve done a bang on job. They delivered in 37 business days and they are the most humble and hardworking people that I have worked with, right. So, that’s a very nice piece. Internally, yes you have to force who is going to do it and why is it on the operations team and the others and how do we integrate this back. But we have backing from senior management on it to meet our target deadlines and onboard that SD-WAN service. And that’s always helpful.
28:20 You know it’s not too often in a technology project that the internal problems are bigger than the technology problems.
Peter: That’s the reality is around this and the hurdles internally. I am currently working towards the vCPE’s services and NFV orchestration etc. And the problem that causes internally from who is talking to whom and who is taking responsibility for which part and so forth. That’s a much larger, much more difficult conversation to have done with the technology vendors.
28:59 Sadly we can’t orchestrate and automate that. [laughing]. Well, I think that’s just about all we have time for speaking with you today Peter. Thanks very much for joining us. And I really appreciate that you take the time to talk about your experience the getting system Anuta system into your network. I do sort of get the sense that you’ve actually haven’t had too much trouble making the Anuta go. And most of your efforts are being much more spen