Awaiting with bated breath for the Finale Of The Decade — the GoT, we too have summed up the 'User-Edge Operations' blog series Finale, by our very own Vice President of Server Engineering, Support and IT of CounterPath — Jim O'Brien. In this blog, Jim concludes the series by summarizing the benefits and best practices of user-edge toolkit techniques and sheds light on how can we leverage it to enhance network operations practices.
Writing this blog series has been a fun experience for me and I hope it's been useful for those of you too following the series. We've received feedback from a few people and I wanted to let those readers know that their feedback was appreciated. In the two earlier posts in this series (What is User-Edge Operations? and The User-Edge Toolkit), we talked about our User-Edge Toolkit and focused on what each of the components offers. In this final post of our series, we'd like to take some time to show real data as well as share some lessons and few best practices.
Real User Data is Insightful
When we prepared for this blog post series, we collected a random sample of our own team's usage. It includes 1000 UEM records. It provided some interesting data that I'm not sure we had realized earlier. There were also some outliers in the data that made us pause, double back and work upon, so as to independently verify the data and confirm its accuracy. It turns out that the data was spot on and we were left a little surprised. Here are some of the baseline data. Things that are interesting, but not very surprising:
The average Call length was 718 seconds long, meaning almost 12 minutes. (Our team at CounterPath likes to talk :)
The average MOS LQ was 3.89. This is very good, especially considering the geographic diversity of our user base.
The average delay was 120 ms.
The average Packet Loss was .93% (including some crazy bad calls with over 85% loss).
22% of the sample calls were inbound to the user meaning that 78% were originated by the user.
Here is a breakdown of the codec used for our calls in the sample: Opus: 61.7%; G711: 33.9%, G722: 0.35%; SILK: 0.03; G729: 0.01%
Operating System breakdown was: 510 Windows Machines, 246 Macs, 165 iOS Devices & 79 Android Devices.
The different kinds of headsets reported included: 376 calls Reported that no headset was used, 57 Jabra LINK 360, 54 Conexant SmartAudio HD, 49 Plantronics C320, 31 Realtek High Definition Audio, 26 Sennheiser USB headset, 25 Plantronics .Audio 478 USB, 23 Plantronics Blackwire 320, 22 Logitech USB Headset, 17 Plantronics Savi 7xx, 17 3- Plantronics C320, 16 Plantronics Blackwire C220, 14 High Definition Audio Device, 14 3- Logitech USB Headset,13 9- HP Digital Stereo Headset, 11 Logitech USB Headset H540, ... and the list went a few more.
Of Video Calls, 45% were 1080P, while 54% were 720P and 1% were VGA resolution. Another interesting note is that of the 720P calls, more than 1 out of 5 were in portrait mode. (Makes one wonder - don't people know video looks better in widescreen?)
Here are the items that we needed to fact-check because the data seemed like it needed independent confirmation:
If 'XYZ' is the device ID that made the most calls — 55 of them; 55/1000 = 5.5%. That's quite a bit for an organization our size. The owner of this device made or received a total of 59 of the calls in our 1000 call sample. The additional 4 calls were from an Android Phone, while the above 55 calls were from the Windows 10 laptop.
And here's what we learnt:
Lesson: A member of our sales team is doing his job and keeping up with his customers!
We double checked. And yes, it turned out that this user regularly makes these many calls. We perhaps caught them on a better than average day.
Lesson: We could leverage the power of the data here. We had call data and were able to correlate this with both user and device data to assess the user experience for what might have looked like an anomaly. Should we want to, we could also deep-dive more and review the average of this user's MOS scores and other IP layer quality metrics as well as correlate this to time of day or compare to other users if there was an issue. It goes without saying that Provisioning, Device management and UEM have enabled a very powerful dataset.
Going further, we also noted that the user with the most reported contacts on their device has 25861. We saw this reported from an iPhone 10/iPhone X. The second most 'contact rich' user reported 7155 contacts on an iPhone 10S Max. Almost shockingly, we did verify that two of our real team members do have this many contacts! I'm guessing more than one person reading this has just looked up at their address books to see how many they have. I sure did. And I was left feeling like a digital shut in with only 3672 contacts on my iPhone!
Lesson? I'm not sure there is one... other than perhaps some of us need more contacts? Or perhaps a few of our team members people need less. Maybe their address book programs have gone crazy and duplicated or quadrupled their address books? In any case, we did let our QA teams know that there are people out there with just this many contacts and they should be checking features against address books with numbers that might be beyond normal expectations.
Real Lessons Learned
At CounterPath we focus much of our time on how we build, operate and support our networks or how the software and services we build will drive success for our customers.
The interesting part of this is that no matter how much preparation we do, no matter how much service, user and network design, we have moments where learn what we need when we need it. It is the joy of life. Reading this you may think that I've taken a bit of a cosmic turn. Maybe true! But what we have learned is that as we deploy services, as we provide trusted customers with demonstration accounts, as we host trials. as we move towards real deployments with real users, we learn; and these learnings are awesome.
And this is where we've started to focus more of our operations and development energy (or devops if you prefer that term). We get a new feature/service/something out to a set of users and we learn, we iterate and we move on to a larger or more demanding set of users. "Wash. Rinse. Repeat." — I'm sure this is the foundational methodology for a number of startup and business books! Thank goodness, we've learned this the hard way.
Getting back to the point: what we've built into the tools of the user-edge toolkit are the experiences of the data we've been looking for, our customers have told us they are looking for. And because of this data, teams can improve operational readiness and feed into their CI / devops / agile or waterfall; features, improvements and fixes that matter.
More specific lessons
User-Edge devices have a wealth of information to provide. This is the transformational element. The devices and applications users leverage for their communications are always in the communications path. They provide the best view of the network and services that we can ask for because the view of these applications is the view that informs the user experience.
Managing user based configurations and understanding user device adoption provides an important backdrop to both troubleshooting and data collection activities. Network performance data can be greatly enhanced with information about the devices/software being leveraged by users, from user location and connectivity, and that provides details of which services are consumed in which user scenarios.
User experience lifecycle is critical and there is a long tail of information that users and their devices emit which can be harnessed by service providers that curate this information. All of this information provides service operations teams with a dataset from which they can understand how their end users leverage the services they are consuming.
Finally, I do hope this series of posts was insightful. If you are interested to learn more about our solutions, please contact our team here. I love talking to our current and prospective customers about how the tools we build can help their teams as well as provide value to their user base. If you'd like to get in touch with me, please share your thoughts in the form below and our team can help setup a call. Learn more about our products and services: