IoT Security and Why It Matters to Your Product!

Screen Shot 2017-06-08 at 12.21.16 PM

You know IoT security is a serious concern when startups are getting funded to the level of $17M by Tier 1 investors like Sequoia Capital and Tenaya Capital.   Armis, an Israeli and Silicon Valley startup, is on a mission to identify compromised IoT devices on the corporate network and quarantine them.   Lest you forget, this is not such a far-fetched concern.   Recently, an army of IoT cameras and WiFi routers, operating as an IoT botnet called Mirai, was used to create a Denial of Service of Attack on Dyn’s DNS service that brought the Internet to its knees.

Said another way, security in IoT products is such a concern that companies are basing their entire business models on protecting from their presence.   If you are a company that makes IoT products, or any product that runs on a customer’s network for that matter, this should scare the bejesus out of you!    You do not want your product to be the cause of the next botnet or worse to compromise your customer’s business.

Cameras in people’s homes are soft targets, but it is only a matter of time before industrial machines, medical devices, ATMs, home security gear and other mission critical infrastructure is targeted – and then it won’t be a Denial of Service attack that the bad guys are trying to accomplish.

Let’s think of the ways that a product of this class can be compromised.

  1. Like the botnet attacks, it can be compromised by an internal attack. There is malicious software running on a network, it finds your machine on that network, and exploits an attack to gain entry.   Anything that gives your machine a footprint on the network is a possible open door for bad guys: telnet service, web service, etc.
  2. Machine is compromised by a poorly designed remote service solution.  Remote access is one of the most useful and potentially troublesome methods for exploiting a company.   Most remote access solutions allow unattended access without the permission of the company that owns the machine.   Furthermore, this unattended access often provides the remote access user, a person outside the company’s security policies, with open network access.
  3. Machine is compromised by the machine vendor’s own employees. Some of the craziest stories we have heard involve a machine vendor’s very own employees using their remote access solution to plant backdoors that they could later exploit for nefarious means.   While it is hard to believe, it is a legitimate threat and must be guarded against.

So how do you have your cake and eat it too?    A product that takes full advantage of the internet, provides support access when it is required, yet protects both the customer and product vendor from the bad guys, inside and outside the company.    In our experience, over 150 Internet and Service enabled products, we recommend the following approach:

  1. Close all open ports on the device:    An open port is an invitation for a bad guy to exploit a device.   We recommend that our customers close all of them.
  2. Utilize indirect access: This approach uses a third-party solution, like RevTwo (shameless plug) to provide access to elements of the machine or device that are required.    For example, if you want to access a web-server present on the machine or device, access it through a remote tunnel on localhost (i.e. ports that are exposed to localhost only – not visible at a network level).    You can use this technique to access web pages, local screens, even terminal screens.
  3. Respect sandbox limitations: Often we see microservice architected solutions that blow away the elegance and security of their solution by utilizing root security level remote service solutions that provide access at the machine level vs. the microservice level.   Respect the sandbox limitation and support from the inside out.
  4. Permission based access: In the end, the customer is king.   We recommend that all access on a customer’s network should initiated only after permission has been explicitly granted for that session.   This protects both the customer from unauthorized access to sensitive equipment, but protects the machine manufacturer from liability associated with that access.
  5. Audit everything: Trust but verify.   A comprehensive audit log not only gives your customer peace of mind that they have a record of all activity on their network but it also protects the machine vendor in case there is a security breach at the customer location.

In the end, while we applaud companies that try to find compromised IoT devices and protect their owners from them, we advocate a more proactive approach.   We believe it is the responsibility of all IoT products to be safe and secure and we view it as our job to make the tools and infrastructure to make sure that happens!


Moving from Axeda to ThingWorx

Adding Voice and Permission-Based Remote Support to ThingWorx

As most of you know, I founded Axeda over 15 years ago and hold quite a few patents on the topic of IoT and Remote Service.   Over that time, I have seen many IoT based remote support programs.   In large part, remote support for complex machines and devices has stayed pretty consistent and work something like this:

  1. A machine operator leaves the machine, goes to a quiet location, and calls for support
  2. The machine operator then navigates the call center until they get the right person to help them
  3. That person opens a remote session and starts troubleshooting
  4. All the while, the machine operator is still far away from the machine, tethered to the phone…

Can you spot the problems with this scenario?  Not only does the operator have to leave the machine to ask for help, but also, remote access can happen without their permission and without their physical proximity to the machine.   In today’s hyper threat world – definitely not an optimal security model.   Furthermore, what happens if the operator provides  a phone number?  Is it their cell phone or the number in a conference room?   What happens when the operator goes home?  For many types of machines this poses a real quandary.

This was the problem that INSTRON faced when they decided to update their existing, Axeda-based, remote support program to ThingWorx.   On one hand, Instron wanted the same remote troubleshooting capability that they enjoyed with their Axeda system, yet, at the same time, they wanted to create a process that dramatically improved the user experience, all while improving site security.

RevTwo provided both.

The new and improved Instron Connect uses ThingWorx to capture machine data and perform cloud-based rules and analytics, while using RevTwo to provide secure remote access and VoIP based machine communication.

The combination delivers a much-improved customer workflow, experience, and security.   The Instron support scenario now looks like this.

  1. Customer requests help right from the machine
  2. The help request gets routed to a support tech who can help them.
  3. The support tech places a “Machine-based” support call using RevTwo’s built-in VoIP technology.  No operator cell phone numbers are necessary.2
  4. The operator must accept the support request call and grant permission for a remote troubleshooting session to occur.IC_Image1-1
  5. The operator and support person are now able to talk with each other while the support agent troubleshoots the machine, ALL AT THE SAME TIME.

The new workflow is much improved for both Instron and their customers.  The support agent is now able to talk to the operator during troubleshooting sessions using the built-in RevTwo-powered VoIP communication.   The operator controls when and if remote sessions occur, improving site security, while the support agent gains a valuable set of eyes and ears improving troubleshooting and safety.

The RevTwo stack seamlessly integrates with all IoT cloud or edge-based data pipes, such as ThingWorx, as well as with the Amazon IoT Platform, the Microsoft Azure IoT Suite, and the EdgeX Foundry.

RevTwo will  attend PTC’s LiveWorx event this  week (May 23-25)..  We will be glad to discuss how RevTwo can be easily added to your remote support program.


The 10 Commandments of Smart Support

“If my product is so smart, why is its support so dumb” – frustrated consumer

We started RevTwo with a simple mission in mind, fix product support.   We were tired of all the things that you experience when trying to solve a product issue:

  • Hunting for the support phone number
  • Waiting on hold for extended lengths of time, cue the elevator music
  • Telling your story over and over again
  • Spending hours researching possible solutions
  • Getting answers that don’t ultimately fix the problem

Products & exceptional support should go together like mom and apple pie, right?  In fact, the last company I founded, Axeda, powered remote support programs from some of the world’s leading companies: GE, Philips, Diebold, Abbott, Roche, Medtronic, to name a few. I have personally been involved with over 300 remote support programs in the IoT space and the one thing I can tell you with absolute earnestness, EVERY COMPANY WANTS TO PROVIDE GREAT SUPPORT.

So why is there a disconnect?

Frankly, it’s pretty simple, support is expensive and hard. With average costs per tickets for some industries hitting almost $70, it is easy to see the difficulty.

When a product issue requires human support intervention the $$ counter starts spinning very fast.  As a result, companies have historically put up defense mechanisms to prevent, or at least limit, human interaction to those who absolutely require it – creating a frustrating process for the very customers they want to delight.  Today’s support paradigm is a lose/lose proposition. Costly for companies, frustrating customers.

It doesn’t need to be this way because today’s products are different:

  • They are connected
  • Most have apps or user interfaces that are also connected
  • They run on standard operating systems and have built-in intelligence

This combination of app, connectivity, and intelligence provides an unprecedented opportunity to do things better.  While we were building RevTwo, we started to consider what “better” really meant.   Ultimately, we codified better, into a set of guiding principles that we affectionately call the 10 Commandments of Smart Support.  

The 10 Commandments of Smart Support:

  1. Customers should never have to describe a problem or ask a question twice: How many times have you called a support desk only to tell person after person about your product issue while trying to get to the magic human who can help?  It is time consuming, frustrating, and costly.
  2. Customers should NEVER wait on hold: We believe that there is no greater insult to a person than to disrespect their time.  When you put a person on hold you are effectively telling them they don’t matter. Callbacks or immediate answers are what customers deserve.
  3. Customer and product context should be always maintained and leveraged: Context! Almost all smart products today know who you are and what you are doing.  When you report the problem from within the product (see commandment #4), you have access to a treasure trove of machine/device data that will help identify what the problem is.
  4. Customers should not have to leave the product to report a problem: Why should customers have to place a call, send an email, tweet, or use messenger to report a problem?   When a problem is reported from outside of the product, the context disappears. I know omnichannel support is a hot topic these days, and should be done, but how much of a customer’s use of alternate support channels is the result of poor support access right at the point of attack?  A customer should be able to initiate and conclude a support session, easily, from within the product itself.
  5. AI and machine learning should be used to solve problems and answer questions, without human intervention, whenever possible: What if a system exists that could solve problems like your best Level 3 engineer?  What if you could deploy that resource with frictionless scale?  What if that resource learned as it went and got better at its job every day!  We believe that the having the actual machine context at the time of an issue makes this possible.
  6. You should communicate with the customer in the best way to solve the problem from within the product itself: Imagine you have reported a problem on a machine in a large factory. However, to communicate with the support agent, you must leave the factory floor to talk on a phone. Does this make sense? As with context, communication at the point of problem is critical to fast and painless resolution.  Chat is lightweight and effective, but there is often no substitute for high fidelity communication.  In fact, leading companies are trending towards voice/video communication to deliver more personalized support.
  7. Agents should have all the information and tools they need to quickly solve problems: Smart products are connected.  This connectivity allows agents to connect directly to the product and leverage their tool set to troubleshoot the problem.  Screen sharing, terminal access, remote program support, and log files – there is no substitute for direct product access
  8. Remote support should be highly secure and utilize permission-based access: The last thing you want is to see that your company’s products were a part of the latest Internet bot attack.   In fact, it is important to remember that security requires both technical and process vigilance. The bad guys are not just outside of the company’s firewall – once a computer gets compromised inside the company, the threat can be on the inside as well.  We recommend a multi-tier approach: Tier 1: Encrypt all communication. Tier 2: Turn off local access – such as SSH services. Tier 3: Require permission to get access.
  9. Support agents should know about all the products in a system and be able to access them quickly: Many consumer smart products involve an app and one or many paired products.  Consider a home automation system where you have one app, and perhaps 20-30 different connected products.  Support agents should be able to identify, connect, and troubleshoot any component in the system at the click of a button.
  10. Think outside of the box: A smart phone in a customer’s hand can turn any product, even ones that aren’t smart yet, into smart product.  Whether it be a retail app selling apparel or an app used by the local landscaping firm, interacting via a smart app can enrich the support experience by providing pictures, deep links, and even video broadcasts of problem.

Whether you make an enterprise app, a smart consumer product, a machine tool, medical device, or even a microservice, we believe that these commandments hold. After all, great customer support is one of the secrets behind some of the world’s most successful brands – it is the real sauce for turning your company into an industry Unicorn.

If you have your own principles of great support that you would like to share or have a comment to add to our list, please add it in the comments – or feel free to reach out to me at dale@revtwo.com to discuss how we can put some of these practices to work for your smart product.