IoT Security and Why It Matters to Your Product!

Screen Shot 2017-06-08 at 12.21.16 PM

You know IoT security is a serious concern when startups are getting funded to the level of $17M by Tier 1 investors like Sequoia Capital and Tenaya Capital.   Armis, an Israeli and Silicon Valley startup, is on a mission to identify compromised IoT devices on the corporate network and quarantine them.   Lest you forget, this is not such a far-fetched concern.   Recently, an army of IoT cameras and WiFi routers, operating as an IoT botnet called Mirai, was used to create a Denial of Service of Attack on Dyn’s DNS service that brought the Internet to its knees.

Said another way, security in IoT products is such a concern that companies are basing their entire business models on protecting from their presence.   If you are a company that makes IoT products, or any product that runs on a customer’s network for that matter, this should scare the bejesus out of you!    You do not want your product to be the cause of the next botnet or worse to compromise your customer’s business.

Cameras in people’s homes are soft targets, but it is only a matter of time before industrial machines, medical devices, ATMs, home security gear and other mission critical infrastructure is targeted – and then it won’t be a Denial of Service attack that the bad guys are trying to accomplish.

Let’s think of the ways that a product of this class can be compromised.

  1. Like the botnet attacks, it can be compromised by an internal attack. There is malicious software running on a network, it finds your machine on that network, and exploits an attack to gain entry.   Anything that gives your machine a footprint on the network is a possible open door for bad guys: telnet service, web service, etc.
  2. Machine is compromised by a poorly designed remote service solution.  Remote access is one of the most useful and potentially troublesome methods for exploiting a company.   Most remote access solutions allow unattended access without the permission of the company that owns the machine.   Furthermore, this unattended access often provides the remote access user, a person outside the company’s security policies, with open network access.
  3. Machine is compromised by the machine vendor’s own employees. Some of the craziest stories we have heard involve a machine vendor’s very own employees using their remote access solution to plant backdoors that they could later exploit for nefarious means.   While it is hard to believe, it is a legitimate threat and must be guarded against.

So how do you have your cake and eat it too?    A product that takes full advantage of the internet, provides support access when it is required, yet protects both the customer and product vendor from the bad guys, inside and outside the company.    In our experience, over 150 Internet and Service enabled products, we recommend the following approach:

  1. Close all open ports on the device:    An open port is an invitation for a bad guy to exploit a device.   We recommend that our customers close all of them.
  2. Utilize indirect access: This approach uses a third-party solution, like RevTwo (shameless plug) to provide access to elements of the machine or device that are required.    For example, if you want to access a web-server present on the machine or device, access it through a remote tunnel on localhost (i.e. ports that are exposed to localhost only – not visible at a network level).    You can use this technique to access web pages, local screens, even terminal screens.
  3. Respect sandbox limitations: Often we see microservice architected solutions that blow away the elegance and security of their solution by utilizing root security level remote service solutions that provide access at the machine level vs. the microservice level.   Respect the sandbox limitation and support from the inside out.
  4. Permission based access: In the end, the customer is king.   We recommend that all access on a customer’s network should initiated only after permission has been explicitly granted for that session.   This protects both the customer from unauthorized access to sensitive equipment, but protects the machine manufacturer from liability associated with that access.
  5. Audit everything: Trust but verify.   A comprehensive audit log not only gives your customer peace of mind that they have a record of all activity on their network but it also protects the machine vendor in case there is a security breach at the customer location.

In the end, while we applaud companies that try to find compromised IoT devices and protect their owners from them, we advocate a more proactive approach.   We believe it is the responsibility of all IoT products to be safe and secure and we view it as our job to make the tools and infrastructure to make sure that happens!

EdgeX Foundry – The Rosetta Stone of IoT

As a pioneer of the IoT industry, I am afforded somewhat of a unique perspective.   Over the past 20 years, I have seen IoT go from the “Why Would We Ever Do That?” stage, when no one understood the benefits of connecting the physical world to the digital, to now, when every company on the face of the earth fancies themselves Google and wants to be the “on-ramp for the digital transformation of the physical world.”   Pick your number, but 50 billion has probably been the most popular, and there is a lot of value to be unlocked by connecting that many things.

However, with every company making their own claim to this bonanza, cooperation between competing standards, approaches, and companies, has been VERY limited – VERY VERY LIMITED.   So, when Dell approached us about a new initiative to drive IoT interoperability through open sourcing via the Linux Foundation, I was intrigued.

The EdgeX Foundry provides a “Rosetta Stone” approach to the disparate approaches around IoT.   Think of it as

A gateway that can talk to anything on one side, do interesting things with those things it talks to, and then send the results and data to any cloud destination.

The best part?  By being an open sourced and a standards-based initiative, companies are incented to add their special brand of connectivity or cloud-service to it, or risk being left out.

EdgeX Foundry
EdgeX Foundry

I was involved in a standard in the mid-90’s called OPC (OLE for Process Control), for those who actually remember Microsoft’s push for an interconnectivity standard for Windows (called OLE at one time).   OPC was and still is the most successful approach for connecting the myriads of different industrial communication standards.   It was a hugely successful initiative with every company in the industrial space supporting it, and it solved a fundamental problem.   Connectivity to the real-world for everything is expensive and hard to do – and most companies make their money on capitalizing on that connectivity – and most end-users accrue benefit only from that capitalization.   So, everyone was incented to cooperate – and cooperate they did.

I see the EdgeX Foundry as a very similar initiative.   The Industrial Internet of Things (IIoT) is a very messy space.   There are sensors, controllers, gizmos of all shapes and sizes. I have never seen a factory with stuff from just one vendor – it is a HETEROGENEOUS soup of connectedness.    If the actual users of those things are going to benefit from the promise of IIoT, it is going to be from high level coordination – FROM MANY VENDORS.    With open connectivity on both sides and open processing in the middle – the EdgeX Foundry is the best approach to this.

Now you may ask, why does RevTwo care about this initiative – aren’t we a support platform?   Well the answer is simple – with millions of EdgeX gateways operating in the wild, there will be the need to securely provide and syndicate support between multiple vendors (Box and Component Vendors) and users.   That is RevTwo’s bread-and-butter.   So, we are in!

Now onto making that 50 billion connected devices and sensors and the value it promises a reality.

The 10 Commandments of Smart Support

“If my product is so smart, why is its support so dumb” – frustrated consumer

We started RevTwo with a simple mission in mind, fix product support.   We were tired of all the things that you experience when trying to solve a product issue:

  • Hunting for the support phone number
  • Waiting on hold for extended lengths of time, cue the elevator music
  • Telling your story over and over again
  • Spending hours researching possible solutions
  • Getting answers that don’t ultimately fix the problem

Products & exceptional support should go together like mom and apple pie, right?  In fact, the last company I founded, Axeda, powered remote support programs from some of the world’s leading companies: GE, Philips, Diebold, Abbott, Roche, Medtronic, to name a few. I have personally been involved with over 300 remote support programs in the IoT space and the one thing I can tell you with absolute earnestness, EVERY COMPANY WANTS TO PROVIDE GREAT SUPPORT.

So why is there a disconnect?

Frankly, it’s pretty simple, support is expensive and hard. With average costs per tickets for some industries hitting almost $70, it is easy to see the difficulty.

When a product issue requires human support intervention the $$ counter starts spinning very fast.  As a result, companies have historically put up defense mechanisms to prevent, or at least limit, human interaction to those who absolutely require it – creating a frustrating process for the very customers they want to delight.  Today’s support paradigm is a lose/lose proposition. Costly for companies, frustrating customers.

It doesn’t need to be this way because today’s products are different:

  • They are connected
  • Most have apps or user interfaces that are also connected
  • They run on standard operating systems and have built-in intelligence

This combination of app, connectivity, and intelligence provides an unprecedented opportunity to do things better.  While we were building RevTwo, we started to consider what “better” really meant.   Ultimately, we codified better, into a set of guiding principles that we affectionately call the 10 Commandments of Smart Support.  

The 10 Commandments of Smart Support:

  1. Customers should never have to describe a problem or ask a question twice: How many times have you called a support desk only to tell person after person about your product issue while trying to get to the magic human who can help?  It is time consuming, frustrating, and costly.
  2. Customers should NEVER wait on hold: We believe that there is no greater insult to a person than to disrespect their time.  When you put a person on hold you are effectively telling them they don’t matter. Callbacks or immediate answers are what customers deserve.
  3. Customer and product context should be always maintained and leveraged: Context! Almost all smart products today know who you are and what you are doing.  When you report the problem from within the product (see commandment #4), you have access to a treasure trove of machine/device data that will help identify what the problem is.
  4. Customers should not have to leave the product to report a problem: Why should customers have to place a call, send an email, tweet, or use messenger to report a problem?   When a problem is reported from outside of the product, the context disappears. I know omnichannel support is a hot topic these days, and should be done, but how much of a customer’s use of alternate support channels is the result of poor support access right at the point of attack?  A customer should be able to initiate and conclude a support session, easily, from within the product itself.
  5. AI and machine learning should be used to solve problems and answer questions, without human intervention, whenever possible: What if a system exists that could solve problems like your best Level 3 engineer?  What if you could deploy that resource with frictionless scale?  What if that resource learned as it went and got better at its job every day!  We believe that the having the actual machine context at the time of an issue makes this possible.
  6. You should communicate with the customer in the best way to solve the problem from within the product itself: Imagine you have reported a problem on a machine in a large factory. However, to communicate with the support agent, you must leave the factory floor to talk on a phone. Does this make sense? As with context, communication at the point of problem is critical to fast and painless resolution.  Chat is lightweight and effective, but there is often no substitute for high fidelity communication.  In fact, leading companies are trending towards voice/video communication to deliver more personalized support.
  7. Agents should have all the information and tools they need to quickly solve problems: Smart products are connected.  This connectivity allows agents to connect directly to the product and leverage their tool set to troubleshoot the problem.  Screen sharing, terminal access, remote program support, and log files – there is no substitute for direct product access
  8. Remote support should be highly secure and utilize permission-based access: The last thing you want is to see that your company’s products were a part of the latest Internet bot attack.   In fact, it is important to remember that security requires both technical and process vigilance. The bad guys are not just outside of the company’s firewall – once a computer gets compromised inside the company, the threat can be on the inside as well.  We recommend a multi-tier approach: Tier 1: Encrypt all communication. Tier 2: Turn off local access – such as SSH services. Tier 3: Require permission to get access.
  9. Support agents should know about all the products in a system and be able to access them quickly: Many consumer smart products involve an app and one or many paired products.  Consider a home automation system where you have one app, and perhaps 20-30 different connected products.  Support agents should be able to identify, connect, and troubleshoot any component in the system at the click of a button.
  10. Think outside of the box: A smart phone in a customer’s hand can turn any product, even ones that aren’t smart yet, into smart product.  Whether it be a retail app selling apparel or an app used by the local landscaping firm, interacting via a smart app can enrich the support experience by providing pictures, deep links, and even video broadcasts of problem.

Whether you make an enterprise app, a smart consumer product, a machine tool, medical device, or even a microservice, we believe that these commandments hold. After all, great customer support is one of the secrets behind some of the world’s most successful brands – it is the real sauce for turning your company into an industry Unicorn.

If you have your own principles of great support that you would like to share or have a comment to add to our list, please add it in the comments – or feel free to reach out to me at to discuss how we can put some of these practices to work for your smart product.