IoT Security and Why It Matters to Your Product!

Screen Shot 2017-06-08 at 12.21.16 PM

You know IoT security is a serious concern when startups are getting funded to the level of $17M by Tier 1 investors like Sequoia Capital and Tenaya Capital.   Armis, an Israeli and Silicon Valley startup, is on a mission to identify compromised IoT devices on the corporate network and quarantine them.   Lest you forget, this is not such a far-fetched concern.   Recently, an army of IoT cameras and WiFi routers, operating as an IoT botnet called Mirai, was used to create a Denial of Service of Attack on Dyn’s DNS service that brought the Internet to its knees.

Said another way, security in IoT products is such a concern that companies are basing their entire business models on protecting from their presence.   If you are a company that makes IoT products, or any product that runs on a customer’s network for that matter, this should scare the bejesus out of you!    You do not want your product to be the cause of the next botnet or worse to compromise your customer’s business.

Cameras in people’s homes are soft targets, but it is only a matter of time before industrial machines, medical devices, ATMs, home security gear and other mission critical infrastructure is targeted – and then it won’t be a Denial of Service attack that the bad guys are trying to accomplish.

Let’s think of the ways that a product of this class can be compromised.

  1. Like the botnet attacks, it can be compromised by an internal attack. There is malicious software running on a network, it finds your machine on that network, and exploits an attack to gain entry.   Anything that gives your machine a footprint on the network is a possible open door for bad guys: telnet service, web service, etc.
  2. Machine is compromised by a poorly designed remote service solution.  Remote access is one of the most useful and potentially troublesome methods for exploiting a company.   Most remote access solutions allow unattended access without the permission of the company that owns the machine.   Furthermore, this unattended access often provides the remote access user, a person outside the company’s security policies, with open network access.
  3. Machine is compromised by the machine vendor’s own employees. Some of the craziest stories we have heard involve a machine vendor’s very own employees using their remote access solution to plant backdoors that they could later exploit for nefarious means.   While it is hard to believe, it is a legitimate threat and must be guarded against.

So how do you have your cake and eat it too?    A product that takes full advantage of the internet, provides support access when it is required, yet protects both the customer and product vendor from the bad guys, inside and outside the company.    In our experience, over 150 Internet and Service enabled products, we recommend the following approach:

  1. Close all open ports on the device:    An open port is an invitation for a bad guy to exploit a device.   We recommend that our customers close all of them.
  2. Utilize indirect access: This approach uses a third-party solution, like RevTwo (shameless plug) to provide access to elements of the machine or device that are required.    For example, if you want to access a web-server present on the machine or device, access it through a remote tunnel on localhost (i.e. ports that are exposed to localhost only – not visible at a network level).    You can use this technique to access web pages, local screens, even terminal screens.
  3. Respect sandbox limitations: Often we see microservice architected solutions that blow away the elegance and security of their solution by utilizing root security level remote service solutions that provide access at the machine level vs. the microservice level.   Respect the sandbox limitation and support from the inside out.
  4. Permission based access: In the end, the customer is king.   We recommend that all access on a customer’s network should initiated only after permission has been explicitly granted for that session.   This protects both the customer from unauthorized access to sensitive equipment, but protects the machine manufacturer from liability associated with that access.
  5. Audit everything: Trust but verify.   A comprehensive audit log not only gives your customer peace of mind that they have a record of all activity on their network but it also protects the machine vendor in case there is a security breach at the customer location.

In the end, while we applaud companies that try to find compromised IoT devices and protect their owners from them, we advocate a more proactive approach.   We believe it is the responsibility of all IoT products to be safe and secure and we view it as our job to make the tools and infrastructure to make sure that happens!

Moving from Axeda to ThingWorx

Adding Voice and Permission-Based Remote Support to ThingWorx

As most of you know, I founded Axeda over 15 years ago and hold quite a few patents on the topic of IoT and Remote Service.   Over that time, I have seen many IoT based remote support programs.   In large part, remote support for complex machines and devices has stayed pretty consistent and work something like this:

  1. A machine operator leaves the machine, goes to a quiet location, and calls for support
  2. The machine operator then navigates the call center until they get the right person to help them
  3. That person opens a remote session and starts troubleshooting
  4. All the while, the machine operator is still far away from the machine, tethered to the phone…

Can you spot the problems with this scenario?  Not only does the operator have to leave the machine to ask for help, but also, remote access can happen without their permission and without their physical proximity to the machine.   In today’s hyper threat world – definitely not an optimal security model.   Furthermore, what happens if the operator provides  a phone number?  Is it their cell phone or the number in a conference room?   What happens when the operator goes home?  For many types of machines this poses a real quandary.

This was the problem that INSTRON faced when they decided to update their existing, Axeda-based, remote support program to ThingWorx.   On one hand, Instron wanted the same remote troubleshooting capability that they enjoyed with their Axeda system, yet, at the same time, they wanted to create a process that dramatically improved the user experience, all while improving site security.

RevTwo provided both.

The new and improved Instron Connect uses ThingWorx to capture machine data and perform cloud-based rules and analytics, while using RevTwo to provide secure remote access and VoIP based machine communication.

The combination delivers a much-improved customer workflow, experience, and security.   The Instron support scenario now looks like this.

  1. Customer requests help right from the machine
  2. The help request gets routed to a support tech who can help them.
  3. The support tech places a “Machine-based” support call using RevTwo’s built-in VoIP technology.  No operator cell phone numbers are necessary.2
  4. The operator must accept the support request call and grant permission for a remote troubleshooting session to occur.IC_Image1-1
  5. The operator and support person are now able to talk with each other while the support agent troubleshoots the machine, ALL AT THE SAME TIME.

The new workflow is much improved for both Instron and their customers.  The support agent is now able to talk to the operator during troubleshooting sessions using the built-in RevTwo-powered VoIP communication.   The operator controls when and if remote sessions occur, improving site security, while the support agent gains a valuable set of eyes and ears improving troubleshooting and safety.

The RevTwo stack seamlessly integrates with all IoT cloud or edge-based data pipes, such as ThingWorx, as well as with the Amazon IoT Platform, the Microsoft Azure IoT Suite, and the EdgeX Foundry.

RevTwo will  attend PTC’s LiveWorx event this  week (May 23-25)..  We will be glad to discuss how RevTwo can be easily added to your remote support program.

EdgeX Foundry – The Rosetta Stone of IoT

As a pioneer of the IoT industry, I am afforded somewhat of a unique perspective.   Over the past 20 years, I have seen IoT go from the “Why Would We Ever Do That?” stage, when no one understood the benefits of connecting the physical world to the digital, to now, when every company on the face of the earth fancies themselves Google and wants to be the “on-ramp for the digital transformation of the physical world.”   Pick your number, but 50 billion has probably been the most popular, and there is a lot of value to be unlocked by connecting that many things.

However, with every company making their own claim to this bonanza, cooperation between competing standards, approaches, and companies, has been VERY limited – VERY VERY LIMITED.   So, when Dell approached us about a new initiative to drive IoT interoperability through open sourcing via the Linux Foundation, I was intrigued.

The EdgeX Foundry provides a “Rosetta Stone” approach to the disparate approaches around IoT.   Think of it as

A gateway that can talk to anything on one side, do interesting things with those things it talks to, and then send the results and data to any cloud destination.

The best part?  By being an open sourced and a standards-based initiative, companies are incented to add their special brand of connectivity or cloud-service to it, or risk being left out.

EdgeX Foundry
EdgeX Foundry

I was involved in a standard in the mid-90’s called OPC (OLE for Process Control), for those who actually remember Microsoft’s push for an interconnectivity standard for Windows (called OLE at one time).   OPC was and still is the most successful approach for connecting the myriads of different industrial communication standards.   It was a hugely successful initiative with every company in the industrial space supporting it, and it solved a fundamental problem.   Connectivity to the real-world for everything is expensive and hard to do – and most companies make their money on capitalizing on that connectivity – and most end-users accrue benefit only from that capitalization.   So, everyone was incented to cooperate – and cooperate they did.

I see the EdgeX Foundry as a very similar initiative.   The Industrial Internet of Things (IIoT) is a very messy space.   There are sensors, controllers, gizmos of all shapes and sizes. I have never seen a factory with stuff from just one vendor – it is a HETEROGENEOUS soup of connectedness.    If the actual users of those things are going to benefit from the promise of IIoT, it is going to be from high level coordination – FROM MANY VENDORS.    With open connectivity on both sides and open processing in the middle – the EdgeX Foundry is the best approach to this.

Now you may ask, why does RevTwo care about this initiative – aren’t we a support platform?   Well the answer is simple – with millions of EdgeX gateways operating in the wild, there will be the need to securely provide and syndicate support between multiple vendors (Box and Component Vendors) and users.   That is RevTwo’s bread-and-butter.   So, we are in!

Now onto making that 50 billion connected devices and sensors and the value it promises a reality.

The 10 Commandments of Smart Support

“If my product is so smart, why is its support so dumb” – frustrated consumer

We started RevTwo with a simple mission in mind, fix product support.   We were tired of all the things that you experience when trying to solve a product issue:

  • Hunting for the support phone number
  • Waiting on hold for extended lengths of time, cue the elevator music
  • Telling your story over and over again
  • Spending hours researching possible solutions
  • Getting answers that don’t ultimately fix the problem

Products & exceptional support should go together like mom and apple pie, right?  In fact, the last company I founded, Axeda, powered remote support programs from some of the world’s leading companies: GE, Philips, Diebold, Abbott, Roche, Medtronic, to name a few. I have personally been involved with over 300 remote support programs in the IoT space and the one thing I can tell you with absolute earnestness, EVERY COMPANY WANTS TO PROVIDE GREAT SUPPORT.

So why is there a disconnect?

Frankly, it’s pretty simple, support is expensive and hard. With average costs per tickets for some industries hitting almost $70, it is easy to see the difficulty.

When a product issue requires human support intervention the $$ counter starts spinning very fast.  As a result, companies have historically put up defense mechanisms to prevent, or at least limit, human interaction to those who absolutely require it – creating a frustrating process for the very customers they want to delight.  Today’s support paradigm is a lose/lose proposition. Costly for companies, frustrating customers.

It doesn’t need to be this way because today’s products are different:

  • They are connected
  • Most have apps or user interfaces that are also connected
  • They run on standard operating systems and have built-in intelligence

This combination of app, connectivity, and intelligence provides an unprecedented opportunity to do things better.  While we were building RevTwo, we started to consider what “better” really meant.   Ultimately, we codified better, into a set of guiding principles that we affectionately call the 10 Commandments of Smart Support.  

The 10 Commandments of Smart Support:

  1. Customers should never have to describe a problem or ask a question twice: How many times have you called a support desk only to tell person after person about your product issue while trying to get to the magic human who can help?  It is time consuming, frustrating, and costly.
  2. Customers should NEVER wait on hold: We believe that there is no greater insult to a person than to disrespect their time.  When you put a person on hold you are effectively telling them they don’t matter. Callbacks or immediate answers are what customers deserve.
  3. Customer and product context should be always maintained and leveraged: Context! Almost all smart products today know who you are and what you are doing.  When you report the problem from within the product (see commandment #4), you have access to a treasure trove of machine/device data that will help identify what the problem is.
  4. Customers should not have to leave the product to report a problem: Why should customers have to place a call, send an email, tweet, or use messenger to report a problem?   When a problem is reported from outside of the product, the context disappears. I know omnichannel support is a hot topic these days, and should be done, but how much of a customer’s use of alternate support channels is the result of poor support access right at the point of attack?  A customer should be able to initiate and conclude a support session, easily, from within the product itself.
  5. AI and machine learning should be used to solve problems and answer questions, without human intervention, whenever possible: What if a system exists that could solve problems like your best Level 3 engineer?  What if you could deploy that resource with frictionless scale?  What if that resource learned as it went and got better at its job every day!  We believe that the having the actual machine context at the time of an issue makes this possible.
  6. You should communicate with the customer in the best way to solve the problem from within the product itself: Imagine you have reported a problem on a machine in a large factory. However, to communicate with the support agent, you must leave the factory floor to talk on a phone. Does this make sense? As with context, communication at the point of problem is critical to fast and painless resolution.  Chat is lightweight and effective, but there is often no substitute for high fidelity communication.  In fact, leading companies are trending towards voice/video communication to deliver more personalized support.
  7. Agents should have all the information and tools they need to quickly solve problems: Smart products are connected.  This connectivity allows agents to connect directly to the product and leverage their tool set to troubleshoot the problem.  Screen sharing, terminal access, remote program support, and log files – there is no substitute for direct product access
  8. Remote support should be highly secure and utilize permission-based access: The last thing you want is to see that your company’s products were a part of the latest Internet bot attack.   In fact, it is important to remember that security requires both technical and process vigilance. The bad guys are not just outside of the company’s firewall – once a computer gets compromised inside the company, the threat can be on the inside as well.  We recommend a multi-tier approach: Tier 1: Encrypt all communication. Tier 2: Turn off local access – such as SSH services. Tier 3: Require permission to get access.
  9. Support agents should know about all the products in a system and be able to access them quickly: Many consumer smart products involve an app and one or many paired products.  Consider a home automation system where you have one app, and perhaps 20-30 different connected products.  Support agents should be able to identify, connect, and troubleshoot any component in the system at the click of a button.
  10. Think outside of the box: A smart phone in a customer’s hand can turn any product, even ones that aren’t smart yet, into smart product.  Whether it be a retail app selling apparel or an app used by the local landscaping firm, interacting via a smart app can enrich the support experience by providing pictures, deep links, and even video broadcasts of problem.

Whether you make an enterprise app, a smart consumer product, a machine tool, medical device, or even a microservice, we believe that these commandments hold. After all, great customer support is one of the secrets behind some of the world’s most successful brands – it is the real sauce for turning your company into an industry Unicorn.

If you have your own principles of great support that you would like to share or have a comment to add to our list, please add it in the comments – or feel free to reach out to me at to discuss how we can put some of these practices to work for your smart product.

INFOGRAPHIC – Top 5 Reasons You Need In-App Support

Apps today have crossed into a place where they are not just for entertainment. We open our house, buy furniture, sell stocks, do our banking, check in on our children and pets, adjust our sprinklers, order pizza, interact with our doctors, and even do our work – all with apps. Apps are a mission critical component of our lives and their support mechanisms need to respect that mission criticality.

Along with the anecdotal evidence, the actual facts paint an even more daunting picture.   Mobile commerce is the fastest growing type of commerce, expanding at 3 times the rate of traditional e-commerce.   Yet, 83% of those transactions require live customer interactions when making a mobile purchase and over 16% will buy from a competitor if they encounter a hiccup at all.

The case is compelling, the need profound, yet still the vast majority of apps today are released without a strategy for support. In the past six months, we have seen case after case where app support is able to significantly improve both the business operations of the publisher and the experience of the actual user.

Still not convinced – check out our infographic for even more mind-blowing detail.

Top 5 Reasons You Need In-App Support

Live from SDC – Part 2

What a week we had out at #iheartSDC!  Our last day didn’t disappoint with an enlightened session by Marv Storey called Don’t Lose App Revenue Over Poor Customer Experience – we couldn’t agree more!

We learned that 75% of users desire help on their smart phones (Synthetix).  Additionally, 72% of app users actually expect the company to support the app (Nuance).  Marv pointed out that customers are willing to pay up to 2x when they know it will lead to a quality product and are much less likely to be dissatisfied with the price when there are no issues.  Encountering just 2 problems can bring price dissatisfaction up to 50% (John A Goodman, ‘customer experience 3.0’ amacom 2014).

Having a real-time support strategy in place could go a long way to ensure customer success & satisfaction, especially since support in apps is typically satisfied with phone support, email support, external forums, and internet searches. 

According to Marv, studies show that 53% of app users prefer phone support over the others for issues they encounter.  These days, app developers tend towards email support because it is easy for them. However, app users are almost always dissatisfied with this because it is impersonal and there is no guarantee of a timely resolution.  Phone calls are much preferred because of the empathy they generate and the customer feels like they can get real time resolution.

This was interesting to me because RevTwo falls right in the sweet spot between these two types of support. We marry the human touch that people love from phone calls with the ease of use of email, all while keeping your users within your app. It is also extremely easy for developers to integrate.

I found Marv’s presentation to be incredibly insightful, what do you think?

Live From SDC

Exciting things are happening in San Francisco this week!  The 2016 Samsung Developer Conference has taken over Moscone West and the RevTwo team and I have been lucky enough to get a first-hand look at several exciting new technologies coming out from Samsung.

Day 0 gave us a chance to get our hands dirty with the Knox API – Samsung’s mobility management platform that gives you the power of full security control over your enterprise devices.  I also had my first experience with virtual reality in their filming for VR workshop.  I eagerly snapped on the Gear VR headset and was instantly transported to Tahiti and immersed in the clear water, overhanging bungalows, and tropical aesthetic.

Needless to say, exposure to such exciting tech had us pumped for Day 1, which opened with an inspiring Keynote that had everyone excited about new areas in VR.  Samsung is aiming for a star trek holodeck immersive experience.


Lots of innovative things on the IoT front and Smart Things platform.  Connected cars are a reality and Samsung wants to make it easier to build apps for your car.  Samsung now offers a small device that can be plugged into your car and provide you with diagnostics about your vehicle and statistics on your driving that can be used by insurance companies for reduced rates & safe driving bonuses.

We also had a chance to attend the Innovation Track, which was a great opportunity to hear from industry leaders about what they are building with Samsung technology and where they see it heading.

Highlights from Day 2 – coming soon!

Android at Samsung SDC

We’re in San Francisco for Samsung’s Developer Conference which coincides with the announcement of RevTwo support of Android. Samsung draws around 5000 developers so this conference is pretty big! Android has the largest market share and with it RevTwo now supports the major platforms that cover 97% of the market.

A lot of developers target Android, with over 1.5M apps, and many need to be cross platform to cover the most users. These are often the most valuable apps that can afford to be on multiple platforms. We believe the most valuable apps have the greatest need for personalized customer support.

Samsung has an important offering with Knox which allows management of policies on phones and tablets, as well as access to some capabilities that are normally not available to an app. It’s all secure and only available to apps with proper credentials, etc. This allows MDM suites to manage a fleet of phones for example, but also makes it possible to create fixed purpose devices such as tablets used by the public in retail, or hotel rooms that have restricted access to apps and settings.

In a book The Invisible Computer the author, Donald Norman, predicted that computers would eventually become “invisible” because they are just part of things we use. Mobile phones are for people, but tablets are self-contained computers with an interactive UI that anyone can use. They are becoming the way you pay in retail, and how you interact with equipment. The form factor makes them finally usable in so many applications. Phones will someday reach saturation and the business will be replacement business. But tablets have the potential to grow into the interactive signs, displays, and controllers of lots of things around us.

And we believe there will always be a need to answer questions and help users with the operation of these invisible computers. That’s our mission.

Debugging with RevTwo

Bugs—developers hate them. Hands down, bugs are the most frustrating and challenging part of the app development process. Crashes suck, but at least then you can see where everything blew up and where the problem happened.

The worst kind of bug is the silent kind or the hairy tarantula as I like to call it–the one that sneaks in as a missing file or unsynced data and usually appears randomly.  Everything works perfectly on your phone and desktop, but as soon as the app is in TestFlight, all hell breaks loose. Our co-workers and customers try to describe what they were seeing on the app.  Frantically, we try to repeat the issue, plug into Xcode and reproduce the error—which usually doesn’t work.

I’ve been there—we had a complicated app that worked in online and offline modes with a rather smart syncing mechanism in between. It all worked great on my high speed Wi-Fi and with our small team during testing, but once it was in the hands of the customer, a bug surfaced.

Fixing it was a nightmare. Users couldn’t clearly explain to me what actually occurred or what caused it (there was no crash, just missing files and data that thought it was synced when it wasn’t), so replicating the problem was difficult. Not being able to see the issue made it seem impossible to make the pesky bug go away—I burned weeks trying to understand what exactly was happening behind the scenes.

A faster, better debugging experience

There is a better way and a faster path to building a successful app. RevTwo makes the debugging process less painful by giving developers tools to create a bug-free app and connecting them with users who need in-app support.

How do the tools work? Three simple steps (really!):

    1. The RevTwo API is integrated into an app in 10 minutes or less, giving users the ability to submit help request tickets describing the issue they’re experiencing.
    2. A developer can initiate a help session to view the user’s screen while talking to them on the phone to see how they’re using the app, and what buttons they’re pressing. Additionally, with RevTwo, developers get insight beyond the surface level, and can get details like:
      • Device system information to see if the user is running the latest iOS and what app version they’re using
      • Network settings such as Wi-Fi and Bluetooth
      • Memory usage and disk space to help pinpoint issues like memory leaks
      • App log details, in real time, just like in Xcode, as well as access to the log history without needing to reproduce the error
      • App file system access (not the whole phone, just the app—we are still bound by the sandbox) to view all the files the app has stored in its documents directory and any subdirectories within (with the capability to download and upload files directly to the app)
      • App SQLite database view, which allows developers to view the table structure and data that the app is operating on
    3. The result is a developer finding and fixing a bug in record time, resulting in a happier user.In the end, I did track down that big, hairy tarantula of a bug, but it cost us time, money and a fair bit of sanity. Debugging can be like wandering a maze in the dark; tools like those from RevTwo let you turn on the lights.

Try it out for yourself; sign up for our free troubleshooting tools to help during TestFlight

Yes, Support Matters!

Over my career, I have seen the impact that well supported products have on their users and the companies that sell them.   Great support creates evangelists, turns newbies into power users, and prevents Twitter-storm disasters!

The harsh reality is that products will have problems, users will get frustrated, and products that don’t have a well-thought out and executed support strategy will fail – sometimes spectacularly.

The last startup I founded, Axeda, created the word’s first IoT (Internet of Things) platform.   At Axeda we connected some of the world’s biggest and baddest machines to the companies that made them.  These companies had a hodgepodge of tools to support their products, and it always seemed like we should “do more” to make this experience better.

We founded RevTwo to “do more”.   The mission of RevTwo is to deliver the next generation of support platforms for next generation products.    Apps, Dockers, and IoT, whose underlying technology is built upon sandboxed operating environments or highly embedded operating systems, have rendered an entire industry of PC based support tools useless.

There are over 3,000,000 Apps on the Google Play and Apple App Stores, over 300,000 dockerized apps in Docker Hub, and over 300 IoT platforms connecting billions of products to thousands of companies.   We want to make supporting those products


As part of this mission, we want to encourage you to communicate with us on any topic of interest regarding support.    To that end, we are have launched a company Slack channel #RevTwoDeveloper.   The #RevTwoDeveloper Slack channel will be the place for discussions about:

  • the RevTwo software stack
  • upcoming features
  • support questions
  • use cases
  • best practices
  • and more

The initial launch of the RevTwo In-App Support Platform for iOS has been a huge success.   We are overwhelmed by the interest and the prospective customers who view support for their Apps as a strategic differentiator.  We have heard your requests and are busy expanding our platform to meet even more use cases and will write a post on that shortly.

Stay tuned and join the conversation!