My IoT Product Journey
UPDATE 01/24/2018: It was pointed out to me in the comments by gv-dev that some of the missing features I mention in the article are due to scaling issues and that each vendor may have alternative ways of accomplishing the same behavior in a scaled way such as using things like Thing Shadows in the AWS case.
As I continue working on the texting dryer project I've started researching how to use MQTT to connect my local infrastructure to the cloud. I've chosen the MQTT protocol for transferring messages to the cloud to be consumed and published to other endpoints. In doing a bunch of research on MQTT I've learned some surprising things. First and foremost if you don't know what MQTT is or only know a little bit about it I highly recommend the blog series from HiveMQ for getting up to speed. It is packed with basically everything you need to know about MQTT.
Once I felt like I understood everything I moved on to finding a cloud MQTT broker I could use. I would use HiveMQ but it seems to be a little too enterprise-y for me. I'm just messing around with hobby projects. So naturally I turned to Amazon.
Amazon AWS IoT
They do, indeed, have an IoT solution including MQTT broker but as I started reading the docs I found that it lacks basic functionality that MQTT provides including some of the more powerful features like QoS 2, Retained Messages and Persistent Sessions. I even found this nugget
"On rare occasions, the message broker might resend the same logical PUBLISH message with a different packet ID."
WTF!?! That sounds like a "We don't really know why but this crazy thing could happen sometimes". Really? Based on the limitations listed I don't see how Amazon's IoT offering could support smaller connected devices reliably. Sure, it will work for basic scenarios but it's missing essential features. [UPDATE: I made the previous statement from a pure MQTT implementation perspective as it is missing core features. There are ways, however, to accomplish the same behavior in AWS that I was unaware of at the time I wrote this. This does require learning Amazon's custom implementation for connected devices but it's unfair to say their IoT offering can't support smaller connected devices.] Here is the complete list of MQTT spec limitations.
In AWS IoT, subscribing to a topic with Quality of Service (QoS) 0 means a message will be delivered zero or more times. A message might be delivered more than once. Messages delivered more than once might be sent with a different packet ID. In these cases, the DUP flag is not set.
AWS IoT does not support publishing and subscribing with QoS 2. The AWS IoT message broker does not send a PUBACK or SUBACK when QoS 2 is requested.
The QoS levels for publishing and subscribing to a topic have no relation to each other. One client can subscribe to a topic using QoS 1 while another client can publish to the same topic using QoS 0.
When responding to a connection request, the message broker sends a CONNACK message. This message contains a flag to indicate if the connection is resuming a previous session. The value of this flag might be incorrect if two MQTT clients connect with the same client ID simultaneously.
When a client subscribes to a topic, there might be a delay between the time the message broker sends a SUBACK and the time the client starts receiving new matching messages.
The MQTT specification provides a provision for the publisher to request that the broker retain the last message sent to a topic and send it to all future topic subscribers. AWS IoT does not support retained messages. If a request is made to retain messages, the connection is disconnected.
The message broker uses the client ID to identify each client. The client ID is passed in from the client to the message broker as part of the MQTT payload. Two clients with the same client ID are not allowed to be connected concurrently to the message broker. When a client connects to the message broker using a client ID that another client is using, a CONNACK message will be sent to both clients and the currently connected client will be disconnected.
The message broker does not support persistent sessions (connections made with the cleanSession flag set to false. The AWS IoT message broker assumes all sessions are clean sessions and messages are not stored across sessions. If an MQTT client attempts to connect to the AWS IoT message broker with the cleanSession set to false, the client will be disconnected.
On rare occasions, the message broker might resend the same logical PUBLISH message with a different packet ID.
The message broker does not guarantee the order in which messages and ACK are received.
So...let's give Microsoft a try.
Micrsoft IoT Hub
Turns out Azure IoT also has an MQTT broker that you can use with their cloud services. Buuuuut, they apparently just followed Amazon's lead. As far as I can tell they limit the topics that you can pub/sub to into a specific box of "/devices/{device_id}/messages/events/{property_bag}". That wouldn't be a deal breaker but Microsoft also doesn't support QoS 2 messages or Retained Messages but does seem to support Persistent Sessions which makes it more friendly to less capable devices and devices with very spotty connections. The list of MQTT broker "don't haves" for Microsoft include:
- IoT Hub does not support QoS 2 messages. If a device app publishes a message with QoS 2, IoT Hub closes the network connection.
- IoT Hub does not persist Retain messages. If a device sends a message with the RETAIN flag set to 1, IoT Hub adds the x-opt-retain application property to the message. In this case, instead of persisting the retain message, IoT Hub passes it to the backend app.
- IoT Hub only supports one active MQTT connection per device. Any new MQTT connection on behalf of the same device ID causes IoT Hub to drop the existing connection.
Man, this is crazy. Doesn't anybody do this right? Google?
Google Cloud IoT
Google Cloud IoT is currently in private beta and based on the form they want you to fill out to be in the beta I don't think they are looking for small time hobbyists like myself to play with it.
Summary
Honestly I was quite disappointed to see that the big players don't fully support what I consider the most powerful features of MQTT. Without things like Retained Messages, Persistent Sessions, and QoS 2 you might as well just use regular old TCP connections or go through the curve of learning the vendor-specific implementation for scaling devices. I'm continuing to research what options are available. There are a lot of open source MQTT v3.1.1 brokers out there and I could just host one in any of the compute cloud infrastructures from Google, Microsoft, or Amazon but I was really hoping for something out of the box. I've also found several MQTT as a Service offerings that I'm going to explore as well. Things like ThingStudio, IBM Bluemix, DIoTY, and CloudMQTT. Look for a future blogpost on those.
