Intro

The Internet of Things has started to take off in recent years with the adoption of cloud and edge technologies and is poised for continued market growth of ~22% per year in the next 5 years. Use cases such as predictive monitoring and digital twin are among the top priority goals for enterprise organizations while smaller organizations are catching up.

There are a myriad of off-the-shelf IoT solutions which are great options if you want to get started quickly. But with the complexity of each organization’s specific environments and requirements, it could pay off to build an IoT platform yourself. Over the past years, cloud platforms such as Microsoft Azure IoT, Amazon AWS IoT, and Google Cloud IoT, have made it increasingly easier for everyone to start building themselves. Connecting devices and sensors, device-to-cloud communication (and vice versa), processing data, storing data, visualizing data, and applying advanced analytics and machine learning - all of it can now be done with cloud services.

Many organizations are now leveraging these cloud platforms to achieve faster time-to-market, scalability, and continuous integration and development. A part of this development involves continuously choosing the right technology stack. And while cloud platforms offer many choices, they may not always be the most suitable, effective, or efficient. In that light, IoT data often comes with a number of scalability challenges:

  • High velocity (often with more than thousands of events per second)
  • High volume (often accumulating tens of GBs per day)
  • High cardinality (often dealing with tens of thousands of devices or sensors)
  • Long retention time requirements (for long-term analyses and building machine learning models)

That is where scalable NoSQL database technologies like Elasticsearch come in, which are known for speed and scale. In itself, Elasticsearch works very well with strong query and aggregation APIs to use in existing applications. And paired with Kibana or Grafana, the Elastic Stack is a comprehensive open stack to:

  • Process and store petabytes of complex IoT data
  • Create (near) real-time dashboards
  • Search for the right information
  • Apply anomaly detection or create your own machine learning models
  • Set up alerts to external systems
  • Collaborate with a case management system

One of the added benefits of Elasticsearch is that it is a search engine, making it very easy and flexible to search for information. Other added benefits include the cross-platform use cases of Elastic like Observability and Security. So while you are building an IoT platform, why not monitor and secure your IoT environment with the same platform?

Many organizations have already built their IoT solutions and applications upon Elasticsearch and Kibana. I’ve compiled a list of use cases for inspiration:

To show how easy it can be to get started with your IoT journey, I will be publishing a series of posts about the use of Elasticsearch together with the three IoT platforms of Azure, AWS, and GCP.