Wednesday
Room 2
17:40 - 18:40
(UTC+02)
Talk (60 min)
1 Million Ways to Break Production: IoT Systems at Scale
When we work with Cloud systems, we are used to controlling the environment. We have fast and reliable networks, continuous deployment, and end-to-end observability. We can trace, monitor and debug services in real time, and roll back any broken deployments. IoT systems at scale are different. The production environment is constrained by real world physics, where devices are deployed into parking cellars with poor cellular connectivity, or outdoor garages with flaky Wi-Fi. At this scale, hardware issues show up, cables degrade and human errors happens. When we have 1 million connected devices, different classes of problems show up. Even if one device operates correctly on an individual level does not imply that the system operates correctly for 1M devices. Synchronized behavior, like devices checking for updates at midnight, quickly turns into a thundering herd that violates your service quotas and connection pools. We will talk about how our rapid growth outgrew the initial system design and caused production to go down for several days. Retry logic and home automation systems became self-inflicted DDoS attacks. Non-essential services became global bottlenecks. This talk is an experience report about the incidents and learnings from running Easee’s IoT Platform through a period where Easee scaled from 0 to 1 million chargers, in a journey of growth, near-collapse, and the way back.
Jan-Erik Carlsen
Jan-Erik is an engineering leader and software engineer with 15+ years of experience building distributed systems and large-scale platforms.
As Head of Software at Easee, he leads the teams building the software, integration platforms, and web and mobile applications that connect 1M+ EV chargers to users, partners and energy services.
He previously worked as a Principal Engineer at GoDaddy, helping lead large-scale cloud migrations and globally distributed systems.
Jan-Erik cares deeply about engineering culture and building high-performing teams that can ship fast and operate reliable systems at scale.
Eirik Ottesen
Eirik Ottesen is Chief Technology Officer at Easee, where he leads the engineering organization behind one of Europe's largest IoT deployments of 1 million connected smart EV chargers.
Joining in 2019 as one of the first software engineers,, Eirik has architected and scaled the software from its earliest days and is now taking ownership of the full technology stack, from hardware, electronics and embedded software, to apps, apis and platform infrastructure.
He has experienced firsthand the challenges of scaling IoT systems through production incidents, platform migrations and the technical constraints of managing distributed hardware at scale.
Before Easee, he worked as a software engineer at an integrated circuits company in the UK and as a consultant in Norway. He now leads the engineering organisation bringing Easee's next generation of charging technology to market.