Advanced

Impact of etcd deployment on Kubernetes, Istio, and application performance

Larsson, Lars ; Tärneberg, William LU ; Klein, Cristian ; Elmroth, Erik and Kihl, Maria LU (2020) In Software - Practice and Experience
Abstract
This experience article describes lessons learned as we conducted experiments in a Kubernetes‐based environment, the most notable of which was that the performance of both the Kubernetes control plane and the deployed application depends strongly and in unexpected ways on the performance of the etcd database. The article contains (a) detailed descriptions of how networking with and without Istio works in Kubernetes, based on the Flannel Container Networking Interface (CNI) provider in VXLAN mode with IP Virtual Server (IPVS)‐backed Kubernetes Services, (b) a comprehensive discussion about how to conduct load and performance testing using a closed‐loop workload generator, and (c) an open source experiment framework useful for executing... (More)
This experience article describes lessons learned as we conducted experiments in a Kubernetes‐based environment, the most notable of which was that the performance of both the Kubernetes control plane and the deployed application depends strongly and in unexpected ways on the performance of the etcd database. The article contains (a) detailed descriptions of how networking with and without Istio works in Kubernetes, based on the Flannel Container Networking Interface (CNI) provider in VXLAN mode with IP Virtual Server (IPVS)‐backed Kubernetes Services, (b) a comprehensive discussion about how to conduct load and performance testing using a closed‐loop workload generator, and (c) an open source experiment framework useful for executing experiments in a shared cloud environment and exploring the resulting data. It also shows that statistical analysis may reveal the data resulting from such experiments to be misleading even when careful preparations are made, and that nondeterministic behavior stemming from etcd can affect both the platform as a whole and the deployed application. Finally, it is demonstrated that using high‐performance backing storage for etcd can reduce the occurrence of such nondeterministic behaviors by a statistically significant (P < .05) margin. The implication of this experience article is that systems researchers studying the performance of applications deployed on Kubernetes cannot simply consider their specific application to be under test. Instead, the particularities of the underlying Kubernetes and cloud platform must be taken into account, in particular because their performance can impact that of etcd. (Less)
Please use this url to cite or link to this publication:
author
; ; ; and
organization
publishing date
type
Contribution to journal
publication status
published
subject
in
Software - Practice and Experience
publisher
John Wiley and Sons
external identifiers
  • scopus:85089109897
ISSN
0038-0644
DOI
10.1002/spe.2885
project
Cyber Security for Next Generation Factory
language
English
LU publication?
yes
id
371464ed-516d-4d89-9eec-4f4efc352a78
date added to LUP
2020-06-11 13:08:26
date last changed
2021-01-06 05:05:49
@article{371464ed-516d-4d89-9eec-4f4efc352a78,
  abstract     = {This experience article describes lessons learned as we conducted experiments in a Kubernetes‐based environment, the most notable of which was that the performance of both the Kubernetes control plane and the deployed application depends strongly and in unexpected ways on the performance of the etcd database. The article contains (a) detailed descriptions of how networking with and without Istio works in Kubernetes, based on the Flannel Container Networking Interface (CNI) provider in VXLAN mode with IP Virtual Server (IPVS)‐backed Kubernetes Services, (b) a comprehensive discussion about how to conduct load and performance testing using a closed‐loop workload generator, and (c) an open source experiment framework useful for executing experiments in a shared cloud environment and exploring the resulting data. It also shows that statistical analysis may reveal the data resulting from such experiments to be misleading even when careful preparations are made, and that nondeterministic behavior stemming from etcd can affect both the platform as a whole and the deployed application. Finally, it is demonstrated that using high‐performance backing storage for etcd can reduce the occurrence of such nondeterministic behaviors by a statistically significant (P &lt; .05) margin. The implication of this experience article is that systems researchers studying the performance of applications deployed on Kubernetes cannot simply consider their specific application to be under test. Instead, the particularities of the underlying Kubernetes and cloud platform must be taken into account, in particular because their performance can impact that of etcd.},
  author       = {Larsson, Lars and Tärneberg, William and Klein, Cristian and Elmroth, Erik and Kihl, Maria},
  issn         = {0038-0644},
  language     = {eng},
  month        = {08},
  publisher    = {John Wiley and Sons},
  series       = {Software - Practice and Experience},
  title        = {Impact of etcd deployment on Kubernetes, Istio, and application performance},
  url          = {http://dx.doi.org/10.1002/spe.2885},
  doi          = {10.1002/spe.2885},
  year         = {2020},
}