KubeCon + CloudNativeCon Europe 2018

Big data applications have been an almost exclusive domain of Java and Scala developers. This not only frustrates engineers who prefer other languages and their ecosystems, but also impedes companies that already have their business logic written on other platforms from achieving the benefits of reuse when they build data-intensive applications.

In this talk we introduce Apache Beam. A unified programming model designed to provide efficient and portable data processing pipelines. We will discuss in detail how Beam achieves portability by relying in two concepts: (1) Runners that translate the Beam's model so it can be executed in existing systems like Apache Spark and Apache Flink and (2) the portability APIs, an architecture of gRPC services that coordinate the execution of pipelines in containers to accomplish language portability.

Load-balancing is a must for everyone building scalable services. This talk will give an overview of gRPC's load balancing options and show how to use them in a containerized environment.

gRPC is gaining traction due to its cross-platform, schema-driven interface design, efficiency, and use of HTTP/2. However, transitioning from REST to gRPC has some pain points due to missing alternatives for commonly-used tools. Sending free-form JSON over cURL does not work. There is no integration with Swagger/OpenAPI. Polished alternatives to GUIs like Postman or Swagger UI don't exist. Providing a REST interface over the gRPC services can solve these problems. We present a brokerless solution, using a Protobuf compiler plugin to generate a library of RESTful controllers with Swagger/OpenAPI documentation. This allows each microservice to expose both a REST and gRPC interface with no extra effort.

Building IoT applications that scales with a variety of devices and capabilities, such as network bandwidth, processor, and power consumption, can be a challenge. Furthermore, this can be complicated with a heterogenous development environment with a multitude of languages, communication frameworks, and protocols.

This talk discusses how Protocol Buffers, a language and platform neutral serialization mechanism, can be used as an efficient protocol for small embedded devices with constrained bandwidth and power. We will demonstrate how gRPC, along with Protocol Buffers, provides a performant communication framework that can be used across diverse languages and device platforms to create realtime IoT applications.