dalamb – An AWS Lambda clone (sort of)

I was going through the AWS Lambda main presentation page yesterday and I started to wonder if it would be possible to create your own stack with the same featureset.

First off, AWS Lambda is a stunning programming model. A cross between stateless distributed functional programming at scale (say that fast ten times if you dare) coupled with zero-administration requirements. In other words, you write a function and AWS Lambda runs it at the appropriate scale for you.

What’s more is that, AWS Lambda can react to events following the event sourcing pattern. So you don’t keep any internal state, AWS Lambda executes your function with the event data.

Our job, as software developer, has just become almost too simple. The good news is that we can now focus on the value we are aiming for rather than the plumbing and machinery underneath. A whole new world.

With that said, AWS Lambda is brilliant but, well, is deeply tied to the AWS services and infrastructure. That’s sort of the point obviously, however, could we create a similar stack and run it ourselves? Because, why not!

Let’s therefore introduce “dalamb – An AWS Lambda clone (sort of)“.

dalamb properties and features

What would be dalamb properties? How could we design such a beast? Interestingly, we have most of the pieces at hand, at least from a technological and architecture perspective.

Let’s rewind a bit, AWS Lambda gives you the main following features:

  • extend AWS services with custom logic
  • perform funky operations triggered by AWS events
  • expose your function as a web service (insert mention to RESTful URL here)

All of this is backed by the powerful AWS infrastructure ensuring:

  • fault-tolerance
  • automatic scalability
  • a rich security model
  • pricing with precision

These features and properties tell us the stories dalamb should pursue:

  • The event-sourcing story: how to map function to events
    • The REST story: this one is just a specific case of the event sourcing story where the events are triggered by requests to a given endpoint
  • The delivery story: can we package up any function in any language?
  • The operational story: deploy, update, scale, keep available…

It’s all about events

The heart of the AWS Lambda platform is its event-sourcing feature. Events are everywhere, they actually define the system in itself.

As developers, what we build, when we forget about them, are fragile stateful monoliths that quickly fail under their own mass and incapacity to accept inevitable changes. Building discrete functions that react to events mean you reduce friction between changes and pressures on your system while, as a developer, keep a comprehensible view of the system.

Events have basic properties:

  • self-contained: the information enclosed within must not depend on moving external resources
  • immutable: events never change once they have been emitted
  • descriptive: events do not command, they describe facts
  • sequenced: event streams ordering must be supported
  • structured: so that rules and patterns can be matched against them

dalamb is not a re-write of the entire set of AWS services. It focuses on mapping discrete, simple, functions to events flowing through the system. For this reason, dalamb will be designed around event-sourcing patterns.

The goal will be to make it simple for external resources to send events and straightforward for functions to be matched to these events following simple rules.

dalamb will likely start using Kafka as a simple event-store backend.

Change is the nature of any live system

A live system is constantly changing due to internal and external stressors. Your functions are one of those stressors. In order for the system not to collapse onto itself, a certain level of automatic elasticity and draining are required.

Draining means that part of the system can be considered like wasteful and the system will find a way to clean them up. Elasticity means that the system must be able to absorb any load stress in a way that keeps the overall system functional.

AWS Lambda owns the responsibility of managing the system’s constant changing nature. So will dalamb. However, since dalamb will not own the infrastructure itself, it will be bound, initially at least, by the fixed size of the underlying infrastructure. However, within those boundaries, dalamb will ensure the system keeps its promises and stays live and well.

dalamb will likely take advantage of Mesos for resources sharing and Kubernetes or Marathon for manging the lifecycle of each function instances.

Geared towards developers

dalamb is meaningless if developers cannot express themselves though it. To achieve this, dalamb will not make any assumption regarding the actual code environment that is executed.

Indeed, in spite of being a powerful platform, AWS Lambda is currently limited by its selective targets. Oddly enough, if you are a Go or Ruby developer (for instance), you must call your code as a subprocess of the executed function.

dalamb takes the bet that the doors should be open to any developers with the same interface. To achieve this, dalamb will expect functions to be packaged up into container images, under the control of the developers. What dalamb should provide is the entry point definition for a container to be executed appropriately on the platform as a dalamb function.

Containers will be short-lived and their state will not be kept around for longer than the time required to run the function.

Developers will store their function on platforms like github or bitbucket and dalamb will simply react to events from those platforms to release to production new versions of a function.

What’s next?

dalamb is by no means properly defined or even designed. It is a story about developers achieving great values by relying on developer-oriented platforms. Whether it is Heroku or AWS Lambda, dealing with the complexity of this world has never been so accessible.