Twelve-Factor ASP.NET Core Apps

I had never done any .NET development until this past year when I started to work with ASP.NET Core. Microsoft says it's cloud-ready and I agree, but I haven't seen anyone describe what a twelve-factor ASP.NET Core app looks like yet. In the Cloud Foundry (CF) and Heroku communities twelve-factor apps are a popular topic. This is my take on it for ASP.NET Core.

The twelve factors are best practices for anyone who works on a software-as-a-service app. They help your app be portable, continuously deployed, horizontally scaled, and easier to develop. Basically cloud-native. I'm most familiar with the CF platform-as-a-service, so assume that's the execution environment of the app in my examples.

• Codebase: The app is tracked in version control.
• Store each app in its own git repository
• Factor shared code into libraries that become dependencies
• Dependencies: The app declares all of it's dependencies and never relies on the implicit existence of system-wide libraries or tools.
• Use NuGet package manager for distribution
• Declare dependencies in project.json
• Dependency isolation is provided by the automatic creation of project.lock.json
• Config: Config for anything that varies between deploys as should be stored as environment variables.
• Cloud Foundry provides service credentials in the VCAP_SERVICES environment variable
• Use environment variables for other config like whether the environment is dev or prod, or logging levels
• Backing services: There is no distinction between local and remote services, they are all just resources.
• Based on the config, add services to the DI container in the ConfigureServices method of Startup.cs
• Build, release, run: A build transports the code repository to an executable bundle, release takes the build output and combines it with the config, and run starts the app.
• I think the build could be a dnu publish or the buildpack compile phase
• I think the release could be pushing the app to CF or the buildpack release phase
• The run (regardless of how you prefer to think of build and release) is CF starting the app and creating a route
• Provide a rollback mechanism by using blue-green deploys and canary testing (there are CF CLI plugins for this)
• Processes: The app is stateless and doesn't rely on sticky sessions. 
• Use a backing service like Redis for HTTP sessions
• Don't write to the local filesystem because it's ephemeral and not shared between the multiple instances of the app
• Port binding: The app is self-contained. It listens on a port specified by the execution environment.
• Declare a dependency on Kestrel
• Kestrel can be configured with the --server.urls command-line option to run on a specific host and port
• Concurrency: The app scales horizontally because nothing is shared. 
• Manual CF CLI scale and auto-scaling based on different metrics
• CF restarts app on crashes and provides an API to start and stop the app
• Run at least 3 app instances to keep the app running during data center and CF updates
• Disposability: The app starts quickly, shuts down gracefully, and handles unexpected non-graceful terminations.
• When an application is deployed the dependencies including the runtime are cached and re-used to create additional instances of the application
• Implement a shutdown handler that can run when CF stops your app
• Use message queues to communicate between applications and keep track of work for non-graceful terminations
• Dev/prod parity: The development, staging, and production environments are as similar as possible.
• Can do development on Linux to match OS in CF
• Can run ASP.NET Core apps in Docker containers even using the cflinuxfs2 image to get really close to CF, for example when running tests
• Can use Kestrel web server in integration tests and production
• Logs: Logs are a stream of events and can be consumed by a data warehouse, log indexing, or log analysis service.
• CF apps can log to STDOUT/STDERR and the log messages are buffered in memory where they can be tailed or dumped
• Log messages can be persisted by providing a syslog URL for a log service and they will be drained there
• Admin processes: One-off admin processes are executed in the same environment and with the same configuration as the app.
• Control the app's start command in a manifest.yml file that is version controlled with the app like dnx ef database update && dnx web ...
• Employ worker apps to run your admin processes and don't assign a route to them

When you dig into these you realize that many of them can actually be handled by a PaaS like Cloud Foundry or by an application's runtime like ASP.NET Core. That's good because then the app can focus on the problem it's trying to solve. The app developers just need to be aware of and know how to leverage what's provided. These folks should study the twelve-factor site as it has much more detail and examples than included here.

Check out the ASP.NET Core Cloud Foundry buildpack mentioned above (full disclosure: I am a contributor to this).

Something else to keep an eye on is Steel Toe OSS. The twelve factors don't cover some microservice patterns like service discovery, or latency and fault tolerance patterns like circuit breakers and bulkheads. The .NET world seems to currently be far behind Java in this area, so it's good to see something in the works.

Reactive Actors

I've been meaning to revisit reactive programming and the actor model for a while now. I first learned about them in the Principles of Reactive Programming Coursera class and then actors came up again in the Seven Concurrency Models in Seven Weeks book. The Scala I picked up is quickly being forgotten and I haven't done a post with code in a while, so here I'll get back into that and create a simple application using Akka and RxScala.

Actor model

The developerWorks article JVM Concurrency: Acting asynchronously with Akka gives a good introduction to the actor model:

The actor model for concurrent computations builds up systems based on primitives called actors. Actors take actions in response to inputs called messages. Actions can include changing the actor's own internal state as well as sending off other messages and even creating other actors. All messages are delivered asynchronously, thereby decoupling message senders from receivers. Because of this decoupling, actor systems are inherently concurrent: Any actors that have input messages available can be executed in parallel, without restriction.

Then JVM Concurrency: Building actor applications with Akka goes on to explain the advantages of this approach:

If you compose your actors and messages correctly, you end up with a system in which most things happen asynchronously. Asynchronous operation is harder to understand than a linear approach, but it pays off in scalability. Highly asynchronous programs are better able to use increased system resources (for example, memory and processors) either to accomplish a particular task more quickly or to handle more instances of the task in parallel. With Akka, you can even extend this scalability across multiple systems, by using remoting to work with distributed actors.

At first the actor model may sound the same as what I described in my Communicating Sequential Processes post because both involve message passing, but the two concurrency models have several differences:

• Actors have identities while CSPs are anonymous
• Actors transmit messages to named actors while CSPs transmit messages using channels
• Actors transmit messages asynchronously while CSPs can't transmit a message until the sender is ready to receive it

My impression, and I could be wrong, is that actors more naturally extend beyond a single machine to a distributed system since the sending and receiving of messages is decoupled. A quick search does turn up distributed channels in pycsp, though, so it seems that both can be distributed.

Reactive applications

The Reactive Manifesto details four qualities of reactive applications:

• responsive - the system responds in a timely manner if at all possible
• resilient - the system stays responsive in the face of failure
• elastic - the system stays responsive under varying workloads
• message driven - the system relies on asynchronous message passing between components

Where Akka describes itself as a toolkit and runtime, RxScala only claims to be a library for composing asynchronous and event-based programs using observable sequences. To me it's not clear how it helps us achieve all four qualities (or if it even intends to).  Nevertheless, the ReactiveX introduction explains their advantages:

The ReactiveX Observable model allows you to treat streams of asynchronous events with the same sort of simple, composable operations that you use for collections of data items like arrays. It frees you from tangled webs of callbacks, and thereby makes your code more readable and less prone to bugs.

This means is that the methods returning Observables can be implemented using thread pools, non-blocking I/O, actors, or anything else. This is how ReactiveX and Akka will be used together: Actors are the concurrency implementation for services communicating with asynchronous messages.

Combining Akka and RxScala

I came up with the following short example. First I wrote a couple of methods returning an Observable to get a feel for it, then added the stockQuote() method which also uses an actor in it's implementation:


Running it produces the expected output, something like:

6

8

broken service

GOOG: 253.22

I can really see the potential in the Observable model, especially after reading more about it at The Netflix Tech Blog. If you were already using actors maybe combining them like this could make sense. I also need to checkout Akka Streams which seems like a similar idea.

UPDATE

Ray is a framework for parallelizing ML workloads. They use the actor model as a way of coordinating work and maintaining state.

Bitwise operations 101

Whenever I see lists of interview questions it seems like bitwise operations are on there. I've never used these at work and I don't even remember needing to do it in school. I wanted to see if I could answer a few of these. After a quick review of what each operation does this is what I came up with.

The first three methods--nthBitSet, countBits, and isPalindrome--are fairly intuitive and I feel like they could be reasonable interview questions.

For adding and multiplying I cheated and looked up the algorithms. They sort of make sense, like the repeated additions for multiplication, but it would have taken a long time to come up with that on my own.