Rust to Python and Fibonacci Numbers

While Python continues to be the language of choice for ML projects, I'm increasingly seeing mentions of Rust in packages I use. Hugging Face fast tokenizers are written in Rust to speed up training and tokenization. They say it "Takes less than 20 seconds to tokenize a GB of text on a server's CPU." I recently used Polars, which is also written in Rust, for a quick task where I wanted to run a SQL query on 20 GB of CSV files. It only took about 60 seconds on my laptop. It's a way of bypassing Python's GIL to write code that is parallelizable. This is in contrast to packages like numpy that traditionally have been written in C/C++ for performance reasons. 

How does one turn Rust code into something you can call from Python? It turns out I've already done something similar, compiling Rust into WebAssembly and using it in a JavaScript project. For Python, the process is similar. I followed Calling Rust from Python using PyO3 and re-created the Fibonacci numbers experiment (apparently I'm not the only one with this idea) from my previous post. It's a toy example, just intended to show the ease with which Rust can be leveraged. It ignores more complex data types and any actual multi-threaded code. Perhaps that will be the topic of a future post.

The function in Rust looks like this:

And the Python code that calls it and times it:

Using the same n=35 as in my JavaScript experiment, I'm seeing about .05 seconds for Rust vs. 7.31 seconds for pure Python. The likelihood of wanting/needing this in a Python project seems greater than with JavaScript. My guess is that the trend continues.

Queueing Simulations

Waiting in lines is something we've all experienced. But how long will the wait for a table really be? How many checkouts should the grocery store have open? Queueing theory gives us the tools to be able to answer questions like these and more. It's also relevant to understanding software under heavy load. I've seen it come up recently in The Amazon Builder's Library Avoiding Insurmountable Queue Backlogs and Site Reliability Engineering's Addressing Cascading Failures chapter. 

Since you are reading this blog it is assumed that you are already familiar with queues and the idea of using them in software systems. To summarize, queues can improve availability and throughput at the expense of latency and resource consumption. Here we will see what the field of queueing theory can teach us about how and when to best use them. 

The Builder's Library article warns us of the bi-modal behavior of queue-based systems: the behavior can be very different based on whether there is a backlog or not. Recovery time after an outage can be dramatically increased and time can be wasted doing work that is no longer useful. We then get the first hints of how to manage this. Using more than one queue helps shape traffic. LIFO-ish behavior can be more desirable than FIFO. The SRE book discusses throttling via small queue sizes so requests are rejected when the incoming request rate is too high to be sustained. Traffic patterns, queue sizes, and the number of threads removing work from the queue are all intertwined. Can we better quantify these recommendations though?

I first learned of queueing theory in The Essential Guide to Queueing Theory e-book from VividCortex and wanted to revisit it in the context of the SRE reading. Right from the beginning we are warned that while queues are linear data structures, queueing systems behave non-linearly. Queueing theory is probabilistic. We won't understand the behavior exactly, but will know about wait times on average or a distribution of wait times.

The first thing they suggest we must understand is that queueing happens even when there is enough capacity to do the work. This is because work arrives to the queue in irregular sizes and at irregular intervals. Queueing gets worse at high utilization, when there is high variability, and when fewer workers are servicing the queues.

Any system can be decomposed into networks of queues and workers. These are the parameters and metrics commonly used to understand these systems:

• arrival rate to the queue (λ or A) - in a stable system this is the same as throughput
• average wait time or residence time in the queue (W or W_q)
• average post-queue service time (S)
• service rate (μ) - inverse of service time
• latency or residence time (R) - sum of wait time and service time
• worker utilization (ρ or U) - 0 to 1
• average number of requests waiting or in service concurrently (L or L_q)
• number of workers (M)

Little's Law states that concurrency is arrival rate times residence time: L = λR and L_{q =} λW_q

The Utilization Law states that utilization is throughput times service time: ρ = λS or ρ = λ/μ or with M workers ρ = λS/M

Kendall's Notation is a shorthand for describing queue systems. Normally we'll be working with M/M/m systems. The first M says events arrive randomly and independently (memoryless) meaning they are generated by a Poisson process. The second M says the service times are exponentially distributed. The last m is the number of workers or M from above. These are safe assumptions unless you know otherwise.

For a M/M/1 system R = S / (1 - ρ) which is a fast-growing curve (the non-linear mentioned above). Residence time is inversely proportional to idle capacity. By rearranging the equations above we can solve for other parameters as well. For systems beyond M/M/1 it gets more complicated. For M/M/m, for example, we need to bring in Erlang's C-formula.

To do that, and to see these laws in action, it's time to write some code. There appear to be several Python packages for simulating queueing systems. I will use Ciw for this simulation. We'll assume we have 1,000 requests per minute, several servers pulling work from a queue, and need enough capacity such that 2 servers could be offline without the system falling behind.

servers	utilization %	latency (sec)
3	83	.35
4	63	.18
5	50	.15

We can use this simulation to figure out what "levers to pull" to get the behavior we want. It could even become a Linear Programming optimization problem. Based on the arrival rate and service rate it's easy to see we need at least 3 servers, but not as easy to see how the latency would change going from 3 to 5 or what the minimum queue size is.

While not always intuitive, especially with multiple queues, there are ways to understand how queue-based systems will behave. If you are interested in the topic, I suggest reading the linked resources as they go into much more detail. If you have other examples of queueing theory in action, please share in the comments.

Java Updates Versions 9-19

The past few years I haven't written much Java code and when I did, it was Java 8. Many projects, it seems, have stuck with Java 8 which was released back in 2014. Per the roadmap, Java 8 is designated as LTS, but so are Java 11 and Java 17. In fact, Java 19 is available as of last month and many interesting features have been introduced in the past 8 years. This post is an overview of what's changed. The highlights, in my opinion, so we're up-to-date. I think it's enough to be interesting but not so much that it can't be picked up quickly if you have experience with older Java versions.

First, some name conventions. Java EE is now Jakarta EE. Definitely don't call it J2EE anymore. And since Java 11 Oracle JDK and OpenJDK are basically the same.

Tooling Updates

I won't go into too much detail here. If we are interested in using any of these they are explained and documented well elsewhere. For awareness:

• Java got a REPL called JShell for learning and prototyping interactively (9)
• Java Module System as a new package abstraction (9) - my impression is that this more for the JDK itself and some libraries while OSGi continues to be applicable for regular, modular apps
• Single-file programs can be executed directly and don't need the intermediate javac step (11)
• Multiple new garbage collection options for improved and more consistent performance (11)
• ZGC for low latency
• Epsilon GC no-op
• Linux container awareness for GC, thread pool sizes, etc. (11)
• Application class data sharing (CDS) for improved startup times and lower memory footprints (12) - this also seems to have implications for apps running in containers

API Updates

We want to start incorporating these into our code where applicable, so I created examples to help get used to some of these updates.

Private interface methods (9). Helps to encapsulate code in default methods and create more reusable code.

Variables can have implicit types, including in lambdas, to reduce the verbosity of code (10 and 11).

Switch expressions to simplify code and prepare for pattern matching in the future (12).

Text blocks as way to simplify code with multi-line strings (13).

Simple pattern matching (14). I was introduced to pattern matching when programming in Scala and this seems to continue a trend of Scala features making their way, in some form, to Java. It looks like more is coming in terms of pattern matching options.

Record keyword for immutable data classes (14). Getters, a public constructor, plus equals, hashCode, and toString methods are generated automatically. Lombok is still more flexible, but this is nice for simple cases.

Sealed classes for fine-grained inheritance control (15). Super-classes that are widely accessible but not widely extensible.

Paradigm Updates

For lack of a better name I'll call these paradigm updates as they relate more to programming models. 

Flow API as an implementation of the Reactive Streams Specification (9). This seems to be a way to get the specification interfaces into the JDK but not necessarily replace libraries with better implementations.

Vector API introduces vectorization to Java (16). It looks like it doesn't happen automatically and requires special code, so to be more useful I think it needs to be included in a common library like NumPy in Python.

Virtual threads and structured concurrency (19). The one-to-one kernel to user thread mapping is broken enabling easier asynchronous programming. Read/watch Project Loom: Revolution in Java Concurrency or Obscure Implementation Detail? The tl;dr is we'll still need a higher level of abstraction like reactive programming unless you want to relearn all the low-level concurrency structures.