Technical Considerations

Protoverse: Engineering

WORK IN PROCESS – The aim here is to think more about the practical aspects: modeling and technical requirements, and to identify helpful tools and skills. If you would like to ponder ideas, I’ll be glad for your input.

Prelude

The Protoverse is not made from things (like particles or objects and such), but processes. What we typically conceive of as entities (“things”) are actually temporary aggregations of Processes and their interactions – it’s Processes all the way down. The Protoverse is relational, co-evolving (darwin-style) and backdrop-independent; hence there’s no a-priori conception of time, space or gravity; although from Processes interacting on certain scales, constants could emerge, resembling a form of “physics”.

The prime abstraction is “Process” (with capital ’P’). Through the interaction, uniform primitive Processes build relations and compose, so that higher-order Processes emerge, exhibiting novel properties. The ultimate goal is pioneering open-ended evolution (in the darwinian sense). Processes constantly become their own “computing substrate”. That’s in principle quite like cellular automata work, where the grid is not a pre-defined “space”, but each cell’s state at a given step is determined by the states of its neighboring cells at the previous step, so that the structure of the grid is inherently linked to the operation of the automaton (this aspect may be overlooked when playing with cellular automata within apps, where cellular automata are always displayed in an area of a certain size).

Dynamic Structures

The Protoverse needs to run on digital computers, since we don’t have anything more sophisticated yet at scale. Digital computers operate inherently and efficiently with discrete structures – so it would be generally reasonable to stick to discrete structures.

Aside from that, I’m quite fond of continuous structures at the lowest level, because necessary successive discretization could elegantly introduce notions of approximation and uncertainty, as mentioned in the metaphysics. I’m thinking of a “level of detail” (LoD) approach, where the discretization happens according to a Process’ perspective or epistemological constraints; the discretization would be defined by relationships between Processes rather than by fixed spatial or temporal grids. Let’s keep that aside right now, and look at some (discrete) ideas first:

Cellular Automata

Cellular automata allow for efficient parallel computation in theory, because each cell’s state depends only on its local neighboorhood, and their states are updated simultaneously.

But cellular automata may not fit all too well to the Protoverse; at least not directly, because cellular automata are pretty low-level, and their expressiveness is inherently tied to discrete steps, a rigid a-priori concept of uniform time progression and locality.

Well, that could be migitated by the rules – it solely depends on how many layers of abstraction we are willing to pile up on another. Put differently, it would be necessary to design a higher-level DSL that compiles down to cellular automaton rulesets. But given that – what’s the point using a cellular automata in the first place?

Graphs / Hypergraphs

Hypergraphs offer higher flexibility to encode also non-local relations and interactions. Eventually, one would not just use “a hypergraph”, but utilize evolving substructures to encode evolutionary dynamics. Graphs lend themselves to model relations, recursive and fractal-like structures, which corresponds well to modeling Nested Scales of Complexity.

Higher-order Functions and Function Composition

The idea here is to put aside structures like cellular automata and hypergraphs, but use just functions for dynamically evolving structures. Eventually, we may end up with some kind of implicit graph anyway (or an unintelligible mess), because we’re just abstracting the notion of structure into the behavior of functions themselves.

Language Oriented Programming (LOP)

The idea here is to have a language representation that is subject to be utilized and evolved directly by the Protoverse Processes (“data is code, code is data”). Everything then would happen at the language level, abstracting from stuff like functions, data structures, graphs, etc. That would certainly add semantic denseness, and brings a clear distinction between high-level design and lower-level implementation. The feedback loop involving design and implementation would have to be very tight, otherwise issues could show up that lead to frequent pivots.

Adopt, Adapt or Design a Process Calculus

A newcomer on this list: As per my current understanding, designing the Process abstraction and dynamics would result in something like a process calculus anyway, just “ad hoc, informally specified, bug-ridden and half-assed”.

Process calculi are considered models of computation; they are formal systems to describe, analyze and express process dynamics, and it seems that they are used to model dynamics in biology.

The idea behind process calculi seems quite appealing, e.g. π-calculus: there are anonymous processes that communicate via named (or typed) channels. These channels can be passed around by processes through other channels, and this “mobility” allows dynamic reconfiguration of the process network. Processes can be composed into larger ones, which maps nicely to the idea of “higher-order” Protoverse Processes. Typed channels might represent various “dimensions” for newly evolved means of relations (and interactions) of higher-order Processes.

There’s a chance that I might conflate something and be mistaken with what I said above, since I’ve looked into process calculi only superficially as of yet. Before deep-diving, it seems rather sensible to advance the design of the Process abstraction by own reasoning first, and reconcile afterwards.

Bigraphs (Robin Milner)

Today I learned that the idea of using channels as “dimensions” is central to bigraphs, which are considered a “generalization of the π-Calculus” – I’m not quite sure why Milner’s bigraph is conisdered a generalization, to me it looks like a special application. Bigraphs consist of two distinct but connected graphs: one modeling the connectivity between processes (communication links) and the other modeling their placement within a spatial or hierarchical context (location).

My intuition suggests that this would be equivalent to using channels of two distinct types, one type representing the hierarchical context and the other representing connectivity. Such a typed approach could accommodate more than just two dimensions. However, I might be confusing things here; the key point seems rather that Robin Milner explicitly defined the formal specification and interactions for each of these two connected graphs, which provides a rigurous formalism.

Requirements

The following requirements are roughly sorted from most immediate neccesity to long-term concerns (optimizations):

Modus Operandi

What type of application will the Protoverse become? There are basically two approaches, and approach calls for a different development style, choice of language and platform, different analysis methods, maintenance- and scaling strategies, and a plethora of other requirements.

Batch-processing

An exponentially growing Protoverse could translate into a “big-bang” computation that runs for a few minutes or hours, with runtime- and subsequent analysis. The batch-processing approach could also be a “big bang/crunch” model: a snapshot of the state is going to be pruned, written to the database, and then fed into the next run. This approach is conceptually simple, contained and portable: we could optimize the entire run as a single, closed computation, potentially leveraging computing resources more efficiently. But it might be likely too simple to be viable: Even though self-contained, complexity is growing anyway. This approach doesn’t encourage resource management, but ignores it; The length of the development cycle (develop-compile-load-run-save-analyze) will become increasingly lengthy, slowing down the iteration process.

Long-running

A long-running app might be much more interesting: real-time interaction and observation becomes possible, and there’s inherent encouragement for fine-grained state- and resource-management from the beginning. The development cycle could be kept short. Computing resources must be continuously available. The infrastructure will introduce additional overhead, in terms of performance and development effort.

Conclusion

Given the nature of the Protoverse as a platform for exploring process-relational dynamics and evolution, the long-running, real-time system seems more aligned with practical goals. This approach facilitates a closer relationship between the developer and the Protoverse, embodying the co-evolutionary aspect of the conceptual framework.

However, early stages or specific experiments within the Protoverse could benefit from the simplicity and control offered by batch processing, especially for testing hypotheses or performing intensive computations that are self-contained.

From Metaphysics to Code

The primary langage used for building the model should enable to express the fundamental dynamics as closely and naturally as possible in order to minimize context-switching. So what are we looking for?

First of all, we might want the language that is concise and clean; resource allocation should be abstracted away, so manual memory management, but also no explicit management of concurrent and parallel execution, as far as possible.

It must lend itself to high-level abstractions, and various sorts of abstractions too; e.g. metalinguistic abstraction – because it seems quite unlikely to achieve a close fit of metaphysics and code with a general-purpose programming language alone, but rather to design and implement a DSL (that may resemble some form of process calculus, for instance).

Certain programming paradigms seem to fit, so it would be nice to have access to functional programming, logic programming, dataflow programming and language-oriented programming powered by metaprogramming (via macros, as they are useful for many things).

There will be no distinct formal specification of the Protoverse model (or put differently, the code will be the specification), hence an exploratory programming style seems like the sanest approach to model the Protoverse dynamics. That sets the focus on primarily interactive use of a language or computing environment.

Pivoting quickly into different directions must be feasible, so we also want an emphasis on refactoring capabilities.

Conversational Programming

The benefit hinges on the anticipated modus operandi of the Protoverse:

  • Batch-processing: This approach would make live-coding abilities not a requirement. We would be free to compile down to machine code and use a statically typed language, where the strict distinction between programming time, compile time and run time wouldn’t be a hindrance.
  • Long-running: The Protoverse as an application that reconfigures itself continuosly leans inherently towards being stateful. It calls for interaction and live-coding at the running system as the primary way of development and maintenance.

The help of a VM- or image-based runtime and language that allows to experiment with the system and to tweak it at runtime would greatly enhance understanding and development efficiency. Unfortunately, this rules out statically typed languages because of their sharp distinctions between programming time, compile time and run time.

Since I asssume the more interesting stuff is likely to happen the longer the Protoverse runs, the ability to step back and forth through the execution history might be desired, too (straightforward state-recreation, or actually circumventing the need for state recreation to begin with).

Massively Interacting Computation

A lot of stuff is going to happen in realtime. Eventually I’m expecting many billions of smallish, interacting computations. In the early stages, where the focus is on modelling the Process abstraction and dynamics, the capacity to handle a couple of millions might be enough, which should be achievable in the cloud, on a small Rasperry Pi cluster or even a bunch of laptops.

Before we come to concurrency and parallelism: the concepts are often mixed up, and understandably so – here’s an explanation and here. Parallelism is the thing that we really want.

The crux with current architectures is that parallelism is tied to the number of physical cores available. Regular CPUs have a dozen up to a couple of dozen cores, GPUs around 16,000 cores. The ideal thing to have would be something like “real-time-soft-scalable” parallelism, but in practical terms with current architecture we need both, the “normal” interleaving or time-slicing concurrency and true parallelism.

Concurrency

Since massive concurrency is a basic need (dealing with a lot of things), this will practically rule out concurrency solely based on “heavyweight” kernel threads.

There are basically two facilities in common use that fit massive concurrency: Either actors (the Erlang model) or CSP-style channels (the Golang model) – breakdown here. My first naive idea is that the primitive Protoverse Processes and user-level threads are mapped one-to-one, where the user-level threads are scheduled by a runtime and multiplexed to kernel threads.

Here is to note that Protoverse Processes compose into higher-order Processes, which means that more complex higher-order Protoverse Processes automatically get more “computational power” (however, the “power” won’t increase entirely linear, of course – there’ll be communication overhead, amongst other factors).

Actor Model vs. Process Calculi

Process calculi are interesting for the sake of pondering about Process dynamics, as they are languages for modeling concurrent computation.

There’s a range of process calculi: π-calculus, Join calculus, Ambient calculus (related: membranes) and several others. I think the π-calculus and Join Calculus are particularly interesting (maybe due to the fact that I didn’t look into others as much).

As far as I know, there are no “production-ready” performant languages that implement π-calculus, and π-calculus wouldn’t be feasible for distributed systems anyway. However, there are at least two implementations of Join-calculus: JoCaml (in OCaml) and Chymyst (in Scala, chemical machine) – both research projects and abanoned. More common are CSP implementations as mentioned before (Communicating Sequential Processes), e.g. concurrent ML, Clojure’s core.async or Go’s go-routines; but even though CSP is conceptually related, its capabilities are quite different as CSP lacks the dynamic parts that makes π-calculus and Join-calculus interesting modeling tools.

The actor model differs from process calculi in several aspects, for instance actors are based on the idea of individual entities; they are not anonymous, but messages are adressed via unique identifiers. The Actor Model is “modeled after physics” (or an understanding thereof) and very identity-centered. At first glance, this seems to run against the process-relational Protoverse metaphysics, where “identity” is a rather fluid concept.

However, we could implement something resembling π-calculus or Join-calculus by a thin layer on top of an existing actor model implementation – interestingly, that has been done before: see JErlang, which implements Join-calculus. Such an implementation based on the yet-to-design Protoverse specifcs could then serve as a DSL to express the Protoverse Process dynamics.

Task-Parallelism vs. Data-Parallelism

The Proptoverse’ model’s focus is on relational dynamics, recursive algorithms (e.g. manipulation of nested graphs) to model potentially infinite Nested Scales of Complexity. Protoverse dynamics are supposedly largely irregular at higher levels and highly interconnected, therefore not strictly hierarchical.

This seems to align more with task parallelism rather than data parallelism. Due to “big data”, data parallelism is more more widespread in use today and generally more efficient (e.g. SIMD). Hence it’s worth considering if parts of the dynamics could be computed in a data-parallel approach.

Generally speaking, encapsulation and locality are good indicators for how effectively computations can be parallelized. For deeply nested and interconnected structures, a hybrid approach might be beneficial.

  • Using task parallel techniques to manage the high-level structure and irregular tasks;
  • Apply data-parallel techniques on subcomponents of the problem where data can be processed uniformly;
  • Recursive tasks decompose into smaller problems anyway, where data parallelism can be exploited;

In any case, it makes sense to have straightforward GPU computing readily available if needed (OpenCL, CUDA or HIP).

Support for Deep Reccursion

Recursion will play a key role (evolutionary dynamics at multiple Nested Scales of Complexity). Specifically, more complex recursive algorithms who use the intermediate values returned by successive recursive calls can’t always be expressed via recursion in tail-call position.

“Never send a human to do a machine’s job.” —Agent Smith about TCO.

However, it’s generally possible to explicitly manage a separate stack as a heap-allocated data structure that collects the outcome of the recursive calls, instead of relying on the call stack, while the stack management logic could be hidden with helper macros, keeping the code clean and readable; or use continuation-passing-style (meh).

While both workarounds are pragmatic, they come with a cognitive overhead and belong to the realms of optimization. Hence, it would be great to have an unlimited or dynamically managed call stack depth to prevent stack overflows, so that we could write recursive procedures in a naive style for convenience, specifically during prototyping.

Scalability

Open-endedness is fundamentally about scaling.

First and foremost, the Protoverse will undergo increasing counsumption of resources. Therefore relatively straightforward scaling should be baked-in from the very beginning. It depends on the principal modus operandi of the Protoverse which scaling model to prefer:

  • Batch-processing: This approach leans toward vertical scaling, throwing more powerful hardware on it, e.g. CPU/GPU arrays. This hardware wouldn’t have to be available all the time.
  • Long-running: The Protoverse as a continuously running application could benefit from a horizontal, distributed scaling model. Permanent access to the hardware infrastructure is required.

Distributed Computing

Since I anticipate the Protoverse rather as a forever-running application than a batch computation (open-ended quasi-darwinian evolution), I lean towards a distributed, horizontal scaling model with redundant nodes, that runs in the cloud, and eventually might even run on commodity hardware (e.g. a heterogenous peer-to-peer network): much like BOINC, or even like Ethereum or Bitcoin.

State Management, Pruning and Data Compression

We probably have to consider resource-saving techniques earlier in the development process; also taking into consideration, how much of Protoverse history will have to be kept around. We have to consider if a Markovian approach (history-less) would be possible, and where.

Clever resource management plays a larger role if he Protoverse takes the path of being implemeted as a long-running application. There could be intrinsic selection (as a part of the Protoverse dynamics) that effectively prunes certain Nested Scales Of Complexity when it is highly probable that nothing interesting happens at these; e.g. when they turn out random and too boring and therefore “non-viable”. On the other side – even inconspicuous, boring dynamics can turn out to be further influential for evolution, and the extent may be unforeseeable.

Computation by Need

The concept of “observer dependency” (which is part of the metaphysics, the universe being participatory) could be understood that the state of a Process is not determined (computed) until it is “observed” – observation as evaluation or observational computation (not to confuse with the “Gang of Four” observer pattern, which is something else).

The analogy becomes stronger, if we buy into the idea that our universe is somehow “computational”. It seems like deferring the resolution of quantum states is similar to how a computer defers computations by lazy evaluation. Although the analogy is not really correct, both concepts deal with the transition from potential to actual states, implying that states are determined as needed, rather than being predetermined and statically defined.

I’ve got a feeling that infinite structures for data representation (open-endedness) might be naturally pervasive in the Protoverse, and dealing with this data easily would simplyfy certain things. This may require non-strict evaluation (laziness).

Theoretically, might turn out practical to follow such an approach to save computing resources; but it might also turn out extremely impractical (piling up thunks). Any way, it deserves consideration and further thought.

Portability

Unforseen complications may likely require to switch certain parts of the implementation during development, e.g. the runtime. It would make sense to write the model in a language that could be ported to something else without too much effort.

The first what comes to mind are C and C++. But these are very low-level or and ugly, and I’m pretty sure I wouldn’t come very far with one of these as a modeling language. But there are other aspects:

  1. Use a high-level language that compiles to either a low-level language or can target multiple backends; for instance, there are a few Scheme dialects that target various platforms, and a growing number of languages target WASM, which could be an interesting runtime, especially with projects like this.
  2. Adher to portable code, that means following a standard closely (e.g. R6RS for the Scheme language).

Observability: Visualization, Analysis and Profiling

Meta-functionality is complicated stuff and imposes overhead that eats away development resources; it’s quite boring but neccessary.

  • So there should be (ideally native) frameworks/libraries available that we can use for data visualization and analysis.
  • Same for performance profiling and tweaking, which is often readily available when building on top of a large-ecosystem virtual machine/platform, such as the BEAM or JVM.

Interfacing With Other Ecosystems

It might turn out beneficial if a bigger language ecosystem is within reach, either by directly building on that very ecosystem/platform, or via straightforward interop. This might be useful for analysis, visualization, measuring; or to offload heavy computations if they cannot be implemented efficiently in the primary language/runtime; or to use specific scientific libraries. Possible targets are the usual suspects: Rust, C, C++, but also Java and Python.

Programming Environment and Language

The following assessment is about choosing the primary platform/language for modeling and implementation. There are gradual pros/cons which are rather tricky to gauge at this point, not knowing in advance what requirements will turn out more or less important over the course of the project; and there are absolute dealbreakers too.

Conventional Programming Languages and Platforms

Erlang Platform (BEAM)

  • Hot code reloading: support for upgrading a running application; but it’s hellish complicated; so it seems that it can’t be leveraged for interactive development (preserving rather than re-creating state is the goal), and further it does not to compare to Common Lisp’s redefining functions and classes within a running system;
  • Erlang projects must be written by adhering to the OTP framework in order to reap the fruits of what Erlang has to provide;
  • Erlang doesn’t provide the same grade of tight-looped interactivity as Common Lisp;
  • Erlang has built-in observability (profilers and analysis tools);
  • Recursion is idiomatic: dynamically growing stack depth, no stack overflows;
  • Each lightweight Erlang process has it’s own stack, heap and garbage collection, no global “stop-the-world”;
  • Responsive under heavy load, fault tolerance facilities;
  • Reduced DevOps mess, a plethora of infrastructure tools can be avoided;
  • Concurrency primitives readily available at the language level; actor model implementation;
  • Ad-hoc parallelism: concurrently executing code is pre-emptively scheduled and will be distributed automatically to all available cores;
  • Ad-hoc distributed computing built-in; however distributed Erlang isn’t suitable for connecting nodes over the internet in a plug-and-play fashion, because there are no security facilities at all. So it’s best used only to connect nodes within an isolated cluster.
  • Passing around a lot (or large) messages is inefficient, because shared-nothing message passing involves copying that data;
  • Is said to be “slow for CPU-heavy” stuff (probably a matter of concern?), however CPU-heavy code could be written as a NIF, e.g. in Rust;
  • Practical cluster size up to ~150 nodes, larger clusters possible with hierarchical clustering and other trickery (see also theoretical system limits);
  • Large-scale simulations: Urban traffic

Sentiment: The question would be if the Protoverse model will eventually translate well to the actor model – on the BEAM, everything is done via actors, there’s nothing else. Physical simulations, as far as they might be related, are usually numeric where each data point affects the others.

BEAM: Erlang language
  • Simple, functional actor-oriented language with pervasive pattern matching;
  • Advantages over Elixir/LFE:
    • Simple, consistent and readable syntax;
    • Lingua franca on the BEAM;
    • Lots of in-depth documentation, 3rd-party books and howtos available;
  • No Lisp-style metaprogramming (macros) but template system;
  • No algebraic data types;
  • Fewer libraries than Elixir and less diverse landscape;
  • Less suitable for interactive Programming, although Livebook supports Erlang;
  • No easy reusing libraries from the larger Elixir ecosystem;
  • Collab opportunities: Engaged community, focus on infrastructure, industrial-engineering-mindset;

Sentiment: Erlang has no Lisp-style macros, but I think I will need them. And there are some rocks on the road to flowy interactive, explorative programming.

BEAM: Elixir language
  • Functional actor-oriented language with pervasive pattern matching;
  • Advantages over Erlang/LFE
    • More DSL friendly (there’s a DSL for logic programming in Elixir)
    • Nx (tensor operations) and Axon (framework for neuronal networks);
    • Elixir-native interop: Rustler (Rust/C/C++ via NIFs);
    • Good interactive development with Emacs and Livebook, incl. plotting, visualization, etc.;
    • Largest library ecosystem on the BEAM, very active;
    • Gradual type system in the making;
  • Opinionated and inconsistent language design (mostly syntax), way too much special cases and gotchas for a modern high-level language;
  • Steeper learning curve than Erlang or LFE;
  • Makes extensive use of metaprogramming (hygienc macros, compile-time only), although it looks quite awkward and verbose compared to Lisp;
  • Lazy streams;
  • No ADTs, which would be great for modeling;
  • Most 3rd-party documentation and howtos are related to the Phoenix web framework and web development;
  • Collab opportunities: Community and ecosystem centered around web backend and business apps; building stuff outside of this domain is slooowly taking off (e.g. machine learning);

Sentiment: I find Elxir rather inconsistent, verbose and a bit clumsy – syntax-wise, mostly. There are quite a lot WTFs, but Elixir is more interactive-friendly and has at least compile-time macros.

LFE (Lisp Flavored Erlang)
  • Simple actor-oriented Lisp with pervasive pattern matching;
  • Advantages over Elixir/Erlang:
    • Elegant and consistent language design, easy to get into;
    • Much closer to Erlang than Elixir, Erlang documentation translates quite easily;
    • Has real Lisp macros (unhygienic), but compile-time only;
    • Macros are not awkward and verbose like in Elixir;
    • Prototyping: interactive experience on par with Elixir with some differences, e.g. defining named functions directly in the REPL;
  • LFE is basically a 2-person hobby project, slow development cycles;
  • Documentation is incomplete, fragmented, unpolished and inconsistent, riddled with 404 not found links;
  • Emacs integration is outdated and inferior-lfe needs a brush-up (probably I could fix most usability issues in a weekend or two);
  • No algebraic data types;
  • LFE can’t use Elixir libraries seamlessly, so there’s no real interop with the largest BEAM ecosystem;
  • Erlang libs can be used without problems;
  • Collab opportunities: intersection of Lisp/Erlang afficinados (very tiny), far lower momentum than Elixir or Gleam;

Sentiment: Lisp on the BEAM would be an interesting match if LFE wasn’t just a 2-person hobby project.

Guile Scheme

  • Clean, highly general, expressive;
  • Astonishingly “general purporse” for an extension language;
  • Schene is great for prototyping abstractions and algorithms;
  • Full metaprogramming (syntax-rules, syntax-case, defmacro);
  • Capable for metalinguistig abstraction (see also LOP);
  • Good for interactive development (Emacs with Geiser);
  • Relies on Guix for package management (MacOS unsupported) or homebrew;
  • There’s a tiny library ecosystem;
  • JIT compiler;
  • Unlimited stack depth for Scheme code, TCO available;
  • Concurrency and multicore parallelism via fibers (manual), Concurrent ML / CSP style, preemptive scheduler;
  • Distributed: Goblins actor-framework in the making (docs), for P2P;
  • Overall performance for CPU-heavy tasks is unclear right now;
  • Collab opportunities: GNU and Scheme enthusiasts, academic;

Sentiment: Due to the Fibers library and its scheduler, Guile is quite interesting; but I have no idea how it compares to Erlang, for instance.

Racket

  • Clean, higly general and the most expressive I guess;
  • Racket is great for prototyping abstractions and algorithms;
  • Full metaprogramming (syntax-rules, syntax-case, defmacro);
  • Built for metalinguistig abstraction (see also LOP);
  • Not so bad for interactive development (in Emacs);
  • Various concurrency facilities, green threads;
  • Parallelism: Places looks quite awkward and hands-on, fully manual;
  • Distributed: has a Syndicated Actors implementation and a Spritely Goblins implementation that’s behind the canonical one; both are experimental.

Gambit / Gerbil Scheme

  • “Distributed” actor-model centric Scheme implementation based on Ganbit Scheme
  • Clean, higly general, expressive;
  • Scheme is great for prototyping abstractions and algorithms;
  • Powerful metaprogramming;
  • Connect REPL to running program (actor ensemble server);
  • Very tiny library ecosystem;
  • No LSP implementation yet, planned for v0.19 release;
  • Interactive development: Emacs integration is thrown together and buggy;
  • Multiprocessor-parallelism (SMP) not working yet, depends on Gambit Scheme implementation;
  • Built-in concurrency (actor model) based on light-weight Gambit threads;
  • Ad-hoc distributed computing with actor ensembles;
  • Compiles to C, overall performance on CPU-heavy tasks probably better than Guile’s;
  • Small documentation, but gives a well-rounded overview;
  • Collab opportunities: Scheme enthusiasts, academic;

Sentiment: Gerbil Scheme is a promising project but in a very early stage (v18.0), and doesn’t support multiple CPU cores. The Emacs integration isn’t comparable to Guile’s, e.g. documentation or code autocompletion.

Julia

  • The only conventional imperative language considered so far;
  • Pattern matching and other niceties are available via MLStyle.jl library;
  • Recursion is not idiomatic, you’ll have to use imperative loops everywhere, Julia’s stack depth is capped;
  • For deep recursion, you would have to manage a separate stack as a data structure, or resort to other workarounds.
  • Metaprogramming via macros is available;
  • A running program can be debugged and modified, new functions can be defined and added (with caveats; not extensively as in Common Lisp);
  • Dynamic typing, but allows type annotations (mostly for performance reasons); Algebraic Data Types via MLStyle.jl
  • Parallel and distributed computing out-of-the-box (cluster-oriented), but it seems not very high-level;
  • Shiny convenience libraries for all kinds of scientific stuff, visualization and plotting;
  • Direct calling of C functions and access to Python libraries;
  • Has native access to GPU computing (CUDA only);
  • Low-level access to the language itself;
  • Collab opportunities: data-, science and simulation folks of all sorts;

Sentiment: Julia is an interesting choice and checks many boxes – the ecosystem fits the domain, which is scientific computing and simulation, and Julia is a good fit for CPU-bound workloads. But I’m not convinced because conceptually, deep recursion will play a crucial role in the Protoverse model, which doessn’t translate well to how things are done in Julia.

OCaml

  • Functional language with pervasive pattern matching, elegant and very readable syntax;
  • There has been a process calculus implemented in OCaml, but abandoned;
  • No true interactive programming; cannot connect to, or modify a running program; program would have to run in the in the REPL to begin with;
  • Static type system makes refactoring fun and documentation effective;
  • ADTs are great for modeling;
  • No real macros but just template preprocessing of sorts, but it has a very powerful module system (which is something entirely different compared to module systems in other languages);
  • Concurrency: There’s an early-stage actor library Riot based on algebraic effects and effect handlers, OCaml’s goto-concurrency Lwt implements promises instead;
  • Parallelism since 5.0 using domainslib;
  • No ad-hoc distributed computing
  • There’s Mirage OS that lets you compile your application directly into a full “micro operating system”, it can be used to deploy nodes for a distributed system;
  • Overall performant, super fast compiler;
  • Messy tooling and workflow – there’s always a new surprise tbat something stopped working;
  • Colaboration opportunities: engaged community, language ecosystem under active development, FP-pragmatics;

Sentiment: It’s my favorite language actually, but the tooling (Opam, Dune build system, Ocaml REPL and Utop REPL) is such a mess. Well, and there’s no real interactive development.

Haskell

  • Functional language, expressive and low-noise, lovely syntax;
  • No support to modify a running program, only REPL (what about rapid?);
  • Non-strict evaluation is elegant, simplifies handling of infinite structures; but laziness is available in other languages when you need it;
  • Static typing simplifies refactoring and improves reasoning, type system is great for modeling;
  • Concurrency: Cloud Haskell (actors) allows also message passing using typed channels, where the SendPort of a channel can be passed to other processes via message; processes don’t share state; seems unpolished, not much used and and there are few how-tos;
  • While excellent Haskell learning material is abundant, libraries lack of howtos, examples and documentation;
  • Libraries tend to use numerous language extensions, many WTFs;
  • Purely functional code is good for parallelization (in principle); not sure how actor model message passing (wich is basically IO), the type system and purity go together; it might turn out very awkward, basically most of the code will live in the IO monad;
  • Collab opportunities: engaged community, FP-enthusiasts;

Sentiment: I’m afraid of lazy-per-default, that reasoning about resource usage may become too complicated and a lot time is spent on fighting Haskell’s opinions. Seems no great choice for explorative programming, except maybe you work basically on the type level.

NIM

  • Multi-paradigm, expressive and low-noise;
  • There’s a REPL for interactive development;
  • Metaprogramming via templates and real compile-time macros;
  • Hot code reloading available to alter a running program;
  • Static typing + algebraic data types;
  • Supports multicore (SMP) parallelism using a thread pool;
  • Portability: compiles to C, JavaScript, and several other backends;
  • Recursion depth is adjustable via compiler options, but the option accepts int16, which means no more than 32,767 recursion depth;
  • The language is both high-and low-level, has multiple GC options and also manual memory management;
  • Both concurrency and parallelism primitives seem rather low level and target native OS threads rather than lightweight threads;
  • There are some high-level concurrency implementations, e.g. “Syndicated Actors” (discussion), however this is something entirely different from Erlang’s communicating lightweight processes; in its application possibilities it seems related to Spritely Goblins;
  • Has many interesting libraries, e.g. a state-machine generator;
  • The language seems kinda stable, but there are a lot of deprecations and even a rewrite of the compiler going on;
  • Collab opportunities: Even though the language doesn’t draw a lot of attention, there’s already a very diverse landscape of libraries and projects written in NIM, and attracts even academics.

JVM: Clojure

  • Functional paradigm, in principle a nice and clean take on Lisp;
  • Recursion support: no TCO, but acknowledged by loop/recur workaround due to JVM characteristics;
  • Interactive and live-coding, working at the running program is possible;
  • Great Emacs integration and prototyping ergonomics;
  • Has metaprogramming (hygienic), but that seems rather discouraged;
  • Optional type system Typed Clojure (?); clojure.spec isn’t a type system but for runtime validation;
  • Lazy sequences;
  • Concurrency: Several concurrency abstractions; core.async implements CSP-style concurrency and allows channels to be passed as values; but memory is shared (unlike Erlang’s processes);
  • I ran into various errors when I did trivial benchmarks involving 100k concurrent computations via core.async “go blocks”; I did a few trivial benchmarks and depending on which concurrency facility you use, I was always running into some limitations when it comes to large numbers of concurrent operations;
  • Portability: Clojure targets several host platforms officially, e.g. JVM, JavaScript, CLR; and there are inofficial implementations, e.g. Erlang/Beam, but these are highly experimental;
  • Observability: JVM has lots of profilers and tools;
  • Clojure has a non-copyleft license (Eclipse License)
  • Collab opportunities: rather business-oriented community and a few lispers;

Sentiment: Dealing with the Java stuff and lingo is unpleasant yet unavoidable. Clojure is a functional language, but since it runs on the JVM, it doesn’t offer the full spectrum of what makes a functional language, but employs workarounds instead. It resorts to Java interop for all kinds of (even basic) functionality, which is confusing.

Common Lisp (SBCL)

  • Supports functional programming and logic programming, anything goes but not everything goes well;
  • Covers a huge range of abstractions via libraries, e.g. pattern matching with trivia;
  • Very moldable to the domain via metaprogramming (DSLs);
  • Algebraic data types available, and there is also Coalton, an ML-style, internal DSL with type classes);
  • Conversational programming at the running system is great for prototyping/exploration;
  • Image-based: The running program can be paused, suspended, debugged, modified and continued without losing state;
  • The state of a running image can be dumped and restored, but that won’t work out-of-the box for a distributed system;
  • Great Emacs integration, simple tooling (sly, quicklisp, asdf and roswell)
  • Concurrency: Sento actor framework is the most promising; and there are Lisp-Actors or ChanL. Most concurrency facilities seem to rely directly on OS-threads instead of something like “lightweight processes”, though.
  • No concurrent garbage collector but stop-the-world;
  • Ther are Petalisp (data parallelism), Lparallel and OpenMPI bindings (likely an overkill, see OpenMPI);
  • SBCL is performant (Java ballpark);
  • Low-level access to the language itself, plenty of optimizations;
  • Collab opportunities: Community is diverse yet enthusiastic, fragmented, less collaborative (many one-person-projects rather than common goals realized);

Sentiment: I’m afraid that Common Lisp could be a dead end when it comes to massive concurrency & parallelism: “(..) high level parallelism in lisp just sucks”. But it might be sensible to set up some benchmarks, (e.g. with Sento) before dismissing CL, because otherwise CL seems pretty nice for such a project – at least for prototyping.

Specialized Programming Languages and Runtimes

While low-level languages like CUDA or OpenCL for GPU computing are well established, high-level languages for massive parallelism are a more recent invention:

Bend

  • Two surface syntaxes: Python-style and ML-style;
  • High-level: higher-order functions, closures, unrestricted recursion, continuations;
  • Untyped language – types seem to be mere documentation, ADT syntax sugar is there to define constructor functions (see features);
  • Aims to provide fully “transparent” parallelism, no manual managment of parallelization (no thread creation, locks, mutexes, etc.);
  • Compiler can generate massively parallel GPU kernels from high-level code;
  • Based on interaction nets (Wikipedia, paper);
  • Still experimenal/research state;
  • https://github.com/HigherOrderCO;
  • Discussion in Julia forum;
  • Gained some traction, has good marketing;

Futhark

Chapel

  • Old-school object-oriented, not very high-level;
  • Task parallism, data parallism and nested parallelism;
  • Portable: orignially designed for Cray supercomputers and not with GPUs in mind, it is portable and runs on Laptops as well;
  • Supports GPU programming (NVIDIA and AMD);
  • What is Chapel: https://developer.hpe.com/platform/chapel/home/;

Conclusion

The Protoverse project ventures into uncharted territories, and my gut feeling says that I should counterbalance that notion with adhering to mature, “industrial-grade” technology where possible.

I’m not expecting to plug libraries together (how most “software engineering” is done today), but to go through a journey of discovery and exploration, and to think through the whole process “from scratch”, and the language should support that (it’s pretty close to the paradigm embodied by the Scheme language).

I’ll be “re-inventing the wheel” several times, which certainly will lead to inefficient, naive implementations (evolutionary algorithms are such a candidate, for instance). In such cases it will be desirable to draw from established algorithms/libraries, after I have developed an understanding beforehand.

Covering much ground quickly by focussing on the model and abstractions is crucial, therefore I want to work as high-level as possible, avoiding clutter in order to free mental capacity.

The Finalists

I’d like to pick two languages/platforms that are somewhat orthogonal:

  1. The Erlang platform, using either Elixir and/or Erlang, follows a connectivity-based approach (message passing): The BEAM is not very performant for CPU-bound-, but for IO-bound computing. However, CPU-intensive parts could be later ported to high-performance implementations if that’s necessary.

    The actor model, available at the language level, makes it convenient to write concurrent (and parallel) programs. As per my current view, it seems close to how I picture the Protoverse model (I might be wrong).

    The actor model basically nudges you into decoupling your code, and the scheduler parallizes automatically where it sees fit. But it doesn’t stop there: each Erlang process has it’s own stack, heap and garbage collector - and you can have a million or more of these processes in a single node.

  2. And what should be the other finalist?
Published by Dan Dee