Open source projects involvement

midipix_build: build/cross-compilation infrastructure for midipix, a POSIX/Linux-compatible development/runtime environment for Windows (February 2016-2024)

Midipix[1] is a POSIX/Linux-compatible environment for Windows XP/Server 2003 or later, facilitating cross-compilation and execution of applications written for POSIX/Linux without suffering substantial performance loss. Unlike Cygwin[2], Midipix does not require interaction with the Windows environment subsystem in order to implement its system calls. Unlike Interix[3], Midipix is not an environment subsystem itself and does not introduce its own subsystem server and client DLL(s). Neither virtualisation nor kernel-mode drivers are required in order to use Midipix: instead, a small set of “runtime components” mediate communication between Musl, the libc chosen for this project, and the Windows NT executive (NTOSKRNL.EXE.)

At the time the midipix_build[4] project was started, [cross-]compiling the toolchain, the runtime components, and any number of 3rd party packages was handled by ad-hoc Bourne shell scripts. This ultimately proved unscalable and often unreliable. Hence, a sufficiently comprehensive build/cross-compilation infrastructure was required. As Midipix has been and still remains a moving target, an iterative development model was chosen: architectural considerations were thus of considerably low importance. Code refactoring has taken place only two times, and the present design has proven to be sufficient for the purposes of Midipix. The following goals and requirements emerged as part of this process:

1. Portability: midipix_build, similarly to NetBSD’s build.sh sans make(1), had and to some extent still has to support cross-compiling a large number of packages on any number of platforms that proves to be sufficiently POSIX-conforming. Thus, Bourne shell was chosen as implementation language, albeit a small number of non-POSIX features are leveraged, such as function-local variables. This permits building Midipix on Linux, Cygwin, natively (Midipix) itself, and most varieties of BSD.
2. Flexibility: the architecture of midipix_build has to be sufficiently simple and mutable in order to allow for the addition and/or removal of features at a reasonably fast pace, e.g. dependency resolving, parallelisation, signed distribution tarballs, automated package version checks, etc. pp. Furthermore, even the mere addition of 3rd party packages may often necessitate introducing new or revisiting current generalisations concerning particularly GNU autotools and Cmake-based build systems.
3. Convenience and reliability: midipix_build is usually how people are introduced and become acquainted with Midipix, largely via the distribution tarballs it produces. Both developers inasmuch as end users will want to be able to build or rebuild a single package, a set of packages, the runtime components, or everything. Therefore, virtually all common tasks are accomplished by entering a short command line invoking “build.sh” and without any other required user interaction.
   Furthermore, the highly inefficient fork-exec-write/read patterns all too prevalent in Bourne/-derived shell code have been virtually eliminated save for when actual external commands as opposed to internal functions are called, resulting in a fairly fast and low latency implementation, including the dependency resolving code.
4. Simplicity: last but not least, low code complexity and size, especially in relation to flexibility, remain a priority: presently, the code sans package- specific subroutines/variables amounts to about 6700 SLOC, whereas package-specific code covers approximately 4100 SLOC. Reusable subroutines, in the form of “build steps” (e.g. fetch, extract, build, install, ...) and pre/post-steps (e.g. setup_env, prereqs and sha256sums, tarballs, resp.,) reside in their own source modules.

Tcl TIP #458: design and implementation of epoll/kqueue support in the Tcl notifier on Linux/*BSD, resp. (FlightAware bounty programme) (November 2016-December 2016)

Tcl (pronounced "tickle" or tee cee ell, /ˈtiː siː ɛl/) is a high-level, general-purpose, interpreted, dynamic programming language[5]. Contributions to Tcl that alter public interfaces are submitted and processed through the “Tcl Improvement Proposal (TIP)” mechanism[6]. In early November of 2016, a bounty programme “for improvements to Tcl and certain Tcl packages” was published on GitHub[7] on part of FlightAware[8]. I chose to implement “[s]upport for epoll()/kqueue() to replace select() in socket handling,” which was finished by the end of December 2016[9].

Tcl implements an event-based architecture concerning I/O and particularly employs callback for I/O completion notification and handling, both of which are implemented in the platform-specific “notifier.” Originally, the only notifier available employed the select(2) system call for I/O multiplexing, which proved to be a major hindrance concerning scalability. The drawbacks of select(2) are well-known and elaborated on in my TIP[10] and shall not be discussed here, but include in particular the limit on file descriptors of 1024 imposed by reliance on select(2). The new notifiers for Linux, employing epoll(7), and *BSD, employing kqueue(2), do not suffer either of those defects.

Lastly, the original select(2)-based notifier implemented event notification and inter-thread IPC via the notifier thread. Multithreading, especially if unjustified, by itself introduces additional complexity and can often lead to equally complex issues such as race conditions, priority inversions, etc. pp. Therefore, the new notifier code calls epoll(7)/kqueue(2) from within the thread context of the caller without itself introducing any new threads. The inter-thread IPC problem, after some amount of discussion on the mailing list and IRC, has been solved by introducing a per-thread trigger pipe(2) on *BSD and eventfd(2) on Linux. As a consequence, one file descriptor per thread that interacts with the notifier and two/one file descriptors for inter-thread IPC on *BSD/Linux, resp., are added. This is considered to be acceptable considering the substantial loss of code complexity due to the removal of the notifier thread. Most importantly, threads, as far as the notifier is concerned, do not implicitly share file descriptors or state with each other.

References

[1] midipix
[2] Cygwin
[3] Windows Services for UNIX Version 3.0
[4] GitHub - lalbornoz/midipix_build: Unified build Bourne shell script for midipix
[5] Tcl – Wikipedia
[6] Tcl Improvement Proposal
[7] GitHub - flightaware/Tcl-bounties: Bounty program for improvements to Tcl and certain Tcl packages
[8] FlightAware - Flug-Tracker / Flugstatus / Flugverfolgung
[9] [TIP] Intent to implement epoll()/kqueue() support for sockets on Linux/FreeBSD · Issue #14 · flightaware/Tcl-bounties · GitHub
[10] TIP #458: Add Support for epoll() and kqueue() in the Notifier