My experience with the 2 language problem

Long post warning; TLDR at the end.

I am a scientist who needs to write high performance code. I started with Julia around 2017 (julia v0.5), and so far everything I needed has worked. I love the language and I want to continue using it for everything.

Even though Julia solves the two language problem, I often question whether I should regress backwards. Lately I have a gut feeling that changing to R/C++, Python/Rust, or something similar could make my life easier in the long run. I haven’t decided for sure yet. Regardless of my decision, I’m writing this post right now so future me can decide whether I made the wrong call or not.

The main problem is that I don’t like to “hack around”, and increasingly I feel like using Julia will require me to do so.

Here are some examples of what I mean:

1. I needed to call some C++ code in Julia. The Julian way of doing so, afaik, is (a) manually write the wrapper code all in C++, (b) use CxxWrap.jl to call the shared library, (c) create a jll package via BinayBuilder.jl, (d) upload to Yggdrasil, (e) write a regular Julia package that depends on said jll, and FINALLY I can call C++ code from julia. This is significantly more complex than I initially imagined, and many of the intermediate steps had little documentions/tutorials. If I encounter any difficulties in any of these steps, like I did here and here, I feel like only 1-2 people in the world would be able to help me (thankfully, they did help).
2. I wanted to create binaries using PackageCompiler.jl. It is one of the most popular packages in Julia, so I figured it would be safe to rely on it. Unfortunately I encountered some troubles, which is fine, because popular packages can have rough edges too. But it took me 1-2 weeks just to get to the bottom of the issue, because the problems I encounter (e.g. here and here) were too ambiguous/complex to describe/reproduce. So my posts were ignored. My current solution is very non-ideal, but I had no choice. Did I mention one can compile cross-platform binaries with BinaryBuilder.jl but cannot with PackageCompiler.jl? Feels like if you can do one you should be able to do the other.
3. I recently picked up an old project which required solving a large non-linear optimization problem. It internally used MathProgBase.jl (deprecated since 2020) to interface with the Ipopt solver, so naturally the first step is to replace it with MathOptInterface.jl. While a tutorial exists, I was immedaitely greeted with a big fat danger sign that basically says I probably don’t want to do this. Fortunately, the translation went smoothly, so that was a big win. However, I definitely want my packages to function without too much maintenance effort. This is pretty hard when something internal like MathProgBase.jl or something as big as LoopVectorization.jl can basically just become deprecated within a few years. I mean, I wrote a paper whose (impressive) benchmark relies heavily on LoopVectorization.jl. What do I do now that it’s becoming deprecated in Julia v1.11+?
4. I wrote an R and another python package to wrap one of my julia package, following the example of DifferentialEquations.jl. As much as I like Julia, I want my work to be relevant to my collaborators who mostly live in R/Python world. Unfortunately JuliaCall (which handles the interfacing between R and Julia) appears to be abandoned, and even basic installation issues will likely never get resolved. As a result, half the time my collaborators cannot successfully install these wrappers without my help. I really thought it’ll be pretty cool to suggest R/Julia rather than R/C++, but I guess this is more of a dream than reality.
5. I have another optimization project involving computation of some nasty gradient/Hessians. I am thinking about relying on automatic differentiation tools to compute gradient/Hessians for me, an area that I basically know nothing about. Then I saw julia have 10-20 packages all doing basically the same thing. From what I can gather, it seems I should go with Enzyme.jl. It seems like a cool concept, but I’m hesitant. I definitely do NOT have the expertise to fix problems in these AD tools shall I encounter any. If I were in Python world, this wouldn’t even be a question because I’ll just use JAX.

Now I find myself wanting my Julia code to be accessible from R/Python. I also want easy access to C++ from Julia without too much effort. I want julia binaries that can run on machines without Julia installed. I want some authoritative figure to tell me, when there are too many options to choose from, which one I should use and why. If I encounter problems, I hope I’m not the first person to have to deal with it, but even if I am, boy I hope I don’t have to deal with it by myself.

As a scientist, I want to spend most of my time thinking about science, algorithms, and datasets. I came to Julia because it promises to solve the 2 language problem. But ironically, now I find myself needing to double dip in multiple langauges regardless, and therefore I’m still spending most of my time crafting workarounds to whatever technical difficulties the 2 language problem entails. If I need to deal with this one way or another, it just doesn’t feel right to do so in Julia, because I’ll need Bart/Mose/some other important person to specifically look at my problem.

Are my worries/concerns justified? Am I expecting too much? Is the grass greener on the other side?

TL;DR I like julia but I’m not sure if switching to R/Python/something else will make my life easier in the long run. Putting this post here so that future me can decide if what I did was right.