Transitioning Code From Closed To Open: A JuliaCon 2020 Discussion Between Julia Users In Industry

16 September 2020 | Jarrett Revels

Despite its virtual nature, this year's JuliaCon was just as energetic as every previous JuliaCon I've attended - absolutely bustling with fresh ideas and lively discussion amongst key leaders from across the open-source technical computing community. While JuliaCon has always been a hit with academic scientists/engineers that require large-scale simulation and modeling capabilities, recent iterations of the conference have seen a dramatic increase in industry user attendance.

Keeping pace with this growth, this year's JuliaCon featured more industry-focused content than ever before, from Jacob Quinn's workshop on building microservices in Julia to the tried-and-true annual Julia In Production BoF ("Birds-of-a-Feather" discussion session).

This blog post is a summary of a similarly industry-targeted session: the "Transitioning Code From Closed To Open" BoF, attended by 15+ community members from 10+ companies. The basic premise of the session was:

Within private companies/institutions, it can be difficult to implement effective internal practices that enable internal code to smoothly transition into high quality, open-source contributions. In this BoF, we’ll swap techniques for maximizing open-source impact in the Julia ecosystem while minimizing refactor time/effort and code churn.

How The BoF Came To Be

Allow me to provide some background before diving in - I'm Jarrett 👋 Formerly a research software developer with the Julia Lab at MIT, I left in 2019 to start Beacon Biosignals with some incredibly talented neuroscience + Julia friends. We’re building technology to render brain monitoring more accessible, interpretable, and actionable, and the Julia ecosystem has played a critical role in many of our early wins.

By early 2020, Beacon had accumulated a lot of interesting work internally that I knew we should open-source/upstream, but perhaps unsurprisingly, it was difficult to actually prioritize doing so when there were so many other pressing concerns competing for our finite bandwidth. I imagined that we weren't the only private entity in the Juliaverse in which the phrase “ah, we should definitely open-source <internal tool> at some point” is commonly uttered but seldom acted upon.

I also recognized that every organization is unique, and the contribution challenges we faced at Beacon might not be the same challenges faced by other industry users. "Wouldn't it be great," I thought, "if us industry folks could get together and hash out strategies for tipping the OSS cost/benefit scales for our organizations?" And thus, the BoF proposal was born.

The rest of this post summarizes the thoughts, ideas, and conclusions that came out of the ensuing discussion at JuliaCon.

Why Companies Contribute To Julia's OSS Ecosystem

Many early adopters of Julia in industry are already prolific contributors (and employers!) within the Julia open-source software (OSS) ecosystem. Here are a few motivating factors that have pushed these companies to contribute:

Barriers To OSS Contribution From Industry

Industry users face many unique technical and non-technical challenges when making open-source contributions. Here are a few of the items discussed at the BoF.

"Clopen" Dependency Chains

This problem is introduced when open-sourceable code has a nontrivial (sometimes transitive) dependency that needs to remain closed-source. For example, package A is mostly open-sourceable, but depends on internal package B which is not.

One solution (that may require a small or large amount of effort, depending on the implementations involved) is to refactor the code-to-be-open-sourced's API to feature some form of control inversion that allows internal users of the code to pass in values that encapsulate private dependencies. Not only will doing this solve your open-sourcing problem; it'll also make your code more extensible/composable, so that both internal and external users can more easily extract value from the contribution.

Another flavor of this problem can exist when open-sourced code features many downstream closed-source dependencies. Multiple attendees noted that organizations would sometimes require the implementation/maintenance of an additional internal wrapper overtop the open-source version of their package.

Organizations That Don't Use OSS In The First Place

One attendee brought up that, for many organizations, the first barrier to allowing employee OSS contributions is to allow internal usage of OSS at all. It can be hard for decision makers at organizations that have no prior OSS experience to come to terms with the traditional motivations and arguments for OSS. They may even harbor common misconceptions about how the process actually works. Luckily, there's a plethora of material available online that targets exactly this audience that organization members can bundle up in a manner that's most effective for their organization.

Licensing, Patents, and IP

One difficulty reported during the BoF was getting organizational approval to release software under permissive licenses. This can be tricky, indeed, as there isn't a one-size-fits-all solution for every business, and non-technical decision makers might not have prior understanding of the legal aspects of OSS. A few different suggestions/ideas came up on this topic during the BoF:

Open Sourcing Packages That Previously Used Internal CI/CD

One important element of successful open-source projects is that CI logs should generally be available to all contributors. This can be a burdensome criteria to meet if a previously closed-source package utilized internal CI/CD pipelines with nicer properties than publicly available CI pipelines. In some instances, certain flavors of this problem can be solved with better tooling, e.g. self-hosted (but publicly accessible) package registries can vendor open-sourced packages just as nicely as Julia's General Registry, but might leverage custom CI/CD for pre-registration checks.

Open Source Efforts Could Reveal Strategic Intent

One attendee proposed a very interesting challenge that they've faced in industry: In theory, their company was fine with contributing their internal implementation of a technique back to the OSS community, but couldn't actually do so because it would reveal that the company was working in a specific area of strategic interest to their market that they didn't want to reveal publicly. One solution to this problem would be to anonymously open source the implementation; however, this is obviously nonideal and the risk to business-critical strategic information will often be too high.

Git History/Metadata Preservation

Commit history and associated PRs, issues, etc. provide important context about design decisions that can ideally be preserved in the transition from closed-source to open-source. However, this can be a significant challenge depending on the provenance of the code and documentation practices. It appears that this process is easier (or less necessary) if...

Conclusions

The BoF ultimately touched on a few high-level takeaways:

This last point is, in my opinion, one of the most challenging obstacles faced by industry users who wish to participate in OSS. To help overcome this issue, I'm announcing The Annual Industry Julia Users Contributhon, an annual Julia community hackathon where participating industry organizations can come together to contribute back to the Julia ecosystem. Via the Contributhon, I hope we can...

Have thoughts/questions about this blog post, the Contributhon, and/or the use of Julia in industry? Come join the discussion in the Julia Slack's #industry-users channel!