A recent curated test of computational experiments pulled from published papers could only re-execute 47% of them end-to-end, and the culprits were mundane: missing dependencies, incompatible environments, and documentation that no longer matched the code. Sit with that number for a second. More than half of what gets cited in a methods section cannot be rerun by a motivated stranger with the repository URL in hand.
Three years later, the problem compounds. A reviewer opens the archive, follows the README, and lands on a machine that no longer exists: a Python version that has moved on, a package that unpinned itself into a breaking release, a GPU driver assumption that was true in the author's lab and nowhere else. The paper still reads cleanly, even as the software underneath has stopped working.
Why Does the Code Stop Running So Fast?
Research code is almost never self-contained. It sits on top of an interpreter, a package manager, a stack of libraries, a kernel, sometimes a specific accelerator, and a handful of system tools the author never thought to write down. Each of those layers has its own release cadence. Each release can change a default, deprecate an argument, or shift a numerical result in a way that will not raise an error.
The README was written for the machine on the author's desk the week of submission. That machine is already gone. The virtual environment was rebuilt, the laptop was reimaged, and the cluster image was updated twice while nobody was looking. What ships with the paper is a snapshot of intent, not a snapshot of the environment that produced the figures.
What Breaks First, and in What Order?
The failure modes arrive on a predictable schedule, and Neliti's coverage of how academic code breaks long before its paper is forgotten walks the same timeline most maintainers recognize. Expect roughly this sequence:
- Unpinned dependencies drift. A top-level library bumps a minor version, pulls in a new transitive dependency, and the install that worked on Monday fails on Friday.
- Runtime upgrades change results. A new BLAS, a new CUDA, a new default dtype, and the numbers move inside the tolerance nobody checked.
- Repository URLs go dark. A lab moves off a university server, a personal GitHub account is renamed, a hosted notebook service shuts down a tier.
- Installation becomes the experiment. By the third or fourth year, a reviewer spends more time getting the code to run than reading what it does.
What Is the Reviewer Actually Looking At?
A reviewer coming back to a repository years later is doing archaeology. They have the paper, a frozen artifact, and they have the repository, a living thing that kept moving without them. The README promises an environment. The lockfile, if there is one, describes it, and the container image, if there is one, embodies it. Most repositories have none of these.
The documentation itself is often the weakest link. A dataset description that lists field names but omits collection dates, licenses, or preprocessing steps leaves the reviewer guessing. The Datasheets for Datasets proposal from Microsoft Research pushed for standardized documentation covering motivation, composition, collection, recommended uses, and maintenance precisely because an undocumented dataset ages into something nobody can safely reuse.
What Habits Keep Code Usable for a Decade?
The practices that keep research software alive are not glamorous, and they do not require a platform team. They require deciding, before the paper goes out, that the code is part of the submission.
- Lockfiles over loose requirements. Pin the exact versions that produced the figures. A requirements file that says a library is needed is not the same as one that says which release.
- Containers that carry the environment. A Dockerfile or a Singularity image bundles the interpreter, system packages, and library versions in one artifact a reviewer can pull and run.
- Archived releases with DOIs. Mint a persistent identifier for the exact commit tied to the paper. The GitHub URL will rot. A DOI pointed at a deposited snapshot will not.
- CI that rebuilds old versions. A scheduled job that reinstalls and reruns the pipeline each quarter catches dependency drift while it is still a one-line fix instead of a dead repository.
- A README that names assumptions. Write down the operating system, the hardware, the interpreter, and any environment variables the code silently reads. Future-you is the reviewer.
Who Pays for the Extra Hour?
Someone has to do this work, and that is the real friction. A graduate student is rewarded for the next paper, not for the previous one still building cleanly five years on. A principal investigator is rewarded for grants, not for container images. Journals are starting to lean on authors to produce reproducible artifacts, which helps, but the day-to-day habit still falls on whoever wrote the code.
The hour spent writing a lockfile, pushing a container, and minting a DOI at submission is cheap. The week spent three years later trying to reconstruct an environment from memory, after a reviewer or a collaborator asks, is not. Make the repository part of the publication, and the machine the paper was written on stops being the only one that can run it.