Molecular origami

In this article

I wanted a post-training experiment where the model had to make something a simulator could disagree with. RNA gave me a particularly satisfying version of that: draw a pattern of stems and loops, then ask a language model to write a sequence that folds into it.

The sequence is just a string of four letters: A, C, G, and U. The interesting part is what those letters allow the strand to do. Positions can pair with other positions, bringing distant parts of the sequence together. Choosing compatible letters for the shape you want is only the beginning. Those same letters can also support a different shape.

I started with Qwen3-8B, taught a small set of adapters from examples, and then rewarded sequences using a folding program. On a held-out set of more complex structures, the supervised model found an exact predicted fold for 7 of 128 targets within eight attempts. After reinforcement learning, the primary run found one for 110.

The established RNAinverse solver solved all 128. A simple heuristic solved 87. Those controls are part of what makes the result useful: the adapter learned a much better distribution of sequences, while the classical tools still set a higher practical bar.

Everything here concerns computational predictions of short RNA secondary structures. I did not synthesize molecules, measure their folds, or establish a biological function. The original training also has a numerical limitation that I describe in detail below.

What the model actually does#

A convenient way to specify a secondary structure is dot-bracket notation. A dot means an unpaired position. Matching parentheses mean those positions should pair. This small target asks for two stems with loops at their ends:

Requested structure · 32 nucleotides

((((....))))....((((....))))....

Sequence selected from eight model candidates

CGCCUUUCGGCGAAAACCGCACAAGCGGAAAC

Predicted structure · ViennaRNA 2.7.2 at 37°C

((((....))))....((((....))))....

This is an actual exported design from the experiment. Its returned minimum-free-energy structure matches the target. Under the same folding model, the target has a predicted ensemble probability of about 94.7%. That second number matters because a strand can have several plausible folds; an exact match to the returned optimum does not by itself say how much probability the rest of the ensemble receives.

The model gets the target string, its length, and an indexed list of requested pairs. It then chooses one nucleotide at a time. I restrict the action space to four existing tokens and enforce the target length. At a closing parenthesis, the decoder allows only a nucleotide compatible with the opening partner, including GU wobble.

That division of labor is important. The decoder supplies valid letters, valid length, and compatible requested pairs. The learned policy chooses among those legal sequences. The difficult part is making the desired pairing pattern win against competing folds. Every learned condition, including the untouched base model, gets the same constraints.

The folding oracle is ViennaRNA, using its thermodynamic model rather than a learned reward model or a human preference judgment.1Lorenz et al., ViennaRNA Package 2.0 (2011). This experiment used version 2.7.2, Turner 2004 parameters, 37°C, and dangles=2. Credit to the authors and the Institute for Theoretical Chemistry, University of Vienna.

A teacher made of physics#

There is a cheap way to make supervised examples for this task: generate random RNA sequences and fold them forward. Each predicted structure becomes an input, and the sequence that produced it becomes one known answer. The answer is generally not unique, which is exactly why reinforcement learning is interesting here.

1. Make examples. Fold random sequences on a CPU and retain 6,000 unique training targets with one or two hairpins.
2. Learn a starting policy. Run 800 supervised steps on rank-16 adapters, keeping the pretrained model frozen.
3. Reward better designs. Run two RL seeds for 300 steps each. Each step compares eight proposals for each of two targets.
4. Freeze and test. Select checkpoints on development reward, then evaluate held-out targets and classical controls.

The trainable portion is 15.3 million parameters in attention adapters. Supervised fine-tuning teaches the model to reproduce the forward-folded examples. For RL, each candidate receives a score with three terms:

R = 0.5 × pair F1
+ 0.4 × (1 − normalized ensemble defect)
+ 0.1 × exact-fold agreement

Pair F1 rewards partial progress toward the requested pairs. Ensemble defect penalizes expected mistakes across the folding ensemble. The final term rewards an exact match to the returned optimum.

Within each group of eight, the code centers and scales the rewards, then makes one clipped policy-gradient update. There is no KL penalty. This is the one-update, beta-zero case of group-relative policy optimization. Each RL run samples 4,800 sequences in total. The two seeds share the same supervised starting checkpoint, so they are two RL repeats, not two independent replications of the whole pipeline.

Using supervision followed by RL for RNA design is an established approach. Gautam and colleagues describe it in Designing RNAs with Language Models. My experiment asks how much of that idea I could make work in a small, inspectable setting, with a general pretrained model, short synthetic targets, and a strict rental budget.2Gautam et al., Designing RNAs with Language Models (2026). The overall SFT-plus-RL method is prior work. This study is not a reproduction of that paper’s benchmark claims.

The gain, and the controls#

I split the synthetic targets by exact dot-bracket structure before training. Training and the familiar test set contain lengths 32, 48, and 64 with at most two hairpins. The harder set contains 128 length-64 structures with at least three hairpins. The forward-folded source sequence proves each target is designable under the primary oracle, but inference never receives that sequence.

The main endpoint is simple: did any of the eight sampled sequences produce exactly the requested structure? All four frozen model conditions were evaluated using the same full-precision inference procedure.

The improvement over supervised training is large on both sets. RL17, the predeclared primary seed, gains 59.0 percentage points on familiar targets and 80.5 points on complex targets. Paired bootstrap intervals over targets are [52.7, 64.8] and [73.4, 86.7] points. Those intervals describe these frozen runs and these target collections; they do not include uncertainty from rerunning every stage of training.

Part of the apparent difficulty gap comes from length: the complex set is entirely length 64. Restricting the familiar set to its 98 length-64 targets leaves a gap of 93.9% versus 85.9% for RL17. That comparison makes the length difference less of an explanation, although hairpin count is not the only structural property that can differ.

The cheap heuristic is worth dwelling on. It favors G and C in paired positions and A in unpaired positions. With eight attempts, it solves 86.7% of familiar targets and 68.0% of complex targets. A random compatible sequence does much worse. There is substantial structure here that a small amount of domain knowledge exploits immediately.

The RL outputs also become more GC-heavy at paired positions and more A-heavy at unpaired positions. On the complex set, their respective fractions move from about 56% and 31% after supervision to 75% and 76% after RL17. That is a descriptive observation about the sequences. It is not evidence that I have identified the model’s algorithm, or that the model has learned a general theory of RNA folding.

Even so, the learned policy and the heuristic do not solve exactly the same targets:

Targets	Heuristic solves	Heuristic misses
RL17 solves	83	27
RL17 misses	4	14

RNAinverse remains stronger on exact success. Its CPU baseline runs were also much shorter than the model evaluations, although the machines, algorithms, and search budgets differ. The result supports an improvement from post-training. It does not establish that an 8B language model is the best tool for these small design problems.

Eight chances to get it right#

“Solved within eight attempts” can hide a bad first guess. Here, RL17’s first candidate alone solves 49.2% of the complex targets, compared with 2.3% for supervision and 25.8% for the heuristic. Taking the first four saved candidates raises RL17 to 81.2%; taking all eight raises it to 85.9%.

This is one reason to keep every candidate instead of saving only the winner. The same ledger can answer whether training improved the first proposal, whether retries helped, and how much of the headline result depends on selection. It also preserves the failed attempts for inspection.

Change the judge#

Training and evaluating with the same folding program leaves an obvious concern: perhaps the policy mainly learns what that particular program rewards. I checked the selected designs at a different temperature, with an older parameter set, and with EternaFold, a different computational predictor.3EternaFold, pinned at revision 87b9aac. Its default maximum-expected-accuracy prediction differs from ViennaRNA’s minimum-free-energy objective, so disagreement includes an objective change as well as a model change.

For this check, I took the one sequence already selected by the primary reward for each target. I did not give each new predictor eight chances to choose its own favorite.

The improvement over the supervised model survives, but exact success falls under the second predictor. RL17 drops from 85.9% to 68.0% on the complex set; RNAinverse drops from 100% to 75.0%. Agreement across computational models is useful evidence about sensitivity. It still does not tell me how a physical molecule will behave.

I also tried a small public challenge set: the 30 Eterna100-V2 puzzles of length at most 80. That cutoff was fixed before test scoring, and the targets have no exact overlap with the fine-tuning data. Both RL seeds solved 15, the supervised model solved six, and RNAinverse solved 20.4The Eterna100 benchmark repository supplies the public puzzles and benchmark definitions. This subset uses ViennaRNA 2.7.2, not the official full benchmark’s specified setup.

The public targets are a useful counterweight to the very high synthetic scores. They expose much more room for improvement. Their upstream reference solutions were used only to verify compatibility with our folding oracle, never for training or initializing a candidate.

Look at the failures#

The opening illustration is the most dramatic improvement in the complex set. It deserves that label. Across the two primary test sets, RL17 still misses 31 targets within eight samples: 13 familiar and 18 complex.

The viewer below uses saved candidates. You can compare the opening success, the first complex failure by target ID, and a target the heuristic solves but RL17 misses. For each method, it shows the candidate selected by the original reward. The paired arcs make a small but consequential structural error easier to see than a long string of parentheses.

This is a viewer of frozen experimental outputs. It does not run a model or a folding simulation in your browser. Download all 31 RL17 failures, including sequences and predicted folds.

A method that improves the average can still lose on a particular target. Keeping those cases makes the next experiment more specific. For example, the heuristic-only successes suggest testing whether the policy needs more diverse proposals, better supervision on particular motifs, or simply a different way to select candidates. Those are hypotheses for follow-up, not conclusions from this viewer.

The numerical surprise#

The most instructive implementation problem had nothing specifically to do with RNA. During generation, the model uses a cache and produces one token at a time. During the policy update, it evaluates a completed sequence in a teacher-forced forward pass. For the same parameters and prefix, the constrained action probabilities should agree closely.

They did not always agree closely enough in BF16. An initial pilot failed the probability-consistency check and stopped before optimization. Promoting the four-action output projection to FP32 reduced the discrepancy and allowed training to proceed. A later audit of the frozen RL17 checkpoint failed again.

Inference precision	Maximum log-probability discrepancy
BF16 base, FP32 action projection	0.19245
Full FP32, TF32 disabled	0.00003481

I froze the training record and repeated final inference for every model in full precision. No RL test scores had been produced when that decision was made, and I did not change checkpoints or training choices. Earlier BF16 base-model and supervised evaluations had already been seen; those attempts remain in the evidence record but are excluded from the primary tables.

That repair makes the final comparison much better controlled. It does not retroactively repair training. A policy-gradient update compares probabilities from the rollout and the training pass; disagreement between those paths can perturb the importance ratios. The results establish what these particular frozen adapters do under verified final inference. They are weaker evidence about a numerically exact implementation of the intended RL procedure.

I independently rescored all 13,008 final model candidates on the local CPU and checked the exported adapter hashes. Those checks support the reported sequence outcomes. They address a different question from whether the training arithmetic was ideal.

What $3.72 bought#

I gave the project a $100 GPU ceiling. The completed rental occupied one A100 80 GB for about 2.08 hours, measured from creation through confirmed deletion, at the quoted rate of $1.79 per hour. That comes to approximately $3.72 in GPU rental.

That estimate includes setup, downloads, failed pilots, supervised training, both RL runs, numerical investigations, inference reruns, evaluation, and export. Storage was tracked separately with a five-cent planning allowance. The provider had posted only part of the bill when the record was captured, so $3.72 is a time-based rental estimate, not a finalized invoice.

The small price comes with a very specific scope. The pretrained 8B model already exists. Its pretraining cost is not included. Data generation and folding analysis ran on a local CPU. The sequences are short, the adapters are small, the runs are brief, and there is no paid teacher API. This is a measured cost for one completed study, not a promised price for every reproduction.

It was enough to exercise a substantial part of a post-training workflow: a constrained action space, an external reward, a supervised starting policy, two RL seeds, checkpoint selection, held-out evaluation, classical baselines, predictor sensitivity, and an audit that caught a real implementation issue.

Technical recipe and reproducibility notes

Base model

Qwen/Qwen3-8B, revision b968826d9c46dd6066d109eabc6255188de91218.

Adapters

Rank 16, alpha 32, attention q/k/v/o projections; 15,335,424 trainable parameters.

Supervision

One seed, 800 steps, batch 16, learning rate 1e−4. 6,000-target pool; 5,281 unique targets sampled in 12,800 draws.

RL

Seeds 17 and 29; 300 steps each; two targets × eight sequences per step; learning rate 5e−6; one clipped update per rollout; no KL penalty.

Selection

Fixed 128-target development set. Checkpoints frozen before final held-out inference.

Main tests

256 familiar structures and 128 more complex structures. All exact dot-bracket targets are separated across primary splits.

Inference

Temperature 1, eight constrained samples per target; FP32 model computation with TF32 disabled for the final reported results.

Evidence

Candidate ledgers, sealed data hashes, source snapshots, protocol amendments, training logs, and adapter checksums retained.

What I would do next#

The first follow-up I want is a precision-controlled repeat of both RL seeds from the existing supervised checkpoint. Sampling and training should use probability-evaluation paths that agree closely throughout the run. I would publish that as a separate follow-up, including an unfavorable result, because these test sets have now been examined. It is proposed work; it has not been run.

After that, a stronger experiment would ask whether a smaller policy can capture the useful behavior, whether a better simple heuristic closes the remaining gap, and which structural families cause the failures. Those questions would tell me more than making the language model larger and collecting another impressive-looking score.

What I like most about this project is the feedback loop. A target starts as a little drawing encoded in parentheses. The model turns it into letters. A simulator turns the letters back into a shape. The distance between those two shapes becomes a signal the model can learn from, and an artifact I can inspect when it fails.

For roughly four dollars of GPU rental, I got a working version of that loop, a large improvement over the supervised starting point, and a clear view of where the claim ends. That is a good experiment to build the next one on.

Code, protocol, and evidence repository
All chart data · Figure source hashes · Complete failure table

This article describes the completed October 18, 2026 experiment. The precision follow-up is proposed, not completed. The classical baselines, predictor limitations, cost qualifications, and training-precision disclosure are part of the result.