Modeling Dependency

Modeling site-site dependencies with heterogeneous Hidden Markov Model and bidirectional LSTM

Read the full thesis | GitHub repository

The Problem: Site-Level Summaries Lose Read-Level Structure

Bisulfite sequencing measures methylation on individual DNA fragments. After the reads are aligned, a conventional analysis compresses them into one methylation level for each CpG site: the fraction of covering reads on which that site is methylated. This site-level profile is convenient, but the compression removes information about which methylation states occur together on the same DNA fragment.

Two distinct read-level methylation patterns with identical methylation levels at every site
Two different read-level patterns produce the same site-level methylation profile.

The example above captures the central problem. Every site has a methylation level of 0.5 in both patterns, yet the biological structures are very different. Pattern 1 is mixed, whereas Pattern 2 contains coordinated blocks of methylated and unmethylated states. Once the reads are reduced to four site-level averages, that distinction is lost.

In probability terms, the usual summary retains each marginal probability $P(Y_t=1)$ but discards joint information such as $P(Y_t,Y_{t+1})$. A model can therefore reproduce every site’s methylation level while generating unrealistic read-level patterns.

Why Independent Sampling Is Not Enough

Most bisulfite-sequencing simulators assign the state of each CpG independently using its site-level methylation probability. This independent Bernoulli model preserves the marginals by construction, but it cannot represent co-methylation—the tendency of nearby CpGs on the same read to share a state.

Box plots comparing dependency statistics in real WGBS data and independent Bernoulli samples
Real WGBS reads and independent Bernoulli samples differ in entropy, mutual information, state correlation, and the probability of adjacent sites sharing the same state.

The Bernoulli samples retain the site-level methylation probabilities used to generate them, yet their dependency statistics remain clearly different from the real reads. This shows that the structure in WGBS data is not merely a consequence of neighboring sites having similar marginal levels; there is genuine site-site dependency that an independent model does not capture.

The modeling task is therefore to generate a binary methylation pattern for each read while satisfying two constraints:

  • the average generated state at each site should match its observed methylation level
  • the generated joint patterns should reproduce how dependency changes with genomic distance and sequence context

Two Complementary Models

Diagrams of independent Bernoulli, heterogeneous HMM, and bidirectional LSTM methylation models
Three approaches to read-level methylation generation: independent Bernoulli sampling, a distance-aware heterogeneous HMM, and a feature-rich bidirectional LSTM.

The heterogeneous Hidden Markov Model makes the transition probability between two CpG states a function of their genomic distance. Nearby sites are encouraged to remain in the same state, and that coupling decays as their distance grows. The model also separates the latent biological state from errors introduced during bisulfite conversion and sequencing. Its main advantage is interpretability: the fitted transition function directly describes the distance-dependent relationship.

The bidirectional LSTM learns dependencies from both directions along a read. In addition to methylation levels and genomic distances, it can use local sequence context and flanking bases. This relaxes the HMM’s first-order, distance-only restriction and allows the model to learn more complex patterns.

Results at a Glance

The simulation study evaluates the same two questions under controlled conditions: whether the generated sequences preserve the target marginal methylation profile and reproduce the known distance-dependent site-site relationship.

Comparison of true and predicted marginal methylation distributions across four models on synthetic data
Marginal distribution preservation across models on synthetic data.
Probability of adjacent sites sharing the same state across synthetic data and four model predictions
Distance-dependent site-site relationships in the synthetic test data and model predictions.

Together, the plots show the central trade-off: matching site-level averages is not the same as preserving their joint structure. The simulation provides a known reference against which both properties can be evaluated directly.

For the model derivations, training procedures, simulation design, numerical comparisons, and complete discussion, see the full thesis on eScholarship.