Hyena Model (deep learning)

The Hyena model is a neural network architecture that was developed to address the scalability issues associated with traditional self‐attention mechanisms. It is designed to efficiently handle very long sequences by replacing the quadratic-complexity self‐attention with a sub-quadratic operator that interleaves implicit long convolutions with data-controlled gating.

Architecture

At the core of the Hyena model is the concept of implicit long convolutions. Traditional convolutions use fixed kernels that are explicitly defined and stored, resulting in a parameter count that scales linearly with the kernel size. In contrast, Hyena generates convolutional filters implicitly using a parameterized function—typically implemented as a small feed-forward network. This allows the model to synthesize long filters on the fly, effectively decoupling the filter length from the number of parameters.

In addition to implicit convolutions, the Hyena operator incorporates data-controlled multiplicative gating. In this mechanism, each token is modulated by gating signals that are derived from learned linear projections of the input. The gating operation is performed element-wise and serves to dynamically adjust the influence of the convolutional output, effectively tailoring the operator to the specific input context.

The overall Hyena operator is defined as a recurrence that alternates between implicit long convolutions and element-wise gating. For an order-N Hyena operator, the recurrence is expressed as follows:

1. $z_{1}[t]=v[t]$, where $v$ is one of the linear projections of the input.
2. For $n=1,\dots ,N$:
   • $z_{n+1}[t]=x_{n}[t]\cdot {\Bigl (}(h_{n}\ast z_{n})[t]{\Bigr )}$, where $x_{n}$ represents a gating projection and $h_{n}$ is an implicitly parameterized long convolution filter.
3. The final output is given by $y[t]=z_{N+1}[t]$.

, where

• $z_{n}[t]$ is the intermediate state at recurrence step $n$ and time position $t$.
• $v[t]$ is a linear projection of the input at time position $t$, analogous to the "value" in self-attention.
• $x_{n}[t]$ is the gating projection at recurrence step $n$.
• $h_{n}$ is the implicit long convolution filter for step $n$.
• The operator $*$ denotes convolution, so $(h_{n}*z_{n})[t]$ is the result of convolving filter $h_{n}$ with the signal $z_{n}$ at time $t$.
• The dot "$\cdot$" indicates element-wise multiplication.

Mathematical Formulation

The implicit convolution filters in Hyena are typically parameterized as functions of time. For each filter $h_{n}$, the response at time is given by:

$h_{n}[t]={\text{Window}}(t)\cdot ({\text{FFN}}\circ {\text{PositionalEncoding}})(t)$

, where $\circ$ is the composition operator, meaning that the positional encoding is first applied to $t$ and then processed by the FFN.

Here, the window function serves to modulate the filter (for example, by imposing an exponential decay), and the feed-forward network (FFN) together with positional encodings generate the filter values. This implicit parameterization is a key design choice that allows Hyena to capture long-range dependencies without a proportional increase in parameter count.

Efficiency and scalability

By replacing the quadratic self-attention mechanism with a sequence of FFT-based convolutions and element-wise multiplications, the Hyena operator achieves an overall time complexity of $O(NL\log L)$, where $N$ is the number of recurrence steps. This subquadratic scaling is particularly advantageous for long sequences, allowing the model to process inputs that are orders of magnitude longer than those feasible with conventional attention.

The operations in the Hyena model—both the implicit convolutions and the gating functions—are highly parallelizable and amenable to optimization on modern hardware accelerators. Techniques such as fast Fourier transforms (FFT) further enhance the efficiency, making the model well-suited for large-scale applications where both speed and memory efficiency are critical.