If you've ever trained a neural network on seasonal time series data only to watch it diverge dramatically when forecasting beyond the training period, you've encountered a fundamental limitation of standard deep learning architectures. Despite their reputation as universal function approximators, conventional neural networks with activation functions like ReLU, tanh, and sigmoid possess a critical weakness: they fail to learn periodic functions effectively, especially when extrapolating beyond the training distribution.

This isn't merely a practical annoyance—it's a theoretical gap with significant implications for fields as diverse as meteorology, finance, and signal processing. In their 2020 NeurIPS paper, Ziyin et al. tackle this problem head-on, identifying the root cause and proposing an elegant solution that challenges our default choices in activation functions .

At first glance, the problem seems counterintuitive. After all, the universal approximation theorem suggests that neural networks should represent any function given sufficient capacity. The catch lies in the word "represent"—while large networks might approximate periodic functions within the training range, their extrapolation behavior remains fundamentally ill-suited for periodic patterns. As we'll see, this isn't a simple optimization issue but rather a consequence of the inductive biases baked into common activation functions.

The Extrapolation Problem: Why Standard Networks Fail

The Theoretical Evidence

Ziyin et al. begin with a rigorous examination of the extrapolation properties of common activation functions . Their findings reveal a consistent pattern:

• ReLU and its variants extrapolate linearly as inputs move far from the training distribution

• Tanh and sigmoid networks converge to constant values during extrapolation

• All standard activations lack the necessary inductive bias for periodic behavior

Figure 1: Exploration of how different activation functions extrapolate various basic function types: \(y = x\) (first column), \(y = \tanh(x)\) (second column), \(y = \sin(x)\) (third column), and \(y = x^2\) (last column). The red curves represents the median model prediction and the shaded regions show the 90% credibility interval from 21 independent runs. Note that the horizontal range is re-scaled so that the training data lies between -1 and 1.

The mathematics behind this is revealing. The authors prove that for a neural network with ReLU activations, the limiting behavior becomes linear: \(f(z \cdot \mathbf{u}) \approx z \cdot \mathbf{W} \cdot \mathbf{u} + \mathbf{b}\) for large \(z\) . Similarly, bounded functions like tanh and sigmoid lead to constant extrapolation: \(f(z \cdot \mathbf{u}) \approx \text{Const}\). Neither linear nor constant behavior captures the essence of periodic functions, which by definition oscillate indefinitely.

This theoretical analysis explains why—no matter how carefully we train our networks—they default to these simplistic extrapolation patterns. The activation function itself dictates the network's asymptotic behavior, creating a fundamental mismatch with periodic data.

Practical Demonstration

Consider a simple experiment: training a standard feedforward network with ReLU activations to approximate a sine wave. Within the training range, the network appears successful, but venture beyond those bounds, and the truth becomes apparent:

The output of our ReLU network turns into a line the further we go away from our dataset .

This isn't an isolated case—it's the expected behavior dictated by the activation function's properties. The same pattern emerges across domains, from financial forecasting to temperature prediction .

Table: Extrapolation Behavior of Common Activation Functions

The Snake Activation Function: A Periodic Fix

Introduction to Snake

The core contribution of Ziyin et al. is the snake activation function, defined as:

\(f(x) = x + \sin^2(x)\)

A parameterized version is also offered: \(\mathrm{Snake}_a(x) = x + \frac{\sin^2(ax)}{a}\).

At first glance, snake might appear to be a simple modification, but its design is deliberate. The "x" term preserves the favorable optimization properties of ReLU-like functions—specifically, avoiding the vanishing gradient problem that plagues pure periodic activations. The \(\sin^2(x)\) term introduces the crucial periodic inductive bias that enables learning oscillatory patterns.

This combination creates a clever balance: the linear component ensures the network can still represent non-periodic trends (common in real-world data), while the periodic component captures cyclical patterns. The result is an activation that learns when to be periodic rather than forcing periodicity everywhere.

Why x + sin²(x) and not sin(x) or x + sin(x)?

A natural question is why the authors chose the specific form \(x + \sin^2(x)\) instead of a simpler periodic function like \(\sin(x)\) or a more direct combination like \(x + \sin(x)\).

The pure \(\sin(x)\) activation is problematic for optimization. Its derivative, \(\cos(x)\), is zero at regular intervals (e.g., \(x = \pi/2, 3\pi/2, ...\)), which can lead to vanishing gradients and make training unstable or impossible for deep networks. Furthermore, a network built solely from \(\sin(x)\) units is a bounded function, which prevents it from modeling any underlying trend in the data: real-world time series are rarely purely periodic.

The alternative, \(x + \sin(x)\), seems more promising as it combines a linear trend with a periodic component. However, its derivative is \(1 + \cos(x)\), which can still be zero (specifically, when \(\cos(x) = -1\)). These points of zero gradient can again stall learning.

The \(\sin^2(x)\) term in the Snake function is the key innovation. Using the trigonometric identity \(\sin^2(x) = (1 - \cos(2x))/2\), we can see that its derivative is \(\sin(2x)\). Therefore, the derivative of the full Snake function is: \[f'(x) = 1 + \sin(2x)\]

This derivative is always non-negative and, crucially, never zero for long stretches. Its value oscillates between 0 and 2, but the constant 1 from the linear term ensures that the gradient has a persistent, non-vanishing component. This provides the stable, ReLU-like gradient flow needed for reliable optimization while simultaneously injecting the periodic inductive bias. This careful balance is what makes Snake both trainable and powerful for periodic tasks.

By the law of large numbers, preactivations are roughly normally distributed, making values near 0 common. The activation \(x + \sin(x)\) is inferior to \(x + \sin^2(x)\) around 0 because its Taylor expansion,

\[ x + \sin(x) = 2x - \frac{x^3}{6} + \cdots \]

\[ x + \sin^2(x) = x + x^2 - \frac{x^4}{3} + \cdots, \]

shows that its first non-linear term is only third-order. In contrast, \(x + \sin^2(x)\) has a non-vanishing second-order term, which more effectively drives non-linear learning.

Why Snake Works Where Others Fail

The genius of snake lies in how it addresses the optimization challenges that made pure periodic activations impractical:

1. Non-zero derivatives: Unlike pure sine functions which can have zero gradients across entire regions, snake's derivative \(f'(x) = 1 + \sin(2x)\) always maintains a non-zero component, ensuring continuous gradient flow during training .

2. Progressive periodicity: The parameter \(a\) in \(\mathrm{Snake}_a(x)\) controls the frequency, allowing networks to learn the appropriate periodicity for the data rather than requiring this to be pre-specified .

3. Compatible optimization: Snake networks optimize similarly to ReLU networks on standard benchmarks like MNIST and CIFAR-10, demonstrating they don't sacrifice general performance for periodic capability .

As the authors note, snake provides the "periodic inductive bias to learn a periodic function while maintaining a favorable optimization property of the ReLU-based activations" .

Theoretical Foundations: The Why Behind the What

The Extrapolation Theorem

The paper establishes a formal foundation through what we might call the Extrapolation Theorem: the asymptotic behavior of a neural network is determined by its activation function . This isn't merely an empirical observation but a provable mathematical property.

For activation functions with linear asymptotes (like ReLU), the network itself behaves linearly far from the training data. For bounded functions (like tanh), the network approaches a constant. Snake's particular functional form—specifically its periodic component—ensures it can maintain oscillatory behavior during extrapolation.

Universal Approximation with Periodic Bias

Snake activation doesn't compromise the fundamental approximation capabilities of neural networks. The authors demonstrate that snake networks maintain the universal approximation property, similar to conventional activation functions .

However, they gain an additional advantage: a Fourier-like flexibility in representing periodic functions without requiring exponentially large networks. This makes them particularly parameter-efficient for domains with inherent periodicity.

Experimental Evidence: Snake in Action

Synthetic Data Validation

The paper first validates snake on controlled synthetic experiments, demonstrating its capability to learn simple periodic functions like sine waves where conventional activations fail . These experiments cleanly isolate the periodic learning problem from other confounding factors.

Real-World Applications

Beyond synthetic tests, snake demonstrates compelling performance on practical problems:

• Financial data prediction: Snake networks effectively capture market seasonality and cyclical trends

• Temperature forecasting: Meteorological data with daily and yearly cycles represents an ideal use case

• Time series with trends: The additive structure of snake naturally handles series with both trend and seasonal components

Figure 2: Experiment on the atmospheric data. (a) Regressing the mean weekly temperature evolution of Minamitorishima with different activation functions. For Snake, \(a\)is treated as a learnable parameter and the red contour shows the 90% credibility interval. (b) Comparison of tanh, ReLU, and Snake on a regression task with learnable \(a\).

Figure 3: Prediction of Wilshire 5000 index, an indicator of the US and global economy.

In one striking example, the authors show that fitting "a superposition of two periodic signals (blue) with a simple recurrent neural network" yields dramatically better predictions with snake compared to other activations .

Performance on Standard Benchmarks

Critically, snake doesn't sacrifice general capability for periodic specialization. The authors validate that snake-based networks achieve competitive performance on standard benchmarks like MNIST and CIFAR-10 . This addresses an important practical concern: snake can serve as a general-purpose activation while excelling on periodic problems.

Practical Implementation and Considerations

Implementing Snake

For practitioners eager to experiment with snake, the implementation is straightforward:

class Snake(nn.Module):
    def __init__(self, a=1.0):
        super().__init__()
        self.a = a

    def forward(self, x):
        return x + torch.sin(self.a * x)**2 / self.a

The parameter \(a\) controls the frequency of the periodic component and can be fixed or learned during training .

Architectural Patterns

The paper explores several architectural strategies:

1. Full conversion: Replacing all activations with snake in a network

2. Hybrid approach: Using snake only in specific layers or components

3. Additive decomposition: Separating trend and seasonality components explicitly

For time series forecasting, the additive trend-seasonality model proves particularly effective:

class AdditiveTrendAndSeasonalityModel(nn.Module):
    def __init__(self):
        super().__init__()

        self.trend_model = nn.Sequential(
            nn.Linear(1, 5),
            nn.ReLU(),
            nn.Linear(5, 1)
        )

        # Seasonality model with snake activation
        self.seasonality_model = nn.Sequential(
            nn.Linear(1, 5),
            Snake(a=1.0),
            nn.Linear(5, 1, bias=False)
        )

    def forward(self, x):
        return self.trend_model(x) + self.seasonality_model(x)

This architecture explicitly separates trend learning (handled by ReLU) from seasonality learning (handled by snake), providing both modeling clarity and performance benefits .

Practical Considerations

Successful application of snake requires attention to several factors:

• Initialization matters: Sine-based networks can be sensitive to weight initialization

• Frequency parameter: The \(a\) parameter in \(\mathrm{Snake}_a(x)\) may need tuning for specific applications. This parameter directly controls the frequency of the periodic component. For data with a known dominant frequency (e.g., daily cycles in temperature), \(a\) can be initialized to a value that reflects this prior knowledge. In the absence of such knowledge, a common and effective strategy is to make \(a\) a learnable parameter. By treating \(a\) as a trainable weight (often initialized to 1.0), the network can automatically discover the most relevant frequency for the task during training, as demonstrated in the paper's experiments on real-world data like the Wilshire 5000 index (fig. 3).

• Overfitting risk: As with any flexible model, snake networks can overfit to spurious periodicities if not properly regularized

For the activation \(x + \frac{\sin^2(ax)}{a}\) under a standard normal distribution, the variance is maximized at \(a_{max} \approx 0.56045\).

To counteract the bias from the non-linear term, authors divide the post-activation by \(\sigma_a\). This correction improves training speed and final accuracy in deep networks like ResNet-101. For standard tasks (e.g., image classification), they recommend \(0.2 \leq a \leq a_{max}\) (default \(a=0.5\)). For tasks with expected periodicity, larger values (\(a=5\) to \(50\)) are more effective.

Broader Implications and Future Directions

Rethinking Activation Functions

The snake activation challenges the conventional wisdom that activation functions should be chosen primarily for their optimization properties. Instead, it demonstrates the value of domain-aware inductive biases in activation design.

This suggests a future where activation functions might be selected or even learned based on the specific characteristics of the problem domain—periodic biases for seasonal data, sparse activations for feature selection, etc.

Applications Beyond Forecasting

While the paper focuses on forecasting applications, the implications extend to numerous domains:

• Signal processing: Periodic patterns abound in audio, communications, and biomedical signals

• Scientific computing: Physical simulations often involve oscillatory phenomena

• Anomaly detection: Deviations from expected periodic patterns can signal problems in cyclical processes

Open Questions and Reproduction

The machine learning community has begun building on this work. A reproducibility study confirmed the paper's central claims, successfully replicating experiments showing that "the proposed snake nonlinearity can learn periodic functions" . However, they noted challenges with certain experiments and initialization details, highlighting opportunities for further refinement.

Independent implementations have emerged across frameworks , making snake more accessible to practitioners.

Conclusion: Embracing Domain-Aware Architectures

The story of snake activation represents more than just a technical improvement—it signals a maturation in deep learning methodology. As the field progresses beyond one-size-fits-all architectures, we're seeing the value of domain-informed inductive biases carefully designed into network components.

Ziyin et al. have identified a genuine limitation in standard neural networks and provided an elegant, practical solution that doesn't compromise general performance. For practitioners working with seasonal data, periodic patterns, or any oscillatory phenomenon, snake deserves a place in your toolkit alongside ReLU and tanh.

Perhaps most importantly, this work reminds us that even in the age of apparently universal function approximators, thoughtful architectural choices based on mathematical understanding and domain knowledge still matter profoundly. The next breakthrough in your application might come not from a larger model, but from a better alignment between your architecture and your problem's fundamental nature.

The paper "Neural Networks Fail to Learn Periodic Functions and How to Fix It" was published at NeurIPS 2020 by Liu Ziyin and colleagues. Code implementations are available in PyTorch and TensorFlow through various community repositories .