neural networks can't figure out Fourier transforms?

neural-network fft dft
Brannon picture Brannon · Apr 26, 2012 · Viewed 10.1k times · Source

I'm trying to understand a few things about neural networks. First, after looking around on the web, it seems that there is no way to compute a (discrete) Fourier transform through a neural network. You can hack it by hard-coding the thing to include the Fourier constants for the transform and then get a decent result. Why can the machine not figure these out by itself?

Answer

hotpaw2 picture hotpaw2 · Apr 27, 2012

A DFT is a linear operator. Some neural networks have a sigmoid, RLU, or other non-linear element in the computation path, which might make it harder to simulate a linear operator closely enough.

Added: A full DFT is an N by N matrix multiplication. A neural net has to be big enough to represent that many multiplications (at minimum O(NlogN)).

Related questions

What is the role of the bias in neural networks?

I'm aware of the gradient descent and the back-propagation algorithm. What I don't get is: when is using a bias important and how do you use it? For example, when mapping the AND function, when I use 2 inputs and 1 output, …

Read More
Epoch vs Iteration when training neural networks

What is the difference between epoch and iteration when training a multi-layer perceptron?

Read More
Why binary_crossentropy and categorical_crossentropy give different performances for the same problem?

I'm trying to train a CNN to categorize text by topic. When I use binary cross-entropy I get ~80% accuracy, with categorical cross-entropy I get ~50% accuracy. I don't understand why this is. It's a multiclass problem, doesn't that mean that I …

Read More