Technology

Neural networks: how they really work

By Recaplica Lab · Updated on

The Recap 30 seconds

A neural network is a system of thousands of simple computing units connected in layers by millions (or billions) of weighted connections, loosely inspired by the brain's neurons. Every connection has an adjustable weight: during training the network tweaks the weights until the examples produce the right answers. With many layers it becomes deep learning, the technology behind image recognition, speech and large language models.

Key Points

  • The network is made of nodes (artificial neurons) organized in layers: input, hidden layers, output.
  • Each connection has a weight: the node sums its weighted inputs and 'fires' if the total passes a threshold.
  • Weights start out random: training adjusts them little by little on the examples.
  • Deep learning = networks with many layers: each layer detects increasingly abstract features.
  • The idea dates to 1944 (McCullough and Pitts); the first trainable model is the 1957 Perceptron.
  • The modern breakthrough came with GPUs, which made huge networks trainable.

Key figures

  • 1957 the year of Frank Rosenblatt's Perceptron, the first trainable neural network Source: MIT News
  • 1944 the first theoretical proposal of a neural network, by McCullough and Pitts Source: MIT News
  • 175bn the weights (parameters) of GPT-3: the scale reached by modern networks Source: arXiv, 2020

Deep Dive

An orchestra of tiny calculators

Forget the electronic brain of the movies. A neural network is something humbler and more surprising: thousands of dead-simple computing units, connected in layers by millions (in modern models, billions) of weighted connections. As MIT explains, each node does one thing: it receives numbers over its incoming connections, multiplies them by their weights, adds everything up and compares the total against a threshold. Above the threshold, the node “fires” and passes the signal to the next layer; below it, silence.

None of these units knows anything. The intelligence — if we want to call it that — lives entirely in the weights: millions, nowadays billions, of dials deciding how much every connection counts.

Training: tuning the dials

Here’s the part that surprises everyone who first learns it: at the start, the weights are random numbers. A newborn network can do literally nothing.

Then the studying begins. It’s shown an example — a photo labeled “cat” — and the output layer’s answer is checked. Wrong? A correction algorithm nudges the weights, ever so slightly, in the direction that would have made the answer less wrong. Repeat with the second example, the thousandth, the millionth. Micro-nudge after micro-nudge, the network comes to give consistently right answers even on inputs it has never seen.

Practical example: it’s like tuning a thousand-string instrument by ear. You play a test note, hear it’s off, slightly turn the pegs that matter most, try again. After millions of tries the instrument plays in tune — though nobody ever wrote down the tuning score.

Why “deep”: the power of layers

With few layers a network learns simple relationships. The real leap comes as the layers multiply — that’s the deep in deep learning. In an image network, the early layers learn to catch edges and contrasts; the middle ones combine edges into shapes; the last ones combine shapes into objects. Nobody designs this hierarchy of abstractions: it emerges on its own from training.

The same principle, applied to text at colossal scale, produces the models behind generative AI: GPT-3 is a network of 175 billion weights. The dials have multiplied beyond counting; the principle is still 1957’s.

A rollercoaster history

Yes, 1957: neural networks are anything but new. The theory was born in 1944 with Warren McCullough and Walter Pitts; in 1957 Frank Rosenblatt unveiled the Perceptron, the first trainable network. Then an influential critique (Minsky and Papert, 1969) froze funding for years; the 1980s brought a renaissance, the 2000s another winter as other methods took over.

The definitive turning point came with GPUs, the graphics cards: built for video games, they turned out to be perfect for the billions of parallel multiplications training requires. From there: networks tens of layers deep, internet-scale data, and deep learning leaving the labs for your phone.

To place the piece in the puzzle: neural networks are the most powerful engine of machine learning, and what a network executes is still an algorithm, with all its well-defined steps. The network doesn’t know it’s intelligent. It executes, and that’s enough.

Common myths

  • ✗ Myth Neural networks work like the human brain.

    ✓ Reality The inspiration is biological, the mechanics aren't: an artificial neuron is a simple mathematical operation, and training doesn't resemble how a brain learns. The metaphor aids intuition, but it's a distant kinship, not a copy.

  • ✗ Myth A neural network understands what it processes.

    ✓ Reality The network turns numbers into numbers through weights tuned on examples. It recognizes a cat without knowing what a cat is: it maps pixel patterns to a label, often with superb accuracy and zero comprehension.

  • ✗ Myth Neural networks are a recent invention.

    ✓ Reality The theory is over eighty years old (1944) and the first trainable model dates to 1957. What's recent is the computing power: GPUs made it feasible to train deep networks on huge data, waking up a field that had repeatedly gone dormant.

Concept map

Drag the background to move around and the nodes to reposition them; use − and + to collapse and expand branches.

Test yourself

Answer the questions to check what you have learned: you get instant feedback and a short explanation.

Grade 0/10 0/5
1 How is a neural network organized?
2 What are weights?
3 How do the weights start out before training?
4 What does 'deep' mean in deep learning?
5 True or false: the 1957 Perceptron was the first trainable neural network.

Flashcards

Tap the card to flip it and check whether you remember the answer, then move to the next one.

1 / 7

Explain it in your own words

The ultimate test: if you can explain it in simple words, you've truly understood it. Write your explanation, then compare it with the Recap.

Your explanation is saved only on this device.

FAQ

Why call them 'neural' if they don't work like the brain?

For the original inspiration: the 1940s pioneers modeled units that activate past a threshold, like biological neurons. The name stuck; the resemblance to the brain stayed superficial.

What does each layer of a deep network do?

It detects features at a growing level of abstraction: in an image network, early layers catch edges and contrasts, middle ones shapes and textures, the last ones whole objects. Nobody programs this hierarchy: it emerges from training.

What's the link between neural networks and ChatGPT?

Large language models are gigantic neural networks trained on text: GPT-3, for instance, is a network with 175 billion weights. The same basic architecture, at industrial scale.

Why were GPUs the turning point?

Training a network takes billions of repetitive multiplications — exactly the kind of math graphics cards were born for. With GPUs it became practical to train networks tens of layers deep on huge datasets: hence the deep learning explosion.