Learning Representations by Back-propagating Errors

Problem

Framing

Single-layer perceptrons could not learn internal features for non-linearly separable structure. The paper closes this gap with multilayer gradient descent that propagates output error through hidden units, enabling distributed representations and tasks like XOR and structured family-role inference.

Currently Used Methods

Foundational

• @rosenblattPerceptron1958 — single-layer perceptron learns linear discriminants.
  • Limitation in context: no hidden layers, so XOR and compositional structure remain unreachable.

Direct antecedents

• "Learning internal representations by error propagation" — early multilayer error-propagation idea.
  • Limitation in context: lacked the clear, practical derivation that made deep supervised learning usable.
• "Parallel distributed processing: Explorations in the microstructure of cognition" — distributed representation program for hidden features.
  • Limitation in context: needed an efficient rule to train internal units from task loss.

Proposed Method

Architecture

The paper studies feedforward multilayer networks with differentiable hidden and output units. Learning adjusts every weight by chaining local derivatives from output loss back to earlier layers.

Evaluation

Datasets

• XOR toy problem.
• Family-tree relational patterns over named individuals.

Metrics

• Qualitative task success.
• Emergent hidden-unit representation structure.

Headline results

• XOR: multilayer backprop learns a solution unreachable by a perceptron.
• Family relations: hidden units encode reusable latent structure across individuals.
• Internal representations: similar roles cluster through shared hidden activations.

Method Strengths and Weaknesses

Strengths

• Solves XOR by learning hidden features, not hand-crafting them.
• Gives a general gradient rule for all network weights.
• Learns distributed internal codes useful across related inputs.

Weaknesses

• No large-scale benchmark or standardized quantitative metric.
• Relies on differentiable activations and supervised targets.
• Optimization details are sparse by modern standards.

Suggestions from the authors

• Test richer multilayer hierarchies for more complex structured domains.
• Study the kinds of internal representations hidden units discover.
• Extend the method to broader cognitive and perception tasks.

Links

Prior Papers

• @rosenblattPerceptron1958 — establishes the linear-threshold baseline that backpropagation decisively surpasses.

Further Papers

• @hintonDeepBeliefNets2006 — revives multilayer representation learning at scale after the original backpropagation era.
• @hochreiterLSTM1997 — extends gradient-based learning to recurrent networks with long-range credit assignment.
• @bengioRepresentationLearning2013 — synthesizes the representation-learning agenda that this paper helped launch.