Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun

2015 · NeurIPS

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Problem

Framing

Region-based detectors were accurate but proposal generation still cost seconds per image. Faster R-CNN replaces hand-engineered proposals with an RPN that shares convolutional features with Fast R-CNN, reducing proposal time to about $10\,\mathrm{ms}$ and reaching $73.2\%$ mAP on VOC 2007 with VGG-16.

Currently Used Methods

Foundational

@krizhevskyAlexNet2012 — deep CNN features make large-scale recognition practical.
- Limitation in context: not a detector or proposal mechanism.
"Rich feature hierarchies for accurate object detection and semantic segmentation" — R-CNN scores warped external proposals.
- Limitation in context: proposal generation and per-region CNN passes are slow.
"Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition" — shares convolutional features across regions.
- Limitation in context: still relies on external proposal algorithms.
"Fast R-CNN" — RoI pooling and shared features speed region classification.
- Limitation in context: Selective Search remains the runtime bottleneck.
"OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks" — sliding-window detection with joint classification and regression.
- Limitation in context: one-stage localization trails two-stage accuracy by about $6\%$ mAP.

Proposed Method

Architecture

The model places an RPN on the shared convolutional map and reuses those features for Fast R-CNN detection. A $3 \times 3$ sliding window feeds sibling $1 \times 1$ heads for objectness and box regression over $k=9$ anchors from $3$ scales and $3$ aspect ratios.

Verified figure: three multi-scale schemes, contrasting image pyramids, filter pyramids, and Faster R-CNN's anchor-box reference pyramid.

Loss / Objective

The RPN optimizes joint anchor classification and box regression.

L(\{p_i\}, \{t_i\}) = \frac{1}{N_{\mathrm{cls}}} \sum_i L_{\mathrm{cls}}(p_i, p_i^*) + \lambda \frac{1}{N_{\mathrm{reg}}} \sum_i p_i^* L_{\mathrm{reg}}(t_i, t_i^*)

L_{\mathrm{cls}}(p_i, p_i^*) = - p_i^* \log p_i - (1-p_i^*) \log (1-p_i)

L_{\mathrm{reg}}(t_i, t_i^*) = R(t_i - t_i^*)

Sampling Rule / Algorithm

Anchor labels and proposal pruning are set by IoU thresholds and NMS.

p_i^* = 1 \quad \text{if anchor } i \text{ has highest IoU with a ground-truth box or IoU } > 0.7

p_i^* = 0 \quad \text{if anchor } i \text{ has IoU } < 0.3 \text{ for all ground-truth boxes}

Training Procedure

Anchors per location: $k=9$ .
Anchor scales: $128^2, 256^2, 512^2$ .
Anchor aspect ratios: $1{:}1, 1{:}2, 2{:}1$ .
RPN mini-batch: $256$ anchors per image.
Positive fraction: up to $128$ positives.
Proposal NMS IoU: $0.7$ .
Proposals after NMS: $2000$ train, $300$ test.
Feature-sharing optimization: 4-step alternating training.

Evaluation

Datasets

PASCAL VOC 2007
PASCAL VOC 2012
MS COCO

Metrics

mAP on VOC
mAP@0.5 on COCO
mAP@[0.5,0.95] on COCO
Proposal time
End-to-end inference rate

Headline results

VOC 2007 test (VGG-16, 07+12): $73.2\%$ mAP.
VOC 2007 test (VGG-16, 07): $69.9\%$ mAP.
VOC 2012 test (VGG-16, 07++12): $70.4\%$ mAP.
COCO test-dev (VGG-16): $42.7\%$ mAP@0.5.
VGG-16 runtime: about $5$ fps end-to-end; proposals cost about $10\,\mathrm{ms}$ .

Table 1: Proposal methods under Fast R-CNN on VOC 2007 test set

train-time region proposals method	# boxes	test-time region proposals method	# proposals	mAP (%)
SS	2000	SS	2000	58.7
EB	2000	EB	2000	58.6
RPN+ZF, shared	2000	RPN+ZF, shared	300	59.9

Ablations

Anchor design: $3$ scales and $3$ aspect ratios reaches $69.9\%$ mAP.
Single-anchor setting: mAP drops by $8.8$ points.
Proposal count: top $100$ RPN proposals already remain competitive.
Shared features: proposal quality improves from RPN+ZF to RPN+VGG.

Method Strengths and Weaknesses

Strengths

Eliminates external proposals, reducing proposal time to about $10\,\mathrm{ms}$ .
Shared convolutional features preserve accuracy while removing duplicated compute.
Reaches $73.2\%$ mAP on VOC 2007 with VGG-16.
High-quality proposals need only $300$ boxes at test time.

Weaknesses

Four-step alternating training is cumbersome versus true joint optimization.
Performance depends on anchor scales, ratios, and IoU heuristics.
VGG-16 inference is only about $5$ fps.
Two-stage design is less direct than single-stage detectors.

Suggestions from the authors

Replace alternating optimization with approximate joint training.
Improve proposal quality with deeper shared backbones.
Extend learned proposal mechanisms beyond object detection.
Handle wider scale and aspect-ratio variation with stronger proposal designs.

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Problem

Framing

Currently Used Methods

Foundational

Proposed Method

Architecture

Loss / Objective

Sampling Rule / Algorithm

Training Procedure

Evaluation

Datasets

Metrics

Headline results

Ablations

Method Strengths and Weaknesses

Strengths

Weaknesses

Suggestions from the authors

Links

Prior Papers

Further Papers