Video Stitching from Multiple Moving Cameras

Contributions

First method for stitching videos from more than two freely moving cameras.
Regularization-free formulation for joint alignment across views and time.
Annotated multi-view video dataset for unconstrained camera motion.
Benchmark for video stitching from multiple moving cameras.
Variance-based geometric metric for multi-view alignment.

Why Existing Methods Fall Short

Video Stitching: At Most Three Cameras, in a Fixed Order

Most moving-camera video stitching methods are limited to two cameras. Among existing video stitching methods, StabStitch++ is the only one supporting more than two inputs, but it is limited to N = 3 and requires a fixed spatial ordering of the cameras. This assumes that the cameras’ relative ordering remains fixed, whereas in freely moving multi-camera capture the view-intersection graph can change over time. As shown below, when cameras swap order, its output degrades.

Image Stitching: Spatially Coherent, Temporally Inconsistent

In multiple video stitching from freely moving cameras, Multiple Image Stitching (MIS) methods produce good panoramas at individual time steps but yield temporally incoherent results when applied frame by frame. This failure stems from the fact that, due to independent camera motions, different panoramic regions usually move differently between times t and t + 1, which per-frame stitching does not model.

N = 4

N = 5

MIS methods and MViS on MMCD scenes with N = 4 and 5 cameras.

Adding Stabilization Helps, but it is Not Enough

We augment key MIS methods with two stabilization strategies: (1) temporal regularization penalizing changes in estimated transformations between consecutive frames (TR), and (2) post-hoc smoothing of per-frame transformations (PS). Both improve stability over per-frame baselines but still lose to MViS. This is expected: in multiple moving-camera videos, the optimal stabilization strength is not uniform, since different cameras, and hence different panoramic regions, can have different and possibly time-dependent motions.

Stabilized MIS variants and MViS.

Takeaway

This calls for a video-specific formulation that jointly enforces spatial and temporal consistency across all cameras.

Multiple Video Stitching (MViS)

MViS is an unsupervised optimization method that aligns videos from any number of freely moving cameras, with no fixed or known camera ordering and a view-intersection graph that may change over time. It only assumes that this graph is connected at each time and that the videos are approximately synchronized.

MViS turns N videos from moving cameras into one panoramic video. Schematic on a synthetic scene with N = 3, used in the figures below.

The Method at a Very High Level

MViS has three steps. Preprocessing estimates noisy pairwise homographies (ORTs); then a lightweight network, which takes a sliding window of 2N frames and predicts a homography per frame, is optimized at test time, first with a geometric loss (Stage 1) and then with an appearance loss (Stage 2). Both losses compare pairs of frames only, yet every frame lands aligned in a shared panorama.

Preprocessing

First, we match keypoints and estimate pairwise homographies, which we call Observed Relative Transformations (ORTs). These will guide both stages of optimization.

Stage 1: The ORT-guided Geometric Loss

For each pair, Stage 1 maps a regular grid G through both the ORT and the composition of the predicted homographies, and penalizes the distance between the two warped grids.

◆ the grid G mapped by the ORT × the grid G mapped by the composition of the predictions

Multi-view pairs at time t

Temporal pairs between t and t + 1

Identity Converged

In each pair, the composition (×) is pulled toward the ORT (◆).

Stage 2: The Appearance Loss

Stage 2 refines the residual misalignment of Stage 1 with an appearance loss on the same pairs. Frame i is warped into the view of frame j by the composition (H_j)⁻¹H_i and compared with it, only inside the pair’s overlap mask.

Sliding window S_t

Video 1Video 2Video 3

tt+1

Pick a pair:

From the Prediction to a Panoramic Video

After optimization, each frame Fti is warped into the panorama by its homography Hti. Because the losses depend only on relative transformations, the homographies are defined only up to a common global homography, while the correspondences they induce between frames are unique. Rendering therefore needs a reference: video 1 at each time, applying (Ht1)⁻¹ to all frames at time t, or video 1 at the first time, applying (H11)⁻¹ to all frames. This choice affects only visualization; MViS gives video 1 no special weight.

Video 1 Video 2 Video 3

t = 1

Video 1 fixed: the other cameras move around it. Scene fixed: the scene stays in place and all three cameras move over it.

The MMCD Dataset

To our knowledge, no available dataset supports multiple video stitching with N > 2 freely moving cameras. Thus, we created MMCD, a new multiview dataset. MMCD contains 53 scenes, each with 3–6 synchronized handheld videos captured on diverse mobile devices under varied environmental and lighting conditions.

Scenes: 53
Videos: 221
Frames: 47,040
Cameras per scene: 3–6

To facilitate quantitative evaluation, we manually annotated keypoints in the overlap regions of the first frames and propagated them through the sequences using Co-Tracker3.

Very high and very low camera motion

Indoor scene

The Multi-Camera Benchmark

For N > 2, no existing video stitching method handles freely moving cameras, so we built a benchmark on the annotated MMCD dataset. It measures cross-view alignment with GV, our new metric (below); stability with the stability score of keypoint trajectories; and temporal smoothness with the mean square acceleration of video-corner trajectories (CMSA).

Method	`N` = 3				`N` = 4				`N` = 5				`N` = 6
Method	GV	Stab.	CMSA	Succ.	GV	Stab.	CMSA	Succ.	GV	Stab.	CMSA	Succ.	GV	Stab.	CMSA	Succ.
Multiple image stitching (MIS)
AutoStitch	3.3	.847	635	14/22	2.4	.901	217	6/10	3.1	.834	>1k	9/11	3.3	.924	111	5/10
APAP	2.9	.850	>1k	19/22	30.3	.832	>1k	10/10	2.4	.846	>1k	11/11	4.0	.875	>1k	8/10
GSP	2.8	.648	>1k	19/22	2.5	.649	>1k	9/10	2.1	.625	>1k	11/11	3.0	.672	>1k	9/10
SPW	4.8	.807	>1k	22/22	10.4	.716	>1k	10/10	2.5	.835	>1k	10/11	14.8	.745	>1k	10/10
GES-GSP	2.9	.618	>1k	19/22	2.5	.668	>1k	9/10	2.4	.614	>1k	10/11	3.2	.670	>1k	9/10
OBJ-GSP	2.9	.621	>1k	19/22	2.5	.634	>1k	9/10	2.2	.683	>1k	8/11	3.1	.681	>1k	6/10
MIS with temporal regularization (TR)
GSP+TR	3.1	.863	62.9	19/22	2.8	.898	52.2	9/10	3.2	.817	93.6	11/11	3.5	.879	38.1	9/10
GES-GSP+TR	3.3	.849	69.4	19/22	3.1	.884	57.2	9/10	3.8	.800	100	11/11	3.9	.866	39.5	9/10
OBJ-GSP+TR	3.2	.853	75.3	19/22	3.0	.879	71.6	9/10	3.5	.808	106	11/11	3.7	.872	42.0	9/10
MIS with post-hoc smoothing (PS)
GSP+PS	2.7	.816	216	19/22	2.4	.841	251	9/10	2.1	.783	577	11/11	2.8	.867	174	9/10
GES-GSP+PS	2.8	.802	206	19/22	2.5	.853	154	9/10	2.3	.796	644	10/11	3.0	.887	90.7	9/10
OBJ-GSP+PS	2.8	.814	177	19/22	2.4	.836	196	9/10	2.1	.822	73.6	8/11	2.9	.884	101	6/10
Video stitching
StabStitch++	27.8	.799	20.0	22/22	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
MViS (ours)	3.1	.917	2.5	22/22	2.8	.925	14.2	10/10	2.3	.874	11.0	11/11	3.0	.930	2.4	10/10

The multi-camera benchmark on MMCD; N is the number of videos per scene. Stab. (↑), GV (↓) and CMSA (↓) are averaged over the scenes on which each method succeeded without collapsing, counted in Succ.; the best values are in bold. StabStitch++’s high GV on N = 3 reflects severe misalignment in scenes where its assumed camera ordering is violated; its code does not support N > 3.

The Geometric Variance Metric (GV)

GV measures cross-view alignment with the annotated keypoints. At each time t, every keypoint visible in all videos is mapped into the panorama by each video’s transformation T^t,i. GV is computed from the variance of these copies, averaged over keypoints and frames and scaled to the frame size. Lower is better: perfectly aligned views give GV = 0.

Frames at time t, with the same keypoint annotated in each

Panorama: the keypoint’s copies, mapped by T^t,i

Zoom on the keypoint

GV = 0.0

Aligned: one sharp sceneMisaligned: ghosting, copies spread

Note: GV needs no distance threshold and evaluates all views together in a shared space. We believe it is a more informative alternative to PCK (Percentage of Correct Keypoints) for any task that aligns multiple images into a common frame.

More Visual Results of MViS

N = 6

N = 5

N = 4

N = 3

N = 5

MViS outputs for scenes with varying numbers of cameras, all from the MMCD dataset.

BibTeX

If you find MViS, the MMCD dataset, the Multi-Camera Benchmark, or the GV metric useful in your research, please cite:

@inproceedings{Ifergane:NeurIPS:2026:MViS,
  title     = {Video Stitching from Multiple Moving Cameras},
  author    = {Ifergane, Shira and Amoyal, Roy and Benishay, Shahar and Freifeld, Oren},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}