Depth Anything 3:
Recovering the Visual Space from Any Views

Predicting a valid rotation matrix $\mathbf{R}_{i}$ is challenging due to the orthogonality constraint. To avoid this, we represent camera pose implicitly with a per-pixel ray map, aligned with the input image and depth map. For each pixel $\mathbf{p}$, the camera ray $\mathbf{r}\in\mathbb{R}^{6}$ is defined by its origin $\mathbf{t}\in\mathbb{R}^{3}$ and direction $\mathbf{d}\in\mathbb{R}^{3}$: $\mathbf{r}=(\mathbf{t},\mathbf{d})$. The direction is obtained by backprojecting $\mathbf{p}$ into the camera frame and rotating it to the world frame: $\mathbf{d}=\mathbf{R}\mathbf{K}^{-1}\mathbf{p}.$ The dense ray map $\mathbf{M}\in\mathbb{R}^{H\times W\times 6}$ stores these parameters for all pixels. We do not normalize $\mathbf{d}$, so its magnitude preserves the projection scale. Thus, a 3D point in world coordinates is simply ${\mathbf{P}}={\mathbf{t}}+\mathbf{D}(u,v)\cdot\mathbf{d}.$ This formulation enables consistent point cloud generation by combining predicted depth and ray maps through element-wise operations.

Given an input image $\mathbf{I}\in\mathbb{R}^{H\times W\times 3}$, the corresponding ray map is denoted by $\mathbf{M}\in\mathbb{R}^{H\times W\times 6}$. This map comprises per-pixel ray origins, stored in the first three channels ($\mathbf{M}(:,:,:3)$), and ray directions, stored in the last three ($\mathbf{M}(:,:,3:)$).

First, the camera center $\mathbf{t}_{c}$ is estimated by averaging the per-pixel ray origin vectors:

To estimate the rotation $\mathbf{R}$ and intrinsics $\mathbf{K}$, we formulate the problem as finding a homography $\mathbf{H}$. We begin by defining a “identity” camera with an identity intrinsics matrix, $\mathbf{K}_{I}=\mathbf{I}$. For a given pixel $\mathbf{p}$, its corresponding ray direction in this canonical camera’s coordinate system is simply $\mathbf{d}_{I}=\mathbf{K}_{I}^{-1}\mathbf{p}=\mathbf{p}$. The transformation from this canonical ray $\mathbf{d}_{I}$ to the ray direction $\mathbf{d}_{\text{cam}}$ in the target camera’s coordinate system is given by $\mathbf{d}_{\text{cam}}=\mathbf{K}\mathbf{R}\mathbf{d}_{I}$. This establishes a direct homography relationship, $\mathbf{H}=\mathbf{K}\mathbf{R}$, between the two sets of rays. We can then solve for this homography by minimizing the geometric error between the transformed canonical rays and a set of pre-computed target rays, $\mathbf{M}(h,w,3:)$. This leads to the following optimization problem:

This is a standard least-squares problem that can be efficiently solved using the Direct Linear Transform (DLT) algorithm [abdel2015direct]. Once the optimal homography $\mathbf{H}^{*}$ is found, we recover the camera parameters. Since the intrinsic matrix $\mathbf{K}$ is upper-triangular and the rotation matrix $\mathbf{R}$ is orthonormal, we can uniquely decompose $\mathbf{H}^{*}$ using RQ decomposition to obtain $\mathbf{K}$, $\mathbf{R}$.

Recent works aim to build unified models for diverse 3D tasks, often using multitask learning with different targets—for example, point maps alone [dust3r], or redundant combinations of pose, local/global point maps, and depth [wang2025vggt, cut3r, fast3r]. While point maps are insufficient to ensure consistency, redundant targets can improve pose accuracy but often introduce entanglement that compromises it. In contrast, our experiments (Tab.˜6) show that a depth-ray representation forms a minimal yet sufficient target set for capturing both scene structure and camera motion, outperforming alternatives like point maps or more complex outputs. However, recovering camera pose from the ray map at inference is computationally costly. We address this by adding a lightweight camera head, $\mathcal{D}_{C}$. This transformer operates on camera tokens to predict the field of view ($\mathbf{f}\in\mathbb{R}^{2}$), rotation as a quaternion ($\mathbf{q}\in\mathbb{R}^{4}$), and translation ($\mathbf{t}\in\mathbb{R}^{3}$). Since it processes only one token per view, the added cost is negligible.

Following the formulation in Sec.˜3.1, our model $\mathcal{F}_{\theta}$ maps an input $\mathcal{I}$ to a set of outputs comprising a depth map $\hat{\mathbf{D}}$, a ray map $\hat{\mathbf{R}}$, and an optional camera pose ${\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\hat{\mathbf{c}}}$: $\mathcal{F}_{\theta}:\mathcal{I}\mapsto\{\hat{\mathbf{D}},\hat{\mathbf{R}},{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\hat{\mathbf{c}}}\}$. The gray color indicates that ${\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\hat{\mathbf{c}}}$ is an optional output, included primarily for practical convenience. Prior to loss computation, all ground-truth signals are normalized by a common scale factor. This scale is defined as the mean $\ell_{2}$ norm of the valid reprojected point maps $\mathbf{P}$, a step that ensures consistent magnitude across different modalities and stabilizes the training process. The overall training objective is defined as a weighted sum of several terms:

where $D_{c,p}$ denotes the confidence of depth $D_{p}$. All loss terms are based on the $\ell_{1}$ norm, with weights set to $\alpha=1$ and $\beta=1$. The gradient loss, $\mathcal{L}_{\text{grad}}$, penalizes the depth gradients:

where $\nabla_{x}$ and $\nabla_{y}$ are the horizontal and vertical finite difference operators. This loss preserves sharp edges while ensuring smoothness in planar regions. In practice, we set $\alpha=1$ and $\beta=1$.

We provide our training datasets in Table 1. Note that for datasets with potential overlap between training and testing (ScanNet++), we ensure a strict separation at the scene level, i.e., scenes in training and testing are mutually exclusive. Note that using Scannet++ for training is fair to other methods, as it is widely used for training in [wang2025vggt, dust3r].

We train our model on 128 H100 GPUs for 200k steps, using an 8k-step warm-up and a peak learning rate of $2\times 10^{-4}$. The base resolution is $504\times 504$, which is divisible by 2, 3, 4, 6, 9 and 14, making it more compatible with common photo aspect ratios such as 2:3, 3:4, and 9:16. Training image resolutions are randomly sampled from $504\times 504$, $504\times 378$, $504\times 336$, $504\times 280$, $336\times 504$, $896\times 504$, $756\times 504$, $672\times 504$. For the $504\times 504$ resolution, the number of views is sampled uniformly from [2, 18]. The batch size is dynamically adjusted to keep the token count per step approximately constant. Supervision transitions from ground-truth depth to teacher-model labels at 120k steps. Pose conditioning is randomly activated during training with probability 0.2.

We trained our metric depth model on 14 datasets, including Taskonomy [zamir2018taskonomy], DIML (Outdoor) [cho2021diml], DDAD [ddad], Argoverse [wilson2023argoverse], Lyft [], PandaSet [xiao2021pandaset], Waymo [Sun_2020_CVPR], ScanNet++ [yeshwanth2023scannet++], ARKitScenes [baruch2021arkitscenes], Map-free [arnold2022map], DSEC [Gehrig21ral], Driving Stereo [yang2019drivingstereo] and Cityscapes [cordts2016cityscapes] datasets. For stereo datasets, we leverage the prediction of FoundationStereo [wen2025foundationstereo] as training labels.

The training largely follows that of the monocular teacher model. All images are trained at a base resolution of 504 with varying aspect ratios (1:1, 1:2, 16:9, 9:16, 3:4, 1:1.5, 1.5:1, 1:1.8). We employ AdamW optimizer and set the learning rate for encoder and decoder to 5e-6 and 5e-5, respectively. We apply random rotation augmentation where training images are rotated at 90 or 270 degree with 5% probability. We set canonical focal length $f^{c}$ to 300. We use the aligned prediction from teacher model as supervision. With a probability of 20%, we use the original ground-truth labels for training. We train with batch size of 64 for 160K iterations. The training objective is a weighted sum of depth loss $\mathcal{L}_{depth}$, $\mathcal{L}_{grad}$ and sky-mask loss $\mathcal{L}_{sky}$.

For training the NVS model, we leverage the large-scale DL3DV dataset [ling2024dl3dv], which provides diverse real-world scenes with camera poses estimated by COLMAP. We use 10,015 scenes from DL3DV for training the feed-forward 3DGS model. To ensure fair evaluation, we strictly maintain exclusivity between training and testing splits: the 140 DL3DV scenes used for benchmarking are completely disjoint from the training set, preventing any data leakage.

For each scene, we select all available images; if the total number exceeds the limit, we randomly sample 100 images using a fixed random seed. The selected images are then processed through a feed-forward model to generate consistent pose and depth estimations, after which the pose accuracy is computed.

For the same image set, we perform reconstruction using the predicted poses together with the predicted depths. To align the reconstructed point cloud with the ground-truth, we employ evo [umeyama2002least] to align the predicted poses to the ground-truth poses, obtaining a transformation that maps the reconstruction into the ground-truth coordinate system. To improve robustness, we adopt a RANSAC-based alignment procedure. Specifically, we repeatedly apply evo on randomly sampled pose subsets and evaluate each candidate transformation by counting the number of inlier poses, where inliers are defined as those with translation errors below the median of the overall pose deviations. The transformation with the largest inlier set is then chosen and applied to fuse the aligned predicted point cloud with the predicted depth maps by TSDF fusion. Finally, reconstruction quality is assessed by comparing the aligned reconstruction with the ground-truth point cloud using the metrics described in Sec. 6.2.

For each testing scene, the number of images typically ranges from 300 to 400 across all benchmark datasets. We sample one out of every 8 images as target novel views for evaluation. From the remaining viewpoints, we use COLMAP camera poses provided by each dataset and apply farthest point sampling, considering both camera translation and rotation distances, to select 12 images as input context views. For DL3DV, we use the official Benchmark set for testing. For Tanks and Temples, all Training Data scenes are included except Courthouse. For MegaDepth, we select scenes numbered from 5000 to 5018, as these are most suitable for NVS.

For assessing pose estimation, we follow the evaluation protocol introduced in [wang2025vggt, wang2023posediffusion] and report results using the AUC. This metric is derived from two components: Relative Rotation Accuracy (RRA) and Relative Translation Accuracy (RTA). RRA and RTA quantify the angular deviation in rotation and translation, respectively, between two images. Each error is compared against a set of thresholds to obtain accuracy values. AUC is then computed as the integral of the accuracy–threshold curve, where the curve is determined by the smaller of RRA and RTA at each threshold. To illustrate performance under different tolerance levels, we primarily report results at thresholds of 3 and 30.

Let $\mathcal{G}$ denote the ground-truth point set and $\mathcal{R}$ the reconstructed point set under evaluation. We measure accuracy using $\text{dist}(\mathcal{R}\rightarrow\mathcal{G})$ and completeness using $\text{dist}(\mathcal{G}\rightarrow\mathcal{R})$ following [aanaes2016dtu]. The Chamfer Distance (CD) is then defined as the average of these two terms. Based on these distances, we define the precision and recall of the reconstruction $\mathcal{R}$ with respect to a distance threshold $d$. Precision is given by $\frac{1}{|\mathcal{R}|}\sum\big[\text{dist}(\mathcal{R}_{i}\rightarrow\mathcal{G})<d\big]$, and recall by $\frac{1}{|\mathcal{G}|}\sum\big[\text{dist}(\mathcal{G}_{i}\rightarrow\mathcal{R})<d\big]$, where $\left[\cdot\right]$ denotes the Iverson bracket [knapitsch2017tnt]. To jointly capture both measures, we report the F1-score, computed as $\text{F1}=\frac{2\times\text{precision}\times\text{recall}}{\text{precision}+\text{recall}}$.

We evaluate visual rendering quality on diverse large-scale scenes. We introduce a new NVS benchmark built from three datasets, including DL3DV [ling2024dl3dv] with 140 scenes, Tanks and Temples [knapitsch2017tanks] with 6, and MegaDepth [li2018megadepth] with 19, each spanning around 300 sampled frames. Ground truth camera poses, estimated with COLMAP, are used directly to ensure accurate and fair comparison across diverse models. We report PSNR, SSIM, and LPIPS metrics on rendered novel views using given camera poses.

VGGT [wang2025vggt] is an end-to-end transformer that jointly predicts camera parameters, depth, and 3D points from one or many views. Pi3 [wang2025pi3] further adopts a permutation-equivariant design to recover affine-invariant cameras and scale-invariant point maps from unordered images. MapAnything [keetha2025mapanything] provides a feed-forward framework that can also take camera pose as input for dense geometric prediction. Fast3R [yang2025fast3r] extends point-map regression to hundreds or even thousands of images in a single forward pass. Finally, DUSt3R [wang2024dust3r] tackles uncalibrated image pairs by regressing point maps and aligning them globally. Our method is similar to VGGT [wang2025vggt], but adopts a new architecture and a different camera representation, and it is orthogonal to Pi3 [wang2025pi3].

As shown in Tab.˜2 and Fig.˜5, comparing against five baselines [dust3r, wang2025vggt, fast3r, wang2025pi3, keetha2025mapanything], our DA3-Giant model attains the best performance on nearly all metrics, with the only exception being Auc30 on the DTU dataset. Notably, on Auc3 our model delivers at least an $8\%$ relative improvement over all competing methods, and on ScanNet++ it achieves a $33\%$ relative gain over the second-best model.

As shown in Tab.˜3, our DA3-Gaint establishes a new SOTA in nearly all scenarios, outperforming all competitors in all five pose-free settings. On average, DA3-Gaint achieves a relative improvement of 25.1% over VGGT and 21.5% over Pi3. Fig.˜7 and Fig.˜6 visualize our predicted depth and recovered point clouds. The results are not only clean, accurate, and complete, but also preserve fine-grained geometric details, clearly demonstrating a superiority over other methods. Even more notably, our much smaller DA3-Large (0.30B parameters) demonstrates remarkable efficiency. Despite being 3$\times$ smaller, it surpasses the prior SOTA VGGT (1.19B parameters) in five out of the ten settings, with particularly strong performance on the ETH3D.

When camera poses are available, both our method and MapAnything can exploit them for improved results, and other methods also benefit from ground-truth pose fusion. Our model shows clear gains on most datasets except 7Scenes, where the limited video setting already saturates performance and reduces the benefit of pose conditioning. Notably, with pose conditioning, performance gains from scaling model size are smaller than in pose-free models, indicating that pose estimation scales more strongly than depth estimation and requires larger models to fully realize improvements.

Monocular depth accuracy also reflects geometry quality. As shown in Tab.˜4, on the standard monocular depth benchmarks reported in [yang2024depthv2], our model outperforms VGGT and Depth Anything 2. For reference, we also include the results of our teacher model.

To fairly evaluate feed-forward novel view synthesis (FF-NVS), we compare against three recent 3DGS models—pixelSplat [charatan2024pixelsplat], MVSplat [chen2024mvsplat], and DepthSplat [xu2025depthsplat]—and further test alternative frameworks by replacing our geometry backbone with Fast3R [yang2025fast3r], MV-DUSt3R [Mv-dust3r+], and VGGT [wang2025vggt]. All models are trained on DL3DV-10K training set under a unified protocol and evaluated on our benchmark (Sec.˜6.3).

As reported in Tab.˜5, all models perform substantially better on DL3DV than on the other datasets, suggesting that 3DGS-based NVS is sensitive to trajectory and pose distributions standardized by DL3DV, rather than scene content. Comparing the two groups, geometry-model-based frameworks consistently outperform specialized feed-forward models, demonstrating that a simple backbone plus DPT head can surpass complex task-specific designs. The advantage stems from large-scale pretraining, which enables better generalization and scalability than approaches relying on epipolar transformers, cost volumes, or cascaded modules. Within this group, NVS performance correlates with geometry estimation capability, making DA3 the strongest backbone. Looking forward, we expect FF-NVS can be effectively addressed with simple architectures leveraging pretrained geometry backbones, and that the strong spatial understanding of DA3 will benefit other 3D vision tasks.

We present visual comparisons with other models in Fig.˜11 under novel view synthesis settings. As illustrated, simply augmenting our DA3 model with a 3D Gaussian DPT head yields significantly improved rendering quality over existing state-of-the-art approaches. Our model demonstrates particular strength in challenging regions, such as thin structures (e.g., columns in the first and third scenes) and large-scale outdoor environments with wide-baseline input views (last two scenes), as shown in Fig.˜11. These results underscore the importance of a robust geometry backbone for high-quality visual rendering, consistent with our quantitative findings in Tab.˜5. We anticipate that the strong geometric understanding of DA3 will also benefit other 3D vision tasks.