实验室研究方向

Size: px

Start display at page:

Download "实验室研究方向"

聊包茅
5 years ago
Views:

1 多视图几何与运动恢复结构章国锋浙江大学 CAD&CG 国家重点实验室

2 视频场景重建的流程运动恢复结构深度恢复三维重建

3 针孔相机模型投影方程 : 齐次坐标表示 : Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, Second Edition 2004.

4 Z Y X f f Z fy fx 针孔相机模型 K [R t] Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, Second Edition 2004.

5 主点的偏移 ~ 1 / / Z Y X y f x f Z Zy fy Zx fx y Z fy x Z fx Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, Second Edition 2004.

6 相机的外部参数 Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, Second Edition 2004.

7 透视相机模型 K f x x 0 f y cy 0 s 0 c 1 P K R t 11 DoF (5+3+3)

径向畸变比如鱼眼镜头 : 数学模型 : R R 2 2 2 2 2 ( x, y) (1

8 径向畸变比如鱼眼镜头 : 数学模型 : R R ( x, y) (1 K ( x y ) K ( x y )...) x 1 2 y (Marc Pollefeys)

9 径向畸变矫正例子 (Marc Pollefeys)

10 Multi-View Geometry Structure-from-Motion Automatically recover the camera parameters and 3D structure from multiple images or video sequences. Noah Snavely, Steven M. Seitz, Richard Szeliski. "Photo tourism: Exploring photo collections in 3D"

11 Two-View Geometry 3D???

12 Two-View Geometry 3D???

13 Two-View Geometry 3D: Epipolar Geometry

14 极线几何

15 基础矩阵只跟两个视图的相对相机姿态和内参有关 F 是一个 3 3 秩为 2 的矩阵 Fe = 0 7 个自由度最少 7 对匹配点就可以求解 F 七点法八点法 K [ t T 2 ] RK 1 1 OpenCV: cvfindfundamentalmat()

16 八点法求解基础矩阵根据对极几何关系, 基本矩阵 F 满足若设那么对极几何关系又可以写作 : 若存在 n 对对应点,F 应满足如下的线性系统 :

17 八点法求解基础矩阵 f 为 9 维向量, 若要有解,rank(A) 至多为 8 在 rank(a) = 8 时,f 的方向是唯一的通过至少 8 对对应点, 可恰好得到使 f 方向唯一的 A f 为 A 的右零空间的基向量, 可用 svd(a) 求得真实数据存在噪音, 大于 8 组对应点得到的 A 满秩即 rank(a) = 9 此时同样可计算 (U,Σ,V) = svd(a) 令 f 为 V 中对应最小奇异值的列向量

18 多视图几何投影函数

19 Structure from Motion Pipeline Feature Tracking Obtain a set of feature tracks Structure from Motion Solve the camera parameters and 3D points of tracks

20 图像特征图像中显著容易区分和匹配的内容不变性点角点线 : 直线, 曲线, 边 : 二维边, 三维边形状 : 长方形, 圆, 椭圆, 球, 纹理视角不变 ( 尺度, 方向, 平移 ) 光照不变物体变形部分遮挡

21 Harris 角点检测核心思想 : 统计图像梯度的分布平滑区域 : 梯度不明显边缘区域 : 梯度明显, 方向一致角落区域 : 梯度明显, 方向不一致方法 : 计算像素邻域的梯度二阶矩计算上述矩阵的角点响应指标对 R 进行阈值过滤和非极大值抑制

如果是角点, 四个位置中应当有三个满足上面的条件速度快, 但对噪音不鲁棒 Edward Rosten, Tom Drummond.

22 FAST 通过直接的阈值和判断来加速角点提取考虑中心点周围的 16 个像素, 设中心点亮度为 p 如果有连续 n 个像素亮度都大于 p+t, 或者都小于 p-t ( 如图中的 14~16, 1 ~ 6) 检查四个位置, 如果是角点, 四个位置中应当有三个满足上面的条件速度快, 但对噪音不鲁棒 Edward Rosten, Tom Drummond. Machine Learning for High-Speed Corner Detection. ECCV (1) 2006:

23 SIFT Scale-Invariant Feature Transform SIFT 通过在不同级别的图像 DoG 上寻找极大 / 极小值来确定特征的位置和对应的尺度, 后续的特征提取在与其尺度最邻近的图像 DoG 上进行这使它有良好的尺度不变性 David G. Lowe.Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision 60(2): (2004).

24 More Invariant Features SIFT 之后陆续出现了各种尺度不变特征描述量提取算法如 RIFT GLOH SURF 等其中 SURF 性能上接近 SIFT SURF 使用了 Haar 小波卷积替代 SIFT 中的高斯核用积分图像进行了加速, 使得计算速度达到 SIFT 的 3~7 倍 ORB 由于其良好的匹配性能和极快的提取速度也得到了广泛使用

25 特征提取精度高 SIFT SURF ORB 速度快 SIFT 极佳的尺度不变性, 能一定程度上适应视角变化和亮度变化 SURF 能够处理严重的图像模糊, 速度要高于 SIFT, 但精度不如 SIFT ORB 极快的提取速度, 在实时应用中常用来替代 SIFT 以上三种特征提取算法均在 OpenCV 中有实现

26 特征匹配模板匹配直接在目标图像中寻找给定的图像块

27 特征匹配在小运动假设下, 可以采用 KLT 跟踪方法 : I(x,y,t) I(x,y,t+1) 一个等式, 两个未知量

28 特征匹配进一步假设 : 相邻像素运动一致 ( 单个像素 ) ( 邻域窗口 )

29 特征匹配大运动情况下的匹配通过比较特征描述量的距离进行匹配 SIFT = 128 维 SURF = 64 维 ORB = 256bits 暴力匹配快速最近邻匹配 OpenCV 中提供了相应的匹配算法

30 Loopback Sequences and Multiple Sequences How to efficiently match the common features among different subsequences?

31 Non-Consecutive Feature Tracking

32 Framework Overview 1. Detect SIFT features over the entire sequence. 2. Consecutive point tracking: 2.1 Match features between consecutive frames with descriptor comparison. 2.2 Perform the second-pass matching to extend track lifetime. 3. Non-consecutive track matching: 3.1 Use hierachical k-means to cluster the constructed tracks. 3.2 Estimate the matching matrix with the grouped tracks. 3.3 Detect overlapping subsequences and join the matched tracks.

33 Two-Pass Matching for Consecutive Tracking SIFT Feature Extraction First-Pass Matching by Descriptor Comparison Global distinctive

35 Two-View Geometry 3D???

36 Two-View Geometry 3D: Epipolar Geometry

37 Not enough! How to handle image distortion? Naïve window-based matching becomes unreliable! How to give a good position initializaton? Whole line searching is still time-consuming and ambiguous with many potential correspondences.???

38 Second-Pass Matching by Planar Motion Segmentation Estimate a set of homographies Using inlier matches in first-pass matching frame t 1 2 H t H, t1 t, t1 frame t+1 Alignment 3 H t, t1 4 H t, t1

39 Second-Pass Matching by Planar Motion Segmentation Guided matching Epipolar constraint Homography constraint

40 Second-Pass Matching with Multi- Homographies First-Pass Matching (53 matches) Direct Searching (11 matches added) Our Second-Pass Matching (346 matches added)

41 Non-Consecutive track matching Fast Matching Matrix Estimation Detect overlapping subsequences and join the matched tracks.

42 Fast Matching Matrix Estimation Each track has a group of description vectors Track descriptor Use a hierarchical K-means approach to cluster the track descriptors

43 Fast Matching Matrix Estimation

matching the common features of the selected image pairs.

44 Non-Consecutive Track Matching Simultaneously Match Images and Refine Matching Matrix Refine the matching matrix after matching the common features of the selected image pairs. More reliably find the best matching images with the updated matching matrix.

45 Traditional SfM Framework Feature tracking over whole sequence Structure & motion initialization Compute F between two initial images Compute P 1 and P 2 Triangulate 3D points of the matched features For each additional view Compute the camera pose Refine and extend 3D points Self-Calibration Upgrade the projective reconstruction to metric one. Refine structure and motion Bundle adjustment

46 三角化已知 F, 计算 P 和 P 已知 x 和 x 计算 X: x= PX x'= P 'X Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, Second Edition 2004.

47 有噪声情况下的三角化由于存在噪声, 反投到三维空间上的射线并不会严格相交优化投影点到对应极线的距离 Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, Second Edition 2004.

48 线性三角化方法给定方程 x= PX x'= P 'X p it 表示 P 的第 i 行. 写成矩阵和向量相乘的形式直接解析求解. 没有几何意义不是最优.

49 优化几何误差 Cost function 用 Levenberg-Marquart 算法求解

50 Knowing 3D points, Compute Camera Motion Compute Projection Matrix Decomposition for Metric Projection Matrix P K[ R t] [ KR Kt] [ M Kt] Decompose M into K, R by QR decomposition 1 t K p, p, p ) ( T

Bundle Adjustment Definition Refining a visual reconstruction to produce jointly optimal 3D structure and viewing parameter (camera pose and/or calibration)

51 Bundle Adjustment Definition Refining a visual reconstruction to produce jointly optimal 3D structure and viewing parameter (camera pose and/or calibration) estimates. B. Triggs, P. F. McLauchlan, R. I. Hartley, and A. W. Fitzgibbon. Bundle adjustment - a modern synthesis. In Workshop on Vision Algorithms, pages , 1999.

52 Geometric Ambiguities Projective Self-Calibration Metric Reconstruction Reconstruction Marc Pollefeys. Visual 3D Modeling from Images

53 Self-Calibration State-of-the-Art References R.I. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, second ed. Cambridge Univ. Press, M. Pollefeys, L.J. Van Gool, M. Vergauwen, F. Verbiest, K. Cornelis, J. Tops, and R. Koch, Visual Modeling with a Hand-Held Camera, Int l J. Computer Vision, vol. 59, no. 3, pp , G. Zhang, X. Qin, W. Hua, T.-T. Wong, P.-A. Heng, and H. Bao, Robust Metric Reconstruction from Challenging Video Sequences, Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2007.

54 推荐 SfM 开源系统 ENFT-SFM or LS-ACTS OpenMVG VisualSFM

视频场景重建的流程运动恢复结构深度恢复三维重建

视频场景重建的流程运动恢复结构深度恢复三维重建运动恢复结构章国锋浙江大学 CAD&CG 国家重点实验室视频场景重建的流程运动恢复结构深度恢复三维重建针孔相机模型投影方程 : 齐次坐标表示 : Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, Second Edition