Jetson
jetson 每个产品都有一套 developer kit 和一套 module，引用自 FAQ，developer kit 用于研发阶段，module 用于生产环境。
FYI:
名词解释
- l4t（Linux for Tegra），为 tegra 处理器发行的 linux，基于 ubuntu 定制。文档：NVIDIA Jetson Linux Developer Guide : Introduction | NVIDIA Docs
- jetpack，nvidia 的软件开放工具包，包括 l4t和各种工具。NVIDIA JetPack Documentation
- cuda
- tensorrt
Flash
刷机有两种方式，一种是通过 nvidia sdk mamager 刷入 jetpack，一种是手动刷入 l4t。
Nvidia Sdk manager
NVIDIA SDK Manager，提供比较简单、自动化的刷机过程，有 gui，同时也支持 Command-Line Install :: NVIDIA SDK Manager Documentation。
简单尝试了下，因为网络的坑比较多，所以最后选择手动刷入 l4t。
Manual
手动刷机主要分为两步，刷入 l4t 后，进入系统再安装 jetpack。
刷 l4t，官方提供了比较详细的文档：NVIDIA Jetson Linux Developer Guide : Quick Start Guide | NVIDIA Docs
安装环境
Host
- Mac mini
- macOS 11.1
- VirtualBox 6.1.16
- VirtualBox Extension Pack 6.1.16
- Ubuntu-18.04.5-live-server-amd64.iso
Vbox 需要支持 USB2/USB3，所以需要安装扩展包：Downloads – Oracle VM VirtualBox
nvidia sdkmanager 有声明需要 ubuntu 16 或 18。手动刷 l4t 没提及发行版的要求。根据这些天踩过 nvidia 的坑，还是稳妥点，乖乖 ubuntu。手动刷机更不需要 gui 了，所以选择 server 版本的 ubuntu，省点资源：Ubuntu 18.04.5 LTS (Bionic Beaver)
Jetson
- Jetson agx xavier 32GB(P2888-004)
- L4T R32.4.4 archive | NVIDIA Developer
安装 Ubuntu
创建虚拟机需要注意:
- 选择 usb3 的支持
- 硬盘镜像选择 30GB，基于 sdkmanager 安装 jetson 的文档提及可用空间需要大于 25GB，所以 30 GB 肯定是够的。
手动安装只安装 l4t，实际肯定不用 30GB，但 10 GB 肯定是不够的，别问为什么。
成功刷机后的空间情况：
$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 462M 0 462M 0% /dev
tmpfs 99M 736K 98M 1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv 20G 19G 76M 100% /
tmpfs 493M 0 493M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 493M 0 493M 0% /sys/fs/cgroup
/dev/sda2 976M 78M 832M 9% /boot
tmpfs 99M 0 99M 0% /run/user/1000
$ lsblk
lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 30G 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 1G 0 part /boot
└─sda3 8:3 0 29G 0 part
└─ubuntu--vg-ubuntu--lv 253:0 0 20G 0 lvm /
sr0 11:0 1 1024M 0 rom
添加光驱镜像 ubuntu-18.04.5-live-server-amd64.iso 后，启动虚拟机便开始安装流程。主要注意把镜像替换为
https://mirrors.tuna.tsinghua.edu.cn/ubuntu
进入系统后，需要安装
sudo apt-get install qemu-user-static
sudo apt-get install python # 需要 python 可执行，不确定是否要 python2
Host 便配置完成。
下载 l4t archive
下载页 L4T R32.4.4 archive | NVIDIA Developer
下载
配置环境变量，可配可不配，主要是文档提及的文件名容易造成混淆。
export L4T_RELEASE_PACKAGE=Tegra186_Linux_R32.4.4_aarch64.tbz2 # 就是下载的 BSP 文件
export SAMPLE_FS_PACKAGE=Tegra_Linux_Sample-Root-Filesystem_R32.4.4_aarch64.tbz2
export export BOARD=jetson-agx-xavier-devkit
解压文件
$ tar xf ${L4T_RELEASE_PACKAGE}
$ cd Linux_for_Tegra/rootfs/
$ sudo tar xpf ../../${SAMPLE_FS_PACKAGE}
$ cd ..
$ sudo ./apply_binaries.sh
然后准备刷机，
- Jetson 关机再进入 Force Recovery Mode (RCM)
- usb 链接 mac mini
- vbox 添加 usb 到虚拟机，可能需要重启虚拟机、重新把擦才能认到。
lsusb确认 usb 已经链接到虚拟机并处在 RCM ：Bus <bbb> Device <ddd>: ID 0955: <nnnn> Nvidia Corp.- usb 需要重启虚拟机才能认到
- 文档未提及需要
python可执行。ubuntu 只有 python3，需要手动安装一些 python,或者链接一下python3
正常情况下，就可以开始刷机了，不过文档未提及需要 python 可执行。第一次执行 flash.sh，因为本地找不到 python 可执行文件，所以报错。安装 python2 后重新执行，提示
Error: probing the target board failed.
Flash Jetson TX2 : Make sure the target board is connected through
Flash Jetson TX2 : USB port and is in recovery mode.
看了下 flash.sh 的代码发现问题是因为 sudo ./bootloader/tegrarcm_v2 --uid 报错：
Failed to read UID
根据这个回帖，https://forums.developer.nvidia.com/t/sdk-manager-could-not-detect-target-hardware/71797/24 猜测是第一次因为 python 报错导致，但 RCM 的信息被读取，下一次 flash 就会读不到信息最后超时报错。 断电重新进入一次 Force Recovery Mode。便可以正常刷机。差点以为是不支持 VM：
不过貌似也有办法支持其他 linux：https://forums.developer.nvidia.com/t/using-jetpack-from-arch-linux/44979
确保所有条件没问题，便可以开始刷机。
- 进入
Force Recovery mode，设备关机，按住 Force Recovery 键（与电源键相邻），再开机。 - 链接 USB
sudo ./flash.sh ${BOARD} mmcblk0p1
刷机成功，jetson 便会自动重启直接进入系统。
换源
备份
sudo cp /etc/apt/sources.list /etc/apt/sources.list.origin
内容替换为：
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ bionic main multiverse restricted universe
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ bionic-security main multiverse restricted universe
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ bionic-updates main multiverse restricted universe
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ bionic-backports main multiverse restricted universe
deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ bionic main multiverse restricted universe
deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ bionic-security main multiverse restricted universe
deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ bionic-updates main multiverse restricted universe
deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ bionic-backports main multiverse restricted universe
更新系统
sudo apt-get update
sudo apt-get upgrade
见 How to Install Debian Packages with OTA Update
display manager 更新为 lightdm
dpkg --configure lightdm
补完 jetpack
sudo apt install nvidia-jetpack
见 1.3. Package Management Tool
最后 32 GB 的空间剩下：
/dev/mmcblk0p1 28G 12G 15G 46% /
其他必备
Headless Jetson
tigervnc 等一般 vncserver 方案
一个 vncserver 对应一个 xsession，不与当前显示器共享 xsession。
当是这方案不支持 EGL
Unfortunately, if there is no monitor case, then VNC cannot work. You could try to use RDP (remote desktop protocol) application and it should work. However, EGL seems not able to run with rdp backend. Thus, if you want to render something by using EGL, it would not work.
x11vnc
- Dummy Monitor 不能生效，且不能使用 nvidia 驱动
- 尝试 hdmi 欺骗器 TODO
开机自动启用 x11vnc
-
login manager 自动登陆
-
先连到 loginmanager 的 xsession(Dispaly=:0)，手动登陆后再，连到桌面（Display=:1）:https://askubuntu.com/a/1036507
X11 Forwarding
转发远端的 xclient 链接到本地的 xserver
macos
- xquartz
Vino
折腾了好久之后才发现官方有文档 Setting Up VNC | NVIDIA Developer
官方用的是 ubuntu 自带的 vino。与 x11vnc 一样是屏幕共享。需要等本地登陆后才能连，也可配置自动登陆。 不插显示器也支持，不过默认分辨率有点小
问题：卡，没有 x11vnc 流畅
切换到 lightdm 后流畅度改善不少。
默认分辨率太小，需要修改 /etc/X11/xorg.conf
Section "Screen"
Identifier "Default Screen"
Monitor "Configured Monitor"
Device "Default Device"
SubSection "Display"
Depth 24
Virtual 1440 900
EndSubSection
EndSection
ZED
确保安装了
Cython
距离识别模式
- STANDARD，效率好，检测边界，存在黑洞（无法获取数据的点）
- FILL，生产稠密的距离图，不支持 Point Cloud，
Dr=Z^2*alpha, where Dr is depth resolution, Z the distance and alpha a constant.
zed-gstreamer
gir1.2-gst-rtsp-server-1.0 gstreamer1.0-doc gstreamer1.0-qt5 libglib2.0-doc libgstrtspserver-1.0-0 libgstrtspserver-1.0-dev
pytorch
cmake python3
https://github.com/pytorch/vision/issues/2565
cmake
系统自带版本太旧，编译 torchvison 不支持 python3。删除系统的版本，再从源码重新编译。
sudo apt remove cmake
确保安装 openssl 依赖
sudo apt install libssl-dev
下载
opencv
自带的 opencv 并不支持 CUDA，cv2.getBuildInfomation()：
General configuration for OpenCV 4.1.1 =====================================
Version control: 4.1.1-2-gd5a58aa75
Platform:
Timestamp: 2019-12-13T17:25:11Z
Host: Linux 4.9.140-tegra aarch64
CMake: 3.10.2
CMake generator: Unix Makefiles
CMake build tool: /usr/bin/make
Configuration: Release
CPU/HW features:
Baseline: NEON FP16
required: NEON
disabled: VFPV3
....
OpenCV modules:
To be built: calib3d core dnn features2d flann gapi highgui imgcodecs imgproc ml objdetect photo python2 python3 stitching ts video videoio
Disabled: world
Disabled by dependency: -
Unavailable: java js
Applications: tests perf_tests examples apps
Documentation: NO
Non-free algorithms: NO
需要自己重新编译：Install OpenCV 4.5 on Jetson Nano - Q-engineering
# reveal the CUDA location
sudo sh -c "echo '/usr/local/cuda/lib64' >> /etc/ld.so.conf.d/nvidia-tegra.conf"
sudo ldconfig
# third-party libraries
sudo apt-get install build-essential cmake git unzip pkg-config
sudo apt-get install libjpeg-dev libpng-dev libtiff-dev
sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev
sudo apt-get install libgtk2.0-dev libcanberra-gtk*
sudo apt-get install python3-dev python3-numpy python3-pip
sudo apt-get install libxvidcore-dev libx264-dev libgtk-3-dev
sudo apt-get install libtbb2 libtbb-dev libdc1394-22-dev
sudo apt-get install libv4l-dev v4l-utils
sudo apt-get install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev
sudo apt-get install libavresample-dev libvorbis-dev libxine2-dev
sudo apt-get install libfaac-dev libmp3lame-dev libtheora-dev
sudo apt-get install libopencore-amrnb-dev libopencore-amrwb-dev
sudo apt-get install libopenblas-dev libatlas-base-dev libblas-dev
sudo apt-get install liblapack-dev libeigen3-dev gfortran
sudo apt-get install libhdf5-dev protobuf-compiler
sudo apt-get install libprotobuf-dev libgoogle-glog-dev libgflags-dev
# qt5
sudo apt-get install qt5-default
下载源码：
# download the latest version
$ cd ~
$ wget -O opencv.zip https://github.com/opencv/opencv/archive/4.5.0.zip
$ wget -O opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/4.5.0.zip
# unpack
$ unzip opencv.zip
$ unzip opencv_contrib.zip
$ mv opencv-4.5.0 opencv
$ mv opencv_contrib-4.5.0 opencv_contrib
# clean up the zip files
$ rm opencv.zip
$ rm opencv_contrib.zip
$ cd ~/opencv
$ mkdir build
$ cd build
确保 CUDA_ARCH_BIN 改成 agx xavier 的版本
默认安装位置是 /usr/local，改成/usr是为了和本机版本共存。
cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr \
-D OPENCV_EXTRA_MODULES_PATH=~/Playground/opencv_contrib/modules \
-D EIGEN_INCLUDE_PATH=/usr/include/eigen3 \
-D WITH_CUDA=ON \
-D CUDA_ARCH_BIN=7.2 \
-D CUDA_ARCH_PTX="" \
-D WITH_CUDNN=ON \
-D WITH_CUBLAS=ON \
-D ENABLE_FAST_MATH=ON \
-D CUDA_FAST_MATH=ON \
-D OPENCV_DNN_CUDA=ON \
-D ENABLE_NEON=ON \
-D WITH_QT=ON \
-D WITH_OPENMP=ON \
-D WITH_OPENGL=ON \
-D BUILD_TIFF=ON \
-D WITH_FFMPEG=ON \
-D WITH_GSTREAMER=ON \
-D WITH_TBB=ON \
-D BUILD_TBB=ON \
-D BUILD_TESTS=OFF \
-D WITH_EIGEN=ON \
-D WITH_V4L=ON \
-D WITH_LIBV4L=ON \
-D OPENCV_ENABLE_NONFREE=ON \
-D INSTALL_C_EXAMPLES=OFF \
-D INSTALL_PYTHON_EXAMPLES=OFF \
-D BUILD_NEW_PYTHON_SUPPORT=ON \
-D BUILD_opencv_python3=TRUE \
-D OPENCV_GENERATE_PKGCONFIG=ON \
-D BUILD_EXAMPLES=ON ..
编译
make -j4
$ sudo rm -r /usr/include/opencv4/opencv2 # 清理 old packages
$ sudo make install
$ sudo ldconfig
# cleaning (frees 300 KB)
$ make clean
测试一下：
General configuration for OpenCV 4.5.1 =====================================
Version control: unknown
Extra modules:
Location (extra): /home/tx/Playground/opencv_contrib/modules
Version control (extra): unknown
Platform:
Timestamp: 2020-12-29T03:01:07Z
Host: Linux 4.9.140-tegra aarch64
CMake: 3.19.2
CMake generator: Unix Makefiles
CMake build tool: /usr/bin/make
Configuration: RELEASE
CPU/HW features:
Baseline: NEON FP16
required: NEON
...
OpenCV modules:
To be built: alphamat aruco bgsegm bioinspired calib3d ccalib core cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev cvv datasets dnn dnn_objdetect dnn_superres dpm face features2d flann freetype fuzzy gapi hdf hfs highgui img_hash imgcodecs imgproc intensity_transform line_descriptor mcc ml objdetect optflow phase_unwrapping photo plot python3 quality rapid reg rgbd saliency sfm shape stereo stitching structured_light superres surface_matching text tracking ts video videoio videostab xfeatures2d ximgproc xobjdetect xphoto
Disabled: world
Disabled by dependency: -
Unavailable: cnn_3dobj java julia matlab ovis python2 viz
Applications: perf_tests examples apps
Documentation: NO
Non-free algorithms: YES
cv2.cuda.getCudaEnabledDeviceCount()
1
gstreamer
安装需要参考 Accelerated GStreamer
sudo add-apt-repository universe
sudo add-apt-repository multiverse
sudo apt-get update
sudo apt-get install gstreamer1.0-tools gstreamer1.0-alsa \
gstreamer1.0-plugins-base gstreamer1.0-plugins-good \
gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly \
gstreamer1.0-libav
sudo apt-get install libgstreamer1.0-dev \
libgstreamer-plugins-base1.0-dev \
libgstreamer-plugins-good1.0-dev \
libgstreamer-plugins-bad1.0-dev
opencv 才能支持 CAP_GSTREAMER，jetcam 才能正常使用
cv2.getBuildInfomation() 输出有：
Video I/O:
FFMPEG: YES
avcodec: YES (57.107.100)
avformat: YES (57.83.100)
avutil: YES (55.78.100)
swscale: YES (4.8.100)
avresample: NO
GStreamer: YES (1.14.5)
v4l/v4l2: YES (linux/videodev2.h)
- videoparse
- videoconvert
- queue
ipython
与 jedi 不兼容
python3.6 支持的 ipython 7.16.x，与 jedi 0.18 不兼容报错：
File "../venv/lib/python3.8/site-packages/IPython/core/completer.py", line 2029, in _complete
completions = self._jedi_matches(
File "../venv/lib/python3.8/site-packages/IPython/core/completer.py", line 1373, in _jedi_matches
interpreter = jedi.Interpreter(
File "../venv/lib/python3.8/site-packages/jedi/api/__init__.py", line 725, in __init__
super().__init__(code, environment=environment,
TypeError: __init__() got an unexpected keyword argument 'column'
需要降级 jedi，emacs lsp-mode 需要的 jedi-language-server 0.22.0，又将 jedi pin 在 0.18。 需要安装指定版本：
pip3 install 'jedi==0.17.2'
pip3 install 'jedi_language_server==0.21.0'
jupyter 无法显示图片
https://stackoverflow.com/questions/61834868/ipython-display-display-not-showing-image-in-jupyter-notebook
python3 -m pip install ipywidgets
jupyter nbextension enable --py widgetsnbextension
# 需要安装 jupyter-lab 和 nodejs > 12
jupyter labextension install @jupyter-widgets/jupyterlab-manager