Why the software stack is frozen in 2021

The original Jetson Nano is end-of-life. NVIDIA’s last image for it is JetPack 4.5.1 / 4.6.x (Ubuntu 18.04, CUDA 10.2, Python 3.6). JetPack 5/6 dropped Nano support entirely. The whole stack is vertically welded together — the GPU driver, CUDA, cuDNN and the OS ship as one Board Support Package — so you can’t just apt upgrade your way to a modern Python or TensorFlow. That forces:

• JetPack 4.5.1, Python 3.6, TensorFlow 2.3.x (NVIDIA’s special Jetson wheels)
• DonkeyCar 4.5.x (the last line that runs on Python 3.6)

So the first lesson: on this board you don’t choose old versions, the chip’s ceiling chooses them for you.

Roadblock 1 — SSH: “Too many authentication failures”

First contact over SSH died before a password prompt even appeared:

$ ssh jeroen@192.168.0.5
Received disconnect from 192.168.0.5 port 22:2: Too many authentication failures
Disconnected from 192.168.0.5 port 22

This is not a wrong password. The client offers every key in your agent, and each rejected key counts against the server’s MaxAuthTries (default 6) — so it hits the limit before reaching password auth. The fix is to stop offering keys:

ssh -o IdentitiesOnly=yes -o PubkeyAuthentication=no \
    -o PreferredAuthentications=password jeroen@192.168.0.5

The durable fix is a dedicated key plus an SSH config entry. Generate one key just for the car:

ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_donkeycar -N "" -C "jeroen@donkeycar-nano"

ssh-copy-id then hit the same “too many failures” wall (it offers all 6 keys before installing the new one), so it needs the same flags:

ssh-copy-id -o IdentitiesOnly=yes -o PubkeyAuthentication=no \
  -o PreferredAuthentications=password -i ~/.ssh/id_ed25519_donkeycar.pub jeroen@192.168.0.5

Roadblock 2 — the terminal type the Nano had never heard of

top, nano, anything full-screen, failed instantly:

$ top
'xterm-ghostty': unknown terminal type.

Ubuntu 18.04’s terminfo database predates the Ghostty terminal. Tell the Nano to use a TERM it knows:

echo 'export TERM=xterm-256color' >> ~/.bashrc

Roadblock 3 — apt lock on first boot

$ sudo apt-get upgrade -y
E: Could not get lock /var/lib/dpkg/lock-frontend - open (11: Resource temporarily unavailable)

unattended-upgrades grabs the dpkg lock at boot. Don’t force it — wait it out:

while sudo fuser /var/lib/dpkg/lock-frontend >/dev/null 2>&1; do
  echo "apt is busy, waiting..."; sleep 5
done
sudo apt-get update && sudo apt-get upgrade -y

The DonkeyCar dependencies install fine as one command (apt-get happily takes a long list), and grpcio then compiles from source because there’s no aarch64 / Python 3.6 wheel — expect 10–25 minutes of a spinner that looks frozen but isn’t. That’s gcc working, not a hang.

Calibration: finding the PCA9685, then its PWM limits

The steering servo and ESC are driven by a PCA9685 over I²C. First confirm the kernel sees it:

$ sudo i2cdetect -y -r 1
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
40: 40 -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: 70 -- -- -- -- -- -- --

40 is the PCA9685’s control address (and 70 its all-call). The first scan came up empty — a wiring fault; reseating the I²C leads brought it back. With the board visible, the DonkeyCar default pins already matched the hardware:

DRIVE_TRAIN_TYPE = "PWM_STEERING_THROTTLE"
PWM_STEERING_THROTTLE = {
    "PWM_STEERING_PIN": "PCA9685.1:40.1",   # steering servo → channel 1
    "PWM_THROTTLE_PIN": "PCA9685.1:40.0",   # ESC → channel 0
}

To calibrate interactively over SSH I wrote a tiny helper that sets a single PWM pulse using DonkeyCar’s own PCA9685 driver (so it behaves exactly like the real car), and exploits the fact that the chip holds the last value after the script exits:

# cal_set.py
import sys
from donkeycar.parts.actuator import PCA9685
ch, pulse = int(sys.argv[1]), int(sys.argv[2])
PCA9685(ch, address=0x40, busnum=1).set_pulse(pulse)
print("channel %d -> pulse %d" % (ch, pulse))

$ python -c 'import donkeycar as dk; d=dk.load_config("myconfig.py").PWM_STEERING_THROTTLE; \
             print(d["STEERING_LEFT_PWM"], d["THROTTLE_FORWARD_PWM"])'
460 430

Roadblock 4 — the controller the kernel refuses

I’d bought an “off-brand Xbox controller.” Plugged in, dmesg told a different story:

usb 1-2.3: New USB device found, idVendor=054c, idProduct=05c4
usb 1-2.3: Product: Wireless controller, Manufacturer: Sony Computer Entertainment
sony 0003:054C:05C4.0001: failed to retrieve feature report 0x81 with the DualShock 4 MAC address
sony: probe of 0003:054C:05C4.0001 failed with error -110

It’s not an Xbox pad at all — it’s a counterfeit DualShock 4. It spoofs a genuine DS4’s USB ID (054c:05c4) but doesn’t implement feature report 0x81 (the MAC address) that the kernel’s hid-sony driver demands. The probe fails with -110 (timeout), so no /dev/input/js0 is ever created — jstest just gets:

$ jstest /dev/input/js0
jstest: No such file or directory

The obvious workaround — force it onto the generic HID driver — is blocked on this kernel. hid-sony is built in (CONFIG_HID_SONY=y), so it can’t be unloaded or blacklisted, and a manual bind to hid-generic is refused:

$ echo -n "0003:054C:05C4.0003" | sudo tee /sys/bus/hid/drivers/hid-generic/bind
bind: FAILED   # the HID core won't let hid-generic claim a device a special driver owns

Verdict: a counterfeit DS4 cannot work on this kernel without rebuilding it without CONFIG_HID_SONY. The lesson — buy a genuine Sony DS4, a real Xbox pad (the xpad driver is far more forgiving), or a Logitech F710. For now, the web controller (below) needs no gamepad at all.

Roadblock 5 — the camera that wasn’t sending data

The CSI camera was the longest hunt. The sensor’s I²C control channel worked — the driver bound and /dev/video0 existed — but every capture segfaulted:

$ gst-launch-1.0 nvarguscamerasrc num-buffers=1 ! \
    'video/x-raw(memory:NVMM),width=1280,height=720' ! nvjpegenc ! filesink location=/tmp/camtest.jpg
(Argus) Error EndOfFile: ... Caught SIGSEGV
$ ls -l /tmp/camtest.jpg
-rw-rw-r-- 1 jeroen jeroen 0 ...   # zero bytes

The smoking gun was in dmesg and the Argus daemon log:

isp 54680000.isp:     TIMEOUT     10000        # ISP waited for frames, got none
nvargus-daemon: CSI_DEBUG_COUNTER_2_0 = 0x00000000   # ZERO bytes received on the CSI bus

So: I²C fine, but zero data on the MIPI/CSI lanes. A reboot later it got worse — even the I²C read started failing:

imx219 6-0010: imx219_board_setup: error during i2c read probe (-110)
imx219: probe of 6-0010 failed with error -110

That the symptom changed between boots is the tell: this is an intermittent physical connection, not software. A useful piece of logic narrowed it down — I²C and the high-speed data lanes share the same ribbon, so the fact I²C ever worked proved the orientation was correct; only the contact was unreliable.

The fix was a careful, full reseat of the ribbon at both ends with the latches firmly closed. Immediately:

$ dmesg | grep imx219
imx219 6-0010: tegracam sensor driver:imx219_v2.0.6
vi 54080000.vi: subdev imx219 6-0010 bound          # I²C bind clean, no -110

$ gst-launch-1.0 nvarguscamerasrc num-buffers=1 ! \
    'video/x-raw(memory:NVMM),width=1280,height=720' ! nvjpegenc ! filesink location=/tmp/camtest.jpg
GST_ARGUS: Done Success
$ ls -l /tmp/camtest.jpg
-rw-rw-r-- 1 jeroen jeroen 51866 /tmp/camtest.jpg   # a real 51 KB frame

A genuine 51 KB JPEG of the room — the camera works. Moral: “the ribbon looks fine” means nothing; CSI ribbons fail intermittently on the data lanes while I²C still limps along.

While the camera was down, CAMERA_TYPE = "MOCK" let everything else (web UI, actuators) be tested with blank frames — a useful way to keep moving. Once fixed, back to CSIC.

Roadblock 6 — power gremlins (three of them)

(a) Reboots under load. Running manage.py drive made the Nano vanish from the network and reboot. The reflex is “out of memory,” but free -h showed a 4 GB board with ~8 GB of swap and only 359 MB used — not OOM. It was a power transient: the Nano was in MAXN mode pulling current spikes a marginal supply couldn’t hold. A solid 5 V/4 A barrel supply plus 5 W mode fixed it:

sudo nvpmodel -m 1     # 5 W mode (MAXN is -m 0) — lower current draw

(b) USB-PD over-voltage — the scary one. The power bank’s USB-C port is USB-PD: it negotiates 9–20 V. Fed into the Nano’s 5 V barrel jack via a USB-C-to-DC cable, the green LED lit for ~10 s, then died as the bank ramped voltage — risking a fried board. Rule: only ever use the power bank’s plain USB-A (5 V) output for the barrel jack. The Nano survived; it was lucky.

(c) DHCP musical chairs. After all the reboots, ssh nano suddenly connected to… nothing useful:

$ ping 192.168.0.5
64 bytes from 192.168.0.5: ttl=64 time=0.026 ms   # loopback speed — that's *this PC*
$ ip -4 addr show | grep 192.168.0.5
inet 192.168.0.5/24 ... wlp0s20f3                 # the laptop grabbed the Nano's old IP

DHCP had reassigned 192.168.0.5 to the laptop; the Nano moved to .4. The fix is to stop hard-coding the IP and use mDNS:

$ getent hosts donkeycar.local
192.168.0.4     donkeycar.local

…and bake it into ~/.ssh/config so it follows the Nano forever:

Host nano
    HostName donkeycar.local
    User jeroen
    IdentityFile ~/.ssh/id_ed25519_donkeycar
    IdentitiesOnly yes
    StrictHostKeyChecking accept-new