Why the software stack is frozen in 2021

The original Jetson Nano is end-of-life. NVIDIA’s last image for it is JetPack 4.5.1 / 4.6.x (Ubuntu 18.04, CUDA 10.2, Python 3.6). JetPack 5/6 dropped Nano support entirely. The whole stack is vertically welded together, the GPU driver, CUDA, cuDNN and the OS ship as one Board Support Package, so you can’t just apt upgrade your way to a modern Python or TensorFlow. That forces:

• JetPack 4.5.1, Python 3.6, TensorFlow 2.3.x (NVIDIA’s special Jetson wheels)
• DonkeyCar 4.5.x (the last line that runs on Python 3.6)

So the first lesson: on this board you don’t choose old versions, the chip’s ceiling chooses them for you.

Roadblock 1, SSH: “Too many authentication failures”

First contact over SSH died before a password prompt even appeared:

$ ssh jeroen@192.168.0.5
Received disconnect from 192.168.0.5 port 22:2: Too many authentication failures
Disconnected from 192.168.0.5 port 22

This is not a wrong password. The client offers every key in your agent, and each rejected key counts against the server’s MaxAuthTries (default 6), so it hits the limit before reaching password auth. The fix is to stop offering keys:

ssh -o IdentitiesOnly=yes -o PubkeyAuthentication=no \
    -o PreferredAuthentications=password jeroen@192.168.0.5

The durable fix is a dedicated key plus an SSH config entry. Generate one key just for the car:

ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_donkeycar -N "" -C "jeroen@donkeycar-nano"

ssh-copy-id then hit the same “too many failures” wall (it offers all 6 keys before installing the new one), so it needs the same flags:

ssh-copy-id -o IdentitiesOnly=yes -o PubkeyAuthentication=no \
  -o PreferredAuthentications=password -i ~/.ssh/id_ed25519_donkeycar.pub jeroen@192.168.0.5

Roadblock 2: the terminal type the Nano had never heard of

top, nano, anything full-screen, failed instantly:

$ top
'xterm-ghostty': unknown terminal type.

Ubuntu 18.04’s terminfo database predates the Ghostty terminal. Tell the Nano to use a TERM it knows:

echo 'export TERM=xterm-256color' >> ~/.bashrc

Roadblock 3: apt lock on first boot

$ sudo apt-get upgrade -y
E: Could not get lock /var/lib/dpkg/lock-frontend - open (11: Resource temporarily unavailable)

unattended-upgrades grabs the dpkg lock at boot. Don’t force it, wait it out:

while sudo fuser /var/lib/dpkg/lock-frontend >/dev/null 2>&1; do
  echo "apt is busy, waiting..."; sleep 5
done
sudo apt-get update && sudo apt-get upgrade -y

The DonkeyCar dependencies install fine as one command (apt-get happily takes a long list), and grpcio then compiles from source because there’s no aarch64 / Python 3.6 wheel, expect 10–25 minutes of a spinner that looks frozen but isn’t. That’s gcc working, not a hang.

Calibration: finding the PCA9685, then its PWM limits

The steering servo and ESC are driven by a PCA9685 over I²C. First confirm the kernel sees it:

$ sudo i2cdetect -y -r 1
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
40: 40 -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: 70 -- -- -- -- -- -- --

40 is the PCA9685’s control address (and 70 its all-call). The first scan came up empty: a wiring fault; reseating the I²C leads brought it back. With the board visible, the DonkeyCar default pins already matched the hardware:

DRIVE_TRAIN_TYPE = "PWM_STEERING_THROTTLE"
PWM_STEERING_THROTTLE = {
    "PWM_STEERING_PIN": "PCA9685.1:40.1",   # steering servo → channel 1
    "PWM_THROTTLE_PIN": "PCA9685.1:40.0",   # ESC → channel 0
}

To calibrate interactively over SSH I wrote a tiny helper that sets a single PWM pulse using DonkeyCar’s own PCA9685 driver (so it behaves exactly like the real car), and exploits the fact that the chip holds the last value after the script exits:

# cal_set.py
import sys
from donkeycar.parts.actuator import PCA9685
ch, pulse = int(sys.argv[1]), int(sys.argv[2])
PCA9685(ch, address=0x40, busnum=1).set_pulse(pulse)
print("channel %d -> pulse %d" % (ch, pulse))

$ python -c 'import donkeycar as dk; d=dk.load_config("myconfig.py").PWM_STEERING_THROTTLE; \
             print(d["STEERING_LEFT_PWM"], d["THROTTLE_FORWARD_PWM"])'
460 430

Roadblock 4: the controller the kernel refuses

I’d bought an “off-brand Xbox controller.” Plugged in, dmesg told a different story:

usb 1-2.3: New USB device found, idVendor=054c, idProduct=05c4
usb 1-2.3: Product: Wireless controller, Manufacturer: Sony Computer Entertainment
sony 0003:054C:05C4.0001: failed to retrieve feature report 0x81 with the DualShock 4 MAC address
sony: probe of 0003:054C:05C4.0001 failed with error -110

It’s not an Xbox pad at all, it’s a counterfeit DualShock 4. It spoofs a genuine DS4’s USB ID (054c:05c4) but doesn’t implement feature report 0x81 (the MAC address) that the kernel’s hid-sony driver demands. The probe fails with -110 (timeout), so no /dev/input/js0 is ever created, jstest just gets:

$ jstest /dev/input/js0
jstest: No such file or directory

The obvious workaround, force it onto the generic HID driver, is blocked on this kernel. hid-sony is built in (CONFIG_HID_SONY=y), so it can’t be unloaded or blacklisted, and a manual bind to hid-generic is refused:

$ echo -n "0003:054C:05C4.0003" | sudo tee /sys/bus/hid/drivers/hid-generic/bind
bind: FAILED   # the HID core won't let hid-generic claim a device a special driver owns

Verdict: a counterfeit DS4 cannot work on this kernel without rebuilding it without CONFIG_HID_SONY. The lesson, buy a genuine Sony DS4, a real Xbox pad (the xpad driver is far more forgiving), or a Logitech F710. For now, the web controller (below) needs no gamepad at all.

Roadblock 5: the camera that wasn’t sending data

The CSI camera was the longest hunt. The sensor’s I²C control channel worked, the driver bound and /dev/video0 existed, but every capture segfaulted:

$ gst-launch-1.0 nvarguscamerasrc num-buffers=1 ! \
    'video/x-raw(memory:NVMM),width=1280,height=720' ! nvjpegenc ! filesink location=/tmp/camtest.jpg
(Argus) Error EndOfFile: ... Caught SIGSEGV
$ ls -l /tmp/camtest.jpg
-rw-rw-r-- 1 jeroen jeroen 0 ...   # zero bytes

The smoking gun was in dmesg and the Argus daemon log:

isp 54680000.isp:     TIMEOUT     10000        # ISP waited for frames, got none
nvargus-daemon: CSI_DEBUG_COUNTER_2_0 = 0x00000000   # ZERO bytes received on the CSI bus

So: I²C fine, but zero data on the MIPI/CSI lanes. A reboot later it got worse, even the I²C read started failing:

imx219 6-0010: imx219_board_setup: error during i2c read probe (-110)
imx219: probe of 6-0010 failed with error -110

That the symptom changed between boots is the tell: this is an intermittent physical connection, not software. A useful piece of logic narrowed it down, I²C and the high-speed data lanes share the same ribbon, so the fact I²C ever worked proved the orientation was correct; only the contact was unreliable.

The fix was a careful, full reseat of the ribbon at both ends with the latches firmly closed. Immediately:

$ dmesg | grep imx219
imx219 6-0010: tegracam sensor driver:imx219_v2.0.6
vi 54080000.vi: subdev imx219 6-0010 bound          # I²C bind clean, no -110

$ gst-launch-1.0 nvarguscamerasrc num-buffers=1 ! \
    'video/x-raw(memory:NVMM),width=1280,height=720' ! nvjpegenc ! filesink location=/tmp/camtest.jpg
GST_ARGUS: Done Success
$ ls -l /tmp/camtest.jpg
-rw-rw-r-- 1 jeroen jeroen 51866 /tmp/camtest.jpg   # a real 51 KB frame

A genuine 51 KB JPEG of the room, the camera works. Moral: “the ribbon looks fine” means nothing; CSI ribbons fail intermittently on the data lanes while I²C still limps along.

While the camera was down, CAMERA_TYPE = "MOCK" let everything else (web UI, actuators) be tested with blank frames, a useful way to keep moving. Once fixed, back to CSIC.

Roadblock 6: power gremlins (three of them)

(a) Reboots under load. Running manage.py drive made the Nano vanish from the network and reboot. The reflex is “out of memory,” but free -h showed a 4 GB board with ~8 GB of swap and only 359 MB used, not OOM. It was a power transient: the Nano was in MAXN mode pulling current spikes a marginal supply couldn’t hold. A solid 5 V/4 A barrel supply plus 5 W mode fixed it:

sudo nvpmodel -m 1     # 5 W mode (MAXN is -m 0), lower current draw

(b) USB-PD over-voltage: the scary one. The power bank’s USB-C port is USB-PD: it negotiates 9–20 V. Fed into the Nano’s 5 V barrel jack via a USB-C-to-DC cable, the green LED lit for ~10 s, then died as the bank ramped voltage, risking a fried board. Rule: only ever use the power bank’s plain USB-A (5 V) output for the barrel jack. The Nano survived; it was lucky.

(c) DHCP musical chairs. After all the reboots, ssh nano suddenly connected to… nothing useful:

$ ping 192.168.0.5
64 bytes from 192.168.0.5: ttl=64 time=0.026 ms   # loopback speed, that's *this PC*
$ ip -4 addr show | grep 192.168.0.5
inet 192.168.0.5/24 ... wlp0s20f3                 # the laptop grabbed the Nano's old IP

DHCP had reassigned 192.168.0.5 to the laptop; the Nano moved to .4. The fix is to stop hard-coding the IP and use mDNS:

$ getent hosts donkeycar.local
192.168.0.4     donkeycar.local

…and bake it into ~/.ssh/config so it follows the Nano forever:

Host nano
    HostName donkeycar.local
    User jeroen
    IdentityFile ~/.ssh/id_ed25519_donkeycar
    IdentitiesOnly yes
    StrictHostKeyChecking accept-new

Driving it: the web controller

No gamepad needed, DonkeyCar serves a browser UI:

cd ~/mycar && python manage.py drive
# → http://donkeycar.local:8887/drive

One gotcha cost ten minutes: after a Nano reboot, the page open on the phone was a stale tab pointing at the dead server’s WebSocket. The clue was server-side:

$ ss -tn | grep :8887 | grep -v 127.0.0.1
# (empty), the phone wasn't actually connected

A hard refresh re-established the 101 GET /wsDrive WebSocket upgrade and the wheels responded.

Version control on the edge

The calibration and camera config live in a fork (BioGeek/mycar). GitHub no longer accepts passwords, so it’s a Personal Access Token cached once:

git config credential.helper store   # then enter the PAT once on first push
git checkout -b jeroen_nano          # the repo was in detached HEAD at tag 4.5.1
git push -u fork jeroen_nano

What’s next

The camera works, the car drives, and it runs untethered. The next milestone is the real point: record laps → train a behavioural-cloning model on the RTX 4070 → hand control to the autopilot:

python manage.py drive              # record "tubs" of (image, steering, throttle)
donkey train --tub data --model models/pilot.h5   # on the PC, not the Nano
python manage.py drive --model models/pilot.h5

…and before training anything, look at the data, the steering-angle balance decides where the car will fail long before the model architecture does. But that’s the next post.

The transferable lessons

• Read the kernel log first. dmesg named every real fault here, the counterfeit controller (probe failed -110), the dead CSI lanes (CSI_DEBUG_COUNTER = 0), the power resets, while the surface symptoms lied.
• Distinguish “accepted” from “effective.” I²C writes succeeded into an unpowered servo; the camera bound but sent no data. Acknowledged ≠ working.
• On EOL hardware, the chip sets your software ceiling, not your preferences.
• Buy genuine peripherals. A counterfeit DS4 turned into an unwinnable fight with a built-in kernel driver.