Skip to content

Larger RL models cause locationd errors on long drives #38505

Description

@tfpgh

This issue can be seen in this segment from @Karnbir.

Not 100% confident but seems like:

  • camerad's msgq endpoint for each camera stream has 15 reader slots. These aren't reclaimed when a VisionIpcClient is destroyed
  • When the car passes through 5m/s, the UI switches the displayed camera by creating a new VisionIpcClient, which consumes another reader slot for the target stream
  • Before this error, there had been 13 fcam to ecam transitions during the drive. Those + modeld reader + encoderd reader filled all 15 ecam slots
  • When the car slowed below 5m/s again, the UI registered a new client which reset all all existing reader slots, including modeld's ecam reader
  • With the RL model, modeld finished processing frame 28087 ~4ms after the reset. The smaller, 7ms faster model, probably would have let modeld recieve the ecam frame before the reset
  • When modeld tried to read the next ecam frame it had to re-register it's reader. This skipped over frame 28088 so modeld ran on ecam frame 28089 and fcam frame 28088
  • On the next iteration, modeld ran on frame 28090 from both streams. It detected the dropped fcam frame 28089 and marked cameraOdometry with valid=false
  • locationd tolerates one bad camera-odometry input as long as it reject it itself. If it comes in marked as valid=false, locationd immediately makes inputsOK=false

Seems like a lot went wrong here. This might cause issues on big models as well but not sure how specific the timing issue is.

Metadata

Metadata

Assignees

Labels

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions