Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

There was a problem when using mavros2 to connect multiple drones with QGC . #1993

Open
XXLiu-HNU opened this issue Sep 19, 2024 · 7 comments
Open

There was a problem when using mavros2 to connect multiple drones with QGC . #1993

XXLiu-HNU opened this issue Sep 19, 2024 · 7 comments

Comments

@XXLiu-HNU
Copy link

XXLiu-HNU commented Sep 19, 2024 •
edited
Loading

Background

I want to use rtk for positioning to connect six drones and ground stations through mavros

Issue details

2246584440

When I used mavros2 to connect multiple drones to QGC, I found that when a certain drone started mavros_node, the other drones would automatically exit. I don't know if this is a bug or a problem.

Here is the way I start mavros:
ros2 run mavros mavros_node --ros-args --params-file ./mavros_param.yaml
and the mavros_param.yaml is

# mavros_param.yaml
mavros:
  ros__parameters: {}

mavros_router:
  ros__parameters: {}

mavros_node:
  ros__parameters:
    fcu_url: /dev/ttyACM0:57600
    gcs_url: udp-b://@
    tgt_system: 2

/**:
  ros__parameters:
    plugin_denylist: [odometry]
    # plugin_allowlist: []

and I change the tgt_system for different drones

MAVROS version and platform

Mavros: mavros2
ROS: galactic
Ubuntu: 20.04
MAVLink package version: 2022.8.8

Autopilot type and version

PX4

Version: 1.15.0rc

Node logs

uav1 and uav3 errors when uav5 is start, and the error is fellow:

uav1@uav3:~$ ros2 run mavros mavros_node --ros-args --params-file ./mavros_param.yaml
[INFO] [1726756281.922862848] [mavros_node]: Starting mavros_node container
[INFO] [1726756281.922909126] [mavros_node]: FCU URL: /dev/ttyACM0:57600
[INFO] [1726756281.922916008] [mavros_node]: GCS URL: udp-b://@
[INFO] [1726756281.922920488] [mavros_node]: UAS Prefix: /uas4
[INFO] [1726756281.922924413] [mavros_node]: Starting mavros router node
[INFO] [1726756281.925748876] [mavros_router]: Built-in SIMD instructions: SSE, SSE2
[INFO] [1726756281.925766617] [mavros_router]: Built-in MAVLink package version: 2022.8.8
[INFO] [1726756281.925771396] [mavros_router]: Known MAVLink dialects: common ardupilotmega ASLUAV AVSSUAS all development icarous matrixpilot paparazzi standard storm32 uAvionix ualberta
[INFO] [1726756281.925775217] [mavros_router]: MAVROS Router started
[INFO] [1726756281.925788927] [mavros_router]: Requested to add endpoint: type: 0, url: /dev/ttyACM0:57600
[INFO] [1726756281.925803037] [mavros_router]: Endpoint link[1000] created
[INFO] [1726756281.927090833] [mavros_router]: link[1000] opened successfully
[INFO] [1726756281.927112521] [mavros_router]: Requested to add endpoint: type: 1, url: udp-b://@
[INFO] [1726756281.927122361] [mavros_router]: Endpoint link[1001] created
[INFO] [1726756281.927202061] [mavros_router]: link[1001] opened successfully
[INFO] [1726756281.927218447] [mavros_router]: Requested to add endpoint: type: 2, url: /uas4
[INFO] [1726756281.927220520] [mavros_router]: link[1000] detected remote address 0.1
[INFO] [1726756281.927241251] [mavros_router]: Endpoint link[1002] created
[INFO] [1726756281.927441718] [mavros_router]: link[1002] opened successfully
[INFO] [1726756281.927454924] [mavros_node]: Starting mavros uas node
[INFO] [1726756281.927534791] [mavros_router]: link[1000] detected remote address 4.1
[INFO] [1726756281.957240586] [mavros]: UAS Executor started, threads: 16
[INFO] [1726756281.975721015] [mavros]: Plugin actuator_control created
[INFO] [1726756281.975757845] [mavros]: Plugin actuator_control initialized
[INFO] [1726756281.976581462] [mavros]: Plugin altitude created
[INFO] [1726756281.976603198] [mavros]: Plugin altitude initialized
[INFO] [1726756281.977995046] [mavros]: Plugin command created
[INFO] [1726756281.978017750] [mavros]: Plugin command initialized
[INFO] [1726756281.979978187] [mavros]: Plugin ftp created
[INFO] [1726756281.980002933] [mavros]: Plugin ftp initialized
[INFO] [1726756281.981305449] [mavros]: Plugin geofence created
[INFO] [1726756281.981355501] [mavros]: Plugin geofence initialized
[INFO] [1726756281.985906824] [mavros]: Plugin global_position created
[INFO] [1726756281.986009033] [mavros]: Plugin global_position initialized
[INFO] [1726756281.988579025] [mavros]: Plugin home_position created
[INFO] [1726756281.988613618] [mavros]: Plugin home_position initialized
[INFO] [1726756281.991209040] [mavros]: Plugin imu created
[INFO] [1726756281.991259109] [mavros]: Plugin imu initialized
[INFO] [1726756281.994265236] [mavros]: Plugin local_position created
[INFO] [1726756281.994308309] [mavros]: Plugin local_position initialized
[INFO] [1726756282.074782970] [mavros_router]: link[1001] detected remote address 255.190
[INFO] [1726756282.310627835] [mavros]: Plugin manual_control created
[INFO] [1726756282.310805111] [mavros]: Plugin manual_control initialized
[INFO] [1726756282.316189927] [mavros]: Plugin nav_controller_output created
[INFO] [1726756282.316266108] [mavros]: Plugin nav_controller_output initialized
[INFO] [1726756282.320137985] [mavros]: Plugin param created
[INFO] [1726756282.320189626] [mavros]: Plugin param initialized
[INFO] [1726756282.322850031] [mavros]: Plugin rallypoint created
[INFO] [1726756282.322895096] [mavros]: Plugin rallypoint initialized
[INFO] [1726756282.324689943] [mavros]: Plugin rc_io created
[INFO] [1726756282.324712623] [mavros]: Plugin rc_io initialized
[INFO] [1726756282.326007651] [mavros]: Plugin setpoint_accel created
[INFO] [1726756282.326029816] [mavros]: Plugin setpoint_accel initialized
[INFO] [1726756282.327890839] [mavros]: Plugin setpoint_attitude created
[INFO] [1726756282.327963454] [mavros]: Plugin setpoint_attitude initialized
[INFO] [1726756282.330185892] [mavros]: Plugin setpoint_position created
[INFO] [1726756282.330278112] [mavros]: Plugin setpoint_position initialized
[INFO] [1726756282.334626696] [mavros]: Plugin setpoint_raw created
[INFO] [1726756282.334659661] [mavros]: Plugin setpoint_raw initialized
[INFO] [1726756282.336718922] [mavros]: Plugin setpoint_trajectory created
[INFO] [1726756282.336743802] [mavros]: Plugin setpoint_trajectory initialized
[INFO] [1726756282.338202145] [mavros]: Plugin setpoint_velocity created
[INFO] [1726756282.338222068] [mavros]: Plugin setpoint_velocity initialized
[INFO] [1726756282.341886677] [mavros]: Plugin sys_status created
[INFO] [1726756282.341931856] [mavros]: Plugin sys_status initialized
[INFO] [1726756282.343267584] [mavros.time]: TM: Timesync mode: MAVLINK
[INFO] [1726756282.343709014] [mavros]: Plugin sys_time created
[INFO] [1726756282.343736637] [mavros]: Plugin sys_time initialized
[INFO] [1726756282.358709375] [mavros]: Plugin waypoint created
[INFO] [1726756282.358772551] [mavros]: Plugin waypoint initialized
[INFO] [1726756282.360270290] [mavros]: Plugin wind_estimation created
[INFO] [1726756282.360298178] [mavros]: Plugin wind_estimation initialized
[INFO] [1726756282.360532042] [mavros]: Built-in SIMD instructions: SSE, SSE2
[INFO] [1726756282.360541890] [mavros]: Built-in MAVLink package version: 2022.8.8
[INFO] [1726756282.360547198] [mavros]: Known MAVLink dialects: common ardupilotmega ASLUAV AVSSUAS all development icarous matrixpilot paparazzi standard storm32 uAvionix ualberta
[INFO] [1726756282.360552035] [mavros]: MAVROS UAS via /uas4 started. MY ID 4.191, TARGET ID 4.1
[INFO] [1726756282.373216550] [mavros.imu]: IMU: High resolution IMU detected!
[INFO] [1726756282.398556390] [mavros.imu]: IMU: Attitude quaternion IMU detected!
[WARN] [1726756282.411778863] [mavros.global_position]: GP: No GPS fix
[INFO] [1726756282.448558674] [mavros_router]: link[1002] detected remote address 4.191
[INFO] [1726756282.944918461] [mavros]: CON: Got HEARTBEAT, connected. FCU: PX4 Autopilot
[INFO] [1726756282.945112595] [mavros.mission]: WP: detected enable_partial_push: 0
[INFO] [1726756282.984019731] [mavros.imu]: IMU: High resolution IMU detected!
[INFO] [1726756282.998194387] [mavros.imu]: IMU: Attitude quaternion IMU detected!
[INFO] [1726756283.953640891] [mavros.geofence]: GF: Using MISSION_ITEM_INT
[INFO] [1726756283.953733248] [mavros.rallypoint]: RP: Using MISSION_ITEM_INT
[INFO] [1726756283.953759662] [mavros.mission]: WP: Using MISSION_ITEM_INT
[INFO] [1726756283.953788627] [mavros.sys]: VER: 4.1: Capabilities         0x000000000000e8ff
[INFO] [1726756283.953823090] [mavros.sys]: VER: 4.1: Flight software:     010f00c0 (30e763b678000000)
[INFO] [1726756283.953839556] [mavros.sys]: VER: 4.1: Middleware software: 010f00c0 (30e763b678000000)
[INFO] [1726756283.953852338] [mavros.sys]: VER: 4.1: OS software:         0b0000ff (5d74bc138955e6f0)
[INFO] [1726756283.953866991] [mavros.sys]: VER: 4.1: Board hardware:      0000004c
[INFO] [1726756283.953884336] [mavros.sys]: VER: 4.1: VID/PID:             3163:004c
[INFO] [1726756283.953897633] [mavros.sys]: VER: 4.1: UID:                 3532510131373432
[WARN] [1726756283.954070517] [mavros.cmd]: CMD: Unexpected command 520, result 0
[INFO] [1726756285.555390144] [mavros.sys]: VER: 4.1: Capabilities         0x000000000000e8ff
[INFO] [1726756285.555483863] [mavros.sys]: VER: 4.1: Flight software:     010f00c0 (30e763b678000000)
[INFO] [1726756285.555514334] [mavros.sys]: VER: 4.1: Middleware software: 010f00c0 (30e763b678000000)
[INFO] [1726756285.555536545] [mavros.sys]: VER: 4.1: OS software:         0b0000ff (5d74bc138955e6f0)
[INFO] [1726756285.555558460] [mavros.sys]: VER: 4.1: Board hardware:      0000004c
[INFO] [1726756285.555580667] [mavros.sys]: VER: 4.1: VID/PID:             3163:004c
[INFO] [1726756285.555602129] [mavros.sys]: VER: 4.1: UID:                 3532510131373432
[INFO] [1726756292.944532846] [mavros.home_position]: HP: requesting home position
[INFO] [1726756297.945619652] [mavros.mission]: WP: mission received
[INFO] [1726756302.946006256] [mavros.rallypoint]: RP: mission received
[INFO] [1726756307.948471095] [mavros.geofence]: GF: mission received
[WARN] [1726756312.501674653] [mavros.global_position]: GP: No GPS fix
[WARN] [1726756342.598439215] [mavros.global_position]: GP: No GPS fix
[ERROR] [1726756347.992185007] [mavros.sys]: FCU: Preflight Fail: Accel 2 inconsistent - check cal
[WARN] [1726756372.696094726] [mavros.global_position]: GP: No GPS fix
[INFO] [1726756388.179963394] [mavros.sys]: VER: 4.1: Capabilities         0x000000000000e8ff
[INFO] [1726756388.180059470] [mavros.sys]: VER: 4.1: Flight software:     010f00c0 (30e763b678000000)
[INFO] [1726756388.180098820] [mavros.sys]: VER: 4.1: Middleware software: 010f00c0 (30e763b678000000)
[INFO] [1726756388.180133746] [mavros.sys]: VER: 4.1: OS software:         0b0000ff (5d74bc138955e6f0)
[INFO] [1726756388.180161537] [mavros.sys]: VER: 4.1: Board hardware:      0000004c
[INFO] [1726756388.180188447] [mavros.sys]: VER: 4.1: VID/PID:             3163:004c
[INFO] [1726756388.180213958] [mavros.sys]: VER: 4.1: UID:                 3532510131373432
[WARN] [1726756388.187023166] [mavros.cmd]: CMD: Unexpected command 520, result 0
terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
  what():  failed to send response: error not set, at /tmp/binarydeb/ros-galactic-rcl-3.1.4/src/rcl/service.c:287
@vooon vooon mentioned this issue Sep 19, 2024
Closed
@ajshank
Copy link
Contributor

ajshank commented Oct 11, 2024

Bumping this up. This is indeed a problem when trying to connect multiple vehicles/instances. Essentially what seems to be happening is that there is a ton of multicast collisions when a bunch of nodes boot up, and this can cause several packets/topic updates to be missed. We have tested this repeatably on Humble/Jazzy, but interestingly, cannot replicate on Foxy so far.

@XXLiu-HNU One way to "fix" the problem is to use ROS_DOMAIN_ID and containerize ROS nodes. Can also use permissions and discovery to limit chatter across nodes. This works perfectly (tested), but will cut all topic-based communication between nodes.

@XXLiu-HNU
Copy link
Author

We always encounter some problems when using ros2, and finally choose to return to ROS1

@ajshank
Copy link
Contributor

ajshank commented Dec 28, 2024

@XXLiu-HNU yes, but I don't think that's a good long-term solution, and, at the very least, does not address this particular issue. If you don't mind, I'd recommend reopening this so that it gets attention!

@XXLiu-HNU
Copy link
Author

XXLiu-HNU commented Dec 29, 2024 •
edited
Loading

Yes, we found many problems when using ROS2 for drone cluster experiments, one of which was the QGC link. In addition, when the number of ROS2 devices increased (six drones), the network often crashed.

@XXLiu-HNU yes, but I don't think that's a good long-term solution, and, at the very least, does not address this particular issue. If you don't mind, I'd recommend reopening this so that it gets attention!

@XXLiu-HNU XXLiu-HNU reopened this Dec 29, 2024
@vooon
Copy link
Member

vooon commented Jan 2, 2025

Just a guess - you use the same node name (and so namespace for all the plugins). For multidrone you'd better to manually run Router and UAS nodes.

@lonk10
Copy link

lonk10 commented Jan 9, 2025

@vooon Just a guess - you use the same node name (and so namespace for all the plugins). For multidrone you'd better to manually run Router and UAS nodes.

Hello, not sure about OP, but I'm running in the same problem as @ajshank and @XXLiu-HNU. While I have the same node name for various vehicles, they are all in different namespaces (eg. /dom1/v1/nameX and /dom1/v2/nameX)

@ajshank
Copy link
Contributor

ajshank commented Jan 10, 2025

@vooon No, the namespaces are all different (and so the fully-resolved node name should also be). To be clear, this happens when N nodes under different namespaces are active on the same wireless network (same host or different host). It appears that VER broadcast request happens sporadically (and connection timeouts) and the Router tries to connect to a different mavlink vehicle. Each of our FCU URLs are tty devices on serial wire (one for each mavros instance, different machines), so my guess is that the Router is trying to connect/discover over wireless/UDP. I understand such routing is essential in mavlink, but why it succeeds in connecting/reconnecting is a mystery. It would be nice to be able to limit the Router to find/route mavlink packets only over certain interfaces/ports (?).

That being said, I think my solution above using ROS discovery limiting does work -- I will try larger numbers next week, but my tests with 2-3 connections work as expected (Jazzy).

My N instances are on N different host machines, so I can use discovery limiting. If those were all on the same machine, one could use ROS_DOMAIN_ID to compartmentalise them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@vooon @ajshank @lonk10 @XXLiu-HNU