[Bug] Latest Docker Release Breaks Foreign Key Constraint in Database

[xcjs]

Is this a support request?

• This is not a support request

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Upon Watchtower updating the Headscale container, the following log messages repeat until the container dies:

2025-01-23T16:16:11Z INF Opening database database=sqlite3 path=/var/lib/headscale/db.sqlite
2025-01-23T16:16:11Z ERR Foreign key constraint violated parent=pre_auth_keys row_id=3 table=nodes
2025-01-23T16:16:11Z FTL Migration failed: foreign key constraints violated error="foreign key constraints violated"

The container updates from the latest tag, but it looks like several other container versions were pushed at the same time. I'm assuming (but haven't verified yet) that this correlates with 0.24.1 release, but definitely the release that occurred ~20 minutes ago.

Expected Behavior

I would expect the database migration to succeed.

Steps To Reproduce

Updating the Docker container from a prior release to the release triggered 20 minutes ago on the latest tag should recreate this error.

Environment

- OS: Ubuntu 22.04 LTS
- Headscale version: Docker `:latest`
- Tailscale version: N/A

Runtime environment

• Headscale is behind a (reverse) proxy
• Headscale runs in a container

Anything else?

No response

[simonccc]

I also had this issue ( in sqlite ) but running direct with no docker - I deleted the node using cmd line and it started up ok

[kradalby]

Which release did you upgrade from? This is unfortunate, this part of the code seem to be a minefield as there has been years of no constraint, allowing a lot of stale broken nodes/routes/keys being written to the database.

The biggest challenge for me is figuring out what will blow up or needs to be removed before and what will not.

[xcjs]

@kradalby I was/am, on the :latest Docker tag, so I'm assuming whatever the prior major version was. Watchtower keeps my containers updated for me so I force myself to stay up to date on things. Based on other Docker tags out there, it was probably 0.24.0 or 0.23.0?

Whatever version was assigned to the latest tag earlier today would be the version I was on.

[xcjs]

I can also say that pulling a backup and using any other container image updated today results in the same issue, so I was not able to go back to an earlier version unless it was really old.

[LucasJanin]

The same issue, on my side, I try to upgrade my Headscale from 0.23 to 0.24.1

[kradalby]

If anyone have a database they could strip of personal info that they can send to me, then that would be helpful for me to include in a test case.

My email is in my profile.

[kradalby]

I have not found any of this issue on any of my "long running instances", so I suspect that there is an old node that has been around from before certain constraint was introduced and now they are violating.

[xcjs]

@kradalby It depends on what "long running" means in this context, but my server has been up for 2-3 years.

[LucasJanin]

Thanks @xcjs for your solution!

sudo apt install sqlite3
sudo sqlite3 db.sqlite
sqlite3> UPDATE nodes SET auth_key_id = NULL WHERE id = insert_row_id_here;
sqlite3> .exit

My Headscale v0.24.1 works perfectly!!!

[xcjs]

If anyone have a database they could strip of personal info that they can send to me, then that would be helpful for me to include in a test case.

My email is in my profile.

I can probably do this with my backup copy of my database, but allow me a few days to look into scrubbing it.

[xcjs]

Thanks @xcjs for your solution!

sudo apt install sqlite3
sudo sqlite3 db.sqlite
sqlite3> UPDATE nodes SET auth_key_id = NULL WHERE id = insert_row_id_here;
sqlite3> .exit

My Headscale v0.24.1 works perfectly!!!

Thanks for publishing this here! I was going to try and mention it in passing at least for other people facing the issue.