Merge pull request #2547 from sebix/doc-apt

docs: deb install: signed-by and wget, update postgres/eventdb docs

CHANGELOG.md

@@ -42,6 +42,7 @@
   - Add a deprecation warning (PR#2544 by Sebastian Wagner)
 
 ### Documentation
+- `docs/admin/installation/linux-packages`: Add `[signed-by=]` options, add wget command as alternative to curl (PR#2547 by Sebastian Wagner).
 
 ### Packaging
 

docs/admin/database/postgresql.md

@@ -9,10 +9,21 @@ The EventDB is a database (usually [PostgreSQL](https://postgresql.org/)) that
 gets filled with with data from IntelMQ using the SQL Output
 Bot.
 
-## intelmq_psql_initdb
+## Setup
+
+You have two basic choices to run PostgreSQL:
+
+1. on the same machine as intelmq, then you could use Unix sockets if available on your platform
+2. on a different machine. In which case you would need to use a TCP connection and make sure you give the right
+   connection parameters to each psql or client call. On the networking side, make sure the connections are allowed, postgresql-server is listening on the correct interface and the user is allowed to connect (`pg_hba.conf`).
+
+### PostgreSQL Server Version
+
+Any supported version of PostgreSQL should work (v>=13 as of January 2025) [[1]](https://www.postgresql.org/support/versioning/).
+### events table definition (`intelmq_psql_initdb`)
 
 IntelMQ comes with the `intelmq_psql_initdb` command line tool designed to help with creating the
-EventDB. It creates in the first line:
+EventDB tabls. It generates these SQL commands for you:
 
 - A `CREATE TABLE events` statement with all valid IntelMQ fields as columns and correct types
 - Several indexes as examples for a good read & search performance
@@ -21,16 +32,45 @@ Having an `events` table as outlined in the SQL file, IntelMQ's SQL Output Bot c
 
 In addition, the script supports some additional features supporting use cases described later in this document:
 
-- `--partition-key` - for generating schema aligned with TimescaleDB or partitioned tables,
-- `--separate-raws` - for generating views and triggers needed to `eventdb_raws_table` (works also together with adjustments for partitioning).
+- `--partition-key` - for generating schema aligned with TimescaleDB or partitioned tables
+- `--separate-raws` - for generating views and triggers needed to `eventdb_raws_table` (works also together with adjustments for partitioning). This can increase the performance by separating data in two tables. See section [Separating raw values in PostgreSQL using view and trigger](#separating-raw-values-in-postgresql-using-view-and-trigger) below for more information.
 
 For a full list of supported parameters, call the script help using `-h` parameter.
 
 All elements of the generated SQL file can be adapted and extended before running the SQL file against a database, especially the indexes. Please review the generated script before applying.
 
 Be aware that if you create tables using another DB user that is used later by the output bot, you may need to adjust ownership or privileges in the database. If you have problems with database permissions,
-refer to `PostgreSQL documentation <https://www.postgresql.org/docs/current/ddl-priv.html>`.
+refer to [PostgreSQL documentation](https://www.postgresql.org/docs/current/ddl-priv.html).
+
+### Installation
+
+First, install the PostgreSQL on your server using system packages or using any installation variant supported by the project: [PostgreSQL server installation](https://www.postgresql.org/download/)
 
+You need a PostgreSQL database-user to own the resulting database. The recommendation is to use the name `intelmq`
+. There may already be such a user for the PostgreSQL database-cluster to be used by other bots. (For example from
+setting up the expert/certbund_contact bot.)
+
+Therefore if still necessary: create the database-user as postgresql superuser, which usually is done via the system
+user `postgres`:
+
+```bash
+createuser --no-superuser --no-createrole --no-createdb --encrypted --pwprompt intelmq
+```
+
+Create the new database:
+
+```bash
+createdb --encoding='utf-8' --owner=intelmq intelmq-events --template template0
+```
+
+The encoding parameter should ensure the right encoding on platform where this is not the default. Template `template1` has encoding `SQL_ASCII`, so in order to use UTF8, `template0` is required.
+
+Now initialize it as database-user `intelmq` (in this example a network connection to localhost is used, so you would
+get to test if the user `intelmq` can authenticate):
+
+```bash
+psql -h localhost intelmq-events intelmq < /tmp/initdb.sql
+```
 ## EventDB Utilities
 
 Some scripts related to the EventDB are located in the
@@ -181,55 +221,7 @@ The last steps brings us several advantages:
 The complete SQL script can be generated using the `intelmq_psql_initdb`. It does *not* cover step 2 to avoid accidental
 data loss - you need to do this step manually.
 
-
-### Other docs
-
-You have two basic choices to run PostgreSQL:
-
-1. on the same machine as intelmq, then you could use Unix sockets if available on your platform
-2. on a different machine. In which case you would need to use a TCP connection and make sure you give the right
-   connection parameters to each psql or client call.
-
-Make sure to consult your PostgreSQL documentation about how to allow network connections and authentication in case 2.
-
-**PostgreSQL Version**
-
-Any supported version of PostgreSQL should work (v>=9.2 as of Oct 2016) [[1]](https://www.postgresql.org/support/versioning/).
-
-If you use PostgreSQL server v >= 9.4, it gives you the possibility to use the time-zone [formatting string](https://www.postgresql.org/docs/9.4/static/images/functions-formatting.html) "OF" for date-times and the [GiST index for the CIDR type](https://www.postgresql.org/docs/9.4/static/images/release-9-4.html#AEN120769). This may be useful depending on how you plan to use the events that this bot writes into the database.
-
-**How to install**
-
-Use `intelmq_psql_initdb` to create initial SQL statements from `harmonization.conf`. The script will create the
-required table layout and save it as `/tmp/initdb.sql`
-
-You need a PostgreSQL database-user to own the result database. The recommendation is to use the name `intelmq`
-. There may already be such a user for the PostgreSQL database-cluster to be used by other bots. (For example from
-setting up the expert/certbund_contact bot.)
-
-Therefore if still necessary: create the database-user as postgresql superuser, which usually is done via the system
-user `postgres`:
-
-```bash
-createuser --no-superuser --no-createrole --no-createdb --encrypted --pwprompt intelmq
-```
-
-Create the new database:
-
-```bash
-createdb --encoding='utf-8' --owner=intelmq intelmq-events
-```
-
-(The encoding parameter should ensure the right encoding on platform where this is not the default.)
-
-Now initialize it as database-user `intelmq` (in this example a network connection to localhost is used, so you would
-get to test if the user `intelmq` can authenticate):
-
-```bash
-psql -h localhost intelmq-events intelmq </tmp/initdb.sql
-```
-
-**PostgreSQL and null characters**
+## PostgreSQL and null characters
 
 While null characters (`0`, not SQL "NULL") in TEXT and JSON/JSONB fields are valid, data containing null characters can
 cause troubles in some combinations of clients, servers and each settings. To prevent unhandled errors and data which

docs/admin/installation/linux-packages.md

@@ -23,11 +23,16 @@ Native packages are currently provided for the following Linux distributions:
 
 ### Debian 11 and 12
 
-Add the repository to the package manager and install IntelMQ (packages `intelmq-api` and `intelmq-manager` are optional):
+1. First, add the APT repository to the package manager:
 
 ```bash
-echo "deb http://download.opensuse.org/repositories/home:/sebix:/intelmq/Debian_$(lsb_release -rs)/ /" | sudo tee /etc/apt/sources.list.d/intelmq.list
+echo "deb [signed-by=/etc/apt/trusted.gpg.d/intelmq.gpg] http://download.opensuse.org/repositories/home:/sebix:/intelmq/Debian_$(lsb_release -rs)/ /" | sudo tee /etc/apt/sources.list.d/intelmq.list
 curl -fsSL "https://download.opensuse.org/repositories/home:sebix:intelmq/Debian_$(lsb_release -rs)/Release.key" | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/intelmq.gpg > /dev/null
+# if curl is not available:
+wget "https://download.opensuse.org/repositories/home:sebix:intelmq/xUbuntu_$(lsb_release -rs)/Release.key" -O - | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/intelmq.gpg > /dev/null
+```
+2. Install the IntelMQ (packages `intelmq-api` and `intelmq-manager` are optional)
+```bash
 sudo apt update
 sudo apt install intelmq intelmq-api intelmq-manager
 ```
@@ -57,8 +62,10 @@ deb http://[...].archive.ubuntu.com/ubuntu/ focal main universe
 
 3. Next, add the IntelMQ APT Repository for Ubuntu:
 ```bash
-echo "deb http://download.opensuse.org/repositories/home:/sebix:/intelmq/xUbuntu_$(lsb_release -rs)/ /" | sudo tee /etc/apt/sources.list.d/intelmq.list
+echo "deb [signed-by=/etc/apt/trusted.gpg.d/intelmq.gpg] http://download.opensuse.org/repositories/home:/sebix:/intelmq/xUbuntu_$(lsb_release -rs)/ /" | sudo tee /etc/apt/sources.list.d/intelmq.list
 curl -fsSL "https://download.opensuse.org/repositories/home:sebix:intelmq/xUbuntu_$(lsb_release -rs)/Release.key" | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/intelmq.gpg > /dev/null
+# if curl is not available:
+wget "https://download.opensuse.org/repositories/home:sebix:intelmq/xUbuntu_$(lsb_release -rs)/Release.key" -O - | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/intelmq.gpg > /dev/null
 ```
 
 3. Now update the list of available packages and install the IntelMQ packages: