Notes & to-do's post clean-install

[lentinj]

I've been doing some spring cleaning, and as part of it rebuilding my aged OneZoom container. My notes on doing so are:

git clone https://github.com/web2py/web2py /srv/web2py
git -C /srv/web2py checkout v2.26.1
git -C /srv/web2py submodule update --init --recursive
git clone [email protected]:OneZoom/OZtree.git /srv/web2py/applications/OZtree
git clone [email protected]:OneZoom/tours.git /srv/web2py/applications/tours
cp /srv/web2py/applications/OZtree/_COPY_CONTENTS_TO_WEB2PY_DIR/routes.py /srv/web2py/routes.py

apt install nginx ssl-cert
cat <<'EOF' > /etc/nginx/sites-enabled/onezoom
server {
    listen      80;
    listen      443 ssl;
    server_name onezoom.* beta.onezoom.org;
    charset     utf-8;
    gzip        on;

    proxy_intercept_errors on;

    ssl_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem;
    ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key;

    location / {
        proxy_pass          https://localhost:8000;
        proxy_ssl_verify off; # NB: Rocket has no provision for NGINX-terminated SSL, the response is https if the input is.
        proxy_set_header    Host            $host;
        proxy_set_header    X-Real-IP       $remote_addr;
        proxy_set_header    X-Forwarded-for $remote_addr;
        proxy_set_header    X-Forwarded-Proto $scheme;
        proxy_set_header    Upgrade $http_upgrade;
        proxy_set_header    Connection $http_connection;
    }
}
EOF
openssl req -newkey rsa:2048 -x509 -days 365 -nodes -keyout /srv/web2py/oz.key -out /srv/web2py/oz.crt

cat <<EOF > /etc/apt/sources.list.d/nodesource.list
deb [signed-by=/usr/share/keyrings/nodesource.gpg] https://deb.nodesource.com/node_16.x bullseye main
EOF
curl -s https://deb.nodesource.com/gpgkey/nodesource.gpg.key | gpg --dearmor | tee /usr/share/keyrings/nodesource.gpg >/dev/null
apt install nodejs
npm ci
./node_modules/.bin/grunt minimal-install
./node_modules/.bin/grunt dev

apt install lsb-release
wget https://dev.mysql.com/get/mysql-apt-config_0.8.25-1_all.deb
dpkg -i mysql-apt-config_0.8.25-1_all.deb
apt install mysql-server
# NB: Select "Use Legacy Authentication Method (Retain MySQL 5.x Compatibility)"
mysql -p
CREATE DATABASE OneZoom;
CREATE USER 'oz'@'localhost' IDENTIFIED BY 'passwd';
GRANT ALL PRIVILEGES ON OneZoom . * TO 'oz'@'localhost';

apt install python3
apt install python3-pymysql  # TODO: This isn't a good long-term solution, we need https://github.com/OneZoom/OZtree/issues/676
cp private/appconfig.ini.example private/appconfig.ini
edit private/appconfig.ini, migrate=1
python3 web2py.py -S OZtree -M -R applications/OZtree/tests/unit/test_modules_sponsorship.py
# NB: Tests will fail, but migrate will happen
edit private/appconfig.ini, migrate=0

# NB: OneZoom.dump.sql came from the docker image
cp /var/lib/containers/storage/overlay/*/merged/OneZoom.dump.sql /srv/lxc/onezoom/rootfs/OneZoom.dump.sql
mysql -p
USE OneZoom
SOURCE /OneZoom.dump.sql

• An nginx config that's closer to the production one
• I bodged the old web2py so https:// was passed through, either fix the nginx config or re-bodge web2py. (We can't force pretend-to-be-HTTPS whilst using rocket, so the proxy-pass connection has to be HTTPS as well)
• Tidy up package.json, so that npm ci can be used instead (solved in 0d5a068)
• Web2py causing issues with grunt test-server #589 is still problematic, if setting the encoding is our solution then we should at least alter the example to suit. (the web2py copy can be overriden, we need a common way of doing so: A venv for web2py #676)
• A grunt task to generate snake-oil SSL cert if required & run web2py
• I didn't assume python 3.7 in the above, installed python 3.9 and Grunt gets upset.

Obviously feeding this into README.markdown would be a good choice, even if it is a bit Debian-centric:

• Most of the dependencies in README.markdown aren't necessary for running OneZoom, even more so now we have the tree-build repo. I've never installed most of the Python & Perl dependencies. @hyanwong could we tidy this up?
• npm will install grunt at ./node_modules/.bin/grunt, so no need to suggest installing your own.
• README.markdown should link to ./private/appconfig.ini.example, don't copy-paste a separate copy

[hyanwong]

Most of the dependencies in README.markdown aren't necessary for running OneZoom, even more so now we have the tree-build repo. I've never installed most of the Python & Perl dependencies. @hyanwong could we tidy this up?

I am going to try switching over to using the tree-build repo this weekend. I will try to tidy up the dependency list then.

[davidebbo]

I am going to try switching over to using the tree-build repo this weekend. I will try to tidy up the dependency list then.

Note that in the tree-build repo, the dependencies are resolved automatically via setup.cfg (see https://github.com/OneZoom/tree-build/blob/main/setup.cfg#L12-L19). So there is no manual step. Also, it's all Python at this point, with no Perl.

[lentinj]

I'm guessing the remaining Perl scripts are ancient history nowadays, and if we don't delete them then at least their dependencies don't need to be this prominent.

Note that pymysql or mysql-connector-python aren't dependencies of the site, as web2py bundles it's own copy. All the listed python dependencies are for things in OZprivate/ServerScripts/Utilities AFAICS. Maybe there should be a requirements.txt in that directory, and a README.md explaining how to install them? (A package like @davidebbo has done would be better, but I'm guessing that's too much for code that is probably very infrequently used).

[davidebbo]

The new tree-build code supersedes subtree_extract.pl and tree_and_meta_parser.pl (in OZprivate/ServerScripts/TreeBuild). There are various other Perl scripts scattered around that I'm not familiar with and can't comment on.

[lentinj]

@hyanwong According to README.markdown we assume Python 3.7, and the Gruntfile has python3.7 hard-coded in it. I'm guessing this assumption needs to move with the times a bit, 3.7 is about to fall off the security updates radar. Any preferences what to? Are the server(s) still on 3.7?

[hyanwong]

Good point. Let's move to something more modern. I'll ping you on slack. 3.8 is possible I think.

[davidebbo]

FWIW, I've been using 3.10 since I got started on OZ.

[hyanwong]

I guess we should just go with 3.10, to avoid too many further updates? I will try this now.

[davidebbo]

I've been on 3.10 without issues, so that's probably a safe bet. That being said, I see the latest stable is now 3.12, and the 3.11 release notes claim some notable perf improvements, so it may be worth exploring.

And as a side note, it would be nice to move OZtree to a virtual env, like we have for tree-build. It avoids having to install systemwide packages (which can cause conflicts, ...).

[hyanwong]

Yes, both true. I agree about venvs, and perhaps we should just bite the bullet and go for 3.12?

[lentinj]

perhaps we should just bite the bullet and go for 3.12?

Debian stable (which isn't that old atm) is only at 3.11. Whilst we don't actively use Debian, I'd be wary of assuming newer than stable without some justification.

venvs

A venv for OZtree will need to happen soon, when switching operating systems/python versions is the obvious time.

[hyanwong]

Ah, and I see FreeBSD 13.2 (which is what we use on the server) only has 3.8 by default, although I guess I can install this: https://www.freshports.org/lang/python311?

[lentinj]

I don't know much BSD, but presumably so.

[hyanwong]

These are the npm messages I get when I try a clean install on my laptop (OS X). I assume most of these can be ignored (anyway, the node modules are only used to compile static JS code, so are presumably not security risks)

(py311) yan@Yans-New-Air OZtree % npm --v
10.2.5
(py311) yan@Yans-New-Air OZtree % npm install              
npm WARN deprecated [email protected]: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).
npm WARN deprecated [email protected]: This module moved to @hapi/sntp. Please make sure to switch over as this distribution is no longer supported and may contain bugs and critical security issues.
npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).
npm WARN deprecated [email protected]: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).
npm WARN deprecated [email protected]: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).
npm WARN deprecated [email protected]: Use your platform's native performance.now() and performance.timeOrigin.
npm WARN deprecated [email protected]: Chokidar 2 does not receive security updates since 2019. Upgrade to chokidar 3 with 15x fewer dependencies
npm WARN deprecated [email protected]: This module moved to @hapi/hawk. Please make sure to switch over as this distribution is no longer supported and may contain bugs and critical security issues.
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142

added 1215 packages, and audited 1426 packages in 43s

102 packages are looking for funding
  run `npm fund` for details

34 vulnerabilities (9 moderate, 23 high, 2 critical)

To address issues that do not require attention, run:
  npm audit fix

To address all issues possible (including breaking changes), run:
  npm audit fix --force

[hyanwong]

To get grunt to work, I also needed to:

export NODE_OPTIONS=--openssl-legacy-provider

(see https://stackoverflow.com/questions/69692842/error-message-error0308010cdigital-envelope-routinesunsupported; npm audit fix --force did not work for me

[lentinj]

(anyway, the node modules are only used to compile static JS code, so are presumably not security risks)

Yes, we vendor very little, if any, Javascript into the client. And all the server-side javascript we do run is just for builds

export NODE_OPTIONS=--openssl-legacy-provider

Upgrading will be the way to solve this though. Again, not a vast security risk, but eventually this cheating option will disappear as they get bored of maintaining old OpenSSL builds.

[hyanwong]

export NODE_OPTIONS=--openssl-legacy-provider

Upgrading will be the way to solve this though. Again, not a vast security risk, but eventually this cheating option will disappear as they get bored of maintaining old OpenSSL builds.

I'm not sure what to upgrade here? npm is on 10.2.5.

[lentinj]

I'm not sure what to upgrade here?

One of our NPM dependencies will be requiring the option to work, and will need bumping / removing.

[hyanwong]

Tidy up package.json, so that npm ci can be used instead

In #687 I have upgraded to webpack 5.0.0, and it seems to compile the site just fine: I haven't checked npm ci though.

I didn't assume python 3.7 in the above, installed python 3.9 and Grunt gets upset.

Also in #687 I installed it all using a conda install of python 3.11, and it seems fine, so I have changed the hardcoded python version in the Gruntfile to 3.11, on the assumption that we'll be able to install that new version on the new server soon.

[lentinj]

I haven't checked npm ci though.

npm ci is a simpler version of npm install. npm install will update transitive dependencies in package-lock.json if it wants to. npm ci OTOH refuses to do so, it just installs what's in package-lock.json.

So once npm install works without making a change to package-lock.json, npm ci should similarly be happy.

[hyanwong]

So with #687 I get

(py311) yan@Yans-New-Air OZtree % npm ci
npm WARN deprecated [email protected]: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).
npm WARN deprecated [email protected]: This module moved to @hapi/sntp. Please make sure to switch over as this distribution is no longer supported and may contain bugs and critical security issues.
npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).
npm WARN deprecated [email protected]: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).
npm WARN deprecated [email protected]: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).
npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated [email protected]: This module moved to @hapi/hawk. Please make sure to switch over as this distribution is no longer supported and may contain bugs and critical security issues.
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated @babel/[email protected]: 🚨 This package has been deprecated in favor of separate inclusion of a polyfill and regenerator-runtime (when needed). See the @babel/polyfill docs (https://babeljs.io/docs/en/babel-polyfill) for more information.
npm WARN deprecated [email protected]: Package no longer supported. Contact Support at https://www.npmjs.com/support for more info.
npm WARN deprecated [email protected]: core-js@<3.23.3 is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could cause a slowdown up to 100x even if nothing is polyfilled. Some versions have web compatibility issues. Please, upgrade your dependencies to the actual version of core-js.

added 984 packages, and audited 1220 packages in 8s

145 packages are looking for funding
  run `npm fund` for details

7 vulnerabilities (4 moderate, 2 high, 1 critical)

I guess that's fine @lentinj : would you recommend that I try to update anything else (e.g. the ones that are mentioned there), or would this break stuff?