update next major

.github/workflows/lock-threads.yml

@@ -0,0 +1,24 @@
+name: 'Lock Threads'
+
+on:
+  schedule:
+    - cron: '0 * * * *'
+  workflow_dispatch:
+
+permissions:
+  issues: write
+  pull-requests: write
+  discussions: write
+
+concurrency:
+  group: lock-threads
+
+jobs:
+  action:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: dessant/lock-threads@v5
+        with:
+          issue-inactive-days: 30
+          pr-inactive-days: 30
+          log-output: true

CHANGELOG.md

@@ -1,13 +1,96 @@
-# NEXT RELEASE
+# NEXT-MAJOR NEXT RELEASE
 
 ### Enhancements
 * <New feature description> (PR [#????](https://github.com/realm/realm-core/pull/????))
+
+### Fixed
+* <How do the end-user experience this issue? what was the impact?> ([#????](https://github.com/realm/realm-core/issues/????), since v?.?.?)
+* None.
+
+### Breaking changes
+* None.
+
+### Compatibility
+* Fileformat: Generates files with format v24. Reads and automatically upgrade from fileformat v10. If you want to upgrade from an earlier file format version you will have to use RealmCore v13.x.y or earlier.
+
+-----------
+
+### Internals
+* None.
+
+----------------------------------------------
+
+# 14.4.1 Release notes
+
+### Fixed
+* Fix pass a thread safe reference to init subscription callback. ([#7497](https://github.com/realm/realm-core/issues/7497), since v13.16.0)
+
+### Compatibility
+* Fileformat: Generates files with format v24. Reads and automatically upgrade from fileformat v10. If you want to upgrade from an earlier file format version you will have to use RealmCore v13.x.y or earlier.
+
+-----------
+
+### Internals
+* Update Catch2 from v3.3.2 to v3.5.3 ([PR #7297](https://github.com/realm/realm-core/pull/7509)).
+
+----------------------------------------------
+
+# 14.4.0 Release notes
+
+### Enhancements
+* Nested path included in 'OutOfBoundsø error message ([#7438](https://github.com/realm/realm-core/issues/7438))
+* Improve file compaction performance on platforms with page sizes greater than 4k (for example arm64 Apple platforms) for files less than 256 pages in size ([PR #7492](https://github.com/realm/realm-core/pull/7492)).
+
+### Fixed
+* Modifying nested collections left the accessor used to make the modification in a stale state, resulting in some unneccesary work being done when making multiple modifications via one accessor ([PR #7470](https://github.com/realm/realm-core/pull/7470), since v14.0.0).
+* Fix depth level for nested collection in debug mode, set it to the same level as release ([#7484](https://github.com/realm/realm-core/issues/7484), since v14.0.0).
+* Fix opening realm with cached user while offline results in fatal error and session does not retry connection. ([#7349](https://github.com/realm/realm-core/issues/7349), since v13.26.0)
+* Fix disallow Sets in ArrayMixed. ([#7502](https://github.com/realm/realm-core/pull/7502), since v14.0.0)
+
+### Breaking changes
+* Update C-API log callback signature to include the log category, and `realm_set_log_callback` to not take a `realm_log_level_e`. ([PR #7494](https://github.com/realm/realm-core/pull/7494)
+
+### Compatibility
+* Fileformat: Generates files with format v24. Reads and automatically upgrade from fileformat v10. If you want to upgrade from an earlier file format version you will have to use RealmCore v13.x.y or earlier.
+
+----------------------------------------------
+
+# 14.3.0 Release notes
+
+### Enhancements
+* Add support to synchronize collections embedded in Mixed properties and other collections (except sets) ([PR #7353](https://github.com/realm/realm-core/pull/7353)).
+* Improve performance of change notifications on nested collections somewhat ([PR #7402](https://github.com/realm/realm-core/pull/7402)).
+* Improve performance of aggregate operations on Dictionaries of objects, particularly when the dictionaries are empty ([PR #7418](https://github.com/realm/realm-core/pull/7418))
+* Added Resumption delay configuration to SyncClientTimeouts. ([PR #7441](https://github.com/realm/realm-core/pull/7441))
+
+### Fixed
+* Fixed conflict resolution bug which may result in an crash when the AddInteger instruction on Mixed properties is merged against updates to a non-integer type ([PR #7353](https://github.com/realm/realm-core/pull/7353)).
+* Fix a spurious crash related to opening a Realm on background thread while the process was in the middle of exiting ([#7420](https://github.com/realm/realm-core/issues/7420jj))
+* Fix a data race in change notification delivery when running at debug log level ([PR #7402](https://github.com/realm/realm-core/pull/7402), since v14.0.0).
+* Fix a 10-15% performance regression when reading data from the Realm resulting from Obj being made a non-trivial type ([PR #7402](https://github.com/realm/realm-core/pull/7402), since v14.0.0).
+
+### Breaking changes
+* Remove `realm_scheduler_set_default_factory()` and `realm_scheduler_has_default_factory()`, and change the `Scheduler` factory function to a bare function pointer rather than a `UniqueFunction` so that it does not have a non-trivial destructor.
+
+### Compatibility
+* Fileformat: Generates files with format v24. Reads and automatically upgrade from fileformat v10. If you want to upgrade from an earlier file format version you will have to use RealmCore v13.x.y or earlier.
+
+-----------
+
+### Internals
+* The CMake option `REALM_MONGODB_ENDPOINT` for running the object-store-tests against baas has been deprecated in favor of an environment variable of the same name ([PR #7423](https://github.com/realm/realm-core/pull/7423)).
+* The object-store-tests test suite can now launch baas containers on its own by specifying a `BAASAAS_API_KEY` in the environment ([PR #7423](https://github.com/realm/realm-core/pull/7423)).
+
+----------------------------------------------
+
+# 14.2.0 Release notes
+
+### Enhancements
 * Added ability to get current log level via C API (PR [#7419](https://github.com/realm/realm-core/pull/7419))
 * Improve performance of object notifiers with complex schemas and very simple changes to process by as much as 20% ([PR #7424](https://github.com/realm/realm-core/pull/7424)).
 * Improve performance with very large number of notifiers as much as 75% ([PR #7424](https://github.com/realm/realm-core/pull/7424)).
 
 ### Fixed
-* <How do the end-user experience this issue? what was the impact?> ([#????](https://github.com/realm/realm-core/issues/????), since v?.?.?)
 * Fixed an issue when removing items from a LnkLst that could result in invalidated links becoming visable which could cause crashes or exceptions when accessing those list items later on. This affects sync Realms where another client had previously removed a link in a linklist that has over 1000 links in it, and then further local removals from the same list caused the list to have fewer than 1000 items. ([#7414](https://github.com/realm/realm-core/pull/7414), since v10.0.0)
 * Query lists vs lists if the property to check is a path with wildcards would not give correct result. This has for a long time also been a problem for queries with linklist properties in the path ([#7393](https://github.com/realm/realm-core/issues/7393), since v14.0.0)
 

Package.swift

@@ -3,7 +3,7 @@
 import PackageDescription
 import Foundation
 
-let versionStr = "14.1.0"
+let versionStr = "14.4.1"
 let versionPieces = versionStr.split(separator: "-")
 let versionCompontents = versionPieces[0].split(separator: ".")
 let versionExtra = versionPieces.count > 1 ? versionPieces[1] : ""

bindgen/spec.yml

@@ -35,6 +35,7 @@ primitives:
   - void
   - std::string
   - std::string_view
+  - std::chrono::milliseconds
   # TODO: see if we need nullable versions of these. For now null buffers are treated as empty.
   - StringData
   - OwnedBinaryData
@@ -553,6 +554,22 @@ records:
       before_notify: 'Nullable<util::UniqueFunction<(r: SharedRealm)>>'
       schema_did_change: 'Nullable<util::UniqueFunction<(r: SharedRealm)>>' # new schema available as r.schema
 
+  ResumptionDelayInfo:
+    cppName: sync::ResumptionDelayInfo
+    fields:
+      max_resumption_delay_interval:
+        type: std::chrono::milliseconds
+        default: 3000000
+      resumption_delay_interval:
+        type: std::chrono::milliseconds
+        default: 1000
+      resumption_delay_backoff_multiplier:
+        type: int
+        default: 2
+      delay_jitter_divisor:
+        type: int
+        default: 4
+
   SyncClientTimeouts:
     fields:
       connect_timeout:
@@ -570,6 +587,9 @@ records:
       fast_reconnect_limit:
         type: uint64_t
         default: 60000
+      reconnect_backoff_info:
+        type: ResumptionDelayInfo
+        default: {}
 
   SyncClientConfig:
     fields:
@@ -1324,7 +1344,7 @@ classes:
     methods:
       start: '(callback: AsyncCallback<(realm: Nullable<ThreadSafeReference>, error: Nullable<std::exception_ptr>) off_thread>)'
       cancel: ()
-      register_download_progress_notifier: '(callback: (transferred_bytes: uint64_t, transferrable_bytes: uint64_t) off_thread) -> uint64_t'
+      register_download_progress_notifier: '(callback: (transferred_bytes: uint64_t, transferrable_bytes: uint64_t, progress_estimate: double) off_thread) -> uint64_t'
       unregister_download_progress_notifier: '(token: uint64_t)'
 
   SyncSession:
@@ -1335,11 +1355,11 @@ classes:
       path: std::string
       user: SharedSyncUser
       config: SyncConfig
-      full_realm_url: std::optional<std::string>
+      full_realm_url: std::string
     methods:
       wait_for_upload_completion: '(callback: AsyncCallback<(err: Status) off_thread>)'
       wait_for_download_completion: '(callback: AsyncCallback<(err: Status) off_thread>)'
-      register_progress_notifier: '(callback: (transferred_bytes: uint64_t, transferrable_bytes: uint64_t) off_thread, direction: ProgressDirection, is_streaming: bool) -> uint64_t'
+      register_progress_notifier: '(callback: (transferred_bytes: uint64_t, transferrable_bytes: uint64_t, progress_estimate: double) off_thread, direction: ProgressDirection, is_streaming: bool) -> uint64_t'
       unregister_progress_notifier: '(token: uint64_t)'
       register_connection_change_callback: '(callback: (old_state: SyncSessionConnectionState, new_state: SyncSessionConnectionState) off_thread) -> uint64_t'
       unregister_connection_change_callback: '(token: uint64_t)'

dependencies.yml

@@ -1,7 +1,7 @@
 PACKAGE_NAME: realm-core
-VERSION: 14.1.0
+VERSION: 14.4.1
 OPENSSL_VERSION: 3.2.0
 ZLIB_VERSION: 1.2.13
 # https://github.com/10gen/baas/commits
-# dd016 is 2024 Feb 22
-BAAS_VERSION: dd01629d83b86292af9c59ebe2a28673c2e559cf
+# 6bf5e1 is 2024 Mar 20
+BAAS_VERSION: 6bf5e111499ba32d04224e2fc10acca3595edb20

evergreen/config.yml

@@ -78,7 +78,6 @@ functions:
           if [ -z "${disable_tests_against_baas|}" ]; then
               scheme="http"
               set_cmake_var baas_vars REALM_ENABLE_AUTH_TESTS BOOL On
-              set_cmake_var baas_vars REALM_MONGODB_ENDPOINT STRING "$scheme://localhost:9090"
               if [ -n "${baas_admin_port|}" ]; then
                 set_cmake_var baas_vars REALM_ADMIN_ENDPOINT STRING "$scheme://localhost:${baas_admin_port}"
               fi
@@ -156,7 +155,7 @@ functions:
           if [ -n "${curl_base|}" ]; then
               set_cmake_var curl_vars CURL_LIBRARY PATH "$(./evergreen/abspath.sh ${curl_base}/lib/libcurl.dll.a)"
               set_cmake_var curl_vars CURL_INCLUDE_DIR PATH "$(./evergreen/abspath.sh ${curl_base}/include)"
-              set_cmake_var baas_vars REALM_CURL_CACERTS PATH "$(./evergreen/abspath.sh "${curl_base}/bin/cacert.pem")"
+              set_cmake_var baas_vars REALM_CURL_CACERTS PATH "$(./evergreen/abspath.sh "${curl_base}/bin/curl-ca-bundle.crt")"
           fi
 
           set_cmake_var realm_vars REALM_NO_TESTS BOOL ${no_tests|Off}
@@ -208,10 +207,17 @@ functions:
         file: './realm-core/benchmark_results/results.latest.json'
 
   "run tests":
+    - command: expansions.update
+      params:
+        file: realm-core/dependencies.yml
     - command: shell.exec
       params:
         working_dir: realm-core
         shell: bash
+        env:
+          BAASAAS_API_KEY: "${baasaas_api_key}"
+          BAASAAS_REF_SPEC: "${BAAS_VERSION}"
+          BAASAAS_START_MODE: "githash"
         script: |-
           set -o errexit
           set -o verbose
@@ -244,6 +250,12 @@ functions:
 
           TEST_FLAGS="--no-tests=error $TEST_FLAGS ${test_flags|}"
 
+          if [[ -n "${disable_tests_against_baas|}" ]]; then
+              unset BAASAAS_API_KEY
+              unset BAASAAS_REF_SPEC
+              unset BAASAAS_START_MODE
+          fi
+
           if [[ -n "${llvm_symbolizer}" ]]; then
               export ASAN_SYMBOLIZER_PATH="$(./evergreen/abspath.sh ${llvm_symbolizer})"
           fi
@@ -977,18 +989,17 @@ tasks:
       test_label: objstore-local
       test_executable_name: "realm-object-store-tests"
       verbose_test_output: true
+      disable_tests_against_baas: true
   - func: "check branch state"
 
 # These are baas object store tests that run against baas running on a remote host
 - name: baas-integration-tests
   tags: [ "test_suite", "for_pull_requests", "requires_baas" ]
   exec_timeout_secs: 3600
   commands:
-  - func: "launch remote baas"
   - func: "compile"
     vars:
       target_to_build: ObjectStoreTests
-  - func: "wait for remote baas to start"
   - func: "run tests"
     vars:
       test_label: objstore-baas

evergreen/install_baas.sh

@@ -107,6 +107,19 @@ function setup_baas_dependencies() {
                 DISTRO_NAME="$(. /etc/os-release ; echo "${ID}")"
                 DISTRO_VERSION="$(. /etc/os-release ; echo "${VERSION_ID}")"
                 DISTRO_VERSION_MAJOR="$(cut -d. -f1 <<< "${DISTRO_VERSION}")"
+                if [[ "${DISTRO_NAME}" == "linuxmint" ]]; then
+                    CODENAME="$(. /etc/os-release ; echo "${UBUNTU_CODENAME}")"
+                    case "${CODENAME}" in
+                        bionic) DISTRO_VERSION_MAJOR=18;;
+                        focal) DISTRO_VERSION_MAJOR=20;;
+                        jammy) DISTRO_VERSION_MAJOR=22;;
+                        # noble) DISTRO_VERSION_MAJOR=24;;
+                        *)
+                            echo "Error: unsupported version of linuxmint ${DISTRO_VERSION}"
+                            exit 1
+                        ;;
+                    esac
+                fi
             elif [[ -e /etc/redhat-release ]]; then
                 # /etc/redhat-release covers RHEL
                 DISTRO_NAME=rhel

how-to-build.md

@@ -145,39 +145,41 @@ These are the available variables:
    testing process as soon as a check fails or an unexpected exception is thrown
    in a test.
 
-## Running [app] tests against a local MongoDB BAAS
-
-Due to MongoDB security policies, running baas requires company issued AWS account credentials.
-These are for MongoDB employees only, if you do not have these, reach out to #realm-core.
-Once you have them, they need to be set in the shell environment.
-
-First, log in to aws using their command line tool. On mac this requries `brew install awscli`.
-Then login using `aws configure` and input your access key and secret acess key. The other 
-configuration options can be left as none. This creates a correctly formatted file locally at
-`~/.aws/credentials` which we will use later.
-
-If you do not want to install the aws command line tools, you can also create the aws file
-manually in the correct location (`~/.aws/credentials`) with the following contents:
+## Running [app] tests against a local BAAS instance
 
+If you already have a baas instance running, you can specify that directly via the
+`BAAS_BASE_URL` environment variable. You can run baas in a local docker container using
+instructions from [the wiki](https://wiki.corp.mongodb.com/display/10GEN/%28Device+Sync%29+Using+Docker+to+run+a+BAAS+server+instance).
 ```
-AWS_ACCESS_KEY_ID = <your-key-id>
-AWS_SECRET_ACCESS_KEY = <your-secret-key>
+export BAAS_BASE_URL=http://localhost:9090
+mkdir build.sync.ninja
+cmake -B build.sync.ninja -G Ninja -DREALM_ENABLE_AUTH_TESTS=1
+cmake --build build.sync.ninja --target realm-object-store-tests
+./build.sync.ninja/test/object-store/realm-object-store-tests -d=1
 ```
 
-We use a script to fetch the dependencies for and run baas locally. Use the `-b sha` to use a particular version from https://github.com/10gen/baas/
-The script uses the configuration from https://github.com/10gen/baas/blob/master/etc/configs/test_rcore_config.json
+## Running [app] tests against an on-demand BAASAAS container
 
+Due to MongoDB security policies, running baas requires company issued credentials.
+These are for MongoDB employees only, if you do not have these, reach out to
+#appx-device-sync-internal. Once you have a baasaas API key, it needs to be set
+in the shell environment.
 ```
-./evergreen/install_baas.sh -w baas
-```
-
-To run the [app] tests against the local baas, you need to configure a build with some cmake options to tell the tests where to point to.
-```
+export BAASAAS_API_KEY=<your API key here>
 mkdir build.sync.ninja
-cmake -B build.sync.ninja -G Ninja -DREALM_ENABLE_AUTH_TESTS=1 -DREALM_MONGODB_ENDPOINT=http://localhost:9090
+cmake -B build.sync.ninja -G Ninja -DREALM_ENABLE_AUTH_TESTS=1
 cmake --build build.sync.ninja --target realm-object-store-tests
 ./build.sync.ninja/test/object-store/realm-object-store-tests -d=1
 ```
+You can tell the object-store tests to use a specific version of baas with the
+`BAASAAS_START_MODE` environment variable, which can either be `githash`, `patchid`,
+or `branch`. If you specify a start mode, you need to tell it which githash or
+branch name to start with via the `BAASAAS_REF_SPEC` environment variable. Omitting
+these will use the latest available commit from the main branch of baas.
+
+If you've started a baasaas container already via the baasaas CLI, you can tell
+the object-store tests to use that with the `BAASAAS_INSTANCE_ID` environment variable.
+
 
 ### Developing inside a container