DB: get URL to offers

[PaulSut]

I'd like to create a URL for a specific Journey that takes me directly to the DB offers page, where I can select offers like Sparpreis and Flexpreis for a specific journey. I'm unsure about the best approach for this. Any guidance would be greatly appreciated.

I would also be happy to contribute a function which creates the URL if there is an interest for this :)

[derhuerst]

I have looked into how the DB Navigator app does this. We can probably mimic its functionality!

Given that hafas-client (currently only) implements the mobile HAFAS API intended for mobile devices, we will only be able to generate links to the mobile DB shop. (Maybe we can reverse-engineer how to generate desktop booking links.)

Note: Keep in mind that there's also a (currently half-broken, because the DB shop changes frequently) library that, given a full journey – obtained from hafts-client, for example –, tries to "navigate" the shop, filling in all necessary details, to get the URL of the tariff/fare selection page. It has many limitations, but it is independent of hafas-client because it only queries the DB shop, so it has a slightly different use case.

[derhuerst]

I have recorded the HTTP requests of DB Navigator v23.08.01. Its booking flow seems to work as follows:

For TripSearch (journeys()) and Reconstruction (refreshJourney()) requests, the response contains a trfRes object for each journey. This trfRes enumerates all bookable tariffs/fares in fareSetL[], and each tariff/fare has a field addData, which contains Base64-encoded data that is used to display details in the the tariff/fare selection screen.

For example, with the requested journey from Berlin to Wien at 2023-09-11T06:29+02:00 (also linked above), the "Super Sparpreis Europa" fare contains an addData that looks like this decoded:

{
	"AngTyp": "superSparP",
	"IsKBKampagne": "NO",
	"ID": "TCK#3860#3863#0#0#S2#9810#",
	"IDVerbund": "PFRhcmlmZiBUYXJpZmZJRD0iVENLIzM4NjAjMzg2MyMwIzAjUzIjOTgxMCMiIFBBcnREcml0dGU9InVua25vd24iLz4%3D%0D%0A",
	"IsVisible": "YES",
	"KoTxt_kurz.0": "",
	"KoTxt_lang.0": "Through ticket: Your ticket constitutes a continuous contract of carriage in each direction. Should you make a passenger rights claim, the ticket will be considered in its entirety.",
	"KoTxt_kurz.1": "",
	"KoTxt_lang.1": "A 3-D Secure Code may be required for credit card payments.",
	"PrioTxt_kurz.0": "Train-specific travel",
	"PrioTxt_lang.0": "You can use all trains indicated on your ticket. You can use any local train (i.e. RE, RB, S). Passengers on train services with mandatory reservation must reserve a seat.",
	"PrioTxt_icon.0": "202",
	"PrioTxt_kurz.1": "",
	"PrioTxt_lang.1": "If you choose a train service that requires a reservation, your booking includes a free seat reservation for this train service.",
	"UeTxt_kurz": "Cancellation excluded",
	"UeTxt_lang": "Cancellation (exchange or refund) of your ticket is excluded.",
	"UeTxt_icon": "202",
	"OptTxt_kurz.0": "No City-Ticket",
	"OptTxt_lang.0": "Your ticket does not include a City-Ticket (local public transport ticket).",
	"OptTxt_icon.0": "202",
	"OptTxt_key.0": "ohne_city",
	"PATyp": "AP",
	"PArtDpt": "3860",
	"ResIcon": "NO",
	"ResStatus": "O",
	"TarifSystemId": "DB",
	"ZugeordnetZuSpezialablauf": "NO",
	"Zusatznutzen": "NO"
}

It's ticketL[0].addData looks like this decoded:

{
	"dir": "OUTWARD",
	"type": "TICKET",
	"FromText": "Berlin Hbf (tief)",
	"ToText": "Wien Hbf",
	"FromEva": "8098160",
	"ToEva": "8103000"
}

The fare's addData's ID (TCK#3860#3863#0#0#S2#9810#) is then being POSTed url-encoded into https://mobile.bahn.de/bin/mobil/query.exe/eox, along with other details from the journey. These are the request's query parameters url-decoded:

A.1:             27
E:               F
E.1:             2
K:               2
M:               D
RT.1:            E
SS:              8098160
T:               202309110629
VH:              T$A=1@O=Berlin Hbf (tief)@L=8098160@a=128@$A=1@O=Nürnberg Hbf@L=8000284@a=128@$202309110629$202309110952$ICE  503$$1$$$$$$§T$A=1@O=Nürnberg Hbf@L=8000284@a=128@$A=1@O=Wien Hbf@L=8103000@a=128@$202309111031$202309111447$ICE   23$$1$$$$$$
ZS:              8103000
journeyOptions:  0
journeyProducts: 1023
optimize:        1
shpCtx:          PFRhcmlmZiBUYXJpZmZJRD0iVENLIzM4NjAjMzg2MyMwIzAjUzIjOTgxMCMiIFBBcnREcml0dGU9InVua25vd24iLz4=
returnurl:       dbnavigator://restore

• VH is directly taken from HAFAS' jny.ctxRecon (refreshToken in hafas-client).
• shpCtx's value Base64-decoded is <Tariff TariffID="TCK#3860#3863#0#0#S2#9810#" PArtDritte="unknown"/>.
• Other parameters likely follow the semantic model implemented by generate-db-shop-url, so we can probably take some hints from there.

[derhuerst]

What needs to be done:

• In the DB profile, for journeys() & refreshJourney(), implement a new flag opt.dbShopUrls that toggles the generation of DB shop tariff/fare selection URLs.
• If opt.dbShopUrls is true, in the DB profile's existing tariff/fare parsing logic, for each tariff/fare,
  1. decode its addData, obtain the ID;
  2. obtain all other relevant details from the journey (VH a.k.a. ctxRecon, SS a.k.a. the origin stop ID, T a.k.a. departure date+time, etc.);
  3. generate a /bin/mobil/query.exe/eox URL containing all necessary query parameters url-encoded;
  4. expose this URL as e.g. journey.fares[].dbShopUrl.
• Verify that all of this works, and write a test for it.

[derhuerst]

A heads-up: Implementing this feature will require some familiarity with the way the hafas-client code base is structured. Nevertheless, you're welcome to ask questions if you get stuck!

[PaulSut]

Thank you very much for the input! Super nice of you!
I will investigate it next week and get familiar with the code base :)

[PaulSut]

Short update:

journeys() & refreshJourney() seems to return a different fareSetL[].

This:

const testJourneys = await client.journeys('8103000', '8011160', {
	routingMode: routingModes.HYBRID,
    tickets: true,
    polylines: true,
    language: 'de',
    departure: new Date(2023, 9, 11, 6, 0),
},)

resulted in:

...
"trfRes": {
              "statusCode": "OK",
              "fareSetL": [
                {
                  "fareL": [
                    {
                      "isFromPrice": true,
                      "isPartPrice": false,
                      "isBookable": true,
                      "isUpsell": false,
                      "targetCtx": "D",
                      "buttonText": "Zur Angebotsauswahl",
                      "price": {
                        "amount": 8990
                      },
                      "retPriceIsCompletePrice": false,
                      "retPrice": -1
                    }
                  ]
                }
              ]
            },
...

unfortunately there is just a buttonText and no buttonUrl :D
I will have another look into this tomorrow.

[derhuerst]

journeys() & refreshJourney() seems to return a different fareSetL[].

Can you elaborate? Or even provide two raw responses? (You can obtain them by running ./tools/debug-cli/cli.js db … with DEBUG=hafas-client.)

[PaulSut]

Thanks for pointing out the DEBUG mode :)

It seems like the 'fareSetL[]' differ when using TripSearch or Reconstruction (e.g. only Reconstruction containing addData)
I get a similar Result to TripSearch when using journeys(), but no fares when using refreshJourney()

I added the used commands at the top of each file in case I am using them wrong.

[derhuerst]

Did you specify opt.age? Does it depend on that?

[PaulSut]

I had to dig a little deeper but i found the issue now :)

The trfReq with the right params was missing for the "Reconstruction" request.
I will try to create a draft PR which implements the needed changes to get the offers from DB soonish.
Afterwards I will have a look into the URL-Topic.

Edit: PR

[PaulSut]

After looking deeper into the URL topic I am pretty sure that there will be more changes regarding the output format which is why I closed the PR for now. I will incorporate the changes in a future PR :)

[derhuerst]

From #297:

I adjusted the output format, which leads to a breaking change for users of p/db.

Keep in mind that I try to minimise the breaking changes (and thus major-version releases) in hafas-client; I try to do a major release every 2 years (in that order of magnitude, not a strict cycle).

So if it's possible with reasonable effort, try to keep your changes backwards-compatible, by adding all newly parsed information as a new field. Later, we can refactor the format as a breaking change.

[PaulSut]

This [edit: permalink] is a non breaking version that enables to get fares and the url to each fare via refreshJourney(). Everything seems to work fine so far, but I did not fully understand all the params yet. Also, the current url is leading to the "old" DBNavigator. I am not sure if it is also possible to get the link for the new one.
Maybe I will find more time to look into it, therefore i would not merge it right now to prevent breaking changes/workarounds in case of better solutions (suggestions regarding the missings params or the new DBNavigator link are very welcome :))

[derhuerst]

Again: Thanks for you effort in reverse-engineering and implementing this! I think this is a very useful addition to hafas-client. 💛

Unfortunately, the commits currently change too many unrelated things for me to feel comfortable to merge them as-is, for example:

• lots of unrelated whitespace/indentation changes – While these theoretically don't bother much, they later make blaming & diffing much harder. It frequently do this to find out when & why something has been changed, often with my own changes too, simply because I don't remember anymore. Having focused and as-atomic-as-possible commits makes this a lot easier.
• In getDbOfferSelectionUrl(), you should use url.searchParams to add all those query parameters to the offer selection URL. It is safer (because you can't forget to URL-encode them when changing things later) and IMHO a bit more readable.
• As you may have noticed, I tend to keep quite a lot of "todo" comments in the code, to keep track (admittedly in a very unapproachable way) of possible enhancements and half-reverse-engineered bits of logic. Your changes remove two of these (1, 2) which I would like to keep, given that the new code doesn't parse the fields mentioned in them.
• some minor naming things, e.g. naming mutations functions "get…" – I would certainly merge them anyways if they were the sole nitpicks I had about a set of commits! But given that I requested other changes above, I'm letting you know.

If you want, I can try to find time to bring your changes into the shape I want them to be in and merge them. (I will make sure to keep you as the author or co-author in the resulting commits, so that you will appear as a contributor to hafas-client.)

On the other hand, if you're planning to change the code anyways, in one way or another, I will first let you finish iterating on it, and then review it again.

What do you think?

[PaulSut]

Sounds great!
But I will probably need a few weeks for the next iteration!
I will try to implement your suggestions, but of course feel free to do further adjustments afterwards :)

[derhuerst]

@PaulSut Any news? ☺️

[PaulSut]

Unfortunately not really :/
Tickets contain now also a firstClass info, but the ticket-url is still not really usable and I couldn't fix it. I will have a closer look into this before new year.

[PaulSut]

I created a new PR which adds the ticket information.

I am rather pessemistic regarding the URL. The generated URLs 'kind of work'. But not with every device/browser/cookie combination and therefore imo not reliable enough to add it to the hafas client. Even if the link works it is not really user friendly

[derhuerst]

I am rather pessemistic regarding the URL. The generated URLs 'kind of work'. But not with every device/browser/cookie combination and therefore imo not reliable enough to add it to the hafas client. Even if the link works it is not really user friendly

We could put them behind an opt-in flag, e.g. generateUnreliableTicketUrls. What do you think?

[PaulSut]

We could put them behind an opt-in flag, e.g. generateUnreliableTicketUrls. What do you think?

This and your suggested changes are now implemented :)