February 15, 2023: Decrypting Salted Base64 of finance.yahoo.com using Google Apps Script

Gists

This sample script decrypts the salted base64 data of finance.yahoo.com using Google Apps Script.

Recently, it seems that the specification of the key for decrypting the data has been changed on the server side, again. In this update, I looked for the logic for retrieving the key value. But, I cannot still find it. So, in this post, I would like to use a workaround discussed in this thread. In this thread, the valid keys are listed in a text file. Using this, I updated the script as follows.

Sample script

function myFunction() {
  // Load crypto-js.min.js.
  const cdnjs =
    "https://cdnjs.cloudflare.com/ajax/libs/crypto-js/4.1.1/crypto-js.min.js";
  eval(UrlFetchApp.fetch(cdnjs).getContentText());

  // Retrieve HTML and retrieve salted base64.
  const url = "https://finance.yahoo.com/quote/PGEN/press-releases"; // This is a sample URL.
  const html = UrlFetchApp.fetch(url)
    .getContentText()
    .match(/root.App.main = ([\s\S\w]+?);\n/);
  if (!html || html.length == 1) return;
  const tempObj = JSON.parse(html[1].trim().replace(/\n/g, ""));
  let obj;
  if (
    typeof tempObj.context.dispatcher.stores === "string" ||
    tempObj.context.dispatcher.stores instanceof String
  ) {
    const keyFile =
      "https://github.com/ranaroussi/yfinance/raw/main/yfinance/scrapers/yahoo-keys.txt";
    const res = UrlFetchApp.fetch(keyFile);
    const keys = res.getContentText().split("\n").filter(String);
    obj = keys.reduce((ar, key) => {
      try {
        obj = JSON.parse(
          CryptoJS.enc.Utf8.stringify(
            CryptoJS.AES.decrypt(tempObj.context.dispatcher.stores, key.trim())
          )
        );
        ar.push(obj);
      } catch (e) {
        // console.log(e.message)
      }
      return ar;
    }, []);
    if (obj.length == 0) {
      throw new Error(
        "Specification at the server side might be changed. Please check it."
      );
    }
    obj = obj[0];
  } else {
    obj = tempObj.context.dispatcher.stores;
  }
  console.log(obj);
}

Note

IMPORTANT

References

 Share!