Decrypting Salted Base64 of finance.yahoo.com using Google Apps Script

GasTips / jstips
Google Apps Script / Javascript / Yahoo Finance

Gists

This sample script decrypts the salted base64 data of finance.yahoo.com using Google Apps Script.

Recently, it seems that the specification of the key for decrypting the data has been changed at the server side. So. in this post, this post is updated. About this specification, I checked this thread.

Sample script

function myFunction() {
  // Load crypto-js.min.js.
  const cdnjs =
    "https://cdnjs.cloudflare.com/ajax/libs/crypto-js/4.1.1/crypto-js.min.js";
  eval(UrlFetchApp.fetch(cdnjs).getContentText());

  // Retrieve HTML and retrieve salted base64.
  const url = "https://finance.yahoo.com/quote/PGEN/press-releases"; // This is a sample URL.
  const html = UrlFetchApp.fetch(url)
    .getContentText()
    .match(/root.App.main = ([\s\S\w]+?);\n/);
  if (!html || html.length == 1) return;
  const tempObj = JSON.parse(html[1].trim());
  let obj;
  if (
    typeof tempObj.context.dispatcher.stores === "string" ||
    tempObj.context.dispatcher.stores instanceof String
  ) {
    // Decrypt the salted base64.
    var key = Object.entries(tempObj).find(
      ([k]) => !["context", "plugins"].includes(k)
    )[1];
    if (!key) {
      throw new Error(
        "Specification at the server side might be changed. Please check it."
      );
    }
    obj = JSON.parse(
      CryptoJS.enc.Utf8.stringify(
        CryptoJS.AES.decrypt(tempObj.context.dispatcher.stores, key)
      )
    );
  } else {
    obj = tempObj.context.dispatcher.stores;
  }
  console.log(obj);
}
  • About the value of context.dispatcher.stores, this script can be used for both the salted base64 and the JSON object.

Note

  • In this sample, in order to load crypto-js, eval(UrlFetchApp.fetch(cdnjs).getContentText()) is used. But, if you don’t want to use it, you can also use this script by copying and pasting the script of https://cdnjs.cloudflare.com/ajax/libs/crypto-js/4.1.1/crypto-js.min.js to the script editor of Google Apps Script. By this, the process cost can be reduced.

IMPORTANT

  • I can confirm that this method can be used for the current situation (January 14, 2023). But, when the specification in the data and HTML is changed in the future update on the server side, this script might not be able to be used. Please be careful about this.

References

 Share!