This sample script decrypts the salted base64 data of finance.yahoo.com using Google Apps Script.
Recently, when I saw the HTML of finance.yahoo.com, I noticed that the data is converted by the salted base64. In order to decrypt the data, it is required to use the key data. But, unfortunately, I couldn’t find the key data from the HTML. When I searched for it, I found this thread. From the thread, I could retrieve the key data. By this, I could a script for decrypting the salted base64.
Sample script
function myFunction() {
// Load crypto-js.min.js.
const cdnjs = "https://cdnjs.cloudflare.com/ajax/libs/crypto-js/4.1.1/crypto-js.min.js";
eval(UrlFetchApp.fetch(cdnjs).getContentText());
// Retrieve HTML and retrieve salted base64.
const url = "https://finance.yahoo.com/quote/PGEN/press-releases"; // This is a sample URL.
const html = UrlFetchApp.fetch(url).getContentText().match(/root.App.main = ([\s\S\w]+?);\n/);
if (!html || html.length == 1) return;
const tempObj = JSON.parse(html[1].trim());
let obj;
if (typeof tempObj.context.dispatcher.stores === "string" || tempObj.context.dispatcher.stores instanceof String) {
// Decrypt the salted base64.
const { _cs, _cr } = tempObj;
if (!_cs || !_cr) return;
const key = CryptoJS.algo.PBKDF2.create({ keySize: 8 }).compute(_cs, JSON.parse(_cr)).toString();
obj = JSON.parse(CryptoJS.enc.Utf8.stringify(CryptoJS.AES.decrypt(tempObj.context.dispatcher.stores, key)));
} else {
obj = tempObj.context.dispatcher.stores;
}
console.log(obj);
}
- About the value of
context.dispatcher.stores, this script can be used for both the salted base64 and the JSON object.
Note
- In this sample, in order to load
crypto-js,eval(UrlFetchApp.fetch(cdnjs).getContentText())is used. But, if you don’t want to use it, you can also use this script by copying and pasting the script ofhttps://cdnjs.cloudflare.com/ajax/libs/crypto-js/4.1.1/crypto-js.min.jsto the script editor of Google Apps Script. By this, the process cost can be reduced.
IMPORTANT
- I can confirm that this method can be used for the current situation (December 2022). But, when the specification in the data and HTML is changed in the future update on the server side, this script might not be able to be used. Please be careful about this.
References
- crypto-js
- crypto-js of cdnjs.com
- Encrypting and Decrypting with AES using crypto-js with Google Apps Script
- I answered this method to the following threads on Stackoverflow.