Gists
Abstract
A script using resumable upload with file streams is proposed to enhance file handling within the Gemini Generative AI API for Node.js. This script allows uploading from web URLs and local storage, efficiently handles large files, and offers potential reusability with other Google APIs.
Description
The @google/generative-ai library provides a powerful way to interact with the Gemini Generative AI API using Node.js. This enables developers to programmatically generate creative text formats, translate languages, write different kinds of creative content, and answer your questions in an informative way, all powered by Gemini’s advanced AI models. Ref
Gists
Overview
This sample script demonstrates uploading multiple files using split asynchronous processes with resumable upload. It leverages JavaScript and HTML within Google Spreadsheets.
Description
In my previous report, “Resumable Upload of Multiple Files with Asynchronous Process for Google Drive”, I presented an approach for uploading files asynchronously.
This script builds upon that concept, introducing a method for uploading multiple files with split asynchronous processes that utilize resumable upload.
Here’s the process breakdown:
Gists
Abstract
Uploads in Google Apps Script are limited to 50 MB, hindering work with large datasets. This report introduces a script with uploadType=resumable to overcome this limit, enabling uploads over 50 MB to Gemini and other services.
Introduction
This report explores the limitations of data upload size using Google Apps Script and introduces a script to overcome these limitations. In the current stage, Gemini API can generate content using the uploaded data to Gemini. You can find more information on this in a previous report. Ref As mentioned in the report, Google Apps Script uses uploadType=multipart for uploading files. However, the maximum file size is limited to 5 MB with this method for Drive API. Ref While I confirmed that the Gemini API allows for uploading over about 20 MB with uploadType=multipart, Google Apps Script enforces a stricter 50 MB limit for both uploads and downloads. Ref This can be inconvenient when working with larger datasets, such as movie or text data, which often exceed this limit.
ResumableUploadForGoogleDrive_js was updated to v2.0.2.
-
v2.0.2 (May 23, 2024)
- From this version, the files could be also uploaded to the shared drive.
CDN
Class ResumableUploadToGoogleDrive
This Class can achieve the resumable upload of a file by reading the file to the memory.
<script src="https://cdn.jsdelivr.net/gh/tanaikech/ResumableUploadForGoogleDrive_js@2.0.2/resumableupload_js.min.js"></script>
Class ResumableUploadToGoogleDrive2
This Class can achieve the resumable upload of a file by directly reading partially the file from the local Disk. By this, the large file which is over the memory of the local PC can be uploaded.
Gists
Abstract
One day, you might have a situation where you are required to make the shared users upload a file and text using a dialog or sidebar on Google Spreadsheet to your Google Drive and Spreadsheet without authorization by the users. This report introduces a solution for achieving this situation.
Introduction
Google Spreadsheet can run Javascript on a dialog and a sidebar. Ref These can be used as a strong tool for working on Spreadsheet. There might be a situation where you want to make the shared users input a value and upload a file without authorization of the scopes. In this report, I would like to introduce 2 patterns to achieve this goal.
Gists
This is a simple sample script for uploading a video file on Google Drive to YouTube with the resumable upload using Google Apps Script.
When you want to upload a video file to YouTube using Google Apps Script, when YouTube API of Advanced Google services is used, the maximum file size is 5 MB, because, in this case, the video file is uploaded with multipart/form-data. When you want to use a video file with more file size using Google Apps Script, a resumable upload is required to be used. But, unfortunately, in the current stage, the methods of Google Apps Script for uploading large video files are not prepared. And also, when I saw the document of resumable upload on YouTube in the official document, I thought that it might be a bit complicated for understanding the resumable upload process. Ref So, in this post, I would like to introduce a simple sample script for uploading video data of more than 5 MB with the resumable upload using Google Apps Script. In this case, the video file is existing in Google Drive. The video file on Google Drive is uploaded to YouTube using Google Apps Script. When this sample script will help you understand the resumable upload of YouTube, I’m glad.
Gists
This is a sample curl command for uploading a video file to YouTube with the resumable upload using YouTube API.
In order to upload a video file to YouTube with the resumable upload using YouTube API, the following 2 processes are required to be done. The basic process of the resumable upload for YouTube is the same with Drive API. Ref So, I think that this document of Drive API might be useful for understanding the resumable upload process.
Gists
This is a sample script for the resumable upload using Axios with Node.js.
Sample script
In this sample script, as a sample situation in order to explain the resumable upload, the file data is loaded from the local PC, and the data is uploaded to Google Drive with the resumable upload.
const axios = require("axios");
const fs = require("fs").promises;
async function sample() {
const filepath = "./###"; // Please set the filename and file path of the upload file.
const new_access_token = "###"; // Please set your access token.
const name = "###"; // Please set the filename on Google Drive.
const mimeType = "###"; // Please set the mimeType of the uploading file. I thought that when this might not be required to be used.
// 1. Prepare chunks from loaded file data.
const split = 262144; // This is a sample chunk size.
const data = await fs.readFile(filepath);
const fileSize = data.length;
const array = [...new Int8Array(data)];
const chunks = [...Array(Math.ceil(array.length / split))].map((_) =>
Buffer.from(new Int8Array(array.splice(0, split)))
);
// 2. Retrieve endpoint for uploading a file.
const res1 = await axios({
method: "POST",
url: "https://www.googleapis.com/upload/drive/v3/files?uploadType=resumable",
headers: {
Authorization: `Bearer ${new_access_token}`,
"Content-Type": "application/json",
},
data: JSON.stringify({ name, mimeType }),
});
const { location } = res1.headers;
// 3. Upload the data using chunks.
let start = 0;
for (let i = 0; i < chunks.length; i++) {
const end = start + chunks[i].length - 1;
const res2 = await axios({
method: "PUT",
url: location,
headers: { "Content-Range": `bytes ${start}-${end}/${fileSize}` },
data: chunks[i],
}).catch(({ response }) =>
console.log({ status: response.status, message: response.data })
);
start = end + 1;
if (res2?.data) console.log(res2?.data);
}
}
sample();
Testing
When this sample script is run, the following result is obtained.
ResumableUploadForGoogleDrive_js was updated to v2.0.0.
Overview
This is a Javascript library to achieve the resumable upload for Google Drive.
Description
When a file more than 5 MB is uploaded to Google Drive with Drive API, the resumable upload is required to be used. I have already published the sample script for “Resumable Upload for Web Apps using Google Apps Script”. Ref In this case, Web Apps is used. Here, I would like to introduce the script for the resumable upload created by only Javascript. Unfortunately, in the current stage, at google-api-javascript-client, there are no methods for the resumable upload. And, I thought that when this function is created as a Javascript library, it might be useful for users. Also that library is also useful for me. So I created this. If this was useful for your situation, I’m glad.
Gists
This is a simple sample script for achieving the resumable upload to Google Drive using Axios. In order to achieve the resumable upload, at first, it is required to retrieve the location, which is the endpoint of upload. The location is included in the response headers. After the location was retrieved, the file can be uploaded to the location URL.
In this sample, a text data is uploaded with the resumable upload using a single chunk.
Gists
This is a sample script of Node.js for downloading the data and uploading the data to Google Drive with the resumable upload without saving it as a file. The downloaded data is uploaded to Google Drive with the stream.
Sample script
Before you use this, please set the variables of accessToken, url, fileSize, mimeType and filename. In this case, fileSize is required to set because the data is uploaded with the resumable upload.
Gists
This is a sample flow for updating a file with the resumable upload using Drive API.
Sample situation:
In this answer, as a sample situation, it supposes that a text file in Google Drive is updated by the resumable upload with the multiple chunks. And as the method for requesting, I use the curl command.
I prepared 2 files for 2 chunks. As the test situation, the 2 chunks of 262,144 bytes and 37,856 bytes are uploaded. So total upload size is 300,000 bytes. Those filenames are data1.txt and data2.txt, respectively.
Gists
This is a simple sample script for achieving the resumable upload to Google Drive using Python. In order to achieve the resumable upload, at first, it is required to retrieve the location, which is the endpoint of upload. The location is included in the response headers. After the location was retrieved, the file can be uploaded to the location URL.
In this sample, a PNG file is uploaded with the resumable upload using a single chunk.
Gists
This is a simple sample script for achieving the resumable upload to Google Drive using Node.js. In order to achieve the resumable upload, at first, it is required to retrieve the location, which is the endpoint of upload. The location is included in the response headers. After the location was retrieved, the file can be uploaded to the location URL.
In this sample, a PNG file is uploaded with the resumable upload using a single chunk.
Overview
This is a sample script for uploading multiple files with large size (> 50 MB) at the sidebar, dialog of Google Docs and Web Apps using the resumable upload of the asynchronous process with Javascript and Google Apps Script (GAS).
Demo
This is a demonstration of this script. As a demonstration, it uploads 5 files with the size of 100 MB to own Google Drive. When the files were selected and the upload button is clicked, those are uploaded by the resumable upload with the asynchronous process. It is found that the files can be completely uploaded with the asynchronous process. In this demo, as a test case, the chunk size of 20 MB was used.
Overview
This is a Javascript library to achieve the resumable upload for Google Drive.
Description
When a file more than 5 MB is uploaded to Google Drive with Drive API, the resumable upload is required to be used. I have already published the sample script for “Resumable Upload for Web Apps using Google Apps Script”. Ref In this case, Web Apps is used. Here, I would like to introduce the script for the resumable upload created by only Javascript. Unfortunately, in the current stage, at google-api-javascript-client, there are no methods for the resumable upload. And, I thought that when this function is created as a Javascript library, it might be useful for users. Also that library is also useful for me. So I created this. If this was useful for your situation, I’m glad.
News
Overview
This is a sample script for uploading files with large size (> 50 MB) at Web Apps using Google Apps Script (GAS). The resumable upload method is used for uploading files. This script can be also applied to the script using gapi of javascript.