Overview
DownloadLargeFilesByUrl is a GAS library for downloading large files from URL to Google Drive using Google Apps Script (GAS).
Description
I had been thinking of about whether a large file from an URL can be downloaded to Google Drive using GAS. When I have tried to download such large files, I noticed the following limitations. These limitations are due to the specification of GAS.
- When users download a file from URL using GAS, at the most users, it retrieves the blob using
UrlFetchApp.fetch(url).getBlob()and saves it as a file usingDriveApp.createFile(blob). In this case, only files less than 50 MB (52,428,800 bytes) can be created, because of the size limitation of Blob. - There are a limit executing time for Google Apps Script (GAS). That is 6 min/execution.
- There are a limit of total executing time for running scripts by triggers. That is 90 min/day. (Recently, it became from 60 min/day to 90 min/day.)
When it tries to create the application for downloading the large files, it is required to consider above limitations. On the other hand, there are the following relaxations of quotas by the recent Google’s update.
- Quotes of URL Fetch response size and URL Fetch POST size became from 100 MB/day to 50 MB/call.
- By this, I have been able to create a sample script for splitting large files in Google Drive.
- fetchAll method was added to the class UrlFetchApp.
- It has been found that fetchAll method is worked by the asynchronous processing.
From these limitations and relaxed quotas, I could notice a direction for achieving it. I came up with the following flow.
- When a large file is download from URL, the file is downloaded by splitting using the range to the header.
- Use fetchAll method to download the file. Downloading is run by the asynchronous processing.
- Use resumable upload to join the split data to the original file. Joining is run by the synchronous processing. Because the resumable upload has to be done by the synchronous processing.
I could create this library using the flow like this. If this was useful for you, I’m glad.
You can check this at https://github.com/tanaikech/DownloadLargeFilesByUrl.