I have a large history file that I exported from one of my previous Galaxy history (which was stored on the 30 day temporary server and is no longer accessible on Usegalaxy.org). The exported file is a rocrate.zip file and has around 14 Gb (before unzipping). When I try to upload the zipped history file through the “History” - “Import History” - “Upload local file from your computer“ function, it gives a “500” error on the top. Is this because my history size is too large? Is there another way to upload the history with large size? Also: since this history may be too big for my permanent storage location, I want to upload the history to the 1T temporary location, and I have changed the default setting in the “Preference” page. Is this the right way of doing it?
I don’t think 14 GB is overly large. However, I do think it can be considered a large size to transfer from certain internet connections. Example: home connections with slower “up” data transfer speeds.
You can check your connection speeds with a tool like fast dot com or whatever is available in your area. People have also had luck by loading up the data into the closest public server in their geographical region. Meaning, if you are in Europe, load to the UseGalaxy.eu server first (you can always transfer to others via URL after). Same for Americas and UseGalaxy.org. Or Asia/AU and UseGalaxy.org.au.
The other item to check is the integrity of that archive. Are you able to uncompress it locally? Meaning, did the archive get exported intact originally? I’m also curious if you re-recompressed and are uploading that version, of if you are loading the originally downloaded version.
For the storage location, yes, you should be able to change the default storage location to be the 1 T location and load the data into there.. as far as I know, I am double checking this, because it is an interesting corner case. Other data “loaded” whether through the Upload tool or a tool respects this toggle but I’ll confirm. I can let you know that your permanent storage needs to be under the 250 GB quota limit to run new jobs, and that includes data loading jobs, so if you are over quota, move more data into the 1 T space first.
Please give all this a review and we can follow up more! I’d like to keep going until we get this working for you!
I have tried both my own wifi (~100 mbps sending speed) and the lab wifi (~300 mbps sending speed) but I’m getting the same “500” error.
I am currently located in the U.S. so I use usegalaxy.org more often (unless I need to use some tools that are unavailable on the U.S. server).
I also think that the 14-Gb file size does not seem to be large by itself. However, the file size after unzipping is more than 250 Gb (271 Gb to be exact; it is a history for one of my 16s sequencing projects using the Mothur method).
I am able to load the zipped file as “data” to a new history (but it is just shown as a zipped file; I cannot use it as the actual history). I haven’t really tried to upload the unzipped file for importing the history (my current laptop/ lab desktop does not have enough storage space for the unzipped file). I have unzipped the file before on my own desktop and were able to open it.
I will try to access a storage device that is large enough and give that a try (I am traveling for this month and don’t have access to my desktop). Will keep you posted.
I think from here you can try to load to UseGalaxy.eu first, then use the transfer function to move it over to the UseGalaxy.org server by URL. Worth a try!
Then getting the data hosted someplace you can access it by URL seems to be what is working best for others. Data is growing so large! You can get connected to personal cloud storage under User → Preferences → Remote File Sources. Or, if you can host the file with a public URL it can be hosted anywhere Galaxy can “see”.
