Best archive format & tool for large amounts of data (50gb+)

Posted by marcusstarnes on Super User See other posts from Super User or by marcusstarnes
Published on 2011-07-01T15:33:31Z Indexed on 2011/07/01 16:24 UTC
Read the original article Hit count: 178

Filed under:

compression

|

zip

|

archiving

|

rar

I only realised this afternoon that the ZIP format has a limit of what appears to be around 20gb. I am trying to automate an archive process (using Automate) to zip/rar/whatever a collection of folders/files on one of my disks. It always appeared to bomb out with an incomplete archive at about 20gb. So I tried using WinRAR and doing it manually as a ZIP file, but it told me of the limit.

So, I was wondering, what is a recommended zip format (and tool for accomplishing the task) for archiving up a large amount of data (around 50gb)?

Thanks

© Super User or respective owner

Related posts about compression

Converting linear colors to SRGB shows banding in FFmpeg

as seen on Super User - Search for 'Super User'
When I convert an EXR file sequence with x264 using FFmpeg and convert the colorspace from linear to SRGB (with gamma 0.45454545) I get some heavy banding issues (most visible on a dark gradient). Here is the ffmpeg command I use: C:/ffmpeg.exe -y -i C:/seq_v001.%04d.exr -vf lutrgb=r=gammaval(0… >>> More
IIS7 Compression CSS files only compressed when dynamic compression is enabled

as seen on Server Fault - Search for 'Server Fault'
If anyone can help it would be appreciated. I would like to enable compression for static files within IIS7 (for the sake of simplicity I'll just refer to static css files for the time being). The problem I'm getting is that css files are only compressed when both dynamic and static compression… >>> More
Built-in GZip/Deflate Compression on IIS 7.x

as seen on West-Wind - Search for 'West-Wind'
IIS 7 improves internal compression functionality dramatically making it much easier than previous versions to take advantage of compression that’s built-in to the Web server. IIS 7 also supports dynamic compression which allows automatic compression of content created in your own applications (ASP… >>> More
Domino Document data compression and design compression

as seen on Server Fault - Search for 'Server Fault'
I was thinking of turning this on some large databases not just mail files - we have around 8 - 10GB of large databases as well as small databases of couple of hundred MB in size. But after reading this post I am not too sure: http://www-10.lotus.com/ldd/nd85forum.nsf/4b9931b774db788c85256bf0006b5e6d/1f4e67b569720e54852576c0003cb8ac… >>> More
I need to choose a compression algorithm

as seen on Stack Overflow - Search for 'Stack Overflow'
I need to choose a compression algorithm to compress some data. I don't know the type of data I'll be compressing in advance (think of it as kinda like the WinRAR program). I've heard of the following algorithms but I don't know which one I should use. Can anyone post a short list of pros and cons… >>> More

Related posts about zip

More CPU cores may not always lead to better performance – MAXDOP and query memory distribution in spotlight

as seen on SQL Blogcasts - Search for 'SQL Blogcasts'
More hardware normally delivers better performance, but there are exceptions where it can hinder performance. Understanding these exceptions and working around it is a major part of SQL Server performance tuning. When a memory allocating query executes in parallel, SQL Server distributes… >>> More
Cisco Unifed Communication integration for Microsoft Lync crashes on Remote Desktop services 2008 R2!

as seen on Server Fault - Search for 'Server Fault'
Hi everybody i have deployed office communication server 2007 R2 and communicator 2007 R2 and i made integration with Cisco Unified Communication Manager 7.1 in my network, i also uses Remote Desktop Servers 2008 R2 for Thin Client Computers, now that i installed Cisco UC integration client for communicator… >>> More
Cisco Unified Communication integration for Microsoft Lync crashes on Remote Desktop services 2008 R2!

as seen on Server Fault - Search for 'Server Fault'
Hi everybody i have deployed office communication server 2007 R2 and communicator 2007 R2 and i made integration with Cisco Unified Communication Manager 7.1 in my network, i also use Remote Desktop Servers 2008 R2 for Thin Client Computers, now that i installed Cisco UC integration client for communicator… >>> More
zip multiple file on the fly and download it as a zip folder php

as seen on Stack Overflow - Search for 'Stack Overflow'
I understand that I can zip multiple file and download it as zip when the files are already exists on ther server. like this. My question is, I have a dropdown list that prompt user download different section of a big form on the fly so I have header below for download it individually. header("Content-type:… >>> More
Have powershell zip the contents of a bunch of folders, individual zip for each folder

as seen on Super User - Search for 'Super User'
Recently, I asked how to do this with a .bat file and an answer was provided. for /D %%d in (*.*) do "C:\Program Files\7-Zip\7z\7za.exe" a -tzip %%d.zip %%d However, this proved useful only for folders that have no spaces in their name. The reason being that batch will do the following: if the… >>> More