Scalable (half-million files) version control system

Posted by hashable on Stack Overflow See other posts from Stack Overflow or by hashable
Published on 2010-03-31T17:55:43Z Indexed on 2010/03/31 18:03 UTC
Read the original article Hit count: 237

Filed under:

version-control

|

svn

|

cvs

|

git

|

mercurial

We use SVN for our source-code revision control and are experimenting using it for non-source-code files.

We are working with a large set (300-500k) of short (1-4kB) text files that will be updated on a regular basis and need to version control it. We tried using SVN in flat-file mode and it is struggling to handle the first commit (500k files checked in) taking about 36 hours.

On a daily basis, we need the system to be able to handle 10k modified files per commit transaction in a short time (<5 min).

My questions:

Is SVN the right solution for my purpose. The initial speed seems too slow for practical use.
If Yes, is there a particular svn server implementation that is fast? (We are currently using the gnu/linux default svn server and command line client.)
If No, what are the best f/oss/commercial alternatives

Thanks

© Stack Overflow or respective owner

Related posts about version-control

In the context of the TFS version control SDK (Microsoft.TeamFoundation.VersionControl), what exactl

as seen on Stack Overflow - Search for 'Stack Overflow'
In the context of the TFS version control SDK (Microsoft.TeamFoundation.VersionControl), what exactly is deletionID? It is a property of Microsoft.TeamFoundation.VersionControl.Client.Item. It is also a parameter to some of the query methods on Microsoft.TeamFoundation.VersionControl.Client. I'm… >>> More
What are the advantages of version control systems that version each file separately?

as seen on Programmers - Search for 'Programmers'
Over the past few years I have worked with several different version control systems. For me, one of the fundamental differences between them has been whether they version files individually (each file has its own separate version numbering and history) or the repository as a whole (a "commit" or… >>> More
What are the advantages of version control systems that version each file separately?

as seen on Programmers - Search for 'Programmers'
Over the past few years I have worked with several different version control systems. For me, one of the fundamental differences between them has been whether they version files individually (each file has its own separate version numbering and history) or the repository as a whole (a "commit" or… >>> More
R and version control for the solo data analyst

as seen on Stack Overflow - Search for 'Stack Overflow'
Many data analysts that I respect use version control. For example: http://github.com/hadley/ See comments on http://permut.wordpress.com/2010/04/21/revision-control-statistics-bleg/ However, I'm evaluating whether adopting a version control system such as git would be worthwhile. A brief overview: I'm… >>> More
What version control tool can generate this header

as seen on Stack Overflow - Search for 'Stack Overflow'
This is from a sql script. What tool can generate this? Thanks --USE [MY_TABLE] GO /****** Object: StoredProcedure [dbo].[adminIncExp] Script Date: 03/05/2010 09:14:12 ******/ SET ANSI_NULLS ON GO SET QUOTED_IDENTIFIER ON GO -- ============================================= -- Author: … >>> More

Related posts about svn

`svn checkout` on the SVN server causes the repo to break with a 301 error

as seen on Server Fault - Search for 'Server Fault'
We have an nginx server which proxies to a standard set-up of Apache+SVN. The nginx set-up is a very simple proxy: server { server_name svn.ourdomain.tld; location / { proxy_pass http://localhost:8080; } } Apache is set-up as follows: <Location /> DAV svn SVNParentPath… >>> More
Can't create directory named "mysql" in subversion repository

as seen on Server Fault - Search for 'Server Fault'
I have a particular problem with subversion. Environment: subversion (1.6.12dfsg-6), apache2 (2.2.16-6+squeeze7) + mod dav_svn. I can't create a directory named "mysql" or "testmysql" or add and commit a file named "mysql.txt" in my repository. There are many references to "subversion PROPSET 403… >>> More
Using AuthzSVNAccessFile for controlling SVN Access produces HTTP 400 Bad Request

as seen on Server Fault - Search for 'Server Fault'
I have a new repository on an existing subversion server that requires us to perform path based authorization within the repository. I found that the AuthzSVNAccessFile directive in apache is directly responsible for allowing this functionality. After fixing several other problems such as AuthzSVNAccessFile… >>> More
Multi Svn Repositories in Apache

as seen on Server Fault - Search for 'Server Fault'
I have a set up with apache and subversion In the apache configuration i have <Location /svn> DAV svn SVNParentPath c:/svn </Location> Now i have multiple repositories a a_b a_c a_b_c a_b_d b and i want to map them as a/svn a/b/svn a/c/svn a/b/c/svn a/b/d/svn b/svn to do… >>> More
SVN Error 403 Forbidden

as seen on Stack Overflow - Search for 'Stack Overflow'
I can't figure this out. I try to import a new project into a svn repository from Netbeans and get 403 Forbidden. I just setup svn on my serverbox today. I can get to it through a browser just fine, though its empty as I haven't imported my project yet. Apache's path for html files is /var/www I… >>> More