Search Results

Search found 92 results on 4 pages for 'readlines'.

Page 3/4 | < Previous Page | 1 2 3 4  | Next Page >

  • How to convert list into a string?

    - by PARIJAT
    I have extracted some data from the file and want to write it in the file 2 but the program says 'sequence item 1: expected string, list found', I want to know how I can convert buffer[] i.e. string into sequence, so that it could be saved in file 2. file = open('/ddfs/user/data/k/ktrip_01/hmm.txt','r') file2 = open('/ddfs/user/data/k/ktrip_01/hmm_write.txt','w') buffer = [] rec = file.readlines() for line in rec : field = line.split() print '>',field[0] term = field[0] buffer.append(term) print field[1], field[2], field[6], field[12] term1 = field [1] buffer.append(term1) term2 = field[2] buffer.append[term2] term3 = field[6] buffer.append[term3] term4 = field[12] buffer.append[term4] file2.write(buffer) file.close() file2.close()

    Read the article

  • Ruby: Read large data from stdout and stderr of an external process on Windows

    - by BinaryMuse
    Greetings, all, I need to run a potentially long-running process from Ruby on Windows and subsequently capture and parse the data from the external process's standard output and error. A large amount of data can be sent to each, but I am only necessarily interested in one line at a time (not capturing and storing the whole of the output). After a bit of research, I found that the Open3 class would take care of executing the process and giving me IO objects connected to the process's standard output and error (via popen3). Open3.popen3("external-program.bat") do |stdin, out, err, thread| # Step3.profit() ? end However, I'm not sure how to continually read from both streams without blocking the program. Since calling IO#readlines on out or err when a lot of data has been sent results in a memory allocation error, I'm trying to continuously check both streams for available input, but not having much luck with any of my implementations. Thanks in advance for any advice!

    Read the article

  • python need some help

    - by maxt0r
    hi i have my codes i have to do a module def subjects but i cant figure out how can u help me thanks. class Enrol: def init(self, dir): def readFiles(, dir, files): for file in files: if file == "classes.txt": class_list = readtable(dir+"/"+file) for item in class_list: Enrol.class_info_dict[item[0]] = item[1:] if item[1] in Enrol.classes_dict: Enrol.classes_dict[item[1]].append(item[0]) else: Enrol.classes_dict[item[1]] = [item[0]] elif file == "subjects.txt": subject_list = readtable(dir+"/"+file) for item in subject_list: Enrol.subjects_dict[item[0]] = item[1] elif file == "venues.txt": venue_list = readtable(dir+"/"+file) for item in venue_list: Enrol.venues_dict[item[0]] = item[1:] elif file.endswith('.roll'): roll_list = readlines(dir+"/"+file) file = os.path.splitext(file)[0] Enrol.class_roll_dict[file] = roll_list for item in roll_list: if item in Enrol.enrolled_dict: Enrol.enrolled_dict[item].append(file) else: Enrol.enrolled_dict[item] = [file] try: os.walk(dir, _readFiles, None) except: print "directory not properly been recognise error" def subjects(what should i write here??): for name in suject_list: print name, out put should look like import enrol e = enrol.Enrol("/path/to/data") e.subjects() ['bw101', 'bw110', 'bw232', 'bw290', 'bw660']

    Read the article

  • Newbie question about file formatting in Python

    - by user568041
    I'm writing a simple program in Python 2.7 using pycURL library to submit file contents to pastebin. Here's the code of the program: #!/usr/bin/env python2 import pycurl, os def send(file): print "Sending file to pastebin...." curl = pycurl.Curl() curl.setopt(pycurl.URL, "http://pastebin.com/api_public.php") curl.setopt(pycurl.POST, True) curl.setopt(pycurl.POSTFIELDS, "paste_code=%s" % file) curl.setopt(pycurl.NOPROGRESS, True) curl.perform() def main(): content = raw_input("Provide the FULL path to the file: ") open = file(content, 'r') send(open.readlines()) return 0 main() The output pastebin looks like standard Python list: ['string\n', 'line of text\n', ...] etc. Is there any way I could format it so it looks better and it's actually human-readable? Also, I would be very happy if someone could tell me how to use multiple data inputs in POSTFIELDS. Pastebin API uses paste_code as its main data input, but it can use optional things like paste_name that sets the name of the upload or paste_private that sets it private.

    Read the article

  • show output of file on client side using jquery + javascript .

    - by tazimk
    Hi, Written some code in my view function : This code reads a file from server . stores it in a list .passes to client def showfiledata(request): f = open("/home/tazim/webexample/test.txt") list = f.readlines() return_dict = {'list':list} json = simplejson.dumps(list) return HttpResponse(json,mimetype="application/json") On, client side the $.ajax callback function receives this list of lines. Now, My Question is . I have to display these lines in a textarea. But these lines should not be displayed at once . Each line should be appended in textarea with some delay. (Use of setInterval is required as per my knowledge) . Also I am using jquery in my templates. The server used is Django . Please provide some solution as in some sample code will be quite helpful .

    Read the article

  • Most Efficient Way to Write to Fixed Width File (Ruby)

    - by Ruby Novice
    I'm currently working with extremely large fixed width files, sometimes well over a million lines. I have written a method that can write over the files based on a set of parameters, but I think there has to be a more efficient way to accomplish this. The current code I'm using is: def self.writefiles(file_name, positions, update_value) @file_name = file_name @positions = positions.to_i @update_value = update_value line_number = 0 @file_contents = File.open(@file_name, 'r').readlines while line_number < @file_contents.length @read_file_contents = @file_contents[line_number] @read_file_contents[@positions] = @update_value @file_contents[line_number] = @read_file_contents line_number += 1 end write_over_file = File.new(@file_name, 'w') line_number = 0 while line_number < @file_contents.length write_over_file.write @file_contents[line_number] line_number += 1 end write_over_file.close end For example, if position 25 in the file indicated that it is an original file the value would be set to "O" and if I wanted to replace that value I would use ClassName.writefiles(filename, 140, "X") to change this position on each line. Any help on making this method more efficient would be greatly appreciated! Thanks

    Read the article

  • reading specific lines from a file

    - by MacUsers
    What's the best way of reading only the specific lines (based on matching text) from a file? This is what I'm doing now: match_txt = "lhcb" for inFile in os.listdir('.'): readFile = open(inFile, 'r') lines = readFile.readlines() readFile.close() for line in lines: if line.find(match_txt)==0: #< do stuff here > i.e. I'm reading the lines, only with "lhcb" in it, from all the files in the present directory one by one. Is it the best way of doing that? Can it be done without loading the whole file in the memory in the first place?

    Read the article

  • MemoryError when running Numpy Meshgrid

    - by joaoc
    I have 8823 data points with x,y coordinates. I'm trying to follow the answer on how to get a scatter dataset to be represented as a heatmap but when I go through the X, Y = np.meshgrid(x, y) instruction with my data arrays I get MemoryError. I am new to numpy and matplotlib and am essentially trying to run this by adapting the examples I can find. Here's how I built my arrays from a file that has them stored: XY_File = open ('XY_Output.txt', 'r') XY = XY_File.readlines() XY_File.close() Xf=[] Yf=[] for line in XY: Xf.append(float(line.split('\t')[0])) Yf.append(float(line.split('\t')[1])) x=array(Xf) y=array(Yf) Is there a problem with my arrays? This same code worked when put into this example but I'm not too sure. Why am I getting this MemoryError and how can I fix this?

    Read the article

  • regular expression not behaving as expected - Python

    - by philippe
    I have the following function which is supposed to read a .html file and search for <input> tags, and inject a <input type='hidden' > tag into the string to be shown into the page. However, that condition is never met:( e.g the if statement is never executed. ) What's wrong with my regex? def print_choose( params, name ): filename = path + name f = open( filename, 'r' ) records = f.readlines() print "Content-Type: text/html" print page = "" flag = True for record in records: if re.match( '<input*', str(record) ) != None: print record page += record page += "<input type='hidden' name='pagename' value='psychology' />" else: page += record print page Thank you

    Read the article

  • Problem with access to file

    - by phenevo
    Hi, I have winforms application and it has reference to library MyLibrary. MyLibrary has method: string[] GiveMeNamesOfAirports() { string[] lines= File.ReadLines("airports.txt"); foreach(string line in lines) ... } And when I run my Winforms application: I get error: file couldn't be find. I was trying other function: string[] lines = File.ReadAllLines(Path.Combine(System.Environment.CurrentDirectory, "airports.txt")); string[] lines = File.ReadAllLines(Path.Combine(Assembly.GetExecutingAssembly().Location, "airports.txt")); string[] lines = File.ReadAllLines(Path.Combine(Assembly.GetAssembly(typeof(Airport)).Location, "airports.txt"));

    Read the article

  • Writing a unique identifier to script?

    - by dannycab
    I'd like to write a subscript that adds a unique identifier (machine time) to a script everytime that it runs. However, each time I edit the script (in IDLE) the indetifiers are over-written. Is there a elegant way of doing this. The script that I wrote appears below. import os, time f = open('sys_time_append.py','r') lines = f.readlines() f.close() fout = open('sys_time_append.py','w') for thisline in lines: fout.write(thisline) fout.write('\n#'+str(time.time())+' s r\n') fout.close() Thanks for any help.

    Read the article

  • Can I accesss an external file when testing an R package?

    - by Abe
    I am using the testthat package to test an R package that is within a larger repository. I would like to test the contents of a file outside of the R package. Can I reference a file that is located outside of an R package while testing? What I have tried A reproducible example can be downloaded as MyRepo.tar.gz My repository is called "myRepo", and it includes an R package, "myRpkg" and a folder full of miscellaneous scripts ~/MyRepo/ ~/MyRepo/MyRpkg ~/MyRepo/Scripts The tests in "MyRpkg" are in the /tests/ folder ~/myRepo/myRpkg/tests/test.myscript.R And I want to be able to test a file in the Scripts folder: ~/MyRepo/Scripts/myscript.sh I would like to read the script to test the contents of the first line doing something like this: check.script <- readLines("../../../Scripts/myscript.sh")[1] expect_true(grepl("echo", check.script)) This works fine if I start from the MyRepo directory: cd ~/MyRepo R CMD check MyRpkg But if I move to another directory, it fails: cd R CMD check MyRepo/MyRpkg

    Read the article

  • can list be converted into string

    - by PARIJAT
    Actually i have extracted some data from the file and want to write it in the file 2 but the program says 'sequence item 1: expected string, list found', I want to know how i can convert buffer[] ie string into sequence, so that it could be saved in file 2...I am new to the python please help* file = open('/ddfs/user/data/k/ktrip_01/hmm.txt','r') file2 = open('/ddfs/user/data/k/ktrip_01/hmm_write.txt','w') buffer = [] rec = file.readlines() for line in rec : field = line.split() print '>',field[0] term = field[0] buffer.append(term) print field[1], field[2], field[6], field[12] term1 = field [1] buffer.append(term1) term2 = field[2] buffer.append[term2] term3 = field[6] buffer.append[term3] term4 = field[12] buffer.append[term4] file2.write(buffer) file.close() file2.close()

    Read the article

  • How to run a module

    - by Jimmy
    I have a module file containing the following functions: def replace(filename): match = re.sub(r'[^\s^\w]risk', 'risk', filename) return match def count_words(newstring): from collections import defaultdict word_dict=defaultdict(int) for line in newstring: words=line.lower().split() for word in words: word_dict[word]+=1 for word in word_dict: if'risk'==word: return word, word_dict[word] when I do this in IDLE: >>> mylist = open('C:\\Users\\ahn_133\\Desktop\\Python Project\\test10.txt').read() >>> newstrings=replace(mylist) ### This works fine. >>> newone=count_words(newstrings) ### This leads to the following error. I get the following error: Traceback (most recent call last): File "<pyshell#134>", line 1, in <module> newPH = replace(newPassage) File "C:\Users\ahn_133\Desktop\Python Project\text_modules.py", line 56, in replace match = re.sub(r'[^\s^\w]risk', 'risk', filename) File "C:\Python27\lib\re.py", line 151, in sub return _compile(pattern, flags).sub(repl, string, count) TypeError: expected string or buffer Is there anyway to run both functions without saving newstrings into a file, opening it using readlines(), and then running count_words function?

    Read the article

  • How do I do this in Python (File Manipulation)?

    - by ThinkCode
    I have a bunch of HTML files in HTML folder. Those HTML files have unicode characters which I solved by using filter(lambda x: x in string.printable, line). Now how do I write the changes back to the original file? What is the best way of doing it? Each HTML file is of 30 kb in size. 1 import os, string 2 3 for file in os.listdir("HTML/"): 4 print file 5 myfile = open('HTML/' + file) 6 fileList = myfile.readlines() 9 for line in fileList: 10 #print line 11 line = filter(lambda x: x in string.printable, line) 12 myfile.close()

    Read the article

  • I have a tab delimeted file that I want to convert into a mysql table

    - by user320835
    I have a tab delimeted file that I want to convert into a mysql table. there are 25 tab delimeted fields in the text file. I can get the values in when I construct the SQL statement word by word and get each value individually stated in the VALUES part but when I try to get the list as a whole it does not work. Here is the code. I couldn't figure it out. Any ideas? lines=open(path, "r").readlines() for line in lines[1:]: linex=line.strip().split("\t") linex.insert(0,'sometextindex') try: cursor.execute('INSERT INTO variants VALUES(%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)',linex) except: print 'line number=',a,linex

    Read the article

  • process the data after using str.split

    - by juju
    I parse a .txt like this: def parse_file(src): for line in src.readlines(): if re.search('SecId', line): continue else: cols = line.split(',') Time = cols[4] output_file.write('{}\n'.format( Time)) I think cols are lists that I could use index. Although it succeeds in printing out correct result as I want, there exists an out of range error. What's the matter? File "./tdseq.py", line 37, in parse_file Time = cols[4] IndexError: list index out of range make: *** [all] Error 1 Data I use: I10.FE,--,2008-04-16,15:15:00,13450,13488,13450,13470,490,359,16APR2008:09:15:00 I10.FE,--,2008-04-16,15:16:00,13468,13473.8,13467,13467,306,521,16APR2008:09:16:00 ....

    Read the article

  • Iterating through a JSON object.

    - by user327508
    [ { "title": "Baby (Feat. Ludacris) - Justin Bieber", "description": "Baby (Feat. Ludacris) by Justin Bieber on Grooveshark", "link": "http://listen.grooveshark.com/s/Baby+Feat+Ludacris+/2Bqvdq", "pubDate": "Wed, 28 Apr 2010 02:37:53 -0400", "pubTime": 1272436673, "TinyLink": "http://tinysong.com/d3wI", "SongID": "24447862", "SongName": "Baby (Feat. Ludacris)", "ArtistID": "1118876", "ArtistName": "Justin Bieber", "AlbumID": "4104002", "AlbumName": "My World (Part II);\nhttp://tinysong.com/gQsw", "LongLink": "11578982", "GroovesharkLink": "11578982", "Link": "http://tinysong.com/d3wI" }, { "title": "Feel Good Inc - Gorillaz", "description": "Feel Good Inc by Gorillaz on Grooveshark", "link": "http://listen.grooveshark.com/s/Feel+Good+Inc/1UksmI", "pubDate": "Wed, 28 Apr 2010 02:25:30 -0400", "pubTime": 1272435930 } ] That is the current JSON object I have. I am now trying to iterate through it to get the import stuff like title and link. This is where I am having trouble I cant seem to get to the content that is past the ":" i tried doing dictionary way couldn't get it. def getLastSong(user,limit): base_url = 'http://gsuser.com/lastSong/' user_url = base_url + str(user) + '/' + str(limit) + "/" raw = urllib.urlopen(user_url) json_raw= raw.readlines() json_object = json.loads(json_raw[0]) #filtering and making it look good. gsongs = [] print json_object for song in json_object[0]: print song This code prints all the information before ":" Please help. ignore the Justin Bieber track :)

    Read the article

  • python-xmpp and looping through list of recipients to receive and IM message

    - by David
    I can't figure out the problem and want some input as to whether my Python code is incorrect, or if this is an issue or design limitation of Python XMPP library. I'm new to Python by the way. Here's snippets of code in question below. What I'd like to do is read in a text file of IM recipients, one recipient per line, in XMPP/Jabber ID format. This is read into a Python list variable. I then instantiate an XMPP client session and loop through the list of recipients and send a message to each recipient. Then sleep some time and repeat test. This is for load testing the IM client of recipients as well as IM server. There is code to alternately handle case of taking only one recipient from command line input instead of from file. What ends up happening is that Python does iterate/loop through the list but only last recipient in list receives message. Switch order of recipients to verify. Kind of looks like Python XMPP library is not sending it out right, or I'm missing a step with the library calls, because the debug print statements during runtime indicate the looping works correctly. recipient = "" delay = 60 useFile = False recList = [] ... elif (sys.argv[i] == '-t'): recipient = sys.argv[i+1] useFile = False elif (sys.argv[i] == '-tf'): fil = open(sys.argv[i+1], 'r') recList = fil.readlines() fil.close() useFile = True ... # disable debug msgs cnx = xmpp.Client(svr,debug=[]) cnx.connect(server=(svr,5223)) cnx.auth(user,pwd,'imbot') cnx.sendInitPresence() while (True): if useFile: for listUser in recList: cnx.send(xmpp.Message(listUser,msg+str(msgCounter))) print "sending to "+listUser+" msg = "+msg+str(msgCounter) else: cnx.send(xmpp.Message(recipient,msg+str(msgCounter))) msgCounter += 1 time.sleep(delay)

    Read the article

  • Writing to a file in Python inserts null bytes

    - by Javier Badia
    I'm writing a todo list program. It keeps a file with a thing to do per line, and lets the user add or delete items. The problem is that for some reason, I end up with a lot of zero bytes at the start of the file, even though the item is correctly deleted. I'll show you a couple of screenshots to make sure I'm making myself clear. This is the file in Notepad++ before running the program: This is the file after deleting item 3 (counting from 1): This is the relevant code. The actual program is bigger, but running just this part triggers the error. import os TODO_FILE = r"E:\javi\code\Python\todo-list\src\todo.txt" def del_elems(f, delete): """Takes an open file and either a number or a list of numbers, and deletes the lines corresponding to those numbers (counting from 1).""" if isinstance(delete, int): delete = [delete] lines = f.readlines() f.truncate(0) counter = 1 for line in lines: if counter not in delete: f.write(line) counter += 1 f = open(TODO_FILE, "r+") del_elems(f, 3) f.close() Could you please point out where's the mistake?

    Read the article

  • How to isolate a single element from a scraped web page in R

    - by PaulHurleyuk
    Hello, I'm trying to do soemone a favour, and it's a tad outside my comfort zone, so I'm stuck. I want to use R to scrape this page (http://www.fifa.com/worldcup/archive/germany2006/results/matches/match=97410001/report.html ) and others, to get the goal scorers and times. So far, this is what I've got require(RCurl) require(XML) theURL <-"http://www.fifa.com/worldcup/archive/germany2006/results/matches/match=97410001/report.html" webpage <- getURL(theURL, header=FALSE, verbose=TRUE) webpagecont <- readLines(tc <- textConnection(webpage)); close(tc) pagetree <- htmlTreeParse(webpagecont, error=function(...){}, useInternalNodes = TRUE) and the pagetree object now contains a pointer to my parsed html (I think). The part I want is <div class="cont")<ul> <div class="bold medium">Goals scored</div> <li>Philipp LAHM (GER) 6', </li> <li>Paulo WANCHOPE (CRC) 12', </li> <li>Miroslav KLOSE (GER) 17', </li> <li>Miroslav KLOSE (GER) 61', </li> <li>Paulo WANCHOPE (CRC) 73', </li> <li>Torsten FRINGS (GER) 87'</li> </ul></div> but I'm now lost as to how to isolate them, and frankly xpathSApply, xpathApply confuse the beejeebies out of me !. So, does anyone know how to fomulate a command to suck out the element conmtaiend within the tags ? Thanks Paul.

    Read the article

  • Python access an object byref / Need tagging

    - by Aaron C. de Bruyn
    I need to suck data from stdin and create a object. The incoming data is between 5 and 10 lines long. Each line has a process number and either an IP address or a hash. For example: pid=123 ip=192.168.0.1 - some data pid=123 hash=ABCDEF0123 - more data hash=ABCDEF123 - More data ip=192.168.0.1 - even more data I need to put this data into a class like: class MyData(): pid = None hash = None ip = None lines = [] I need to be able to look up the object by IP, HASH, or PID. The tough part is that there are multiple streams of data intermixed coming from stdin. (There could be hundreds or thousands of processes writing data at the same time.) I have regular expressions pulling out the PID, IP, and HASH that I need, but how can I access the object by any of those values? My thought was to do something like this: myarray = {} for each line in sys.stdin.readlines(): if pid and ip: #If we can get a PID out of the line myarray[pid] = MyData().pid = pid #Create a new MyData object, assign the PID, and stick it in myarray accessible by PID. myarray[pid].ip = ip #Add the IP address to the new object myarray[pid].lines.append(data) #Append the data myarray[ip] = myarray[pid] #Take the object by PID and create a key from the IP. <snip>do something similar for pid and hash, hash and ip, etc...</snip> This gives my an array with two keys (a PID and an IP) and they both point to the same object. But on the next iteration of the loop, if I find (for example) an IP and HASH and do: myarray[hash] = myarray[ip] The following is False: myarray[hash] == myarray[ip] Hopefully that was clear. I hate to admit that waaay back in the VB days, I remember being able handle objects byref instead of byval. Is there something similar in Python? Or am I just approaching this wrong?

    Read the article

  • With regards to urllib AttributeError: 'module' object has no attribute 'urlopen'

    - by Matt
    import re import string import shutil import os import os.path import time import datetime import math import urllib from array import array import random filehandle = urllib.urlopen('http://www.google.com/') #open webpage s = filehandle.read() #read print s #display #what i plan to do with it once i get the first part working #results = re.findall('[<td style="font-weight:bold;" nowrap>$][0-9][0-9][0-9][.][0-9][0-9][</td></tr></tfoot></table>]',s) #earnings = '$ ' #for money in results: #earnings = earnings + money[1]+money[2]+money[3]+'.'+money[5]+money[6] #print earnings #raw_input() this is the code that i have so far. now i have looked at all the other forums that give solutions such as the name of the script, which is parse_Money.py, and i have tried doing it with urllib.request.urlopen AND i have tried running it on python 2.5, 2.6, and 2.7. If anybody has any suggestions it would be really welcome, thanks everyone!! --Matt ---EDIT--- I also tried this code and it worked, so im thinking its some kind of syntax error, so if anybody with a sharp eye can point it out, i would be very appreciative. import shutil import os import os.path import time import datetime import math import urllib from array import array import random b = 3 #find URL URL = raw_input('Type the URL you would like to read from[Example: http://www.google.com/] :') while b == 3: #get file name file1 = raw_input('Enter a file name for the downloaded code:') filepath = file1 + '.txt' if os.path.isfile(filepath): print 'File already exists' b = 3 else: print 'Filename accepted' b = 4 file_path = filepath #open file FileWrite = open(file_path, 'a') #acces URL filehandle = urllib.urlopen(URL) #display souce code for lines in filehandle.readlines(): FileWrite.write(lines) print lines print 'The above has been saved in both a text and html file' #close files filehandle.close() FileWrite.close()

    Read the article

  • Save memory in Python. How to iterate over the lines and save them efficiently with a 2million line

    - by skyl
    I have a tab-separated data file with a little over 2 million lines and 19 columns. You can find it, in US.zip: http://download.geonames.org/export/dump/. I started to run the following but with for l in f.readlines(). I understand that just iterating over the file is supposed to be more efficient so I'm posting that below. Still, with this small optimization, I'm using 10% of my memory on the process and have only done about 3% of the records. It looks like, at this pace, it will run out of memory like it did before. Also, the function I have is very slow. Is there anything obvious I can do to speed it up? Would it help to del the objects with each pass of the for loop? def run(): from geonames.models import POI f = file('data/US.txt') for l in f: li = l.split('\t') try: p = POI() p.geonameid = li[0] p.name = li[1] p.asciiname = li[2] p.alternatenames = li[3] p.point = "POINT(%s %s)" % (li[5], li[4]) p.feature_class = li[6] p.feature_code = li[7] p.country_code = li[8] p.ccs2 = li[9] p.admin1_code = li[10] p.admin2_code = li[11] p.admin3_code = li[12] p.admin4_code = li[13] p.population = li[14] p.elevation = li[15] p.gtopo30 = li[16] p.timezone = li[17] p.modification_date = li[18] p.save() except IndexError: pass if __name__ == "__main__": run()

    Read the article

  • Python: slow read & write for millions of small files

    - by Jami
    I am building directory tree which has tons of subdirectories and files. The total directory count is somewhere along 256^32 subdirectories with 256 files in each end which are only a few bytes long. I did this so I would have fast access to these files (since i'm not searching and i'm just directly accessing then via a known file path) I have a python script that builds this filesystem and reads & writes those files. The problem is that when I reach more than 1Gb of total filesize, the read and write methods become extremely slow. Here's the function I have that reads the contents of a file (the file contains an integer string), adds a certain number to it, then writes it back to the original file. def addInFile(path, scoreToAdd): num = scoreToAdd try: shutil.copyfile(path, '/tmp/tmp.txt') fp = open('/tmp/tmp.txt', 'r') num += int(fp.readlines()[0]) fp.close() except: pass fp = open('/tmp/tmp.txt', 'w') fp.write(str(num)) fp.close() shutil.copyfile('/tmp/tmp.txt', path) I previously tried performing linux console commands but it was slower. I copy the file to a temporary file first then access/modify it then copy it back because i found this was faster than directly accessing the file. I think the cause of the slowdown is because there're tons of files. performing this function 1000 times sometimes reach 1 minute now, but before (when there were only a few files, 1000 calls was performed for only less than 1 second) How do you suggest I fix this?

    Read the article

< Previous Page | 1 2 3 4  | Next Page >