Search Results

Search found 14486 results on 580 pages for 'python idle'.

Page 138/580 | < Previous Page | 134 135 136 137 138 139 140 141 142 143 144 145  | Next Page >

  • Memory efficient import many data files into panda DataFrame in Python

    - by richardh
    I import into a panda DataFrame a directory of |-delimited.dat files. The following code works, but I eventually run out of RAM with a MemoryError:. import pandas as pd import glob temp = [] dataDir = 'C:/users/richard/research/data/edgar/masterfiles' for dataFile in glob.glob(dataDir + '/master_*.dat'): print dataFile temp.append(pd.read_table(dataFile, delimiter='|', header=0)) masterAll = pd.concat(temp) Is there a more memory efficient approach? Or should I go whole hog to a database? (I will move to a database eventually, but I am baby stepping my move to pandas.) Thanks! FWIW, here is the head of an example .dat file: cik|cname|ftype|date|fileloc 1000032|BINCH JAMES G|4|2011-03-08|edgar/data/1000032/0001181431-11-016512.txt 1000045|NICHOLAS FINANCIAL INC|10-Q|2011-02-11|edgar/data/1000045/0001193125-11-031933.txt 1000045|NICHOLAS FINANCIAL INC|8-K|2011-01-11|edgar/data/1000045/0001193125-11-005531.txt 1000045|NICHOLAS FINANCIAL INC|8-K|2011-01-27|edgar/data/1000045/0001193125-11-015631.txt 1000045|NICHOLAS FINANCIAL INC|SC 13G/A|2011-02-14|edgar/data/1000045/0000929638-11-00151.txt

    Read the article

  • Python interface to PayPal - urllib.urlencode non-ASCII characters failing

    - by krys
    I am trying to implement PayPal IPN functionality. The basic protocol is as such: The client is redirected from my site to PayPal's site to complete payment. He logs into his account, authorizes payment. PayPal calls a page on my server passing in details as POST. Details include a person's name, address, and payment info etc. I need to call a URL on PayPal's site internally from my processing page passing back all the params that were passed in abovem and an additional one called 'cmd' with a value of '_notify-validate'. When I try to urllib.urlencode the params which PayPal has sent to me, I get a: While calling send_response_to_paypal. Traceback (most recent call last): File "<snip>/account/paypal/views.py", line 108, in process_paypal_ipn verify_result = send_response_to_paypal(params) File "<snip>/account/paypal/views.py", line 41, in send_response_to_paypal params = urllib.urlencode(params) File "/usr/local/lib/python2.6/urllib.py", line 1261, in urlencode v = quote_plus(str(v)) UnicodeEncodeError: 'ascii' codec can't encode character u'\ufffd' in position 9: ordinal not in range(128) I understand that urlencode does ASCII encoding, and in certain cases, a user's contact info can contain non-ASCII characters. This is understandable. My question is, how do I encode non-ASCII characters for POSTing to a URL using urllib2.urlopen(req) (or other method) Details: I read the params in PayPal's original request as follows (the GET is for testing): def read_ipn_params(request): if request.POST: params= request.POST.copy() if "ipn_auth" in request.GET: params["ipn_auth"]=request.GET["ipn_auth"] return params else: return request.GET.copy() The code I use for sending back the request to PayPal from the processing page is: def send_response_to_paypal(params): params['cmd']='_notify-validate' params = urllib.urlencode(params) req = urllib2.Request(PAYPAL_API_WEBSITE, params) req.add_header("Content-type", "application/x-www-form-urlencoded") response = urllib2.urlopen(req) status = response.read() if not status == "VERIFIED": logging.warn("PayPal cannot verify IPN responses: " + status) return False return True Obviously, the problem only arises if someone's name or address or other field used for the PayPal payment does not fall into the ASCII range.

    Read the article

  • Python: x-y-plot with matplotlib

    - by kame
    I want to plot some data. The first column contains the x-data. But matplotlib doesnt plot this. Where is my mistake? #fresnel formula import numpy as np from numpy import cos from scipy import * from pylab import plot, show, ylim, yticks from matplotlib import * from pprint import pprint n1 = 1.0 n2 = 1.5 #alpha, beta, intensity data = [ [10, 22, 4.3], [20, 42, 4.2], [30, 62, 3.6], [40, 83, 1.3], [45, 102, 2.8], [50, 123, 3.0], [60, 143, 3.2], [70, 163, 3.8], ] for i in range(len(data)): rhotang1 = (n1 * cos(data[i][0]) - n2 * cos(data[i][1])) rhotang2 = (n1 * cos(data[i][0]) + n2 * cos(data[i][1])) rhotang = rhotang1 / rhotang2 data[i].append(rhotang) #append 4th value pprint(data) x = data[:][0] y1 = data[:][2] y3 = data[:][3] plot(x, y1, x, y3) show() EDIT: http://paste.pocoo.org/show/205534/ But it doesnt work.

    Read the article

  • Searching a list of tuples in python

    - by Niclas Nilsson
    I'm having a database (sqlite) of members of an organisation (less then 200 people). Now I'm trying to write an wx app that will search the database and return some contact information in a wx.grid. The app will have 2 TextCtrls, one for the first name and one for the last name. What I want to do here is make it possible to only write one or a few letters in the textctrls and that will start to return result. So, if I search "John Smith" I write "Jo" in the first TextCtrl and that will return every single John (or any one else having a name starting with those letters). It will not have an "search"-button, instead it will start searching whenever I press a key. One way to solve this would be to search the database with like " SELECT * FROM contactlistview WHERE forname LIKE 'Jo%' " But that seems like a bad idea (very database heavy to do that for every keystroke?). Instead i thought of use fetchall() on a query like this " SELECT * FROM contactlistview " and then, for every keystroke, search the list of tuples that the query have returned. And that is my problem: Searching a list is not that difficult but how can I search a list of tuples with wildcards?

    Read the article

  • How to read a csv file with python

    - by john
    Hello, I'm trying to read a csv file but it doesn't work. I can read my csv file but when I see what I read, there where white space between values. Here is my code # -*- coding: iso-8859-1 -*- import sql_db, tmpl_macros, os import security, form, common import csv class windows_dialect(csv.Dialect): """Describe the usual properties of unix-generated CSV files.""" delimiter = ',' quotechar = '"' doublequote = 1 skipinitialspace = 0 lineterminator = 'n' quoting = csv.QUOTE_MINIMAL def reco(d): cars = {210:'"', 211:'"', 213:"'", 136:'à', 143:'è', 142:'é'} for c in cars: d = d.replace(chr(c),cars[c]) return d def page_process(ctx): if ctx.req_equals('catalog_send'): if 'catalog_file' in ctx.locals.__dict__: contenu = ctx.locals.catalog_file[0].file.read() #contenu.encode('') p = csv.reader(contenu, delimiter=',') inserted = 0 modified = 0 (cr,db) = sql_db.cursor_get() for line in p: if line: logfile = open('/tmp/test.log', 'a') logfile.write(line[0]) logfile.write('\n') logfile.write('-----------------------------\n') logfile.close()

    Read the article

  • Decoding not reversing unicode encoding in Django/Python

    - by PhilGo20
    Ok, I have a hardcoded string I declare like this name = u"Par Catégorie" I have a # -- coding: utf-8 -- magic header, so I am guessing it's converted to utf-8 Down the road it's outputted to xml through xml_output.toprettyxml(indent='....', encoding='utf-8') And I get a UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128) Most of my data is in French and is ouputted correctly in CDATA nodes, but that one harcoded string keep ... I don't see why an ascii codec is called. what's wrong ?

    Read the article

  • Help a Python newbie with a Django model inheritance problem

    - by Joshmaker
    I'm working on my first real Django project after years of PHP programming, and I am running into a problem with my models. First, I noticed that I was copying and pasting code between the models, and being a diligent OO programmer I decided to make a parent class that the other models could inherit from: class Common(model.Model): self.name = models.CharField(max_length=255) date_created = models.DateTimeField(auto_now_add=True) date_modified = models.DateTimeField(auto_now=True) def __unicode__(self): return self.name class Meta: abstract=True So far so good. Now all my other models extend "Common" and have names and dates like I want. However, I have a class for "Categories" were the name has to be unique. I assume there should be a relatively simple way for me to access the name attribute from Common and make it unique. However, the different methods I have tried to use have all failed. For example: class Category(Common): def __init__(self, *args, **kwargs): self.name.unique=True Spits up the error "Caught an exception while rendering: 'Category' object has no attribute 'name' Can someone point me in the right direction?

    Read the article

  • Can't get MySQL source query to work using Python mysqldb module

    - by Chris
    I have the following lines of code: sql = "source C:\\My Dropbox\\workspace\\projects\\hosted_inv\\create_site_db.sql" cursor.execute (sql) When I execute my program, I get the following error: Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'source C:\My Dropbox\workspace\projects\hosted_inv\create_site_db.sql' at line 1 Now I can copy and past the following into mysql as a query: source C:\\My Dropbox\\workspace\\projects\\hosted_inv\\create_site_db.sql And it works perfect. When I check the query log for the query executed by my script, it shows that my query was the following: source C:\\My Dropbox\\workspace\\projects\\hosted_inv\\create_site_db.sql However, when I manually paste it in and execute, the entire create_site_db.sql gets expanded in the query log and it shows all the sql queries in that file. Am I missing something here on how mysqldb does queries? Am I running into a limitation. My goal is to run a sql script to create the schema structure, but I don't want to have to call mysql in a shell process to source the sql file. Any thoughts? Thanks!

    Read the article

  • feedparser fails during script run, but can't reproduce in interactive python console

    - by Rhubarb
    It's failing with this when I run eclipse or when I run my script in iPython: 'ascii' codec can't decode byte 0xe2 in position 32: ordinal not in range(128) I don't know why, but when I simply execute the feedparse.parse(url) statement using the same url, there is no error thrown. This is stumping me big time. The code is as simple as: try: d = feedparser.parse(url) except Exception, e: logging.error('Error while retrieving feed.') logging.error(e) logging.error(formatExceptionInfo(None)) logging.error(formatExceptionInfo1()) Here is the stack trace: d = feedparser.parse(url) File "C:\Python26\lib\site-packages\feedparser.py", line 2623, in parse feedparser.feed(data) File "C:\Python26\lib\site-packages\feedparser.py", line 1441, in feed sgmllib.SGMLParser.feed(self, data) File "C:\Python26\lib\sgmllib.py", line 104, in feed self.goahead(0) File "C:\Python26\lib\sgmllib.py", line 143, in goahead k = self.parse_endtag(i) File "C:\Python26\lib\sgmllib.py", line 320, in parse_endtag self.finish_endtag(tag) File "C:\Python26\lib\sgmllib.py", line 360, in finish_endtag self.unknown_endtag(tag) File "C:\Python26\lib\site-packages\feedparser.py", line 476, in unknown_endtag method() File "C:\Python26\lib\site-packages\feedparser.py", line 1318, in _end_content value = self.popContent('content') File "C:\Python26\lib\site-packages\feedparser.py", line 700, in popContent value = self.pop(tag) File "C:\Python26\lib\site-packages\feedparser.py", line 641, in pop output = _resolveRelativeURIs(output, self.baseuri, self.encoding) File "C:\Python26\lib\site-packages\feedparser.py", line 1594, in _resolveRelativeURIs p.feed(htmlSource) File "C:\Python26\lib\site-packages\feedparser.py", line 1441, in feed sgmllib.SGMLParser.feed(self, data) File "C:\Python26\lib\sgmllib.py", line 104, in feed self.goahead(0) File "C:\Python26\lib\sgmllib.py", line 138, in goahead k = self.parse_starttag(i) File "C:\Python26\lib\sgmllib.py", line 296, in parse_starttag self.finish_starttag(tag, attrs) File "C:\Python26\lib\sgmllib.py", line 338, in finish_starttag self.unknown_starttag(tag, attrs) File "C:\Python26\lib\site-packages\feedparser.py", line 1588, in unknown_starttag attrs = [(key, ((tag, key) in self.relative_uris) and self.resolveURI(value) or value) for key, value in attrs] File "C:\Python26\lib\site-packages\feedparser.py", line 1584, in resolveURI return _urljoin(self.baseuri, uri) File "C:\Python26\lib\site-packages\feedparser.py", line 286, in _urljoin return urlparse.urljoin(base, uri) File "C:\Python26\lib\urlparse.py", line 215, in urljoin params, query, fragment)) File "C:\Python26\lib\urlparse.py", line 184, in urlunparse return urlunsplit((scheme, netloc, url, query, fragment)) File "C:\Python26\lib\urlparse.py", line 192, in urlunsplit url = scheme + ':' + url File "C:\Python26\lib\encodings\cp1252.py", line 15, in decode return codecs.charmap_decode(input,errors,decoding_table)

    Read the article

  • Opening SSL URLs with Python

    - by RadiantHex
    Hi folks, I'm using mechanize to navigate pages, it works pretty well. Unfortunately I have a random error come up, by random I mean it occasionally appears. URLError at /test/ urlopen error [Errno 1] _ssl.c:1325: error:140943FC:SSL routines:SSL3_READ_BYTES:sslv3 alert bad record mac I really need help on this one :) any ideas?

    Read the article

  • Python: Split by 1 or more occurrences of a delimiter

    - by Adam Matan
    Hi, I have a formatted string from a log file, which looks like: >>> a="test result" That is, the test and the result are split by some spaces - it was probably created using formatted string which gave test some constant spacing. Simple splitting won't do the trick: >>> a.split(" ") ['test', '', '', '', ... '', '', '', '', '', '', '', '', '', '', '', 'result'] split(DELIMITER, COUNT) cleared some unnecessary values: >>> a.split(" ",1) ['test', ' result'] This helped - but of course, I really need: ['test', 'result'] I can use split() followed by map + strip(), but I wondered if there is a more Pythonic way to do it. Thanks, Adam

    Read the article

  • read a binary file (python)

    - by beratch
    Hi, I cant read a file, and I dont understand why: f = open("test/test.pdf", "r") data = list(f.read()) print data Returns : [] I would like to open a PDF, and extract every bytes, and put it in a List. What's wrong with my code ? :( Thanks,

    Read the article

  • Python: Class factory using user input as class names

    - by Sano98
    Hi everyone, I want to add class atttributes to a superclass dynamically. Furthermore, I want to create classes that inherit from this superclass dynamically, and the name of those subclasses should depend on user input. There is a superclass "Unit", to which I can add attributes at runtime. This already works. def add_attr (cls, name, value): setattr(cls, name, value) class Unit(object): pass class Archer(Unit): pass myArcher = Archer() add_attr(Unit, 'strength', 5) print "Strenght ofmyarcher: " + str(myArcher.strength) Archer.strength = 2 print "Strenght ofmyarcher: " + str(myArcher.strength) This leads to the desired output: Strenght ofmyarcher: 5 Strenght ofmyarcher: 2 But now I don't want to predefine the subclass Archer, but I'd rather let the user decide how to call this subclass. I've tried something like this: class Meta(type, subclassname): def __new__(cls, subclassname, bases, dct): return type.__new__(cls, subclassname, Unit, dct) factory = Meta() factory.__new__("Soldier") but no luck. I guess I haven't really understood what new does here. What I want as a result here is class Soldier(Unit): pass being created by the factory. And if I call the factory with the argument "Knight", I'd like a class Knight, subclass of Unit, to be created. Any ideas? Many thanks in advance! Bye -Sano

    Read the article

  • Working with bytes and binary data in Python

    - by ignoramus
    Four consecutive bytes in a byte string together specify some value. However, only 7 bits in each byte are used; the most significant bit is ignored (that makes 28 bits altogether). So... b"\x00\x00\x02\x01" would be 000 0000 000 0000 000 0010 000 0001. Or, for the sake of legibility, 10 000 0001. That's the value the four bytes represent. But I want a decimal, so I do this: >>> 0b100000001 257 I can work all that out myself, but how would I incorporate it into a program?

    Read the article

  • How to Extract data asocaited with attribute of XML file using python 3.2

    - by user1460383
    I have this xml format..... <event timestamp="0.447463" bustype="LIN" channel="LIN 1"> <col name="Time"/> <col name="Start of Frame">0.440708</col> <col name="Channel">LIN 1</col> <col name="Dir">Tx</col> <col name="Event Type">LIN Frame (Diagnostic Request)</col> <col name="Frame Name">MasterReq_DB</col> <col name="Id">3C</col> <col name="Data">81 06 04 04 FF FF 50 4C</col> <col name="Publisher">TestMaster (simulated)</col> <col name="Checksum">D3 &quot;Classic&quot;</col> <col name="Header Duration">2.090 ms (40.1 bits)</col> <col name="Resp. Duration">4.688 ms (90.0 bits)</col> <col name="Time difference">0.049987</col> <empty/> </event> In above xml, i need to extract data associated with attribute 'name' Am able to get all names but am unable to fetch MasterReq_DB< field Please help me ... Thanks in advance

    Read the article

  • Errno socket error in python

    - by Emma
    i wrote this code : import random import sys import urllib openfile = open(sys.argv[1]).readlines() c = random.choice(openfile) i = 0 while i < 5: i=i+1 c = random.choice(openfile) proxies = {'http': c} opener = urllib.FancyURLopener(proxies).open("http://whatismyip.com.au/").read() ::: I put 3 proxy in a txt file . : http://211.161.159.74:8080 http://119.70.40.101:8080 http://124.42.10.119:8080 but when execute it i get this error : IOError: [Errno socket error] (10054, 'Connection reset by peer') what am i going to do ? please help me .

    Read the article

  • Search for string allowing for one mismatches in any location of the string, Python

    - by Vincent
    I am working with DNA sequences of length 25 (see examples below). I have a list of 230,000 and need to look for each sequence in the entire genome (toxoplasma gondii parasite) I am not sure how large the genome is but much more that 230,000 sequences. I need to look for each of my sequences of 25 characters example(AGCCTCCCATGATTGAACAGATCAT). The genome is formatted as a continuous string ie (CATGGGAGGCTTGCGGAGCCTGAGGGCGGAGCCTGAGGTGGGAGGCTTGCGGAGTGCGGAGCCTGAGCCTGAGGGCGGAGCCTGAGGTGGGAGGCTT.........) I don't care where or how many times it is found, just yes or no. This is simple I think, str.find(AGCCTCCCATGATTGAACAGATCAT) But I also what to find a close match defined as wrong(mismatched) at any location but only 1 location and record the location in the sequnce. I am not sure how do do this. The only thing I can think of is using a wildcard and performing the search with a wildcard in each position. ie search 25 times. For example AGCCTCCCATGATTGAACAGATCAT AGCCTCCCATGATAGAACAGATCAT close match with a miss-match at position 13 Speed is not a big issue I am only doing it 3 times. i hope but it would be nice it was fast. The are programs that do this find matches and partial matches but I am looking for a type of partial match that is not available with these applications. Here is a similar post for pearl but they are only comparing sequnces not searching a continuous string Related post

    Read the article

  • python parsing file json

    - by michele
    File json: {"maps":[{"id":"blabla","iscategorical":"0"},{"id":"blabla","iscategorical":"0"}], "masks":["id":"valore"], "om_points":"value", "parameters":["id":"valore"] } I write this script but it only print all the text. json_data=open(file_directory).read() data = json.loads(json_data) pprint(data) How can I parse the file and extract single values? Thanks in advance.

    Read the article

  • Python: unable to inherit from a C extension.

    - by celil
    I am trying to add a few extra methods to a matrix type from the pysparse library. Apart from that I want the new class to behave exactly like the original, so I chose to implement the changes using inheritance. However, when I try from pysparse import spmatrix class ll_mat(spmatrix.ll_mat): pass this results in the following error TypeError: Error when calling the metaclass bases cannot create 'builtin_function_or_method' instances What is this causing this error? Is there a way to use delegation so that my new class behaves exactly the same way as the original?

    Read the article

  • mouse rollover event in Python (VPython)

    - by kame
    Is there something similar to scene.mouse.getclick in the visual module (VPython)? I need it for a rollover. Thanks in advance. EDIT: I need a function for doing something when the mouse moves inside a special area without clicking.

    Read the article

  • Python, a smarter way of string to integer conversion

    - by Hellnar
    Hello I have written this code to convert string in such format "0(532) 222 22 22" to integer such as 05322222222 . class Phone(): def __init__(self,input): self.phone = input def __str__(self): return self.phone #convert to integer. def to_int(self): return int((self.phone).replace(" ","").replace("(","").replace(")","")) test = Phone("0(532) 222 22 22") print test.to_int() It feels very clumsy to use 3 replace methods to solve this. I am curious if there is a better solution?

    Read the article

  • Python metaclass for enforcing immutability of custom types

    - by Mark Lehmacher
    Having searched for a way to enforce immutability of custom types and not having found a satisfactory answer I came up with my own shot at a solution in form of a metaclass: class ImmutableTypeException( Exception ): pass class Immutable( type ): ''' Enforce some aspects of the immutability contract for new-style classes: - attributes must not be created, modified or deleted after object construction - immutable types must implement __eq__ and __hash__ ''' def __new__( meta, classname, bases, classDict ): instance = type.__new__( meta, classname, bases, classDict ) # Make sure __eq__ and __hash__ have been implemented by the immutable type. # In the case of __hash__ also make sure the object default implementation has been overridden. # TODO: the check for eq and hash functions could probably be done more directly and thus more efficiently # (hasattr does not seem to traverse the type hierarchy) if not '__eq__' in dir( instance ): raise ImmutableTypeException( 'Immutable types must implement __eq__.' ) if not '__hash__' in dir( instance ): raise ImmutableTypeException( 'Immutable types must implement __hash__.' ) if _methodFromObjectType( instance.__hash__ ): raise ImmutableTypeException( 'Immutable types must override object.__hash__.' ) instance.__setattr__ = _setattr instance.__delattr__ = _delattr return instance def __call__( self, *args, **kwargs ): obj = type.__call__( self, *args, **kwargs ) obj.__immutable__ = True return obj def _setattr( self, attr, value ): if '__immutable__' in self.__dict__ and self.__immutable__: raise AttributeError( "'%s' must not be modified because '%s' is immutable" % ( attr, self ) ) object.__setattr__( self, attr, value ) def _delattr( self, attr ): raise AttributeError( "'%s' must not be deleted because '%s' is immutable" % ( attr, self ) ) def _methodFromObjectType( method ): ''' Return True if the given method has been defined by object, False otherwise. ''' try: # TODO: Are we exploiting an implementation detail here? Find better solution! return isinstance( method.__objclass__, object ) except: return False However, while the general approach seems to be working rather well there are still some iffy implementation details (also see TODO comments in code): How do I check if a particular method has been implemented anywhere in the type hierarchy? How do I check which type is the origin of a method declaration (i.e. as part of which type a method has been defined)?

    Read the article

  • Python learner needs help spotting an error

    - by Protean
    This piece of code gives a syntax error at the colon of "elif process.loop(i, len(list_i) != 'repeat':" and I can't seem to figure out why. class process: def loop(v1, v2): if v1 < v2 - 1: return 'repeat' def isel(chr_i, list_i): for i in range(len(list_i)): if chr_i == list_i[i]: return list_i[i] elif process.loop(i, len(list_i) != 'repeat': return 'error'()

    Read the article

< Previous Page | 134 135 136 137 138 139 140 141 142 143 144 145  | Next Page >