Optimizing BeautifulSoup (Python) code

Posted by user283405 on Stack Overflow See other posts from Stack Overflow or by user283405
Published on 2010-04-26T09:58:27Z Indexed on 2010/04/26 10:23 UTC
Read the original article Hit count: 247

I have code that uses the BeautifulSoup library for parsing, but it is very slow. The code is written in such a way that threads cannot be used. Can anyone help me with this?

I am using BeautifulSoup for parsing and than save into a DB. If I comment out the save statement, it still takes a long time, so there is no problem with the database.

def parse(self,text):                
    soup = BeautifulSoup(text)
    arr = soup.findAll('tbody')                

    for i in range(0,len(arr)-1):
        data=Data()
        soup2 = BeautifulSoup(str(arr[i]))
        arr2 = soup2.findAll('td')

        c=0
        for j in arr2:                                       
            if str(j).find("<a href=") > 0:
                data.sourceURL = self.getAttributeValue(str(j),'<a href="')
            else:  
                if c == 2:
                    data.Hits=j.renderContents()

            #and few others...

            c = c+1

            data.save()

Any suggestions?

Note: I already ask this question here but that was closed due to incomplete information.

© Stack Overflow or respective owner

Related posts about python

Related posts about optimization