Search Results

Search found 121 results on 5 pages for 'pickle'.

Page 5/5 | < Previous Page | 1 2 3 4 5 

  • Q on Python serialization/deserialization

    - by neil
    What chances do I have to instantiate, keep and serialize/deserialize to/from binary data Python classes reflecting this pattern (adopted from RFC 2246 [TLS]): enum { apple, orange } VariantTag; struct { uint16 number; opaque string<0..10>; /* variable length */ } V1; struct { uint32 number; opaque string[10]; /* fixed length */ } V2; struct { select (VariantTag) { /* value of selector is implicit */ case apple: V1; /* VariantBody, tag = apple */ case orange: V2; /* VariantBody, tag = orange */ } variant_body; /* optional label on variant */ } VariantRecord; Basically I would have to define a (variant) class VariantRecord, which varies depending on the value of VariantTag. That's not that difficult. The challenge is to find a most generic way to build a class, which serializes/deserializes to and from a byte stream... Pickle, Google protocol buffer, marshal is all not an option. I made little success with having an explicit "def serialize" in my class, but I'm not very happy with it, because it's not generic enough. I hope I could express the problem. My current solution in case VariantTag = apple would look like this, but I don't like it too much import binascii import struct class VariantRecord(object): def __init__(self, number, opaque): self.number = number self.opaque = opaque def serialize(self): out = struct.pack('>HB%ds' % len(self.opaque), self.number, len(self.opaque), self.opaque) return out v = VariantRecord(10, 'Hello') print binascii.hexlify(v.serialize()) >> 000a0548656c6c6f Regards

    Read the article

  • How is it that json serialization is so much faster than yaml serialization in python?

    - by guidoism
    I have code that relies heavily on yaml for cross-language serialization and while working on speeding some stuff up I noticed that yaml was insanely slow compared to other serialization methods (e.g., pickle, json). So what really blows my mind is that json is so much faster that yaml when the output is nearly identical. >>> import yaml, cjson; d={'foo': {'bar': 1}} >>> yaml.dump(d, Dumper=yaml.SafeDumper) 'foo: {bar: 1}\n' >>> cjson.encode(d) '{"foo": {"bar": 1}}' >>> import yaml, cjson; >>> timeit("yaml.dump(d, Dumper=yaml.SafeDumper)", setup="import yaml; d={'foo': {'bar': 1}}", number=10000) 44.506911039352417 >>> timeit("yaml.dump(d, Dumper=yaml.CSafeDumper)", setup="import yaml; d={'foo': {'bar': 1}}", number=10000) 16.852826118469238 >>> timeit("cjson.encode(d)", setup="import cjson; d={'foo': {'bar': 1}}", number=10000) 0.073784112930297852 PyYaml's CSafeDumper and cjson are both written in C so it's not like this is a C vs Python speed issue. I've even added some random data to it to see if cjson is doing any caching, but it's still way faster than PyYaml. I realize that yaml is a superset of json, but how could the yaml serializer be 2 orders of magnitude slower with such simple input?

    Read the article

  • Spring MVC with annotations: how to beget that method always is called

    - by TheStijn
    hi, I'm currently migrating a project that is using Spring MVC without annotations to Spring MVC with annotations. This is causing less problems than expected but I did come across one issue. In my project I have set up an access mechanisme. Whether or not a User has access to a certain view depends on more than just the role of the User (e.g. it also depends on the status of the entity, the mode (view/edit), ...). To address this I had created an abstract parent controller which has a method hasAccess. This method calls also other methods like getAllowedEditStatuses which are here and there overridden by the child controllers. The hasAccess method gets called from the showForm method (below code was minimized for your readability): @Override protected ModelAndView showForm(final HttpServletRequest request, final HttpServletResponse response, final BindException errors) throws Exception { Integer id = Integer.valueOf(request.getParameter("ID")); Project project = this.getProject(id); if (!this.hasAccess(project, this.getActiveUser())) { return new ModelAndView("errorNoAccess", "code", project != null ? project.getCode() : null); } return this.showForm(request, response, project, errors); } So, if the User has no access to the view then he gets redirected to an error page. Now the 'pickle': how to set this up when using annotations. There no longer is a showForm or other method that is always called by the framework. My (and maybe your) first thought was: simply call this method from within each controller before going to the view. This would of course work but I was hoping for a nicer, more generic solution (less code duplication). The only other solution I could think of is preceeding the hasAccess method with the @ModelAttribute annotation but this feels a lot like raping the framework :-). So, does anyone have a (better) idea? thanks, Stijn

    Read the article

  • Return a list of imported Python modules used in a script?

    - by Jono Bacon
    Hi All, I am writing a program that categorizes a list of Python files by which modules they import. As such I need to scan the collection of .py files ad return a list of which modules they import. As an example, if one of the files I import has the following lines: import os import sys, gtk I would like it to return: ["os", "sys", "gtk"] I played with modulefinder and wrote: from modulefinder import ModuleFinder finder = ModuleFinder() finder.run_script('testscript.py') print 'Loaded modules:' for name, mod in finder.modules.iteritems(): print '%s ' % name, but this returns more than just the modules used in the script. As an example in a script which merely has: import os print os.getenv('USERNAME') The modules returned from the ModuleFinder script return: tokenize heapq __future__ copy_reg sre_compile _collections cStringIO _sre functools random cPickle __builtin__ subprocess cmd gc __main__ operator array select _heapq _threading_local abc _bisect posixpath _random os2emxpath tempfile errno pprint binascii token sre_constants re _abcoll collections ntpath threading opcode _struct _warnings math shlex fcntl genericpath stat string warnings UserDict inspect repr struct sys pwd imp getopt readline copy bdb types strop _functools keyword thread StringIO bisect pickle signal traceback difflib marshal linecache itertools dummy_thread posix doctest unittest time sre_parse os pdb dis ...whereas I just want it to return 'os', as that was the module used in the script. Can anyone help me achieve this?

    Read the article

  • Return a list of important Python modules in a script?

    - by Jono Bacon
    Hi All, I am writing a program that categorizes a list of Python files by which modules they import. As such I need to scan the collection of .py files ad return a list of which modules they import. As an example, if one of the files I import has the following lines: import os import sys, gtk I would like it to return: ["os", "sys", "gtk"] I played with modulefinder and wrote: from modulefinder import ModuleFinder finder = ModuleFinder() finder.run_script('testscript.py') print 'Loaded modules:' for name, mod in finder.modules.iteritems(): print '%s ' % name, but this returns more than just the modules used in the script. As an example in a script which merely has: import os print os.getenv('USERNAME') The modules returned from the ModuleFinder script return: tokenize heapq __future__ copy_reg sre_compile _collections cStringIO _sre functools random cPickle __builtin__ subprocess cmd gc __main__ operator array select _heapq _threading_local abc _bisect posixpath _random os2emxpath tempfile errno pprint binascii token sre_constants re _abcoll collections ntpath threading opcode _struct _warnings math shlex fcntl genericpath stat string warnings UserDict inspect repr struct sys pwd imp getopt readline copy bdb types strop _functools keyword thread StringIO bisect pickle signal traceback difflib marshal linecache itertools dummy_thread posix doctest unittest time sre_parse os pdb dis ...whereas I just want it to return 'os', as that was the module used in the script. Can anyone help me achieve this?

    Read the article

  • Python subprocess: 64 bit windows server PIPE doesn't exist :(

    - by Spaceman1861
    I have a GUI that launches selected python scripts and runs it in cmd next to the gui window. I am able to get my launcher to work on my (windows xp 32 bit) laptop but when I upload it to the server(64bit windows iss7) I am running into some issues. The script runs, to my knowledge but spits back no information into the cmd window. My script is a bit of a Frankenstein that I have hacked and slashed together to get it to work I am fairly certain that this is a very bad example of the subprocess module. Just wondering if i could get a hand :). My question is how do i have to alter my code to work on a 64bit windows server. :) from Tkinter import * import pickle,subprocess,errno,time,sys,os PIPE = subprocess.PIPE if subprocess.mswindows: from win32file import ReadFile, WriteFile from win32pipe import PeekNamedPipe import msvcrt else: import select import fcntl def recv_some(p, t=.1, e=1, tr=5, stderr=0): if tr < 1: tr = 1 x = time.time()+t y = [] r = '' pr = p.recv if stderr: pr = p.recv_err while time.time() < x or r: r = pr() if r is None: if e: raise Exception(message) else: break elif r: y.append(r) else: time.sleep(max((x-time.time())/tr, 0)) return ''.join(y) def send_all(p, data): while len(data): sent = p.send(data) if sent is None: raise Exception(message) data = buffer(data, sent) The code above isn't mine def Run(): print filebox.get(0) location = filebox.get(0) location = location.__str__().replace(listbox.get(ANCHOR).__str__(),"") theTime = time.asctime(time.localtime(time.time())) lastbox.delete(0, END) lastbox.insert(END,theTime) for line in CookieCont: if listbox.get(ANCHOR) in line and len(line) > 4: line[4] = theTime else: "Fill In the rip Details to record the time" if __name__ == '__main__': if sys.platform == 'win32' or sys.platform == 'win64': shell, commands, tail = ('cmd', ('cd "'+location+'"',listbox.get(ANCHOR).__str__()), '\r\n') else: return "Please use contact admin" a = Popen(shell, stdin=PIPE, stdout=PIPE) print recv_some(a) for cmd in commands: send_all(a, cmd + tail) print recv_some(a) send_all(a, 'exit' + tail) print recv_some(a, e=0) The Code above is mine :)

    Read the article

  • inserting a form to session raises picklingerror - django

    - by shanyu
    I receive an exception when I add a form to the session: PicklingError: Can't pickle <class 'django.utils.functional.__proxy__'>: attribute lookup django.utils.functional.__proxy__ failed The form includes a few simple fields and has some javascript attached to a widget. It might be that Django forms cannot be pickled at all, but the exception seems to point to unicode lazy translation. To test further, I have also tried to insert only the form errors (an errordict) to the session and received the same error. I appreciate some help here, thanks in advance. EDIT: Here's why I insert a form into the session: I have an app that has a form. This form is rendered by a template tag in another app. When posted, if the form is valid, no problem, I do stuff and redirect to "next". However if it is not valid, I want to go back to the posting page to show errors. Recall that the comments app in this case redirects to an intermediate "hey, please fix the errors" page. I am trying to avoid this, and hence redirect back to the posting page with the form and its errors in the session that the template tag will render.

    Read the article

  • Raising events and object persistence in Django

    - by Mridang Agarwalla
    Hi, I have a tricky Django problem which didn't occur to me when I was developing it. My Django application allows a user to sign up and store his login credentials for a sites. The Django application basically allows the user to search this other site (by scraping content off it) and returns the result to the user. For each query, it does a couple of queries of the other site. This seemed to work fine but sometimes, the other site slaps me with a CAPTCHA. I've written the code to get the CAPTCHA image and I need to return this to the user so he can type it in but I don't know how. My search request (the query, the username and the password) in my Django application gets passed to a view which in turn calls the backend that does the scraping/search. When a CAPTCHA is detected, I'd like to raise a client side event or something on those lines and display the CAPTCHA to the user and wait for the user's input so that I can resume my search. I would somehow need to persist my backend object between calls. I've tried pickling it but it doesn't work because I get the Can't pickle 'lock' object error. I don't know to implement this though. Any help/ideas? Thanks a ton.

    Read the article

  • Is there any danger in committing to a component library such as SmartGwt or Swing?

    - by Banang
    Since February this year I have been working on an app that's built using SmartGWT components. Generally, I find the components very nice to work with, and the fact that they're open source and free to use is just fantastic. However, I can't seem to shake the feeling that it's not a durable way of developing, but I can't quite explain why. Maybe it's because I know that any minute now the team developing it could decide to stop, which would leave me and my team in a bit of a pickle, but I'm sure there must be something more. I have been trying to find ways of explaining this feeling to myself, but to no avail. Therefore I turn to you, dear community, to ask if you can come up with a good reason why committing to building your app (that's supposed to be around for many more years to come) using a component library such as SmartGWT is a bad idea? Is there any reason I should just have developed the components myself? Or did I make the right choice when deciding not to reinvent the wheel and just go for what was readily available?

    Read the article

  • Google App Engine: TypeError problem with Models

    - by Rosarch
    I'm running Google App Engine on the dev server. Here is my models file: from google.appengine.ext import db import pickle import re re_dept_code = re.compile(r'[A-Z]{2,}') re_course_number = re.compile(r'[0-9]{4}') class DependencyArcHead(db.Model): sink = db.ReferenceProperty() tails = db.ListProperty() class DependencyArcTail(db.Model): courses = db.ListProperty() It gives this error: Traceback (most recent call last): File "C:\Program Files\Google\google_appengine\google\appengine\tools\dev_appserver.py", line 3192, in _HandleRequest self._Dispatch(dispatcher, self.rfile, outfile, env_dict) File "C:\Program Files\Google\google_appengine\google\appengine\tools\dev_appserver.py", line 3135, in _Dispatch base_env_dict=env_dict) File "C:\Program Files\Google\google_appengine\google\appengine\tools\dev_appserver.py", line 516, in Dispatch base_env_dict=base_env_dict) File "C:\Program Files\Google\google_appengine\google\appengine\tools\dev_appserver.py", line 2394, in Dispatch self._module_dict) File "C:\Program Files\Google\google_appengine\google\appengine\tools\dev_appserver.py", line 2304, in ExecuteCGI reset_modules = exec_script(handler_path, cgi_path, hook) File "C:\Program Files\Google\google_appengine\google\appengine\tools\dev_appserver.py", line 2200, in ExecuteOrImportScript exec module_code in script_module.__dict__ File "main.py", line 19, in <module> from src.Models import Course, findCourse, validateCourse, dictForJSON, clearAndBuildDependencyGraph File "C:\Program Files\Google\google_appengine\google\appengine\tools\dev_appserver.py", line 1279, in Decorate return func(self, *args, **kwargs) File "C:\Program Files\Google\google_appengine\google\appengine\tools\dev_appserver.py", line 1929, in load_module return self.FindAndLoadModule(submodule, fullname, search_path) File "C:\Program Files\Google\google_appengine\google\appengine\tools\dev_appserver.py", line 1279, in Decorate return func(self, *args, **kwargs) File "C:\Program Files\Google\google_appengine\google\appengine\tools\dev_appserver.py", line 1831, in FindAndLoadModule description) File "C:\Program Files\Google\google_appengine\google\appengine\tools\dev_appserver.py", line 1279, in Decorate return func(self, *args, **kwargs) File "C:\Program Files\Google\google_appengine\google\appengine\tools\dev_appserver.py", line 1782, in LoadModuleRestricted description) File "src\Models.py", line 14, in <module> class DependencyArcHead(db.Model): File "src\Models.py", line 17, in DependencyArcHead tails = db.ListProperty() TypeError: __init__() takes at least 2 arguments (1 given) What am I doing wrong?

    Read the article

  • Python: why does this code take forever (infinite loop?)

    - by Rosarch
    I'm developing an app in Google App Engine. One of my methods is taking never completing, which makes me think it's caught in an infinite loop. I've stared at it, but can't figure it out. Disclaimer: I'm using http://code.google.com/p/gaeunitlink text to run my tests. Perhaps it's acting oddly? This is the problematic function: def _traverseForwards(course, c_levels): ''' Looks forwards in the dependency graph ''' result = {'nodes': [], 'arcs': []} if c_levels == 0: return result model_arc_tails_with_course = set(_getListArcTailsWithCourse(course)) q_arc_heads = DependencyArcHead.all() for model_arc_head in q_arc_heads: for model_arc_tail in model_arc_tails_with_course: if model_arc_tail.key() in model_arc_head.tails: result['nodes'].append(model_arc_head.sink) result['arcs'].append(_makeArc(course, model_arc_head.sink)) # rec_result = _traverseForwards(model_arc_head.sink, c_levels - 1) # _extendResult(result, rec_result) return result Originally, I thought it might be a recursion error, but I commented out the recursion and the problem persists. If this function is called with c_levels = 0, it runs fine. The models it references: class Course(db.Model): dept_code = db.StringProperty() number = db.IntegerProperty() title = db.StringProperty() raw_pre_reqs = db.StringProperty(multiline=True) original_description = db.StringProperty() def getPreReqs(self): return pickle.loads(str(self.raw_pre_reqs)) def __repr__(self): return "%s %s: %s" % (self.dept_code, self.number, self.title) class DependencyArcTail(db.Model): ''' A list of courses that is a pre-req for something else ''' courses = db.ListProperty(db.Key) def equals(self, arcTail): for this_course in self.courses: if not (this_course in arcTail.courses): return False for other_course in arcTail.courses: if not (other_course in self.courses): return False return True class DependencyArcHead(db.Model): ''' Maintains a course, and a list of tails with that course as their sink ''' sink = db.ReferenceProperty() tails = db.ListProperty(db.Key) Utility functions it references: def _makeArc(source, sink): return {'source': source, 'sink': sink} def _getListArcTailsWithCourse(course): ''' returns a LIST, not SET there may be duplicate entries ''' q_arc_heads = DependencyArcHead.all() result = [] for arc_head in q_arc_heads: for key_arc_tail in arc_head.tails: model_arc_tail = db.get(key_arc_tail) if course.key() in model_arc_tail.courses: result.append(model_arc_tail) return result Am I missing something pretty obvious here, or is GAEUnit acting up?

    Read the article

  • Building an 'Activation Key' Generator in JAVA

    - by jax
    I want to develop a Key generator for my phone applications. Currently I am using an external service to do the job but I am a little concerned that the service might go offline one day hence I will be in a bit of a pickle. How authentication works now. Public key stored on the phone. When the user requests a key the 'phone ID' is sent to the "Key Generation Service" and the encrypted key key is returned and stored inside a license file. On the phone I can check if the key is for the current phone by using a method getPhoneId() which I can check with the the current phone and grant or not grant access to features. I like this and it works well, however, I want to create my own "Key Generation Service" from my own website. Requirements: Public and Private Key Encryption:(Bouncy Castle) Written in JAVA Must support getApplicationId() (so that many applications can use the same key generator) and getPhoneId() (to get the phone id out of the encrypted license file) I want to be able to send the ApplicationId and PhoneId to the service for license key generation. Can someone give me some pointers on how to accomplish this? I have dabbled around with some java encryption but am definitely no expert and can't find anything that will help me. A list of the Java classes I would need to instantiate would be helpful.

    Read the article

  • Exit and rollback everything in script on error

    - by Jan W.
    Hey guys ! I'm in a bit of a pickle here. I have a TSQL script that does a lot of database structure adjustments but it's not really safe to just let it go through when something fails. to make things clear: using MS SQL 2005 it's NOT a stored procedure, just a script file (.sql) what I have is something in the following order BEGIN TRANSACTION ALTER Stuff GO CREATE New Stuff GO DROP Old Stuff GO IF @@ERROR != 0 BEGIN PRINT 'Errors Found ... Rolling back' ROLLBACK TRANSACTION RETURN END ELSE PRINT 'No Errors ... Committing changes' COMMIT TRANSACTION just to illustrate what I'm working with ... can't go into specifics now, the problem ... When I introduce an error (to test if things get rolled back), I get a statement that the ROLLBACK TRANSACTION could not find a corresponding BEGIN TRANSACTION. This leads me to believe that something when REALLY wrong and the transaction was already killed. what I also noticed is that the script didn't fully quit on error and thus DID try to execute every statement after the error occured. (I noticed this when new tables showed up when I wasn't expecting them because it should have rollbacked) any help in this department is welcome if more speficics are needed, ask! greetz

    Read the article

  • guide for creating addins for Visual Studio 2010?

    - by JMarsch
    Hello: 2 questions actuall: Is there a good comprensive guide out there for creating add-ins for Visual Studio? Here's a weird specific problem -- I'm trying to get my add-in to load. The code that the wizard stubs into the OnConnect method will add it to the toolbar, but only if it passes this "if" statement: if (connectMode == ext_ConnectMode.ext_cm_UISetup) The problem is that connectMode never seems to equal ext_cm_UISetup. It always seems to equal ext_cm_AfterStartup, and My AddIn never appears on the toolbar. I could cheat and short-circuit the if, but since Visual Studio put that code there, I bet it's right, and I have something else wrong. According to the docs, the value should only ever be ext_cm_UISetup once -- the first time VS encounters the addin. So I probably need to somehow make VS forget about the add-in, but I can't find anywhere to do that. And by the way, isn't that exceptionally delicate? The reason I'm in this pickle is because I happened to get an exception on the first try, so now I can never try again?? Can't be right.

    Read the article

  • 500 internal server error on certain page after a few hours

    - by Brian Leach
    I am getting a 500 Internal Server Error on a certain page of my site after a few hours of being up. I restart uWSGI instance with uwsgi --ini /home/metheuser/webapps/ers_portal/ers_portal_uwsgi.ini and it works again for a few hours. The rest of the site seems to be working. When I navigate to my_table, I am directed to the login page. But, I get the 500 error on my table page on login. I followed the instructions here to set up my nginx and uwsgi configs. That is, I have ers_portal_nginx.conf located i my app folder that is symlinked to /etc/nginx/conf.d/. I start my uWSGI "instance" (not sure what exactly to call it) in a Screen instance as mentioned above, with the .ini file located in my app folder My ers_portal_nginx.conf: server { listen 80; server_name www.mydomain.com; location / { try_files $uri @app; } location @app { include uwsgi_params; uwsgi_pass unix:/home/metheuser/webapps/ers_portal/run_web_uwsgi.sock; } } My ers_portal_uwsgi.ini: [uwsgi] #user info uid = metheuser gid = ers_group #application's base folder base = /home/metheuser/webapps/ers_portal #python module to import app = run_web module = %(app) home = %(base)/ers_portal_venv pythonpath = %(base) #socket file's location socket = /home/metheuser/webapps/ers_portal/%n.sock #permissions for the socket file chmod-socket = 666 #uwsgi varible only, does not relate to your flask application callable = app #location of log files logto = /home/metheuser/webapps/ers_portal/logs/%n.log Relevant parts of my views.py data_modification_time = None data = None def reload_data(): global data_modification_time, data, sites, column_names filename = '/home/metheuser/webapps/ers_portal/app/static/' + ec.dd_filename mtime = os.stat(filename).st_mtime if data_modification_time != mtime: data_modification_time = mtime with open(filename) as f: data = pickle.load(f) return data @a bunch of authentication stuff... @app.route('/') @app.route('/index') def index(): return render_template("index.html", title = 'Main',) @app.route('/login', methods = ['GET', 'POST']) def login(): login stuff... @app.route('/my_table') @login_required def my_table(): print 'trying to access data table...' data = reload_data() return render_template("my_table.html", title = "Rundata Viewer", sts = sites, cn = column_names, data = data) # dictionary of data I installed nginx via yum as described here (yesterday) I am using uWSGI installed in my venv via pip I am on CentOS 6 My uwsgi log shows: Wed Jun 11 17:20:01 2014 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 287] during GET /whm-server-status (127.0.0.1) IOError: write error [pid: 9586|app: 0|req: 135/135] 127.0.0.1 () {24 vars in 292 bytes} [Wed Jun 11 17:20:01 2014] GET /whm-server-status => generated 0 bytes in 3 msecs (HTTP/1.0 404) 2 headers in 0 bytes (0 switches on core 0) When its working, the print statement in the views "my_table" route prints into the log file. But not once it stops working. Any ideas?

    Read the article

  • YouTube Scalability Lessons

    - by Bertrand Matthelié
    @font-face { font-family: "Arial"; }@font-face { font-family: "Courier New"; }@font-face { font-family: "Wingdings"; }@font-face { font-family: "Calibri"; }@font-face { font-family: "Cambria"; }p.MsoNormal, li.MsoNormal, div.MsoNormal { margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman"; }h2 { margin: 12pt 0cm 3pt; page-break-after: avoid; font-size: 14pt; font-family: "Times New Roman"; font-style: italic; }a:link, span.MsoHyperlink { color: blue; text-decoration: underline; }a:visited, span.MsoHyperlinkFollowed { color: purple; text-decoration: underline; }span.Heading2Char { font-family: Calibri; font-weight: bold; font-style: italic; }div.Section1 { page: Section1; }ol { margin-bottom: 0cm; }ul { margin-bottom: 0cm; } Very interesting blog post by Todd Hoff at highscalability.com presenting “7 Years of YouTube Scalability Lessons in 30 min” based on a presentation from Mike Solomon, one of the original engineers at YouTube: …. The key takeaway away of the talk for me was doing a lot with really simple tools. While many teams are moving on to more complex ecosystems, YouTube really does keep it simple. They program primarily in Python, use MySQL as their database, they’ve stuck with Apache, and even new features for such a massive site start as a very simple Python program. That doesn’t mean YouTube doesn’t do cool stuff, they do, but what makes everything work together is more a philosophy or a way of doing things than technological hocus pocus. What made YouTube into one of the world’s largest websites? Read on and see... Stats @font-face { font-family: "Arial"; }@font-face { font-family: "Cambria"; }p.MsoNormal, li.MsoNormal, div.MsoNormal { margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman"; }div.Section1 { page: Section1; } 4 billion Views a day 60 hours of video is uploaded every minute 350+ million devices are YouTube enabled Revenue double in 2010 The number of videos has gone up 9 orders of magnitude and the number of developers has only gone up two orders of magnitude. 1 million lines of Python code Stack @font-face { font-family: "Arial"; }@font-face { font-family: "Cambria"; }p.MsoNormal, li.MsoNormal, div.MsoNormal { margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman"; }div.Section1 { page: Section1; } Python - most of the lines of code for YouTube are still in Python. Everytime you watch a YouTube video you are executing a bunch of Python code. Apache - when you think you need to get rid of it, you don’t. Apache is a real rockstar technology at YouTube because they keep it simple. Every request goes through Apache. Linux - the benefit of Linux is there’s always a way to get in and see how your system is behaving. No matter how bad your app is behaving, you can take a look at it with Linux tools like strace and tcpdump. MySQL - is used a lot. When you watch a video you are getting data from MySQL. Sometime it’s used a relational database or a blob store. It’s about tuning and making choices about how you organize your data. Vitess- a  new project released by YouTube, written in Go, it’s a frontend to MySQL. It does a lot of optimization on the fly, it rewrites queries and acts as a proxy. Currently it serves every YouTube database request. It’s RPC based. Zookeeper - a distributed lock server. It’s used for configuration. Really interesting piece of technology. Hard to use correctly so read the manual Wiseguy - a CGI servlet container. Spitfire - a templating system. It has an abstract syntax tree that let’s them do transformations to make things go faster. Serialization formats - no matter which one you use, they are all expensive. Measure. Don’t use pickle. Not a good choice. Found protocol buffers slow. They wrote their own BSON implementation, which is 10-15 time faster than the one you can download. ...Contiues. Read the blog Watch the video

    Read the article

  • Python multiprocessing global variable updates not returned to parent

    - by user1459256
    I am trying to return values from subprocesses but these values are unfortunately unpicklable. So I used global variables in threads module with success but have not been able to retrieve updates done in subprocesses when using multiprocessing module. I hope I'm missing something. The results printed at the end are always the same as initial values given the vars dataDV03 and dataDV04. The subprocesses are updating these global variables but these global variables remain unchanged in the parent. import multiprocessing # NOT ABLE to get python to return values in passed variables. ants = ['DV03', 'DV04'] dataDV03 = ['', ''] dataDV04 = {'driver': '', 'status': ''} def getDV03CclDrivers(lib): # call global variable global dataDV03 dataDV03[1] = 1 dataDV03[0] = 0 # eval( 'CCL.' + lib + '.' + lib + '( "DV03" )' ) these are unpicklable instantiations def getDV04CclDrivers(lib, dataDV04): # pass global variable dataDV04['driver'] = 0 # eval( 'CCL.' + lib + '.' + lib + '( "DV04" )' ) if __name__ == "__main__": jobs = [] if 'DV03' in ants: j = multiprocessing.Process(target=getDV03CclDrivers, args=('LORR',)) jobs.append(j) if 'DV04' in ants: j = multiprocessing.Process(target=getDV04CclDrivers, args=('LORR', dataDV04)) jobs.append(j) for j in jobs: j.start() for j in jobs: j.join() print 'Results:\n' print 'DV03', dataDV03 print 'DV04', dataDV04 I cannot post to my question so will try to edit the original. Here is the object that is not picklable: In [1]: from CCL import LORR In [2]: lorr=LORR.LORR('DV20', None) In [3]: lorr Out[3]: <CCL.LORR.LORR instance at 0x94b188c> This is the error returned when I use a multiprocessing.Pool to return the instance back to the parent: Thread getCcl (('DV20', 'LORR'),) Process PoolWorker-1: Traceback (most recent call last): File "/alma/ACS-10.1/casa/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap self.run() File "/alma/ACS-10.1/casa/lib/python2.6/multiprocessing/process.py", line 88, in run self._target(*self._args, **self._kwargs) File "/alma/ACS-10.1/casa/lib/python2.6/multiprocessing/pool.py", line 71, in worker put((job, i, result)) File "/alma/ACS-10.1/casa/lib/python2.6/multiprocessing/queues.py", line 366, in put return send(obj) UnpickleableError: Cannot pickle <type 'thread.lock'> objects In [5]: dir(lorr) Out[5]: ['GET_AMBIENT_TEMPERATURE', 'GET_CAN_ERROR', 'GET_CAN_ERROR_COUNT', 'GET_CHANNEL_NUMBER', 'GET_COUNT_PER_C_OP', 'GET_COUNT_REMAINING_OP', 'GET_DCM_LOCKED', 'GET_EFC_125_MHZ', 'GET_EFC_COMB_LINE_PLL', 'GET_ERROR_CODE_LAST_CAN_ERROR', 'GET_INTERNAL_SLAVE_ERROR_CODE', 'GET_MAGNITUDE_CELSIUS_OP', 'GET_MAJOR_REV_LEVEL', 'GET_MINOR_REV_LEVEL', 'GET_MODULE_CODES_CDAY', 'GET_MODULE_CODES_CMONTH', 'GET_MODULE_CODES_DIG1', 'GET_MODULE_CODES_DIG2', 'GET_MODULE_CODES_DIG4', 'GET_MODULE_CODES_DIG6', 'GET_MODULE_CODES_SERIAL', 'GET_MODULE_CODES_VERSION_MAJOR', 'GET_MODULE_CODES_VERSION_MINOR', 'GET_MODULE_CODES_YEAR', 'GET_NODE_ADDRESS', 'GET_OPTICAL_POWER_OFF', 'GET_OUTPUT_125MHZ_LOCKED', 'GET_OUTPUT_2GHZ_LOCKED', 'GET_PATCH_LEVEL', 'GET_POWER_SUPPLY_12V_NOT_OK', 'GET_POWER_SUPPLY_15V_NOT_OK', 'GET_PROTOCOL_MAJOR_REV_LEVEL', 'GET_PROTOCOL_MINOR_REV_LEVEL', 'GET_PROTOCOL_PATCH_LEVEL', 'GET_PROTOCOL_REV_LEVEL', 'GET_PWR_125_MHZ', 'GET_PWR_25_MHZ', 'GET_PWR_2_GHZ', 'GET_READ_MODULE_CODES', 'GET_RX_OPT_PWR', 'GET_SERIAL_NUMBER', 'GET_SIGN_OP', 'GET_STATUS', 'GET_SW_REV_LEVEL', 'GET_TE_LENGTH', 'GET_TE_LONG_FLAG_SET', 'GET_TE_OFFSET_COUNTER', 'GET_TE_SHORT_FLAG_SET', 'GET_TRANS_NUM', 'GET_VDC_12', 'GET_VDC_15', 'GET_VDC_7', 'GET_VDC_MINUS_7', 'SET_CLEAR_FLAGS', 'SET_FPGA_LOGIC_RESET', 'SET_RESET_AMBSI', 'SET_RESET_DEVICE', 'SET_RESYNC_TE', 'STATUS', '_HardwareDevice__componentName', '_HardwareDevice__hw', '_HardwareDevice__stickyFlag', '_LORRBase__logger', '__del__', '__doc__', '__init__', '__module__', '_devices', 'clearDeviceCommunicationErrorAlarm', 'getControlList', 'getDeviceCommunicationErrorCounter', 'getErrorMessage', 'getHwState', 'getInternalSlaveCanErrorMsg', 'getLastCanErrorMsg', 'getMonitorList', 'hwConfigure', 'hwDiagnostic', 'hwInitialize', 'hwOperational', 'hwSimulation', 'hwStart', 'hwStop', 'inErrorState', 'isMonitoring', 'isSimulated'] In [6]:

    Read the article

  • Error using paho-mqtt in App Engine Python App

    - by calumb
    I am trying to right a Google Cloud Platform app in python with Flask that makes an MQTT connection. I have included the paho python library by doing pip install paho-mqtt -t libs/. However, when I try to run the app, even if I don't try to connect to MQTT. I get a weird error about IP address checking: RuntimeError: error('illegal IP address string passed to inet_pton',) It seems something in the remote_socket lib is causing a problem. Is this a security issue? Is there someway to disable it? Relevant code: from flask import Flask import paho.mqtt.client as mqtt import logging as logger app = Flask(__name__) # Note: We don't need to call run() since our application is embedded within # the App Engine WSGI application server. #callback to print out connection status def on_connect(mosq, obj, rc): logger.info('on_connect') if rc == 0: logger.info("Connected") mqttc.subscribe('test', 0) else: logger.info(rc) def on_message(mqttc, obj, msg): logger.info(msg.topic+" "+str(msg.qos)+" "+str(msg.payload)) mqttc = mqtt.Client("mqttpy") mqttc.on_message = on_message mqttc.on_connect = on_connect As well as full stack trace: ERROR 2014-06-03 15:14:57,285 wsgi.py:262] Traceback (most recent call last): File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 239, in Handle handler = _config_handle.add_wsgi_middleware(self._LoadHandler()) File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 298, in _LoadHandler handler, path, err = LoadObject(self._handler) File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 84, in LoadObject obj = __import__(path[0]) File "/Users/cbarnes/code/ignite/tank-demo/appengine-flask-demo/main.py", line 24, in <module> mqttc = mqtt.Client("mqtthtpp") File "/Users/cbarnes/code/ignite/tank-demo/appengine-flask-demo/lib/paho/mqtt/client.py", line 403, in __init__ self._sockpairR, self._sockpairW = _socketpair_compat() File "/Users/cbarnes/code/ignite/tank-demo/appengine-flask-demo/lib/paho/mqtt/client.py", line 255, in _socketpair_compat listensock.bind(("localhost", 0)) File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/dist27/socket.py", line 222, in meth return getattr(self._sock,name)(*args) File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/api/remote_socket/_remote_socket.py", line 668, in bind self._SetProtoFromAddr(request.mutable_proxy_external_ip(), address) File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/api/remote_socket/_remote_socket.py", line 632, in _SetProtoFromAddr proto.set_packed_address(self._GetPackedAddr(address)) File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/api/remote_socket/_remote_socket.py", line 627, in _GetPackedAddr AI_NUMERICSERV|AI_PASSIVE): File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/api/remote_socket/_remote_socket.py", line 338, in getaddrinfo canonical=(flags & AI_CANONNAME)) File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/api/remote_socket/_remote_socket.py", line 211, in _Resolve canon, aliases, addresses = _ResolveName(name, families) File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/api/remote_socket/_remote_socket.py", line 229, in _ResolveName apiproxy_stub_map.MakeSyncCall('remote_socket', 'Resolve', request, reply) File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 94, in MakeSyncCall return stubmap.MakeSyncCall(service, call, request, response) File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 328, in MakeSyncCall rpc.CheckSuccess() File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_rpc.py", line 156, in _WaitImpl self.request, self.response) File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 200, in MakeSyncCall self._MakeRealSyncCall(service, call, request, response) File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 234, in _MakeRealSyncCall raise pickle.loads(response_pb.exception()) RuntimeError: error('illegal IP address string passed to inet_pton',) INFO 2014-06-03 15:14:57,291 module.py:639] default: "GET / HTTP/1.1" 500 - Thanks!

    Read the article

  • Setting up a "to-many" relationship value dependency for a transient Core Data attribute

    - by Greg Combs
    I've got a relatively complicated Core Data relationship structure and I'm trying to figure out how to set up value dependencies (or observations) across various to-many relationships. Let me start out with some basic info. I've got a classroom with students, assignments, and grades (students X assignments). For simplicity's sake, we don't really have to focus much on the assignments yet. StudentObj <--->> ScoreObj <<---> AssignmentObj Each ScoreObj has a to-one relation with the StudentObj and the AssignmentObj. ScoreObj has real attributes for the numerical grade, the turnInDate, and notes. AssignmentObj.scores is the set of Score objects for that assignment (N = all students). AssignmentObj has real attributes for name, dueDate, curveFunction, gradeWeight, and maxPoints. StudentObj.scores is the set of Score objects for that student (N = all assignments). StudentObj also has real attributes like name, studentID, email, etc. StudentObj has a transient (calculated, not stored) attribute called gradeTotal. This last item, gradeTotal, is the real pickle. it calculates the student's overall semester grade using the scores (ScoreObj) from all their assignments, their associated assignment gradeWeights, curves, and maxPoints, and various other things. This gradeTotal value is displayed in a table column, along with all the students and their individual assignment grades. Determining the value of gradeTotal is a relatively expensive operation, particularly with a large class, therefore I want to run it only when necessary. For simplicity's sake, I'm not storing that gradeTotal value in the core data model. I don't mind caching it somewhere, but I'm having a bitch of a time determining where and how to best update that cache. I need to run that calculation for each student whenever any value changes that affects their gradeTotal. If this were a simple to-one relationship, I know I could use something like keyPathsForValuesAffectingGradeTotal ... but it's more like a many-to-one-to-many relationship. Does anyone know of an elegant (and KVC correct) solution? I guess I could tear through all those score and assignment objects and tell them to register their students as observers. But this seems like a blunt force approach.

    Read the article

  • PHP database selection issue

    - by Citroenfris
    I'm in a bit of a pickle with freshening up my PHP a bit, it's been about 3 years since I last coded in PHP. Any insights are welcomed! I'll give you as much information as I possibly can to resolve this error so here goes! Files config.php database.php news.php BLnews.php index.php Includes config.php - news.php database.php - news.php news.php - BLnews.php BLnews.php - index.php Now the problem with my current code is that the database connection is being made but my database refuses to be selected. The query I have should work but due to my database not getting selected it's kind of annoying to get any data exchange going! database.php <?php class Database { //------------------------------------------- // Connects to the database //------------------------------------------- function connect() { if (isset($dbhost) && isset($dbuser) && isset($dbpass)) { $con = mysql_connect($dbhost, $dbuser, $dbpass) or die("Could not connect: " . mysql_error()); } }// end function connect function selectDB() { if (isset($dbname) && isset($con)) { $selected_db = mysql_select_db($dbname, $con) or die("Could not select test DB"); } } } // end class Database ?> News.php <?php // include the config file and database class include 'config.php'; include 'database.php'; ... ?> BLnews.php <?php // include the news class include 'news.php'; // create an instance of the Database class and call it $db $db = new Database; $db -> connect(); $db->selectDB(); class BLnews { function getNews() { $sql = "SELECT * FROM news"; if (isset($sql)) { $result = mysql_query($sql) or die("Could not execute query. Reason: " .mysql_error()); } return $result; } ?> index.php <?php ... include 'includes/BLnews.php'; $blNews = new BLnews(); $news = $blNews->getNews(); ?> ... <?php while($row = mysql_fetch_array($news)) { echo '<div class="post">'; echo '<h2><a href="#"> ' . $row["title"] .'</a></h2>'; echo '<p class="post-info">Posted by <a href="#"> </a> | <span class="date"> Posted on <a href="#">' . $row["date"] . '</a></span></p>'; echo $row["content"]; echo '</div>'; } ?> Well this is pretty much everything that should get the information going however due to the mysql_error in $result = mysql_query($sql) or die("Could not execute query. Reason: " .mysql_error()); I can see the error and it says: Could not execute query. Reason: No database selected I honestly have no idea why it would not work and I've been fiddling with it for quite some time now. Help is most welcomed and I thank you in advance! Greets Lemon

    Read the article

  • Where can these be posted besides the Python Cookbook?

    - by Noctis Skytower
    Whitespace Assembler #! /usr/bin/env python """Assembler.py Compiles a program from "Assembly" folder into "Program" folder. Can be executed directly by double-click or on the command line. Give name of *.WSA file without extension (example: stack_calc).""" ################################################################################ __author__ = 'Stephen "Zero" Chappell <[email protected]>' __date__ = '14 March 2010' __version__ = '$Revision: 3 $' ################################################################################ import string from Interpreter import INS, MNEMONIC ################################################################################ def parse(code): program = [] process_virtual(program, code) process_control(program) return tuple(program) def process_virtual(program, code): for line, text in enumerate(code.split('\n')): if not text or text[0] == '#': continue if text.startswith('part '): parse_part(program, line, text[5:]) elif text.startswith(' '): parse_code(program, line, text[5:]) else: syntax_error(line) def syntax_error(line): raise SyntaxError('Line ' + str(line + 1)) ################################################################################ def process_control(program): parts = get_parts(program) names = dict(pair for pair in zip(parts, generate_index())) correct_control(program, names) def get_parts(program): parts = [] for ins in program: if isinstance(ins, tuple): ins, arg = ins if ins == INS.PART: if arg in parts: raise NameError('Part definition was found twice: ' + arg) parts.append(arg) return parts def generate_index(): index = 1 while True: yield index index *= -1 if index > 0: index += 1 def correct_control(program, names): for index, ins in enumerate(program): if isinstance(ins, tuple): ins, arg = ins if ins in HAS_LABEL: if arg not in names: raise NameError('Part definition was never found: ' + arg) program[index] = (ins, names[arg]) ################################################################################ def parse_part(program, line, text): if not valid_label(text): syntax_error(line) program.append((INS.PART, text)) def valid_label(text): if not between_quotes(text): return False label = text[1:-1] if not valid_name(label): return False return True def between_quotes(text): if len(text) < 3: return False if text.count('"') != 2: return False if text[0] != '"' or text[-1] != '"': return False return True def valid_name(label): valid_characters = string.ascii_letters + string.digits + '_' valid_set = frozenset(valid_characters) label_set = frozenset(label) if len(label_set - valid_set) != 0: return False return True ################################################################################ from Interpreter import HAS_LABEL, Program NO_ARGS = Program.NO_ARGS HAS_ARG = Program.HAS_ARG TWO_WAY = tuple(set(NO_ARGS) & set(HAS_ARG)) ################################################################################ def parse_code(program, line, text): for ins, word in enumerate(MNEMONIC): if text.startswith(word): check_code(program, line, text[len(word):], ins) break else: syntax_error(line) def check_code(program, line, text, ins): if ins in TWO_WAY: if text: number = parse_number(line, text) program.append((ins, number)) else: program.append(ins) elif ins in HAS_LABEL: text = parse_label(line, text) program.append((ins, text)) elif ins in HAS_ARG: number = parse_number(line, text) program.append((ins, number)) elif ins in NO_ARGS: if text: syntax_error(line) program.append(ins) else: syntax_error(line) def parse_label(line, text): if not text or text[0] != ' ': syntax_error(line) text = text[1:] if not valid_label(text): syntax_error(line) return text ################################################################################ def parse_number(line, text): if not valid_number(text): syntax_error(line) return int(text) def valid_number(text): if len(text) < 2: return False if text[0] != ' ': return False text = text[1:] if '+' in text and '-' in text: return False if '+' in text: if text.count('+') != 1: return False if text[0] != '+': return False text = text[1:] if not text: return False if '-' in text: if text.count('-') != 1: return False if text[0] != '-': return False text = text[1:] if not text: return False valid_set = frozenset(string.digits) value_set = frozenset(text) if len(value_set - valid_set) != 0: return False return True ################################################################################ ################################################################################ from Interpreter import partition_number VMC_2_TRI = { (INS.PUSH, True): (0, 0), (INS.COPY, False): (0, 2, 0), (INS.COPY, True): (0, 1, 0), (INS.SWAP, False): (0, 2, 1), (INS.AWAY, False): (0, 2, 2), (INS.AWAY, True): (0, 1, 2), (INS.ADD, False): (1, 0, 0, 0), (INS.SUB, False): (1, 0, 0, 1), (INS.MUL, False): (1, 0, 0, 2), (INS.DIV, False): (1, 0, 1, 0), (INS.MOD, False): (1, 0, 1, 1), (INS.SET, False): (1, 1, 0), (INS.GET, False): (1, 1, 1), (INS.PART, True): (2, 0, 0), (INS.CALL, True): (2, 0, 1), (INS.GOTO, True): (2, 0, 2), (INS.ZERO, True): (2, 1, 0), (INS.LESS, True): (2, 1, 1), (INS.BACK, False): (2, 1, 2), (INS.EXIT, False): (2, 2, 2), (INS.OCHR, False): (1, 2, 0, 0), (INS.OINT, False): (1, 2, 0, 1), (INS.ICHR, False): (1, 2, 1, 0), (INS.IINT, False): (1, 2, 1, 1) } ################################################################################ def to_trinary(program): trinary_code = [] for ins in program: if isinstance(ins, tuple): ins, arg = ins trinary_code.extend(VMC_2_TRI[(ins, True)]) trinary_code.extend(from_number(arg)) else: trinary_code.extend(VMC_2_TRI[(ins, False)]) return tuple(trinary_code) def from_number(arg): code = [int(arg < 0)] if arg: for bit in reversed(list(partition_number(abs(arg), 2))): code.append(bit) return code + [2] return code + [0, 2] to_ws = lambda trinary: ''.join(' \t\n'[index] for index in trinary) def compile_wsa(source): program = parse(source) trinary = to_trinary(program) ws_code = to_ws(trinary) return ws_code ################################################################################ ################################################################################ import os import sys import time import traceback def main(): name, source, command_line, error = get_source() if not error: start = time.clock() try: ws_code = compile_wsa(source) except: print('ERROR: File could not be compiled.\n') traceback.print_exc() error = True else: path = os.path.join('Programs', name + '.ws') try: open(path, 'w').write(ws_code) except IOError as err: print(err) error = True else: div, mod = divmod((time.clock() - start) * 1000, 1) args = int(div), '{:.3}'.format(mod)[1:] print('DONE: Comipled in {}{} ms'.format(*args)) handle_close(error, command_line) def get_source(): if len(sys.argv) > 1: command_line = True name = sys.argv[1] else: command_line = False try: name = input('Source File: ') except: return None, None, False, True print() path = os.path.join('Assembly', name + '.wsa') try: return name, open(path).read(), command_line, False except IOError as err: print(err) return None, None, command_line, True def handle_close(error, command_line): if error: usage = 'Usage: {} <assembly>'.format(os.path.basename(sys.argv[0])) print('\n{}\n{}'.format('-' * len(usage), usage)) if not command_line: time.sleep(10) ################################################################################ if __name__ == '__main__': main() Whitespace Helpers #! /usr/bin/env python """Helpers.py Includes a function to encode Python strings into my WSA format. Has a "PRINT_LINE" function that can be copied to a WSA program. Contains a "PRINT" function and documentation as an explanation.""" ################################################################################ __author__ = 'Stephen "Zero" Chappell <[email protected]>' __date__ = '14 March 2010' __version__ = '$Revision: 1 $' ################################################################################ def encode_string(string, addr): print(' push', addr) print(' push', len(string)) print(' set') addr += 1 for offset, character in enumerate(string): print(' push', addr + offset) print(' push', ord(character)) print(' set') ################################################################################ # Prints a string with newline. # push addr # call "PRINT_LINE" """ part "PRINT_LINE" call "PRINT" push 10 ochr back """ ################################################################################ # def print(array): # if len(array) <= 0: # return # offset = 1 # while len(array) - offset >= 0: # ptr = array.ptr + offset # putch(array[ptr]) # offset += 1 """ part "PRINT" # Line 1-2 copy get less "__PRINT_RET_1" copy get zero "__PRINT_RET_1" # Line 3 push 1 # Line 4 part "__PRINT_LOOP" copy copy 2 get swap sub less "__PRINT_RET_2" # Line 5 copy 1 copy 1 add # Line 6 get ochr # Line 7 push 1 add goto "__PRINT_LOOP" part "__PRINT_RET_2" away part "__PRINT_RET_1" away back """ Whitespace Interpreter #! /usr/bin/env python """Interpreter.py Runs programs in "Programs" and creates *.WSO files when needed. Can be executed directly by double-click or on the command line. If run on command line, add "ASM" flag to dump program assembly.""" ################################################################################ __author__ = 'Stephen "Zero" Chappell <[email protected]>' __date__ = '14 March 2010' __version__ = '$Revision: 4 $' ################################################################################ def test_file(path): disassemble(parse(trinary(load(path))), True) ################################################################################ load = lambda ws: ''.join(c for r in open(ws) for c in r if c in ' \t\n') trinary = lambda ws: tuple(' \t\n'.index(c) for c in ws) ################################################################################ def enum(names): names = names.replace(',', ' ').split() space = dict((reversed(pair) for pair in enumerate(names)), __slots__=()) return type('enum', (object,), space)() INS = enum('''\ PUSH, COPY, SWAP, AWAY, \ ADD, SUB, MUL, DIV, MOD, \ SET, GET, \ PART, CALL, GOTO, ZERO, LESS, BACK, EXIT, \ OCHR, OINT, ICHR, IINT''') ################################################################################ def parse(code): ins = iter(code).__next__ program = [] while True: try: imp = ins() except StopIteration: return tuple(program) if imp == 0: # [Space] parse_stack(ins, program) elif imp == 1: # [Tab] imp = ins() if imp == 0: # [Tab][Space] parse_math(ins, program) elif imp == 1: # [Tab][Tab] parse_heap(ins, program) else: # [Tab][Line] parse_io(ins, program) else: # [Line] parse_flow(ins, program) def parse_number(ins): sign = ins() if sign == 2: raise StopIteration() buffer = '' code = ins() if code == 2: raise StopIteration() while code != 2: buffer += str(code) code = ins() if sign == 1: return int(buffer, 2) * -1 return int(buffer, 2) ################################################################################ def parse_stack(ins, program): code = ins() if code == 0: # [Space] number = parse_number(ins) program.append((INS.PUSH, number)) elif code == 1: # [Tab] code = ins() number = parse_number(ins) if code == 0: # [Tab][Space] program.append((INS.COPY, number)) elif code == 1: # [Tab][Tab] raise StopIteration() else: # [Tab][Line] program.append((INS.AWAY, number)) else: # [Line] code = ins() if code == 0: # [Line][Space] program.append(INS.COPY) elif code == 1: # [Line][Tab] program.append(INS.SWAP) else: # [Line][Line] program.append(INS.AWAY) def parse_math(ins, program): code = ins() if code == 0: # [Space] code = ins() if code == 0: # [Space][Space] program.append(INS.ADD) elif code == 1: # [Space][Tab] program.append(INS.SUB) else: # [Space][Line] program.append(INS.MUL) elif code == 1: # [Tab] code = ins() if code == 0: # [Tab][Space] program.append(INS.DIV) elif code == 1: # [Tab][Tab] program.append(INS.MOD) else: # [Tab][Line] raise StopIteration() else: # [Line] raise StopIteration() def parse_heap(ins, program): code = ins() if code == 0: # [Space] program.append(INS.SET) elif code == 1: # [Tab] program.append(INS.GET) else: # [Line] raise StopIteration() def parse_io(ins, program): code = ins() if code == 0: # [Space] code = ins() if code == 0: # [Space][Space] program.append(INS.OCHR) elif code == 1: # [Space][Tab] program.append(INS.OINT) else: # [Space][Line] raise StopIteration() elif code == 1: # [Tab] code = ins() if code == 0: # [Tab][Space] program.append(INS.ICHR) elif code == 1: # [Tab][Tab] program.append(INS.IINT) else: # [Tab][Line] raise StopIteration() else: # [Line] raise StopIteration() def parse_flow(ins, program): code = ins() if code == 0: # [Space] code = ins() label = parse_number(ins) if code == 0: # [Space][Space] program.append((INS.PART, label)) elif code == 1: # [Space][Tab] program.append((INS.CALL, label)) else: # [Space][Line] program.append((INS.GOTO, label)) elif code == 1: # [Tab] code = ins() if code == 0: # [Tab][Space] label = parse_number(ins) program.append((INS.ZERO, label)) elif code == 1: # [Tab][Tab] label = parse_number(ins) program.append((INS.LESS, label)) else: # [Tab][Line] program.append(INS.BACK) else: # [Line] code = ins() if code == 2: # [Line][Line] program.append(INS.EXIT) else: # [Line][Space] or [Line][Tab] raise StopIteration() ################################################################################ MNEMONIC = '\ push copy swap away add sub mul div mod set get part \ call goto zero less back exit ochr oint ichr iint'.split() HAS_ARG = [getattr(INS, name) for name in 'PUSH COPY AWAY PART CALL GOTO ZERO LESS'.split()] HAS_LABEL = [getattr(INS, name) for name in 'PART CALL GOTO ZERO LESS'.split()] def disassemble(program, names=False): if names: names = create_names(program) for ins in program: if isinstance(ins, tuple): ins, arg = ins assert ins in HAS_ARG has_arg = True else: assert INS.PUSH <= ins <= INS.IINT has_arg = False if ins == INS.PART: if names: print(MNEMONIC[ins], '"' + names[arg] + '"') else: print(MNEMONIC[ins], arg) elif has_arg and ins in HAS_ARG: if ins in HAS_LABEL and names: assert arg in names print(' ' + MNEMONIC[ins], '"' + names[arg] + '"') else: print(' ' + MNEMONIC[ins], arg) else: print(' ' + MNEMONIC[ins]) ################################################################################ def create_names(program): names = {} number = 1 for ins in program: if isinstance(ins, tuple) and ins[0] == INS.PART: label = ins[1] assert label not in names names[label] = number_to_name(number) number += 1 return names def number_to_name(number): name = '' for offset in reversed(list(partition_number(number, 27))): if offset: name += chr(ord('A') + offset - 1) else: name += '_' return name def partition_number(number, base): div, mod = divmod(number, base) yield mod while div: div, mod = divmod(div, base) yield mod ################################################################################ CODE = (' \t\n', ' \n ', ' \t \t\n', ' \n\t', ' \n\n', ' \t\n \t\n', '\t ', '\t \t', '\t \n', '\t \t ', '\t \t\t', '\t\t ', '\t\t\t', '\n \t\n', '\n \t \t\n', '\n \n \t\n', '\n\t \t\n', '\n\t\t \t\n', '\n\t\n', '\n\n\n', '\t\n ', '\t\n \t', '\t\n\t ', '\t\n\t\t') EXAMPLE = ''.join(CODE) ################################################################################ NOTES = '''\ STACK ===== push number copy copy number swap away away number MATH ==== add sub mul div mod HEAP ==== set get FLOW ==== part label call label goto label zero label less label back exit I/O === ochr oint ichr iint''' ################################################################################ ################################################################################ class Stack: def __init__(self): self.__data = [] # Stack Operators def push(self, number): self.__data.append(number) def copy(self, number=None): if number is None: self.__data.append(self.__data[-1]) else: size = len(self.__data) index = size - number - 1 assert 0 <= index < size self.__data.append(self.__data[index]) def swap(self): self.__data[-2], self.__data[-1] = self.__data[-1], self.__data[-2] def away(self, number=None): if number is None: self.__data.pop() else: size = len(self.__data) index = size - number - 1 assert 0 <= index < size del self.__data[index:-1] # Math Operators def add(self): suffix = self.__data.pop() prefix = self.__data.pop() self.__data.append(prefix + suffix) def sub(self): suffix = self.__data.pop() prefix = self.__data.pop() self.__data.append(prefix - suffix) def mul(self): suffix = self.__data.pop() prefix = self.__data.pop() self.__data.append(prefix * suffix) def div(self): suffix = self.__data.pop() prefix = self.__data.pop() self.__data.append(prefix // suffix) def mod(self): suffix = self.__data.pop() prefix = self.__data.pop() self.__data.append(prefix % suffix) # Program Operator def pop(self): return self.__data.pop() ################################################################################ class Heap: def __init__(self): self.__data = {} def set_(self, addr, item): if item: self.__data[addr] = item elif addr in self.__data: del self.__data[addr] def get_(self, addr): return self.__data.get(addr, 0) ################################################################################ import os import zlib import msvcrt import pickle import string class CleanExit(Exception): pass NOP = lambda arg: None DEBUG_WHITESPACE = False ################################################################################ class Program: NO_ARGS = INS.COPY, INS.SWAP, INS.AWAY, INS.ADD, \ INS.SUB, INS.MUL, INS.DIV, INS.MOD, \ INS.SET, INS.GET, INS.BACK, INS.EXIT, \ INS.OCHR, INS.OINT, INS.ICHR, INS.IINT HAS_ARG = INS.PUSH, INS.COPY, INS.AWAY, INS.PART, \ INS.CALL, INS.GOTO, INS.ZERO, INS.LESS def __init__(self, code): self.__data = code self.__validate() self.__build_jump() self.__check_jump() self.__setup_exec() def __setup_exec(self): self.__iptr = 0 self.__stck = stack = Stack() self.__heap = Heap() self.__cast = [] self.__meth = (stack.push, stack.copy, stack.swap, stack.away, stack.add, stack.sub, stack.mul, stack.div, stack.mod, self.__set, self.__get, NOP, self.__call, self.__goto, self.__zero, self.__less, self.__back, self.__exit, self.__ochr, self.__oint, self.__ichr, self.__iint) def step(self): ins = self.__data[self.__iptr] self.__iptr += 1 if isinstance(ins, tuple): self.__meth[ins[0]](ins[1]) else: self.__meth[ins]() def run(self): while True: ins = self.__data[self.__iptr] self.__iptr += 1 if isinstance(ins, tuple): self.__meth[ins[0]](ins[1]) else: self.__meth[ins]() def __oint(self): for digit in str(self.__stck.pop()): msvcrt.putwch(digit) def __ichr(self): addr = self.__stck.pop() # Input Routine while msvcrt.kbhit(): msvcrt.getwch() while True: char = msvcrt.getwch() if char in '\x00\xE0': msvcrt.getwch() elif char in string.printable: char = char.replace('\r', '\n') msvcrt.putwch(char) break item = ord(char) # Storing Number self.__heap.set_(addr, item) def __iint(self): addr = self.__stck.pop() # Input Routine while msvcrt.kbhit(): msvcrt.getwch() buff = '' char = msvcrt.getwch() while char != '\r' or not buff: if char in '\x00\xE0': msvcrt.getwch() elif char in '+-' and not buff: msvcrt.putwch(char) buff += char elif '0' <= char <= '9': msvcrt.putwch(char) buff += char elif char == '\b': if buff: buff = buff[:-1] msvcrt.putwch(char) msvcrt.putwch(' ') msvcrt.putwch(char) char = msvcrt.getwch() msvcrt.putwch(char) msvcrt.putwch('\n') item = int(buff) # Storing Number self.__heap.set_(addr, item) def __goto(self, label): self.__iptr = self.__jump[label] def __zero(self, label): if self.__stck.pop() == 0: self.__iptr = self.__jump[label] def __less(self, label): if self.__stck.pop() < 0: self.__iptr = self.__jump[label] def __exit(self): self.__setup_exec() raise CleanExit() def __set(self): item = self.__stck.pop() addr = self.__stck.po

    Read the article

< Previous Page | 1 2 3 4 5