Search Results

Search found 602 results on 25 pages for 'chunks'.

Page 19/25 | < Previous Page | 15 16 17 18 19 20 21 22 23 24 25  | Next Page >

  • Looking for advice on importing large dataset in sqlite and Cocoa/Objective-C

    - by jluckyiv
    I have a fairly large hierarchical dataset I'm importing. The total size of the database after import is about 270MB in sqlite. My current method works, but I know I'm hogging memory as I do it. For instance, if I run with Zombies, my system freezes up (although it will execute just fine if I don't use that Instrument). I was hoping for some algorithm advice. I have three hierarchical tables comprising about 400,000 records. The highest level has about 30 records, the next has about 20,000, the last has the balance. Right now, I'm using nested for loops to import. I know I'm creating an unreasonably large object graph, but I'm also looking to serialize to JSON or XML because I want to break up the records into downloadable chunks for the end user to import a la carte. I have the code written to do the serialization, but I'm wondering if I can serialize the object graph if I only have pieces in memory. Here's pseudocode showing the basic process for sqlite import. I left out the unnecessary detail. [database open]; [database beginTransaction]; NSArray *firstLevels = [[FirstLevel fetchFromURL:url retain]; for (FirstLevel *firstLevel in firstLevels) { [firstLevel save]; int id1 = [firstLevel primaryKey]; NSArray *secondLevels = [[SecondLevel fetchFromURL:url] retain]; for (SecondLevel *secondLevel in secondLevels) { [secondLevel saveWithForeignKey:id1]; int id2 = [secondLevel primaryKey]; NSArray *thirdLevels = [[ThirdLevel fetchFromURL:url] retain]; for (ThirdLevel *thirdLevel in thirdLevels) { [thirdLevel saveWithForeignKey:id2]; } [database commit]; [database beginTransaction]; [thirdLevels release]; } [secondLevels release]; } [database commit]; [database release]; [firstLevels release];

    Read the article

  • File transfer eating alot of CPU

    - by Dan C.
    I'm trying to transfer a file over a IHttpHandler, the code is pretty simple. However when i start a single transfer it uses about 20% of the CPU. If i were to scale this to 20 simultaneous transfers the CPU is very high. Is there a better way I can be doing this to keep the CPU lower? the client code just sends over chunks of the file 64KB at a time. public void ProcessRequest(HttpContext context) { if (context.Request.Params["secretKey"] == null) { } else { accessCode = context.Request.Params["secretKey"].ToString(); } if (accessCode == "test") { string fileName = context.Request.Params["fileName"].ToString(); byte[] buffer = Convert.FromBase64String(context.Request.Form["data"]); string fileGuid = context.Request.Params["smGuid"].ToString(); string user = context.Request.Params["user"].ToString(); SaveFile(fileName, buffer, user); } } public void SaveFile(string fileName, byte[] buffer, string user) { string DirPath = @"E:\Filestorage\" + user + @"\"; if (!Directory.Exists(DirPath)) { Directory.CreateDirectory(DirPath); } string FilePath = @"E:\Filestorage\" + user + @"\" + fileName; FileStream writer = new FileStream(FilePath, File.Exists(FilePath) ? FileMode.Append : FileMode.Create, FileAccess.Write, FileShare.ReadWrite); writer.Write(buffer, 0, buffer.Length); writer.Close(); }

    Read the article

  • Hadoop Map/Reduce - simple use example to do the following...

    - by alexeypro
    I have MySQL database, where I store the following BLOB (which contains JSON object) and ID (for this JSON object). JSON object contains a lot of different information. Say, "city:Los Angeles" and "state:California". There are about 500k of such records for now, but they are growing. And each JSON object is quite big. My goal is to do searches (real-time) in MySQL database. Say, I want to search for all JSON objects which have "state" to "California" and "city" to "San Francisco". I want to utilize Hadoop for the task. My idea is that there will be "job", which takes chunks of, say, 100 records (rows) from MySQL, verifies them according to the given search criteria, returns those (ID's) which qualify. Pros/cons? I understand that one might think that I should utilize simple SQL power for that, but the thing is that JSON object structure is pretty "heavy", if I put it as SQL schemas, there will be at least 3-5 tables joins, which (I tried, really) creates quite a headache, and building all the right indexes eats RAM faster than I one can think. ;-) And even then, every SQL query has to be analyzed to be utilizing the indexes, otherwise with full scan it literally is a pain. And with such structure we have the only way "up" is just with vertical scaling. But I am not sure it's the best option for me, as I see how JSON objects will grow (the data structure), and I see that the number of them will grow too. :-) Help? Can somebody point me to simple examples of how this can be done? Does it make sense at all? Am I missing something important? Thank you.

    Read the article

  • Precision error on matrix multiplication

    - by Wam
    Hello all, Coding a matrix multiplication in my program, I get precision errors (inaccurate results for large matrices). Here's my code. The current object has data stored in a flattened array, row after row. Other matrix B has data stored in a flattened array, column after column (so I can use pointer arithmetic). protected double[,] multiply (IMatrix B) { int columns = B.columns; int rows = Rows; int size = Columns; double[,] result = new double[rows,columns]; for (int row = 0; row < rows; row++) { for (int col = 0; col < columns; col++) { unsafe { fixed (float* ptrThis = data) fixed (float* ptrB = B.Data) { float* mePtr = ptrThis + row*rows; float* bPtr = ptrB + col*columns; double value = 0.0; for (int i = 0; i < size; i++) { value += *(mePtr++) * *(bPtr++); } result[row, col] = value; } } } } } Actually, the code is a bit more complicated : I do the multiply thing for several chunks (so instead of having i from 0 to size, I go from localStart to localStop), then sum up the resulting matrices. My problem : for a big matrix I get precision error : NUnit.Framework.AssertionException: Error at (0,1) expected: <6.4209571409444209E+18> but was: <6.4207619776304906E+18> Any idea ?

    Read the article

  • JQuery Deferred - Adding to the Deferred contract

    - by MgSam
    I'm trying to add another asynchronous call to the contract of an existing Deferred before its state is set to success. Rather than try and explain this in English, see the following pseudo-code: $.when( $.ajax({ url: someUrl, data: data, async: true, success: function (data, textStatus, jqXhr) { console.log('Call 1 done.') jqXhr.pipe( $.ajax({ url: someUrl, data: data, async: true, success: function (data, textStatus, jqXhr) { console.log('Call 2 done.'); }, }) ); }, }), $.ajax({ url: someUrl, data: data, async: true, success: function (data, textStatus, jqXhr) { console.log('Call 3 done.'); }, }) ).then(function(){ console.log('All done!'); }); Basically, Call 2 is dependent on the results of Call 1. I want Call 1 and Call 3 to be executed in parallel. Once all 3 calls are complete, I want the All Done code to execute. My understanding is that Deferred.pipe() is supposed to chain another asynchronous call to the given deferred, but in practice, I always get Call 2 completing after All Done. Does anyone know how to get jQuery's Deferred to do what I want? Hopefully the solution doesn't involve ripping the code apart into chunks any further. Thanks for any help.

    Read the article

  • Mahout - Error when try out wikipedia exmaples

    - by Li'
    Note this post is similar to Caused by: java.lang.ClassNotFoundException: classpath but different error message. When I try to run Wikipedia Bayes Example from https://cwiki.apache.org/confluence/display/MAHOUT/Wikipedia+Bayes+Example When I ran the following command : lis-macbook-pro:mahout-distribution-0.8 Li$ mahout wikipediaXMLSplitter -d examples/temp/enwiki-latest-pages-articles10.xml -o wikipedia/chunks -c 64 I got error message: MAHOUT_LOCAL is set, so we don't add HADOOP_CONF_DIR to classpath. MAHOUT_LOCAL is set, running locally SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/Users/Li/File/Java/mahout-distribution-0.8/examples/target/mahout-examples-0.8-job.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/Users/Li/File/Java/mahout-distribution-0.8/examples/target/dependency/slf4j-jcl-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.JCLLoggerFactory] Oct 21, 2013 4:25:47 PM org.slf4j.impl.JCLLoggerAdapter warn WARNING: Unable to add class: wikipediaXMLSplitter java.lang.ClassNotFoundException: wikipediaXMLSplitter at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:171) at org.apache.mahout.driver.MahoutDriver.addClass(MahoutDriver.java:236) at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:127) I am using Hadoop 1.2 and Mahout 0.8. mahout-distribution-0.8/bin has been added to $PATH. $MAHOUT_LOCAL is set to "True", so it runs locally. I dont know why I got "Unable to add class: wikipediaXMLSplitter"

    Read the article

  • Django Image Upload: IOErrno2 Could not find path -- and yet it's saving the image there anyway?

    - by Rob
    I have an issue where the local version of django is handling image upload as expected but my server is not. Note: I am using a Django Container on MediaTemple.net (grid server) Here is my code. def view_settings(request): <snip> if request.POST: success_msgs = () mForm = MainProfileForm(request.POST, request.FILES, instance = mProfile) pForm = ChangePasswordForm(request.POST) eForm = ChangeEmailForm(request.POST) if mForm.is_valid(): m = mForm.save(commit = False) if mForm.cleaned_data['avatar']: m.avatar = upload_photo(request.FILES['avatar'], settings.AVATAR_SAVE_LOCATION) m.save() success_msgs += ('profile pictured updated',) <snip> def upload_photo(data,saveLocation): savePath = os.path.join(settings.MEDIA_ROOT, saveLocation, data.name) destination = open(savePath, 'wb+') for chunk in data.chunks(): destination.write(chunk) destination.close() return os.path.join(saveLocation, data.name) Here's where it gets whacky and I was hoping someone could shed a light on this error, because either a) it's the wrong error code, or b) something is happening with the file before it's completely handled. To recap, the file was actually uploaded to the server in the intended directory - and yet this err msg continues to persist. IOError at /user/settings [Errno 2] No such file or directory: u'/home/user66666/domains/example.com/html/media/images/avatars/DSC03852.JPG' Environment: Request Method: POST Request URL: http://111.111.111.111:2011/user/settings Django Version: 1.0.2 final Python Version: 2.4.4 Installed Applications: ['django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.sites', 'ctrlme', 'usertools', 'easy_thumbnails'] Installed Middleware: ('django.middleware.common.CommonMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware') Traceback: File "/home/user6666/containers/django/leonidas/usertools/views.py" in view_settings m.avatar = upload_photo(request.FILES['avatar'], settings.AVATAR_SAVE_LOCATION) File "/home/user666666/containers/django/leonidas/usertools/functions.py" in upload_photo destination = open(savePath, 'wb+')

    Read the article

  • NetworkStream.Read delay .Net

    - by Gilbes
    I have a class that inherits from TcpClient. In that class I have a method to process responses. In that method I call I get the NetworkStream with MyBase.GetStream and call Read on it. This works fine, excpet the first call to read blocks too long. And by too long I mean that the socket has recieved plenty of data, but won't read it until some arbitrary limit is reached. I can see that it has recieved plenty of data using the packet sniffer WireShark. I have set the recieve buffer to small amounts, and very small amounts (like just a few bytes) to no avail. I have done the same with the buffer byte array I pass to the read method, and it still delays. Or to put it another way. I am download 600k. The download takes 5 seconds (at a little over 100k/second connection to the server which makes sense). The initial Read call takes 2-3 seconds and tells me only 256 bytes are availble (256 is the Recieve buffer and the size of the array I read in to). Then magically, the other few hundred thousand bytes can be read in 256 byte chunks in only a few process ticks each. Using a packet sniffer, I know that during those initial 2-3 seconds, the socket has recieved much more than just 256 bytes. My connection wasn't .25k/second for 3 seconds and then 400k for 2 seconds. How do I get the bytes from a socket as they come in?

    Read the article

  • Multiple threads modifying a collection in Java??

    - by posdef
    Hi, The project I am working on requires a whole bunch of queries towards a database. In principle there are two types of queries I am using: read from excel file, check for a couple of parameters and do a query for hits in the database. These hits are then registered as a series of custom classes. Any hit may (and most likely will) occur more than once so this part of the code checks and updates the occurrence in a custom list implementation that extends ArrayList. for each hit found, do a detail query and parse the output, so that the classes created in (I) get detailed info. I figured I would use multiple threads to optimize time-wise. However I can't really come up with a good way to solve the problem that occurs with the collection these items are stored in. To elaborate a little bit; throughout the execution objects are supposed to be modified by both (I) and (II). I deliberately didn't c/p any code, as it would be big chunks of code to make any sense.. I hope it make some sense with the description above. Thanks,

    Read the article

  • Can I specify the files to commit in subversion in a file rather than on the command line?

    - by René Nyffenegger
    I have renamed (with svn move) a lot of files in a subversion project. Now, I am trying to commit these on Window's cmd.exe. It seems that I hit a limit (probably by cmd.exe) in that the number of files is too long for the command line to swallow. Now, I thought and hoped that I could list the files to commit in a seperate file that I could specify with the commit command (something like svn ci --files-to-commit=renamed-files.txt -m "Renamed a lot of files" Yet, either such an option does not exist or I am unable to find this. Unfortunately, I cannot do a svn ci . as I have done other changes in the project as well. Neither can I do a svn ci *pattern-of-renamed-files* since this would only check in the added files, not the deleted ones. Before I start checking in the files with smaller chunks of files to check in (and thus increase the revision number uneccesserily without giving a hint as to the 'atomicity' of the operation) I thought I ask if this is indeed impossible to do.

    Read the article

  • Reading text files line by line, with exact offset/position reporting

    - by Benjamin Podszun
    Hi. My simple requirement: Reading a huge ( a million) line test file (For this example assume it's a CSV of some sorts) and keeping a reference to the beginning of that line for faster lookup in the future (read a line, starting at X). I tried the naive and easy way first, using a StreamWriter and accessing the underlying BaseStream.Position. Unfortunately that doesn't work as I intended: Given a file containing the following Foo Bar Baz Bla Fasel and this very simple code using (var sr = new StreamReader(@"C:\Temp\LineTest.txt")) { string line; long pos = sr.BaseStream.Position; while ((line = sr.ReadLine()) != null) { Console.Write("{0:d3} ", pos); Console.WriteLine(line); pos = sr.BaseStream.Position; } } the output is: 000 Foo 025 Bar 025 Baz 025 Bla 025 Fasel I can imagine that the stream is trying to be helpful/efficient and probably reads in (big) chunks whenever new data is necessary. For me this is bad.. The question, finally: Any way to get the (byte, char) offset while reading a file line by line without using a basic Stream and messing with \r \n \r\n and string encoding etc. manually? Not a big deal, really, I just don't like to build things that might exist already..

    Read the article

  • Sending jpegs by tcp socket...sometimes incomplete.

    - by Guy
    Vb.net Hi I've been working on a project for months now (vb 2008 express). There is one final problem which I can't solve. I need to send images to a client from a 'server'(listener). The code below works most of the time but sometimes the image is incomplete. I believe this might be something to do with the tcp packet sizes varying, maybe limited by how busy it is out there on the net. I have seen examples of code that splits the image into chunks and sends them out, but I can't get them to work maybe because I'm using a different vb version. The pictures to be sent are small 20k max. Any working code examples would be wonderful. I have been experimenting and failing with this final hurdle for weeks. Thanks in anticipation. Client----- Sub GetPic() '------- Connect to Server ClientSocket = New Socket(AddressFamily.InterNetwork, SocketType.Stream, _ ProtocolType.Tcp) ClientSocket.Connect(Epoint) '------- Send Picture Request Dim Bytes() As Byte = System.Text.ASCIIEncoding.ASCII.GetBytes("Send Picture") ClientSocket.Send(Bytes, Bytes.Length, SocketFlags.None) '------- Receive Response Dim RecvBuffer(20000) As Byte Dim Numbytes As Integer Numbytes = ClientSocket.Receive(RecvBuffer) Dim Darray(Numbytes) As Byte Buffer.BlockCopy(RecvBuffer, 0, Darray, 0, Numbytes) '------- Close Connection ClientSocket.Shutdown(SocketShutdown.Both) ClientSocket.Close() '------- Dim MStrm = New MemoryStream(Darray) Picture = Image.FromStream(MStrm) End Sub Listener----- 'Threaded from a listener Sub ClientThread(ByVal Client As TcpClient) Dim MStrm As New MemoryStream Dim Rbuffer(1024) As Byte Dim Tbyte As Byte() Dim NStrm As NetworkStream = Client.GetStream() Dim I As Integer = NStrm.Read(Rbuffer, 0, Rbuffer.Length) Dim Incoming As String = System.Text.Encoding.ASCII.GetString(Rbuffer, 0, I) If Incoming = "Send Picture" then Picture Save(MStrm, Picture.RawFormat) Tbyte = MStrm.ToArray NStrm.Write(Tbyte, 0, Tbyte.Length) End if Client.Close() End Sub

    Read the article

  • Oracle command hangs when using view for "WHILE x IN..." subquery

    - by Calvin Fisher
    I'm working on a web service that fetches data from an oracle data source in chunks and passes it back to an indexing/search tool in XML format. I'm the C#/.NET guy, and am kind of fuzzy on parts of Oracle. Our Oracle team gave us the following script to run, and it works well: SELECT ROWID, [columns] FROM [table] WHERE ROWID IN ( SELECT ROWID FROM ( SELECT ROWID FROM [table] WHERE ROWID > '[previous_batch_last_rowid]' ORDER BY ROWID ) WHERE ROWNUM <= 10000 ) ORDER BY ROWID 10,000 rows is an arbitrary but reasonable chunk size and ROWID is sufficiently unique for our purposes to use as a UID since each indexing run hits only one table at a time. Bracketed values are filled in programmatically by the web service. Now we're going to start adding views to the indexing, each of which will union a few separate tables. Since ROWID would no longer function as a unique identifier, they added a column to the views (VIEW_UNIQUE_ID) that concatenates the ROWIDs from the component tables to construct a UID for each union. But this script does not work, even though it follows the same form as the previous one: SELECT VIEW_UNIQUE_ID, [columns] FROM [view] WHERE VIEW_UNIQUE_ID IN ( SELECT VIEW_UNIQUE_ID FROM ( SELECT VIEW_UNIQUE_ID FROM [view] WHERE ROWID > '[previous_batch_last_view_unique_id]' ORDER BY VIEW_UNIQUE_ID ) WHERE ROWNUM <= 10000 ) ORDER BY VIEW_UNIQUE_ID It hangs indefinitely with no response from the Oracle server. I've waited 20+ minutes and the SQLTools dialog box indicating a running query remains the same, with no progress or updates. I've tested each subquery independently and each works fine and takes a very short amount of time (<= 1 second), so the view itself is sound. But as soon as the inner two SELECT queries are added with "WHERE VIEW_UNIQUE_ID IN...", it hangs. Why doesn't this query work for views? In what important way are they not interchangeable here?

    Read the article

  • ASP.net file operations delay

    - by mtranda
    Ok, so here's the problem: I'm reading the stream from a FileUpload control, reading in chunks of n bytes and writing the array in a loop until I reach the stream's end. Now the reason I do this is because I need to check several things while the upload is still going on (rather than doing a Save(); which does the whole thing in one go). Here's the problem: when doing this from the local machine, I can see the file just fine as it's uploading and its size increases (had to add a Sleep(); clause in the loop to actually get to see the file being written). However, when I upload the file from a remote machine, I don't get to see it until the the file has completed uploading. Also, I've added another call to write the progress to a text file as the progress is going on, and I get the same thing. Local: the file updates as the upload goes on, remote: the token file only appears after the upload's done (which is somewhat useless since I need it while the upload's still happening). Is there some sort of security setting in (or ASP.net) that maybe saves files in a temporary location for remote machines as opposed to the local machine and then moves them to the specified destination? I would liken this with ASP.net displaying error messages when browsing from the local machine (even on the public hostname) as opposed to the generic compilation error page/generic exception page that is shown when browsing from a remote machine (and customErrors are not off) Any clues on this? Thanks in advance.

    Read the article

  • Mysql Database Question about Large Columns

    - by murat
    Hi, I have a table that has 100.000 rows, and soon it will be doubled. The size of the database is currently 5 gb and most of them goes to one particular column, which is a text column for PDF files. We expect to have 20-30 GB or maybe 50 gb database after couple of month and this system will be used frequently. I have couple of questions regarding with this setup 1-) We are using innodb on every table, including users table etc. Is it better to use myisam on this table, where we store text version of the PDF files? (from memory usage /performance perspective) 2-) We use Sphinx for searching, however the data must be retrieved for highlighting. Highlighting is done via sphinx API but still we need to retrieve 10 rows in order to send it to Sphinx again. This 10 rows may allocate 50 mb memory, which is quite large. So I am planning to split these PDF files into chunks of 5 pages in the database, so these 100.000 rows will be around 3-4 million rows and couple of month later, instead of having 300.000-350.000 rows, we'll have 10 million rows to store text version of these PDF files. However, we will retrieve less pages, so again instead of retrieving 400 pages to send Sphinx for highlighting, we can retrieve 5 pages and it will have a big impact on the performance. Currently, when we search a term and retrieve PDF files that have more than 100 pages, the execution time is 0.3-0.35 seconds, however if we retrieve PDF files that have less than 5 pages, the execution time reduces to 0.06 seconds, and it also uses less memory. Do you think, this is a good trade-off? We will have million of rows instead of having 100k-200k rows but it will save memory and improve the performance. Is it a good approach to solve this problem and do you have any ideas how to overcome this problem? The text version of the data is used only for indexing and highlighting. So, we are very flexible. Thanks,

    Read the article

  • parse content away from structure in a binary file

    - by Jeff Godfrey
    Using C#, I need to read a packed binary file created using FORTRAN. The file is stored in an "Unformatted Sequential" format as described here (about half-way down the page in the "Unformatted Sequential Files" section): http://www.tacc.utexas.edu/services/userguides/intel8/fc/f_ug1/pggfmsp.htm As you can see from the URL, the file is organized into "chunks" of 130 bytes or less and includes 2 length bytes (inserted by the FORTRAN compiler) surrounding each chunk. So, I need to find an efficient way to parse the actual file payload away from the compiler-inserted formatting. Once I've extracted the actual payload from the file, I'll then need to parse it up into its varying data types. That'll be the next exercise. My first thoughts are to slurp up the entire file into a byte array using File.ReadAllBytes. Then, just iterate through the bytes, skipping the formatting and transferring the actual data to a second byte array. In the end, that second byte array should contain the actual file contents minus all the formatting, which I'd then need to go back through to get what I need. As I'm fairly new to C#, I thought there might be a better, more accepted way of tackling this. Also, in case it's helpful, these files could be fairly large (say 30MB), though most will be much smaller...

    Read the article

  • How to Use XSLT to Replace Coordinate Separator With List of Tuples?

    - by kuloch
    I have a space-separated list of coordinate tuples. Each tuple consists of a space-separated list of 2-dimensional coordinates. E.g. "1.1 2.8 1.2 2.9" represents a line from POINT(1.1 2.8) to POINT(1.2 2.9). I need this to instead be "1.1,2.8 1.2,2.9". How would I use XSLT to perform the replacement of space-to-comma between pairs of numbers? I have the "string(gml:LinearRing/gml:posList)". This is being used on a Java Web Service that spits out GML 3.1.1 features with geometries. The service supports optional KML output, by using XSLT to transform the GML document into a KML document (at least, the chunks deemed "important"). I am locked into XSLT 1.0, so regex from XSLT 2.0 is not an option. I am aware that GML uses lat/lon while KML uses lon/lat. That's being handled before XSLT, though it would be nice to have that also done with XSLT.

    Read the article

  • .net IHTTPHandler Streaming SQL Binary Data

    - by Yisman
    Hello everybody I am trying to implement an ihttphandeler for streaming files. files may be tiny thumbnails or gigantic movies the binaries r stored in sql server i looked at a lot of code online but something does not make sense isnt streaming supposed to read the data piece by piece and move it over the line? most of the code seems to first read the whole field from mssql to memory and then use streaming for the output writing wouldnt it b more eficient to actually stream from disk directly to http byte by byte (or buffered chunks?) heres my code so far but cant figure out the correct combination of the sqlreader mode and the stream object and the writing system Public Sub ProcessRequest(ByVal context As HttpContext) Implements IHttpHandler.ProcessRequest context.Response.BufferOutput = False Dim FileField=safeparam(context.Request.QueryString("FileField")) Dim FileTable=safeparam(context.Request.QueryString("FileTable")) Dim KeyField=safeparam(context.Request.QueryString("KeyField")) Dim FileKey=safeparam(context.Request.QueryString("FileKey")) Using connection As New SqlConnection(ConfigurationManager.ConnectionStrings("Main").ConnectionString) Using command As New SqlCommand("SELECT " & FileField & "Bytes," & FileField & "Type FROM " & FileTable & " WHERE " & KeyField & "=" & FileKey, connection) command.CommandType = Data.CommandType.Text enbd using end using end sub please be aware that this sql command also returns the file extension (pdf,jpg,doc...) in the second field of the query thank you all very much

    Read the article

  • How to parse text as JavaScript?

    - by Danjah
    This question of mine (currently unanswered), drove me toward finding a better solution to what I'm attempting. My requirements: chunks of code which can be arbitrarily added into a document, without an identifier: [div class="thing"] [elements... /] [/div] the objects are scanned for and found by an external script: var things = yd.getElementsBy(function(el){ return yd.hasClass('thing'); },null,document ); the objects must be individually configurable, what I have currently is identifier-based: [div class="thing" id="thing0"] [elements... /] [script type="text/javascript"] new Thing().init({ id:'thing0'; }); [/script] [/div] So I need to ditch the identifier (id="thing0") so there are no duplicates when more than one chunk of the same code is added to a page I still need to be able to config these objects individually, without an identifier SO! All of that said, I wondered about creating a dynamic global variable within the script block of each added chunk of code, within its script tag. As each 'thing' is found, I figure it would be legit to grab the innerHTML of the script tag and somehow convert that text into a useable JS object. Discuss. Ok, don't discuss if you like, but if you get the drift then feel free to correct my wayward thinking or provide a better solution - please! d

    Read the article

  • How much detail should be in a project plan or spec?

    - by DeanMc
    I have an issue that I feel many programmers can relate to... I have worked on many small scale projects. After my initial paper brain storm I tend to start coding. What I come up with is usually a rough working model of the actual application. I design in a disconnected fashion so I am talking about underlying code libraries, user interfaces are the last thing as the library usually dictates what is needed in the UI. As my projects get bigger I worry that so should my "spec" or design document. The above paragraph, from my investigations, is echoed all across the internet in one fashion or another. When a UI is concerned there is a bit more information but it is UI specific and does not relate to code libraries. What I am beginning to realise is that maybe code is code is code. It seems from my extensive research that there is no 1:1 mapping between a design document and the code. When I need to research a topic I dump information into OneNote and from there I prioritise features into versions and then into related chunks so that development runs in a fairly linear fashion, my tasks tend to look like so: Implement Binary File Reader Implement Binary File Writer Create Object to encapsulate Data for expression to the caller Now any programmer worth his salt is aware that between those three to do items could be a potential wall of code that could expand out to multiple files. I have tried to map the complete code process for each task but I simply don't think it can be done effectively. By the time one mangles pseudo code it is essentially code anyway so the time investment is negated. So my question is this: Am I right in assuming that the best documentation is the code itself. We are all in agreement that a high level overview is needed. How high should this be? Do you design to statement, class or concept level? What works for you?

    Read the article

  • XSL Reuse? YES! But: Element must not contain an xsl:import element! :-(

    - by Fedor Steeman
    I am using a heavy stylesheet with a lot of recurring transformations, so I thought it would be smart to reuse the same chunks of code, so I would not need to make the same changes at a bunch of different places. So I discovered , but -alas- it won't allow me to do it. When trying to run it in Sonic Workbench I get the following error: An xsl:for-each element must not contain an xsl:import element This is my stylesheet code: <xsl:template match="/"> <InboundFargoMessage> <EdiSender> <xsl:value-of select="TransportInformationMessage/SenderId"/> </EdiSender> <EdiReceiver> <xsl:value-of select="TransportInformationMessage/RecipientId"/> </EdiReceiver> <EdiSource>PORLOGIS</EdiSource> <EdiDestination>FARGO</EdiDestination> <Transportations> <xsl:for-each select="TransportInformationMessage/TransportUnits/TransportUnit"> <xsl:import href="TransportCDMtoFDM_V0.6.xsl"/> </xsl:for-each> <xsl:for-each select="TransportInformationMessage/Waybill/TransportUnits/TransportUnit"> <xsl:import href="TransportCDMtoFDM_V0.6.xsl"/> </xsl:for-each> </Transportations> </InboundFargoMessage> </xsl:template> </xsl:stylesheet> I will leave out the child xsl-sheets for now, as the problem appears to be happening at the base. If I cannot use xsl:import, is there any option of reuse?

    Read the article

  • Programming style question on how to code functions

    - by shawnjan
    Hey all! So, I was just coding a bit today, and I realized that I don't have much consistency when it comes to a coding style when programming functions. One of my main concerns is whether or not its proper to code it so that you check that the input of the user is valid OUTSIDE of the function, or just throw the values passed by the user into the function and check if the values are valid in there. Let me sketch an example: I have a function that lists hosts based on an environment, and I want to be able to split the environment into chunks of hosts. So an example of the usage is this: listhosts -e testenv -s 2 1 This will get all the hosts from the "testenv", split it up into two parts, and it is displaying part one. In my code, I have a function that you pass it in a list, and it returns a list of lists based on you parameters for splitting. BUT, before I pass it a list, I first verify the parameters in my MAIN during the getops process, so in the main I check to make sure there are no negatives passed by the user, I make sure the user didnt request to split into say, 4 parts, but asking to display part 5 (which would not be valid), etc. tl;dr: Would you check the validity of a users input the flow of you're MAIN class, or would you do a check in your function itself, and either return a valid response in the case of valid input, or return NULL in the case of invalid input? Obviously both methods work, I'm just interested to hear from experts as to which approach is better :) Thanks for any comments and suggestions you guys have!

    Read the article

  • Catching specific vs. generic exceptions in c#

    - by Scott Vercuski
    This question comes from a code analysis run against an object I've created. The analysis says that I should catch a more specific exception type than just the basic Exception. Do you find yourself using just catching the generic Exception or attempting to catch a specific Exception and defaulting to a generic Exception using multiple catch blocks? One of the code chunks in question is below: internal static bool ClearFlags(string connectionString, Guid ID) { bool returnValue = false; SqlConnection dbEngine = new SqlConnection(connectionString); SqlCommand dbCmd = new SqlCommand("ClearFlags", dbEngine); SqlDataAdapter dataAdapter = new SqlDataAdapter(dbCmd); dbCmd.CommandType = CommandType.StoredProcedure; try { dbCmd.Parameters.AddWithValue("@ID", ID.ToString()); dbEngine.Open(); dbCmd.ExecuteNonQuery(); dbEngine.Close(); returnValue = true; } catch (Exception ex) { ErrorHandler(ex); } return returnValue; } Thank you for your advice EDIT: Here is the warning from the code analysis Warning 351 CA1031 : Microsoft.Design : Modify 'ClearFlags(string, Guid)' to catch a more specific exception than 'Exception' or rethrow the exception

    Read the article

  • Memorystream and Large Object Heap

    - by Flo
    I have to transfer large files between computers on via unreliable connections using WCF. Because I want to be able to resume the file and I don't want to be limited in my filesize by WCF, I am chunking the files into 1MB pieces. These "chunk" are transported as stream. Which works quite nice, so far. My steps are: open filestream read chunk from file into byet[] and create memorystream transfer chunk back to 2. until the whole file is sent My problem is in step 2. I assume that when I create a memory stream from a byte array, it will end up on the LOH and ultimately cause an outofmemory exception. I could not actually create this error, maybe I am wrong in my assumption. Now, I don't want to send the byte[] in the message, as WCF will tell me the array size is too big. I can change the max allowed array size and/or the size of my chunk, but I hope there is another solution. My actual question(s): Will my current solution create objects on the LOH and will that cause me problem? Is there a better way to solve this? Btw.: On the receiving side I simple read smaller chunks from the arriving stream and write them directly into the file, so no large byte arrays involved.

    Read the article

  • Large scale perspective lights casting shadow maps, in the most optimized way?

    - by meds
    I'm using projected texture shadows coupled with lights to light a large sports field at night. To do this I'm using shadow cameras which I place in the position of the stadiums lights and shine it down on the field at the appropriate angle. The problem with this method is the textures to which I render the shadows into have to be very large so they can keep sufficient detail over the entire stadium. This is incredibly under optimized since at any given point the players attention is only directed on a small portion of the field meaning large chunks of the texture just take up space wit no benefits. However the issue is the lights need to be perspective based as they come from actual directional lights hovering over the stadium. The way to solve this, I believe, is to figure out in the shadow cameras view matrix it would be to place the actual camera to render from, and adjust the view matrix accordingly to the position it is. So my question is, how can I calculate the optimal position to put the shadow camera and calculate its view matrix such that the shadows it projects will appear to be coming from the light source rather than the camera?

    Read the article

< Previous Page | 15 16 17 18 19 20 21 22 23 24 25  | Next Page >