Search Results

Search found 58779 results on 2352 pages for 'cleaned data'.

Page 151/2352 | < Previous Page | 147 148 149 150 151 152 153 154 155 156 157 158  | Next Page >

  • Memory efficient import many data files into panda DataFrame in Python

    - by richardh
    I import into a panda DataFrame a directory of |-delimited.dat files. The following code works, but I eventually run out of RAM with a MemoryError:. import pandas as pd import glob temp = [] dataDir = 'C:/users/richard/research/data/edgar/masterfiles' for dataFile in glob.glob(dataDir + '/master_*.dat'): print dataFile temp.append(pd.read_table(dataFile, delimiter='|', header=0)) masterAll = pd.concat(temp) Is there a more memory efficient approach? Or should I go whole hog to a database? (I will move to a database eventually, but I am baby stepping my move to pandas.) Thanks! FWIW, here is the head of an example .dat file: cik|cname|ftype|date|fileloc 1000032|BINCH JAMES G|4|2011-03-08|edgar/data/1000032/0001181431-11-016512.txt 1000045|NICHOLAS FINANCIAL INC|10-Q|2011-02-11|edgar/data/1000045/0001193125-11-031933.txt 1000045|NICHOLAS FINANCIAL INC|8-K|2011-01-11|edgar/data/1000045/0001193125-11-005531.txt 1000045|NICHOLAS FINANCIAL INC|8-K|2011-01-27|edgar/data/1000045/0001193125-11-015631.txt 1000045|NICHOLAS FINANCIAL INC|SC 13G/A|2011-02-14|edgar/data/1000045/0000929638-11-00151.txt

    Read the article

  • Providing dynamic data to webpage

    - by Marius
    Hi, I have a web page that displays dynamic data which changes every 2 seconds. Data is selected from various data sources including Oracle. Currently, the page reloads every 10 seconds and runs a PHP script which retrieves the data and displays the page. I have other pages that gives a different view on the same data. This means the same query is run again for them as well. If I have 4 of these pages with 10 concurrent users each, suddenly the data retrieval happens 40 times every 10 seconds, obviously not ideal. I have some ideas on how to improve this situation, but I thought I would ask from some ideas from other experts that might have come across a similar situation. I'm not bound to PHP, and my server is on a Linux platform. Regards Marius

    Read the article

  • Providing dynamic data to webpage

    - by Marius
    Hi, I have a web page that displays dynamic data which changes every 2 seconds. Data is selected from various data sources including Oracle. Currently, the page reloads every 10 seconds and runs a PHP script which retrieves the data and displays the page. I have other pages that gives a different view on the same data. This means the same query is run again for them as well. If I have 4 of these pages with 10 concurrent users each, suddenly the data retrieval happens 40 times every 10 seconds, obviously not ideal. I have some ideas on how to improve this situation, but I thought I would ask from some ideas from other experts that might have come across a similar situation. I'm not bound to PHP, and my server is on a Linux platform. Regards Marius

    Read the article

  • How to use json object notation to retrieve dbpedia json data

    - by Margi
    In my php code, I am retrieving json data as below. <?php $url = "http://dbpedia.org/data/Los_Angeles.json"; $data = file_get_contents($url); echo $data; ?> The javascript code consumes this json data returned from php and gets the json object as below. var doc = eval('(' + request.responseText + ')'); How to retrieve the following json data using dot notations. The keys contain URLs. "http://dbpedia.org/ontology/populationTotal" : [ { "type" : "literal", "value" : 3792621 , "datatype" : "http://www.w3.org/2001/XMLSchema#integer" } ] , "http://dbpedia.org/ontology/PopulatedPlace/areaTotal" : [ { "type" : "literal", "value" : "1301.9688931491348" , "datatype" : "http://dbpedia.org/datatype/squareKilometre" }

    Read the article

  • Write data to an xml file.

    - by Bobby
    My aim is to write an XML file with few tags whose values are the regional language. I'm using Python to do this and using IDLE(Pythong GUI) for programming. While I try to write the words in an xmls file it gives the following error. UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-4: ordinal not in range(128) For now, I'm not using any xml writer library instead I'm opening a file "test.xml" and writing the data into it. This error is encountered by the line: f.write("\t\" + data + "\"). If I replace the above write statement with print statement then it prints the data properly on Python Shell. I'm reading the data from an excel file which is not in the UTF8,16, or 32 encoding formats. Its in some other format. cp1252 is reading the data properly. Any help in getting this data writtent oan xml would be highly appreciated. Thanks in advance. Bob

    Read the article

  • ItemFileWriteStore: how to change the data?

    - by jeff porter
    Hi, I'd like to change the data in my dojo.data.ItemFileWriteStore. Currently I have... var rawdataCacheItems = [{'cachereq':'14:52:03','name':'Test1','cacheID':'3','ver':'7'}]; var cacheInfo = new dojo.data.ItemFileWriteStore({ data: { identifier: 'cacheID', label: 'cacheID', items: rawdataCacheItems } }); I'd like to make a XHR request to get a new JSON string of the data to render. What I can't work out is how to change the data in the "items" held by ItemFileWriteStore. Can anyone point me in the correct direction? Thanks Jeff Porter

    Read the article

  • selectors-api for data attributes

    - by MJ
    In HTML5, CSS selectors seem to operate well with data-* attributes. For example: <style> div[data-foo='bar'] { background:#eee; } </style> <div data-foo='bar'>colored</div> <div>not colored</div> will properly style the first . But, attempts to select such elements using the selectors-api fail. Examples: var foos = document.querySelectorAll("div[data-foo]='bar'"); or var foos = document.querySelectorAll("div data-foo='bar'"); in Chrome and Safari, this produces a cryptic error: SYNTAX_ERR: DOM Exception 12 Any thoughts on how to use the selectors-api to properly select elements on the basis of data-* attributes?

    Read the article

  • DB2 load partitioned data in parallel

    - by Erik Paulson
    I have a 10-node DB2 9.5 database, with raw data on each machine (ie node1:/scratch/data/dataset.1 node2:/scratch/data/dataset.2 ... node10:/scratch/data/dataset.10 There is no shared NFS mount - none of my machines could handle all of the datasets combined. each line of a dataset file is a long string of text, column delimited. The first column is the key. I don't know the hash function that DB2 will use, so dataset is not pre-partitioned. Short of renaming all of my files, is there any way to get DB2 to load them all in parallel? I'm trying to do something like load from '/scratch/data/dataset' of del modified by coldel| fastparse messages /dev/null replace into TESTDB.data_table part_file_location '/scratch/data'; but I have no idea how to suggest to db2 that it should look for dataset.1 on the first node, etc.

    Read the article

  • data directory in automake

    - by Alex Farber
    I have some data files that should be distributed with my program. Using dist_pkgdata_DATA in Makefile.am, I get these files installed to /usr/local/data/share/package-name. The problem is that data is read-only, and my program needs to modify it. Playing with dist_sharedstate_DATA, dist_localstate_DATA, dist-data_DATA varibles, I got different installation directories, like /usr/local/com, usr/local/var, but data is always read-only. How can I distribute modifiable data files with my package? I need some common directory for all users, or maybe local data in a user directory.

    Read the article

  • Access MP3 audio data independently of ID3 tags?

    - by kyl191
    Hi, this is a 2 part question. First off, is it possible to access the audio data in an MP3 independently of the ID3 tags, and secondly, is there any way to do so using available libraries? I recently consolidated my music collection from 3 computers and ended up with songs which had changed ID3 tags, but the audio data itself was unmodified. Running a search for duplicate files failed because the file changed with the ID3 tag change, but I think it should be possible to identify duplicate files if I just run a deduplication using the audio data for comparison. I know that it's possible to seek to a particular position past the ID3 header in the file, and directly read the data, but was wondering if there's a library that would expose the audio data so I could just extract the data, run a checksum on it, and store the computed result somewhere, then look for identical checksums. (Also, I'd probably have to use some kind of library when you take into account variable length headers.)

    Read the article

  • DataSetProvider - DataSet to ClientDataSet

    - by nomad311
    I am trying to take data from a TMSQuery which is connected to my DB and populate a ClientDataSet with some of that data using a DataSetProvider. My problem is that I will need to modify some of this data before it can go into my ClientDataSet. The ClientDataSet has persistent fields that will not match up with the raw DB data. I can't even get a string from the DB into a memo field in ClientDataSet. The ClientDataSet is a part of my data tier so I will need to conform the data from the DB to the ClientDataSet field by field (well most will be able to go right through, but many will require routing and/or conversion). Does anyone have experience with this?

    Read the article

  • Unrequired property keeps getting data-val-required attribute

    - by frennky
    This is the model with it's validation: [MetadataType(typeof(TagValidation))] public partial class Tag { } public class TagValidation { [Editable(false)] [HiddenInput(DisplayValue = false)] public int TagId { get; set; } [Required] [StringLength(20)] [DataType(DataType.Text)] public string Name { get; set; } //... } Here is the view: <h2>Create</h2> <script src="@Url.Content("~/Scripts/jquery.validate.min.js")" type="text/javascript"></script> <script src="@Url.Content("~/Scripts/jquery.validate.unobtrusive.min.js")" type="text/javascript"></script> @using (Html.BeginForm()) { @Html.ValidationSummary(true) <fieldset> <legend>Tag</legend> <div>@Html.EditorForModel()</div> <p> <input type="submit" value="Create" /> </p> </fieldset> } <div> @Html.ActionLink("Back to List", "Index") </div> And here is what get's renderd: <form action="/Tag/Create" method="post"> <fieldset> <legend>Tag</legend> <div><input data-val="true" data-val-number="The field TagId must be a number." data-val-required="The TagId field is required." id="TagId" name="TagId" type="hidden" value="" /> <div class="editor-label"><label for="Name">Name</label></div> <div class="editor-field"><input class="text-box single-line" data-val="true" data-val-length="The field Name must be a string with a maximum length of 20." data-val-length-max="20" data-val-required="The Name field is required." id="Name" name="Name" type="text" value="" /> <span class="field-validation-valid" data-valmsg-for="Name" data-valmsg-replace="true"></span></div> ... </fieldset> </form> The problem is that TagId validation gets generated althoug thare is no Required attribute set on TagId property. Because of that I can't even pass the client-side validation in order to create new Tag in db. What am I missing?

    Read the article

  • NoSQL and meteorological data

    - by christian studer
    So there's this new cool thing, these NoSQL-databases. And so there's my data: Rows of rows of rows of meteorological data: Values, representing certain measurements at a certain station (Identified by a WMO number, not coordinates), at a certain time. Not every station measures every parameter, not every parameter is measured all the time. I store this data (30 years worth of hourly values, resulting in ~1 billion values) currently in MySQL. The continous growth and the forseeable addition of even more data give me a little headache. Reading about the document based NoSQL systems which seem to scale rather easily, I was wondering if NoSQL is a viable data storage concept for meteorological data too. Do you have any experience with this?

    Read the article

  • Dynamically add new HTML elements on page load after getting JSON data from Web API

    - by Luis D Urraca
    I'm building an hybrid Android App using Phonegap/Apache Cordova. My app should get data from my web api. I'm using JSON to serve the data to the APP. So i got a the following code: function init() { document.addEventListener('deviceready', onDeviceReady, false); var url = "http://23.21.128.153:3000/regions.json";var jsonresults; $.getJSON(url,function(data){ jsonresults = data; $.each(jsonresults, function(i,v){ $('#main-content').append('<li>'+jsonresults[i].name+'</li>'); }); }); } and also on the body of the html i have a div called main-content. Everythings works fine in the Eclipse browser, but on the Android Emulator is not working. Not sure if there's another way to pull data from Web API using JSON and dinamically create HMTL elements after getting the data. https://gist.github.com/2956660

    Read the article

  • data in mysql show after barcode split and matches character

    - by klox
    i need some code for the next step..this my first step: <script> $("#mod").change(function() { var barcode; barCode=$("#mod").val(); var data=barCode.split(" "); $("#mod").val(data[0]); $("#seri").val(data[1]); var str=data[0]; var matches=str.matches(/EE|[EJU]).*(D)/i); }); </script> after matches..i want the result can connect to data base then show data from table inside <div id="value">...how to do that?

    Read the article

  • Table data display in HTML

    - by Hulk
    In the following table how to adjust the height and width automatically for the table. i.e, if the data is more it should display all the data else present data ina fashionable manner <table="mytable"> <tr><td>more data.............................................................................</td> <td>a</td> </tr> In the nothing should be wrapped wither in case of large chuks of data or small chunks of data. Thanks....

    Read the article

  • post data to a thickbox using ajax

    - by sqlchild
    I need to post data to a thickbox using ajax and open it immediately and display the posted data. The user would click on a link/button and the data i.e. value of the selected checkboxes would be posted to "my_thickbox.php" and the thickbox (url : my_thickbox.php) would open with checkbox values displayed. <div id="showthickbox" ><a href="my_thickbox.php" class="thickbox"></div> $('#showthickbox').click(function() { var data = $('input:checkbox:checked').map(function() { return this.value; }).get(); $.ajax({ type: 'POST', url: 'my_thickbox.php', data: data, success: success, dataType: dataType }); });

    Read the article

  • Why android app does not recognize GPRS, WiFi upon changing data connection

    - by Maxood
    If i turn off WiFi, i’m not able to open the app ( this is with no data connection, even without data connection it should display cached items ) If i tun off WiFi and turn on GPRS the app cannot recognize this and gives the error page no Wifi/GPRS connection. if i restart the phone with GPRS on the app works fine till i change data connection type. if i restart the phone with WiFi on the app works fine till i change the data connection type. whenever i change the data connection type i have to restart the phone so the app can recognize the connection type. Why my android app does not recognize WiFi, GPRS upon changing data connection in real time?

    Read the article

  • Data sources and NSTableView

    - by lampShade
    I know that table sources need a data source to hold the data that the tableview will display. Lets' say that I'm going to make my AppController be the data source of my tableview and that I make the connection in interface builder. My question is since my actual data is going to be stored in an array,let's call it myArray, when I set the data source in code should I do this [tableView setDataSource:myArray]; or this [tableView setDataSource:self]; I'm confused about this. setting the data source with the keyword "self" would set it to the AppController if I'm not mistaken.

    Read the article

  • How to implement a lazy loaded Silverlight data grid without using paging

    - by Marc Wittke
    Using the Business Application template from the brand new released RIA Services, you can see lots of examples using the data grid on top of a DomainDataSource in combination with a DataPager. The properties PageSize and LoadSize can be used to adjust the amount of data to be displayed in one page and the data that is prefetched in the background. Now I'd like to have a data grid with a scrollbar and no pager. The underlying DomainDataSource should load only the data that is diplayed in the grid. It should trigger another load, when the user scrolls down to items that are not yet in the data context. Is there any sample implementation how to do this?

    Read the article

  • Intermittent "Specified cast is invalid" with StructureMap injected data context

    - by FreshCode
    I am intermittently getting an System.InvalidCastException: Specified cast is not valid. error in my repository layer when performing an abstracted SELECT query mapped with LINQ. The error can't be caused by a mismatched database schema since it works intermittently and it's on my local dev machine. Could it be because StructureMap is caching the data context between page requests? If so, how do I tell StructureMap v2.6.1 to inject a new data context argument into my repository for each request? Update: I found this question which correlates my hunch that something was being re-used. Looks like I need to call Dispose on my injected data context. Not sure how I'm going to do this to all my repositories without copypasting a lot of code. Edit: These errors are popping up all over the place whenever I refresh my local machine too quickly. Doesn't look like it's happening on my remote deployment box, but I can't be sure. I changed all my repositories' StructureMap life cycles to HttpContextScoped() and the error persists. Code: public ActionResult Index() { // error happens here, which queries my page repository var page = _branchService.GetPage("welcome"); if (page != null) ViewData["Welcome"] = page.Body; ... } Repository: GetPage boils down to a filtered query mapping in my page repository. public IQueryable<Page> GetPages() { var pages = from p in _db.Pages let categories = GetPageCategories(p.PageId) let revisions = GetRevisions(p.PageId) select new Page { ID = p.PageId, UserID = p.UserId, Slug = p.Slug, Title = p.Title, Description = p.Description, Body = p.Text, Date = p.Date, IsPublished = p.IsPublished, Categories = new LazyList<Category>(categories), Revisions = new LazyList<PageRevision>(revisions) }; return pages; } where _db is an injected data context as an argument, stored in a private variable which I reuse for SELECT queries. Error: Specified cast is not valid. Exception Details: System.InvalidCastException: Specified cast is not valid. Stack Trace: [InvalidCastException: Specified cast is not valid.] System.Data.Linq.SqlClient.SqlProvider.Execute(Expression query, QueryInfo queryInfo, IObjectReaderFactory factory, Object[] parentArgs, Object[] userArgs, ICompiledSubQuery[] subQueries, Object lastResult) +4539 System.Data.Linq.SqlClient.SqlProvider.ExecuteAll(Expression query, QueryInfo[] queryInfos, IObjectReaderFactory factory, Object[] userArguments, ICompiledSubQuery[] subQueries) +207 System.Data.Linq.SqlClient.SqlProvider.System.Data.Linq.Provider.IProvider.Execute(Expression query) +500 System.Data.Linq.DataQuery`1.System.Linq.IQueryProvider.Execute(Expression expression) +50 System.Linq.Queryable.FirstOrDefault(IQueryable`1 source) +383 Manager.Controllers.SiteController.Index() in C:\Projects\Manager\Manager\Controllers\SiteController.cs:68 lambda_method(Closure , ControllerBase , Object[] ) +79 System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary`2 parameters) +258 System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters) +39 System.Web.Mvc.<>c__DisplayClassd.<InvokeActionMethodWithFilters>b__a() +125 System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodFilter(IActionFilter filter, ActionExecutingContext preContext, Func`1 continuation) +640 System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodWithFilters(ControllerContext controllerContext, IList`1 filters, ActionDescriptor actionDescriptor, IDictionary`2 parameters) +312 System.Web.Mvc.ControllerActionInvoker.InvokeAction(ControllerContext controllerContext, String actionName) +709 System.Web.Mvc.Controller.ExecuteCore() +162 System.Web.Mvc.<>c__DisplayClass8.<BeginProcessRequest>b__4() +58 System.Web.Mvc.Async.<>c__DisplayClass1.<MakeVoidDelegate>b__0() +20 System.Web.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +453 System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +371

    Read the article

  • Google Visualization Annotated Time Line, removing data points.

    - by Vitaly Babiy
    I am trying to build a graph that will change resolution depending on how far you are zoomed in. Here is what it looks like when you are complete zoomed out. So this looks good so when I zoom in I get a higher resolution data and my graph looks like this: The problem is when I zoom out the higher resolution data does not get cleared out of the graph: The tables below the graphs are table display what is in the DataTable. This is what drawing code looks like. var g_graph = new google.visualization.AnnotatedTimeLine(document.getElementById('graph_div_json')); var table = new google.visualization.Table(document.getElementById('table_div_json')); function handleQueryResponse(response){ log("Drawing graph") var data = response.getDataTable() g_graph.draw(data, {allowRedraw:true, thickness:2, fill:50, scaleType:'maximized'}) table.draw(data, {allowRedraw:true}) } I am try to find a way for it to only displaying the data that is in the DataTable. I have tried removing the allowRedraw flag but then it breaks the zooming operation. Any help would be greatly appreciated. Thanks

    Read the article

  • System.Windows.Forms.DataGridView does not display data

    - by Water Cooler v2
    All I am doing is a simple: // Both the methods, order.GetAllOrderItems() and order.GetOrderedItemsWhereBrandIs("foo") // return an IEnumerable<T> so the assignment to the DataSource property of the DataGridView // should be fine. The problem is in re-assigning the data source property. public void DisplayItems() { // The data appears if I have just this line. dgvOrderedItems.DataSource = order.GetAllOrderItems(); dgvOrderedItems.DataSource = null; // This time the data grid does not take the new data source. Instead, because // of the null assignment in the previous statement, it displays no data at all. dgvOrderedItems.DataSource = order.GetOrderedItemsWhereBrandIs("Lenovo"); } My question is: is there a way to change the data source of a DataGridView control once it has been set? I am using C# 4.0 and Visual Studio 2010 for development.

    Read the article

  • jQuery $.data(): Possible misuse?

    - by Rosarch
    Perhaps I'm using $.data incorrectly. Assigning the data: var course_li = sprintf('<li class="draggable course">%s</li>', course["fields"]["name"]); $(course_li).data('pk', course['pk']); alert(course['pk']); // shows a correct value Moving the li to a different ul: function moveToTerm(item, term) { item.fadeOut(function() { item.appendTo(term).fadeIn(); }); } Trying to access the data later: $.each($(term).children(".course"), function(index, course) { var pk = $(course).data('pk'); // pk is undefined courses.push(pk); }); What am I doing wrong? I have confirmed that the course li on which I am setting the data is the same as the one on which I am looking for it. (Unless I'm messing that up by calling appendTo() on it?)

    Read the article

  • Is there any data missing?

    - by smwikipedia
    Hi experts, Here is some code quoted from Douglas.E.Comer's < Computer Networks and Internets 4th edition. This program will send back any data it received. ... while((len = recv(conn, buff, BUFFERSIZE,0)) >0) // receive data send(conn, buff, len, 0); // send it back ... I am wondering, what if some data arrived when the code is executing in send(..) function, will it miss that data? Because the recv() function is not being executed. If no data is missed, where is the data kept? And by whom? Thanks...

    Read the article

< Previous Page | 147 148 149 150 151 152 153 154 155 156 157 158  | Next Page >