Search Results

Search found 74791 results on 2992 pages for 'remote file inclusion'.

Page 72/2992 | < Previous Page | 68 69 70 71 72 73 74 75 76 77 78 79  | Next Page >

  • mmap() for large file I/O?

    - by Boatzart
    I'm creating a utility in C++ to be run on Linux which can convert videos to a proprietary format. The video frames are very large (up to 16 megapixels), and we need to be able to seek directly to exact frame numbers, so our file format uses libz to compress each frame individually, and append the compressed data onto a file. Once all frames are finished being written, a journal which includes meta data for each frame (including their file offsets and sizes) is written to the end of the file. I'm currently using ifstream and ofstream to do the file i/o, but I am looking to optimize as much as possible. I've heard that mmap() can increase performance in a lot of cases, and I'm wondering if mine is one of them. Our files will be in the tens to hundreds of gigabytes, and although writing will always be done sequentially, random access reads should be done in constant time. Any thoughts as to whether I should investigate this further, and if so does anyone have any tips for things to look out for? Thanks!

    Read the article

  • Specific 'boot file' definition

    - by Jazz
    Hey, I have been given a general explanation of how a computer boots up. However a very loose definition to the term 'boot file' was given. Could someone explain 'boot file' to me in a very simple but concise manner? I have read about the POST, the clearing of registers, BIOS in the CMOS, etc. What I understand is that the boot file is different to the boot program. the boot program gets the system ready to accept an OS while the boot file contains some of the parameters by which the system will operate. The boot program is stored on ROM and the boot file isnt? cheers, jazz

    Read the article

  • Recognizing file - Python

    - by Francisco Aleixo
    Ok, so the title may trick you a bit, and I'm sorry for that but didn't find a better title. This question might be a bit hard to understand so I'll try my best. I have no idea how this works or if it is even possible but what I want to do is for example create a file type (lets imagine .test (in which a random file name would be random.test)). Now before I continue, its obviously easy to do this using for example: filename = "random.test" file = open(filename, 'w') file.write("some text here") But now what I would like to know is if it is possible to write the file .test so if I set it to open with a wxPython program, it recognizes it and for example opens up a Message Dialog automatically. I'm sorry if I'm being vague and in case you don't understand, let me know so I can try my best to explain you.

    Read the article

  • How to code a batch file to copy and rename the most recently dated file?

    - by david.murtagh.keltie.com
    I'm trying to code a batch file to copy only the most recently dated file in a given folder to another directory on the local machine, and simultaneously rename it as it does. I've found a very similar question here http://stackoverflow.com/questions/97371/batch-script-to-copy-newest-file and have managed to cobble together the below code from other forums too, but have hit a brick wall as it only results in the batch file itself being copied to the destination folder. It doesn't matter to me where the batch file itself sits in order for this to run. The source folder is C:! BATCH and the destination folder is C:\DROP The code is below, apologies if this is a glaringly obvious answer but it's literally the first foray into coding batch files for me... Thanks! @echo off setLocal EnableDelayedExpansion pushd C:\! BATCH for /f "tokens=* delims= " %%G in ('dir/b/od') do (set newest=%%G) copy "!newest!" C:\DROP\ PAUSE

    Read the article

  • Best way to choose a random file from a directory in a shell script

    - by jhs
    What is the best way to choose a random file from a directory in a shell script? Here is my solution in Bash but I would be very interested for a more portable (non-GNU) version for use on Unix proper. dir='some/directory' file=`/bin/ls -1 "$dir" | sort --random-sort | head -1` path=`readlink --canonicalize "$dir/$file"` # Converts to full path echo "The randomly-selected file is: $path" Anybody have any other ideas? Edit: lhunath makes a good point about parsing ls. I guess it comes down to whether you want to be portable or not. If you have the GNU findutils and coreutils then you can do: find "$dir" -maxdepth 1 -mindepth 1 -type f -print0 \ | sort --zero-terminated --random-sort \ | sed 's/\d000.*//g/' Whew, that was fun! Also it matches my question better since I said "random file". Honsetly though, these days it's hard to imagine a Unix system deployed out there having GNU installed but not Perl 5.

    Read the article

  • Export GridView to TXT, then upload file to server

    Basically what I want to do is export an array (or GridView) to a file called "getpathin.dat". That is easy, but the method I am using downloads the file to my computer, which is what I don't want. I want to write an array to either a PRE-EXISTING file that is on the server OR create a new file on the server in a folder, and this new file will contain either the array or the gridview, which I have stored in the array. i'll be doing this in... Visual Studio 2008/SQL Server using C#

    Read the article

  • How to use application config file in C#?

    - by badpanda
    I am trying to use a config file in my C# console application. I created the file within the project by going New -- Application Configuration File, and naming it myProjectName.config. My config file looks like this: <?xml version="1.0" encoding="utf-8" ?> <configuration> <appSettings> <add key="SSDirectory" value="D:\Documents and Settings\****\MyDocuments\****" /> </appSettings> </configuration> The code to access it looks like this: private FileValidateUtil() { sSDirFilePath = ConfigurationSettings.AppSettings["SSDirectory"]; if (sSDirFilePath == null) Console.WriteLine("config file not reading in."); } Can anyone lend a hint as to why this is not working? (I am getting the error message.) Thanks!! badPanda

    Read the article

  • Java how to copy part of a file

    - by user3479074
    I have to read a file and depending of the content of the last lines, I have to copy most of its content into a new file. Unfortunately I didn't found a way to copy first n lines or chars of a file in java. The only way I found, is copying the file using nio FileChannels where I can specifiy the length in bytes. However, therefore I would need to know how many bytes the stuff I read needed in the source-file. Does anyone know a solution for one of these problems?

    Read the article

  • Need to get the uploaded file to my local PC

    - by Suhail
    Hi, I have created a test form which will ask users to enter a name and upload the image file: <html lang="en"> <head> <title>Testing image upload</title> </head> <body> <form action="/services/upload" method="POST" enctype="multipart/form-data"> File Description: <input name='fdesc' type='text'><br> File name: <input type="file" name="fname"><br> <div><input type="submit"></div> </form> </body> </html> i need to get the file uploaded by the user and store it on my local PC. can this be done in python ? please let me know.

    Read the article

  • Unknown symbols when I read file

    - by Sergey Gavruk
    I read file, but in the end of file i get unknown symbols: int main() { char *buffer, ch; int i = 0, size; FILE *fp = fopen("file.txt", "r"); if(!fp){ printf("File not found!\n"); exit(1); } fseek(fp, 0, SEEK_END); size = ftell(fp); printf("%d\n", size); fseek(fp, 0, SEEK_SET); buffer = malloc(size * sizeof(*buffer)); while(((ch = fgetc(fp)) != NULL) && (i <= size)){ buffer[i++] = ch; } printf(buffer); fclose(fp); free(buffer); getch(); return 0; }

    Read the article

  • saving file name from html form input

    - by user343934
    I am working on python and biopython right now. I have a file upload form and whatever file is uploaded suppose(abc.fasta) then i want to pass same name in execute (abc.fasta) function parameter and display function parameter (abc.aln). Workflow goes like this. ----If submit is not true then display only header and form part --- if submit is true then call execute() and get file name from form input --- Then display the save file result in the same page. File name is same as input. My raw code is here -- http://pastebin.com/FPUgZSSe Any suggestions, changes and algorithm is appreciated Thanks

    Read the article

  • Reading file data during form's clean method

    - by Dominic Rodger
    So, I'm working on implementing the answer to my previous question. Here's my model: class Talk(models.Model): title = models.CharField(max_length=200) mp3 = models.FileField(upload_to = u'talks/', max_length=200) Here's my form: class TalkForm(forms.ModelForm): def clean(self): super(TalkForm, self).clean() cleaned_data = self.cleaned_data if u'mp3' in self.files: from mutagen.mp3 import MP3 if hasattr(self.files['mp3'], 'temporary_file_path'): audio = MP3(self.files['mp3'].temporary_file_path()) else: # What goes here? audio = None # setting to None for now ... return cleaned_data class Meta: model = Talk Mutagen needs file-like objects - the first case (where the uploaded file is larger than the size of file handled in memory) works fine, but I don't know how to handle InMemoryUploadedFile that I get otherwise. I've tried: # TypeError (coercing to Unicode: need string or buffer, InMemoryUploadedFile found) audio = MP3(self.files['mp3']) # TypeError (coercing to Unicode: need string or buffer, cStringIO.StringO found) audio = MP3(self.files['mp3'].file) # Hangs seemingly indefinitely audio = MP3(self.files['mp3'].file.read()) Is there something wrong with mutagen, or am I doing it wrong?

    Read the article

  • Instant file sharing between users in PHP

    - by Skyfe
    Hi there, Working on a rather complex system in which users can directly exchange files with eachother from the website. However is any of these things possible: EITHER * Have another user download a file which is still being uploaded by another user ( in progress ) OR * Make a user automaticly ( instant ) download a file from another users PC through our website OR * Make a user automaticly (instant) download a file from our server ( so it's directly downloaded to the users pc and the progress shown on our website of the download progress, without the normal internet explorer dialog downloading the file or firefox ). Thank you very much in advanced, Best Regards, Webcodez.net. UPDATE: an example would be MSN's file sharing but then through a website instead of application.

    Read the article

  • Read a file with 2048 bytes

    - by Suresh S
    Guys i have a file which has only one line. The file has no encoding it is a simple text file with single line. For every 2048 byte in a line , there is new record of 151 byte (totally 13*151 byte = 1945 records + 85 byte empty space). similarly for the next 2048 bytes. What is the best file i/o to use? i am thinking of reading 2048 bytes from file and storing it in an array . while (offset < fileLength &&(numRead=in.read(recordChunks, offset,alength)) >= 0) { } how can i get from the read statement only 2048 bytes at a time . i am getting IndexOutofBoundException.

    Read the article

  • Delete file after sharing via intent.

    - by Matt
    I'm trying to delete a temporary file after sharing it via android's Intent.ACTION_SEND feature. Right now I am starting the activity for a result and in OnActivityResult, I am deleting the file. Unfortunately this only works if I am debugging it with a breakpoint, but when I let it run freely and say, email the file, the email has no attachment. I think what is happening is my activity is deleting the file before it had been emailed. What I don't get is why, shouldn't onActivityResult only be called AFTER the other activity is finished? I have also tried deleting the file in onResume, but no luck. Is there a better way to do this?

    Read the article

  • How can I convert a specific .cs file into either a .bat or a .exe file?

    - by user2974969
    I am trying to convert around 1,900 .txp files (a proprietary Sony image format) to .png. However, the .exe program I am using to convert these files only allows me to select one file at a time. I am running Windows 7. There's a TXP.CS file in the 'src' folder of the program, so I figured that if I was able to turn that into an executable file, I'd be able to use it to mass convert these files to PNG. However, whenever I try to convert the file using csc.exe, I get the CS0246 error (the type or namespace name 'Tools' could not be found. Are you missing a using directive or an assembly reference?). I can't use Microsoft Visual Studio right now, so I'm hoping someone can walk me through this, or maybe convert the file to either a .bat or an .exe for me. Thank you. TXP.CS

    Read the article

  • C: Reading file with a starting point

    - by Shinka
    A simple question but I can't find the answer in my book. I want to read a binary file to seed a random number generator, but I don't want to seed my generator with the same seed each time I call the function, so I will need to keep a variable for my position in the file (not a problem) and I would need to know how to read a file starting a specific point in the file (no idea how). The code: void rng_init(RNG* rng) { // ... FILE *input = fopen("random.bin", "rb"); unsigned int seed[32]; fread(seed, sizeof(unsigned int), 32, input); // seed 'rng'... fclose(input); }

    Read the article

  • Accessing a file (for writing) from a JBoss Web Service

    - by Andreas Grech
    Let's say I have this structure of my Java Web Application: TheProject -- [Web Pages] -- -- abc.txt -- -- index.jsp -- [Source Packages] -- -- [wservices] -- -- -- WS.java WS.java is my Web Service, which is situated in a wservices package. Now from this service, I need to access the abc.txt file and write to it. These are my urls: http://127.0.0.1:8080/TheProject/WS <- the webservice http://127.0.0.1:8080/TheProject/abc.txt <- the file I want to access To read the file, I tried with getResourceAsStream and I was successful in reading from it. But now I also want to write to this file, and I tried such a method but failed. Is there a way I can get access to the abc.txt file from WS.java and be able to successfully read from and write to it?

    Read the article

  • Problem with access to file

    - by phenevo
    Hi, I have winforms application and it has reference to library MyLibrary. MyLibrary has method: string[] GiveMeNamesOfAirports() { string[] lines= File.ReadLines("airports.txt"); foreach(string line in lines) ... } And when I run my Winforms application: I get error: file couldn't be find. I was trying other function: string[] lines = File.ReadAllLines(Path.Combine(System.Environment.CurrentDirectory, "airports.txt")); string[] lines = File.ReadAllLines(Path.Combine(Assembly.GetExecutingAssembly().Location, "airports.txt")); string[] lines = File.ReadAllLines(Path.Combine(Assembly.GetAssembly(typeof(Airport)).Location, "airports.txt"));

    Read the article

  • Removing words from a file

    - by user1765792
    I'm trying to take a regular text file and remove words identified in a separate file (stopwords) containing the words to be removed separated by carriage returns ("\n"). Right now I'm converting both files into lists so that the elements of each list can be compared. I got this function to work, but it doesn't remove all of the words I have specified in the stopwords file. Any help is greatly appreciated. def elimstops(file_str): #takes as input a string for the stopwords file location stop_f = open(file_str, 'r') stopw = stop_f.read() stopw = stopw.split('\n') text_file = open('sample.txt') #Opens the file whose stop words will be eliminated prime = text_file.read() prime = prime.split(' ') #Splits the string into a list separated by a space tot_str = "" #total string i = 0 while i < (len(stopw)): if stopw[i] in prime: prime.remove(stopw[i]) #removes the stopword from the text else: pass i += 1 # Creates a new string from the compilation of list elements # with the stop words removed for v in prime: tot_str = tot_str + str(v) + " " return tot_str

    Read the article

  • Multiple file upload with asp.net 4.5 and Visual Studio 2012

    - by Jalpesh P. Vadgama
    This post will be part of Visual Studio 2012 feature series. In earlier version of ASP.NET there is no way to upload multiple files at same time. We need to use third party control or we need to create custom control for that. But with asp.net 4.5 now its possible to upload multiple file with file upload control. With ASP.NET 4.5 version Microsoft has enhanced file upload control to support HTML5 multiple attribute. There is a property called ‘AllowedMultiple’ to support that attribute and with that you can easily upload the file. So what we are waiting for!! It’s time to create one example. On the default.aspx file I have written following. <asp:FileUpload ID="multipleFile" runat="server" AllowMultiple="true" /> <asp:Button ID="uploadFile" runat="server" Text="Upload files" onclick="uploadFile_Click"/> Here you can see that I have given file upload control id as multipleFile and I have set AllowMultiple file to true. I have also taken one button for uploading file.For this example I am going to upload file in images folder. As you can see I have also attached event handler for button’s click event. So it’s time to write server side code for this. Following code is for the server side. protected void uploadFile_Click(object sender, EventArgs e) { if (multipleFile.HasFiles) { foreach(HttpPostedFile uploadedFile in multipleFile.PostedFiles) { uploadedFile.SaveAs(System.IO.Path.Combine(Server.MapPath("~/Images/"),uploadedFile.FileName)); Response.Write("File uploaded successfully"); } } } Here in the above code you can see that I have checked whether multiple file upload control has multiple files or not and then I have save that in Images folder of web application. Once you run the application in browser it will look like following. I have selected two files. Once I have selected and clicked on upload file button it will give message like following. As you can see now it has successfully upload file and you can see in windows explorer like following. As you can see it’s very easy to upload multiple file in ASP.NET 4.5. Stay tuned for more. Till then happy programming. P.S.: This feature is only supported in browser who support HTML5 multiple file upload. For other browsers it will work like normal file upload control in asp.net.

    Read the article

  • That Escalated Quickly

    - by Jesse Taber
    Originally posted on: http://geekswithblogs.net/GruffCode/archive/2014/05/17/that-escalated-quickly.aspxI have been working remotely out of my home for over 4 years now. All of my coworkers during that time have also worked remotely. Lots of folks have written about the challenges inherent in facilitating communication on remote teams and strategies for overcoming them. A popular theme around this topic is the notion of “escalating communication”. In this context “escalating” means taking a conversation from one mode of communication to a different, higher fidelity mode of communication. Here are the five modes of communication I use at work in order of increasing fidelity: Email – This is the “lowest fidelity” mode of communication that I use. I usually only check it a few times a day (and I’m trying to check it even less frequently than that) and I only keep items in my inbox if they represent an item I need to take action on that I haven’t tracked anywhere else. Forums / Message boards – Being a developer, I’ve gotten into the habit of having other people look over my code before it becomes part of the product I’m working on. These code reviews often happen in “real time” via screen sharing, but I also always have someone else give all of the changes another look using pull requests. A pull request takes my code and lets someone else see the changes I’ve made side-by-side with the existing code so they can see if I did anything dumb. Pull requests can facilitate a conversation about the code changes in an online-forum like style. Some teams I’ve worked on also liked using tools like Trello or Google Groups to have on-going conversations about a topic or task that was being worked on. Chat & Instant Messaging  - Chat and instant messaging are the real workhorses for communication on the remote teams I’ve been a part of. I know some teams that are co-located that also use it pretty extensively for quick messages that don’t warrant walking across the office to talk with someone but reqire more immediacy than an e-mail. For the purposes of this post I think it’s important to note that the terms “chat” and “instant messaging” might insinuate that the conversation is happening in real time, but that’s not always true. Modern chat and IM applications maintain a searchable history so people can easily see what might have been discussed while they were away from their computers. Voice, Video and Screen sharing – Everyone’s got a camera and microphone on their computers now, and there are an abundance of services that will let you use them to talk to other people who have cameras and microphones on their computers. I’m including screen sharing here as well because, in my experience, these discussions typically involve one or more people showing the other participants something that’s happening on their screen. Obviously, this mode of communication is much higher-fidelity than any of the ones listed above. Scheduled meetings are typically conducted using this mode of communication. In Person – No matter how great communication tools become, there’s no substitute for meeting with someone face-to-face. However, opportunities for this kind of communcation are few and far between when you work on a remote team. When a conversation gets escalated that usually means it moves up one or more positions on this list. A lot of people advocate jumping to #4 sooner than later. Like them, I used to believe that, if it was possible, organizing a call with voice and video was automatically better than any kind of text-based communication could be. Lately, however, I’m becoming less convinced that escalating is always the right move. Working Asynchronously Last year I attended a talk at our local code camp given by Drew Miller. Drew works at GitHub and was talking about how they use GitHub internally. Many of the folks at GitHub work remotely, so communication was one of the main themes in Drew’s talk. During the talk Drew used the phrase, “asynchronous communication” to describe their use of chat and pull request comments. That phrase stuck in my head because I hadn’t heard it before but I think it perfectly describes the way in which remote teams often need to communicate. You don’t always know when your co-workers are at their computers or what hours (if any) they are working that day. In order to work this way you need to assume that the person you’re talking to might not respond right away. You can’t always afford to wait until everyone required is online and available to join a voice call, so you need to use text-based, persistent forms of communication so that people can receive and respond to messages when they are available. Going back to my list from the beginning of this post for a second, I characterize items #1-3 as being “asynchronous” modes of communication while we could call items #4 and #5 “synchronous”. When communication gets escalated it’s almost always moving from an asynchronous mode of communication to a synchronous one. Now, to the point of this post: I’ve become increasingly reluctant to escalate from asynchronous to synchronous communication for two primary reasons: 1 – You can often find a higher fidelity way to convey your message without holding a synchronous conversation 2 - Asynchronous modes of communication are (usually) persistent and searchable. You Don’t Have to Broadcast Live Let’s start with the first reason I’ve listed. A lot of times you feel like you need to escalate to synchronous communication because you’re having difficulty describing something that you’re seeing in words. You want to provide the people you’re conversing with some audio-visual aids to help them understand the point that you’re trying to make and you think that getting on Skype and sharing your screen with them is the best way to do that. Firing up a screen sharing session does work well, but you can usually accomplish the same thing in an asynchronous manner. For example, you could take a screenshot and annotate it with some text and drawings to illustrate what it is you’re seeing. If a screenshot won’t work, taking a short screen recording while your narrate over it and posting the video to your forum or chat system along with a text-based description of what’s in the recording that can be searched for later can be a great way to effectively communicate with your team asynchronously. I Said What?!? Now for the second reason I listed: most asynchronous modes of communication provide a transcript of what was said and what decisions might have been made during the conversation. There have been many occasions where I’ve used the search feature of my team’s chat application to find a conversation that happened several weeks or months ago to remember what was decided. Unfortunately, I think the benefits associated with the persistence of communicating asynchronously often get overlooked when people decide to escalate to a in-person meeting or voice/video call. I’m becoming much more reluctant to suggest a voice or video call if I suspect that it might lead to codifying some kind of design decision because everyone involved is going to hang up the call and immediately forget what was decided. I recognize that you can record and archive these types of interactions, but without being able to search them the recordings aren’t terribly useful. When and How To Escalate I don’t mean to imply that communicating via voice/video or in person is never a good idea. I probably jump on a Skype call with a co-worker at least once a day to quickly hash something out or show them a bit of code that I’m working on. Also, meeting in person periodically is really important for remote teams. There’s no way around the fact that sometimes it’s easier to jump on a call and show someone my screen so they can see what I’m seeing. So when is it right to escalate? I think the simplest way to answer that is when the communication starts to feel painful. Everyone’s tolerance for that pain is different, but I think you need to let it hurt a little bit before jumping to synchronous communication. When you do escalate from asynchronous to synchronous communication, there are a couple of things you can do to maximize the effectiveness of the communication: Takes notes – This is huge and yet I’ve found that a lot of teams don’t do this. If you’re holding a meeting with  > 2 people you should have someone taking notes. Taking notes while participating in a meeting can be difficult but there are a few strategies to deal with this challenge that probably deserve a short post of their own. After the meeting, make sure the notes are posted to a place where all concerned parties (including those that might not have attended the meeting) can review and search them. Persist decisions made ASAP – If any decisions were made during the meeting, persist those decisions to a searchable medium as soon as possible following the conversation. All the teams I’ve worked on used a web-based system for tracking the on-going work and a backlog of work to be done in the future. I always try to make sure that all of the cards/stories/tasks/whatever in these systems always reflect the latest decisions that were made as the work was being planned and executed. If held a quick call with your team lead and decided that it wasn’t worth the effort to build real-time validation into that new UI you were working on, go and codify that decision in the story associated with that work immediately after you hang up. Even better, write it up in the story while you are both still on the phone. That way when the folks from your QA team pick up the story to test a few days later they’ll know why the real-time validation isn’t there without having to invoke yet another conversation about the work. Communicating Well is Hard At this point you might be thinking that communicating asynchronously is more difficult than having a live conversation. You’re right: it is more difficult. In order to communicate effectively this way you need to very carefully think about the message that you’re trying to convey and craft it in a way that’s easy for your audience to understand. This is almost always harder than just talking through a problem in real time with someone; this is why escalating communication is such a popular idea. Why wouldn’t we want to do the thing that’s easier? Easier isn’t always better. If you and your team can get in the habit of communicating effectively in an asynchronous manner you’ll find that, over time, all of your communications get less painful because you don’t need to re-iterate previously made points over and over again. If you communicate right the first time, you often don’t need to rehash old conversations because you can go back and find the decisions that were made laid out in plain language. You’ll also find that you get better at doing things like writing useful comments in your code, creating written documentation about how the feature that you just built works, or persuading your team to do things in a certain way.

    Read the article

  • Copy/Paste (Clipboard like) functionality from a VNC desktop

    - by goldenmean
    I use TightVNC, RealVNC to access the Remote Linux(CentOS,Ubuntu) desktops by running a vncserver on the remote machine. When I connect to those servers from my Windows host: Can I use the Copy/Paste functionality on the VNC client desktop window. i.e. Copy some text on some terminal open inside the VNC client desktop window and paste it into some text file on my local windows machhines and vice-a-versa? I checked TightVNC options but did not see anything. Can it be done by a)Running vncserver on the remote machine with some options or b)By running some configuration on remote machine to enable this. How could I get it done? Also is this kind of Copy/Paste functionality possible in Microsoft Windows Remote Desktop connection/Terminal session protocol?

    Read the article

  • Oracle Data Integrator 11.1.1.5 Complex Files as Sources and Targets

    - by Alex Kotopoulis
    Overview ODI 11.1.1.5 adds the new Complex File technology for use with file sources and targets. The goal is to read or write file structures that are too complex to be parsed using the existing ODI File technology. This includes: Different record types in one list that use different parsing rules Hierarchical lists, for example customers with nested orders Parsing instructions in the file data, such as delimiter types, field lengths, type identifiers Complex headers such as multiple header lines or parseable information in header Skipping of lines  Conditional or choice fields Similar to the ODI File and XML File technologies, the complex file parsing is done through a JDBC driver that exposes the flat file as relational table structures. Complex files are mapped to one or more table structures, as opposed to the (simple) file technology, which always has a one-to-one relationship between file and table. The resulting set of tables follows the same concept as the ODI XML driver, table rows have additional PK-FK relationships to express hierarchy as well as order values to maintain the file order in the resulting table.   The parsing instruction format used for complex files is the nXSD (native XSD) format that is already in use with Oracle BPEL. This format extends the XML Schema standard by adding additional parsing instructions to each element. Using nXSD parsing technology, the native file is converted into an internal XML format. It is important to understand that the XML is streamed to improve performance; there is no size limitation of the native file based on memory size, the XML data is never fully materialized.  The internal XML is then converted to relational schema using the same mapping rules as the ODI XML driver. How to Create an nXSD file Complex file models depend on the nXSD schema for the given file. This nXSD file has to be created using a text editor or the Native Format Builder Wizard that is part of Oracle BPEL. BPEL is included in the ODI Suite, but not in standalone ODI Enterprise Edition. The nXSD format extends the standard XSD format through nxsd attributes. NXSD is a valid XML Schema, since the XSD standard allows extra attributes with their own namespaces. The following is a sample NXSD schema: <?xml version="1.0"?> <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:nxsd="http://xmlns.oracle.com/pcbpel/nxsd" elementFormDefault="qualified" xmlns:tns="http://xmlns.oracle.com/pcbpel/demoSchema/csv" targetNamespace="http://xmlns.oracle.com/pcbpel/demoSchema/csv" attributeFormDefault="unqualified" nxsd:encoding="US-ASCII" nxsd:stream="chars" nxsd:version="NXSD"> <xsd:element name="Root">         <xsd:complexType><xsd:sequence>       <xsd:element name="Header">                 <xsd:complexType><xsd:sequence>                         <xsd:element name="Branch" type="xsd:string" nxsd:style="terminated" nxsd:terminatedBy=","/>                         <xsd:element name="ListDate" type="xsd:string" nxsd:style="terminated" nxsd:terminatedBy="${eol}"/>                         </xsd:sequence></xsd:complexType>                         </xsd:element>                 </xsd:sequence></xsd:complexType>         <xsd:element name="Customer" maxOccurs="unbounded">                 <xsd:complexType><xsd:sequence>                 <xsd:element name="Name" type="xsd:string" nxsd:style="terminated" nxsd:terminatedBy=","/>                         <xsd:element name="Street" type="xsd:string" nxsd:style="terminated" nxsd:terminatedBy="," />                         <xsd:element name="City" type="xsd:string" nxsd:style="terminated" nxsd:terminatedBy="${eol}" />                         </xsd:sequence></xsd:complexType>                         </xsd:element>                 </xsd:sequence></xsd:complexType> </xsd:element> </xsd:schema> The nXSD schema annotates elements to describe their position and delimiters within the flat text file. The schema above uses almost exclusively the nxsd:terminatedBy instruction to look for the next terminator chars. There are various constructs in nXSD to parse fixed length fields, look ahead in the document for string occurences, perform conditional logic, use variables to remember state, and many more. nXSD files can either be written manually using an XML Schema Editor or created using the Native Format Builder Wizard. Both Native Format Builder Wizard as well as the nXSD language are described in the Application Server Adapter Users Guide. The way to start the Native Format Builder in BPEL is to create a new File Adapter; in step 8 of the Adapter Configuration Wizard a new Schema for Native Format can be created:   The Native Format Builder guides through a number of steps to generate the nXSD based on a sample native file. If the format is complex, it is often a good idea to “approximate” it with a similar simple format and then add the complex components manually.  The resulting *.xsd file can be copied and used as the format for ODI, other BPEL constructs such as the file adapter definition are not relevant for ODI. Using this technique it is also possible to parse the same file format in SOA Suite and ODI, for example using SOA for small real-time messages, and ODI for large batches. This nXSD schema in this example describes a file with a header row containing data and 3 string fields per row delimited by commas, for example: Redwood City Downtown Branch, 06/01/2011 Ebeneezer Scrooge, Sandy Lane, Atherton Tiny Tim, Winton Terrace, Menlo Park The ODI Complex File JDBC driver exposes the file structure through a set of relational tables with PK-FK relationships. The tables for this example are: Table ROOT (1 row): ROOTPK Primary Key for root element SNPSFILENAME Name of the file SNPSFILEPATH Path of the file SNPSLOADDATE Date of load Table HEADER (1 row): ROOTFK Foreign Key to ROOT record ROWORDER Order of row in native document BRANCH Data BRANCHORDER Order of Branch within row LISTDATE Data LISTDATEORDER Order of ListDate within row Table ADDRESS (2 rows): ROOTFK Foreign Key to ROOT record ROWORDER Order of row in native document NAME Data NAMEORDER Oder of Name within row STREET Data STREETORDER Order of Street within row CITY Data CITYORDER Order of City within row Every table has PK and/or FK fields to reflect the document hierarchy through relationships. In this example this is trivial since the HEADER and all CUSTOMER records point back to the PK of ROOT. Deeper nested documents require this to identify parent elements. All tables also have a ROWORDER field to define the order of rows, as well as order fields for each column, in case the order of columns varies in the original document and needs to be maintained. If order is not relevant, these fields can be ignored. How to Create an Complex File Data Server in ODI After creating the nXSD file and a test data file, and storing it on the local file system accessible to ODI, you can go to the ODI Topology Navigator to create a Data Server and Physical Schema under the Complex File technology. This technology follows the conventions of other ODI technologies and is very similar to the XML technology. The parsing settings such as the source native file, the nXSD schema file, the root element, as well as the external database can be set in the JDBC URL: The use of an external database defined by dbprops is optional, but is strongly recommended for production use. Ideally, the staging database should be used for this. Also, when using a complex file exclusively for read purposes, it is recommended to use the ro=true property to ensure the file is not unnecessarily synchronized back from the database when the connection is closed. A data file is always required to be present  at the filename path during design-time. Without this file, operations like testing the connection, reading the model data, or reverse engineering the model will fail.  All properties of the Complex File JDBC Driver are documented in the Oracle Fusion Middleware Connectivity and Knowledge Modules Guide for Oracle Data Integrator in Appendix C: Oracle Data Integrator Driver for Complex Files Reference. David Allan has created a great viewlet Complex File Processing - 0 to 60 which shows the creation of a Complex File data server as well as a model based on this server. How to Create Models based on an Complex File Schema Once physical schema and logical schema have been created, the Complex File can be used to create a Model as if it were based on a database. When reverse-engineering the Model, data stores(tables) for each XSD element of complex type will be created. Use of complex files as sources is straightforward; when using them as targets it has to be made sure that all dependent tables have matching PK-FK pairs; the same applies to the XML driver as well. Debugging and Error Handling There are different ways to test an nXSD file. The Native Format Builder Wizard can be used even if the nXSD wasn’t created in it; it will show issues related to the schema and/or test data. In ODI, the nXSD  will be parsed and run against the existing test XML file when testing a connection in the Dataserver. If either the nXSD has an error or the data is non-compliant to the schema, an error will be displayed. Sample error message: Error while reading native data. [Line=1, Col=5] Not enough data available in the input, when trying to read data of length "19" for "element with name D1" from the specified position, using "style" as "fixedLength" and "length" as "". Ensure that there is enough data from the specified position in the input. Complex File FAQ Is the size of the native file limited by available memory? No, since the native data is streamed through the driver, only the available space in the staging database limits the size of the data. There are limits on individual field sizes, though; a single large object field needs to fit in memory. Should I always use the complex file driver instead of the file driver in ODI now? No, use the file technology for all simple file parsing tasks, for example any fixed-length or delimited files that just have one row format and can be mapped into a simple table. Because of its narrow assumptions the ODI file driver is easy to configure within ODI and can stream file data without writing it into a database. The complex file driver should be used whenever the use case cannot be handled through the file driver. Are we generating XML out of flat files before we write it into a database? We don’t materialize any XML as part of parsing a flat file, either in memory or on disk. The data produced by the XML parser is streamed in Java objects that just use XSD-derived nXSD schema as its type system. We use the nXSD schema because is the standard for describing complex flat file metadata in Oracle Fusion Middleware, and enables users to share schemas across products. Is the nXSD file interchangeable with SOA Suite? Yes, ODI can use the same nXSD files as SOA Suite, allowing mixed use cases with the same data format. Can I start the Native Format Builder from the ODI Studio? No, the Native Format Builder has to be started from a JDeveloper with BPEL instance. You can get BPEL as part of the SOA Suite bundle. Users without SOA Suite can manually develop nXSD files using XSD editors. When is the database data written back to the native file? Data is synchronized using the SYNCHRONIZE and CREATE FILE commands, and when the JDBC connection is closed. It is recommended to set the ro or read_only property to true when a file is exclusively used for reading so that no unnecessary write-backs occur. Is the nXSD metadata part of the ODI Master or Work Repository? No, the data server definition in the master repository only contains the JDBC URL with file paths; the nXSD files have to be accessible on the file systems where the JDBC driver is executed during production, either by copying or by using a network file system. Where can I find sample nXSD files? The Application Server Adapter Users Guide contains nXSD samples for various different use cases.

    Read the article

< Previous Page | 68 69 70 71 72 73 74 75 76 77 78 79  | Next Page >