Search Results

Search found 8354 results on 335 pages for 'welton v3 50'.

Page 219/335 | < Previous Page | 215 216 217 218 219 220 221 222 223 224 225 226  | Next Page >

• Inserting Records in Ascending Order function- C homework assignment

  - by Aaron McRuer

  Good day, Stack Overflow. I have a homework assignment that I'm working on this weekend that I'm having a bit of a problem with. We have a struct "Record" (which contains information about cars for a dealership) that gets placed in a particular spot in a linked list according to 1) its make and 2) according to its model year. This is done when initially building the list, when a "int insertRecordInAscendingOrder" function is called in Main. In "insertRecordInAscendingOrder", a third function, "createRecord" is called, where the linked list is created. The function then goes to the function "compareCars" to determine what elements get put where. Depending on the value returned by this function, insertRecordInAscendingOrder then places the record where it belongs. The list is then printed out. There's more to the assignment, but I'll cross that bridge when I come to it. Ideally, and for the assignment to be considered correct, the linked list must be ordered as: Chevrolet 2012 25 Chevrolet 2013 10 Ford 2010 5 Ford 2011 3 Ford 2012 15 Honda 2011 9 Honda 2012 3 Honda 2013 12 Toyota 2009 2 Toyota 2011 7 Toyota 2013 20 from the a text file that has the data ordered the following way: Ford 2012 15 Ford 2011 3 Ford 2010 5 Toyota 2011 7 Toyota 2012 20 Toyota 2009 2 Honda 2011 9 Honda 2012 3 Honda 2013 12 Chevrolet 2013 10 Chevrolet 2012 25 Notice that the alphabetical order of the "make" field takes precedence, then, the model year is arranged from oldest to newest. However, the program produces this as the final list: Chevrolet 2012 25 Chevrolet 2013 10 Honda 2011 9 Honda 2012 3 Honda 2013 12 Toyota 2009 2 Toyota 2011 7 Toyota 2012 20 Ford 2010 5 Ford 2011 3 Ford 2012 15 I sat down with a grad student and tried to work out all of this yesterday, but we just couldn't figure out why it was kicking the Ford nodes down to the end of the list. Here's the code. As you'll notice, I included a printList call at each instance of the insertion of a node. This way, you can see just what is happening when the nodes are being put in "order". It is in ANSI C99. All function calls must be made as they are specified, so unfortunately, there's no real way of getting around this problem by creating a more efficient algorithm. #include <stdio.h> #include <stdlib.h> #include <string.h> #define MAX_LINE 50 #define MAX_MAKE 20 typedef struct record { char *make; int year; int stock; struct record *next; } Record; int compareCars(Record *car1, Record *car2); void printList(Record *head); Record* createRecord(char *make, int year, int stock); int insertRecordInAscendingOrder(Record **head, char *make, int year, int stock); int main(int argc, char **argv) { FILE *inFile = NULL; char line[MAX_LINE + 1]; char *make, *yearStr, *stockStr; int year, stock, len; Record* headRecord = NULL; /*Input and file diagnostics*/ if (argc!=2) { printf ("Filename not provided.\n"); return 1; } if((inFile=fopen(argv[1], "r"))==NULL) { printf("Can't open the file\n"); return 2; } /*obtain values for linked list*/ while (fgets(line, MAX_LINE, inFile)) { make = strtok(line, " "); yearStr = strtok(NULL, " "); stockStr = strtok(NULL, " "); year = atoi(yearStr); stock = atoi(stockStr); insertRecordInAscendingOrder(&headRecord,make, year, stock); } printf("The original list in ascending order: \n"); printList(headRecord); } /*use strcmp to compare two makes*/ int compareCars(Record *car1, Record *car2) { int compStrResult; compStrResult = strcmp(car1->make, car2->make); int compYearResult = 0; if(car1->year > car2->year) { compYearResult = 1; } else if(car1->year == car2->year) { compYearResult = 0; } else { compYearResult = -1; } if(compStrResult == 0 ) { if(compYearResult == 1) { return 1; } else if(compYearResult == -1) { return -1; } else { return compStrResult; } } else if(compStrResult == 1) { return 1; } else { return -1; } } int insertRecordInAscendingOrder(Record **head, char *make, int year, int stock) { Record *previous = *head; Record *newRecord = createRecord(make, year, stock); Record *current = *head; int compResult; if(*head == NULL) { *head = newRecord; printf("Head is null, list was empty\n"); printList(*head); return 1; } else if ( compareCars(newRecord, *head)==-1) { *head = newRecord; (*head)->next = current; printf("New record was less than the head, replacing\n"); printList(*head); return 1; } else { printf("standard case, searching and inserting\n"); previous = *head; while ( current != NULL &&(compareCars(newRecord, current)==1)) { printList(*head); previous = current; current = current->next; } printList(*head); previous->next = newRecord; previous->next->next = current; } return 1; } /*creates records from info passed in from main via insertRecordInAscendingOrder.*/ Record* createRecord(char *make, int year, int stock) { printf("CreateRecord\n"); Record *theRecord; int len; if(!make) { return NULL; } theRecord = malloc(sizeof(Record)); if(!theRecord) { printf("Unable to allocate memory for the structure.\n"); return NULL; } theRecord->year = year; theRecord->stock = stock; len = strlen(make); theRecord->make = malloc(len + 1); strncpy(theRecord->make, make, len); theRecord->make[len] = '\0'; theRecord->next=NULL; return theRecord; } /*prints list. lists print.*/ void printList(Record *head) { int i; int j = 50; Record *aRecord; aRecord = head; for(i = 0; i < j; i++) { printf("-"); } printf("\n"); printf("%20s%20s%10s\n", "Make", "Year", "Stock"); for(i = 0; i < j; i++) { printf("-"); } printf("\n"); while(aRecord != NULL) { printf("%20s%20d%10d\n", aRecord->make, aRecord->year, aRecord->stock); aRecord = aRecord->next; } printf("\n"); } The text file you'll need for a command line argument can be saved under any name you like; here are the contents you'll need: Ford 2012 15 Ford 2011 3 Ford 2010 5 Toyota 2011 7 Toyota 2012 20 Toyota 2009 2 Honda 2011 9 Honda 2012 3 Honda 2013 12 Chevrolet 2013 10 Chevrolet 2012 25 Thanks in advance for your help. I shall continue to plow away at it myself.

  Read the article

• nested for-each loops in xml

  - by user1748443

  I'm new to XML. I'm trying to create table containing item details and another table to contain customer details for each order on a picklist. It seems like it should be straightforward but I just get a list of all items on all orders repeated by the number of orders. What am I doing wrong? (XSL code below...) <?xml version="1.0" encoding="utf-8"?> <xsl:stylesheet version="1.0" xmlns:xsl = "http://www.w3.org/1999/XSL/Transform"> <xsl:output method="html" doctype-system="about:legacy-compat"/> <xsl:template match="/"> <html xmlns = "http://www.w3.org/1999/xhtml"> <head> <meta charset="utf-8"/> <link rel="stylesheet" type="css/text" href="style.css"/> <title>Orders</title> </head> <body> <xsl:for-each select="//order"> <table> <caption><h3>Order Information</h3></caption> <thead> <th align="left">Item Id</th> <th align="left">Item Description</th> <th align="left">Quantity</th> <th align="left">Price</th> </thead> <xsl:for-each select="//item"> <tr> <td align="left"><xsl:value-of select="itemId"/></td> <td align="left"><xsl:value-of select="itemName"/></td> <td align="left"><xsl:value-of select="quantity"/></td> <td align="left"><xsl:value-of select="price"/></td> </tr> </xsl:for-each> </table> <table> <caption><h3>Customer Information</h3></caption> <thead> <th align="left">Customer Name</th> <th align="left">Street</th> <th align="left">City</th> </thead> <xsl:for-each select="//item"> <tr> <td align="left"><xsl:value-of select="customerName"/></td> <td align="left"><xsl:value-of select="street"/></td> <td align="left"><xsl:value-of select="city"/></td> </tr> </xsl:for-each> </table> </xsl:for-each> </body> </html> </xsl:template> </xsl:stylesheet> This is the XML: <?xml version="1.0" encoding="utf-8"?> <?xml-stylesheet type="text/xsl" href="Orders.xsl"?> <orders xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="Orders.xsd"> <order> <orderId>123</orderId> <items> <item> <itemId>001</itemId> <itemName>Nylon Rope</itemName> <quantity>1</quantity> <price>3.50</price> </item> <item> <itemId>002</itemId> <itemName>Shovel</itemName> <quantity>1</quantity> <price>24.95</price> </item> </items> <customerAddress> <customerName>Larry Murphy</customerName> <street>Shallowgrave Lane</street> <city>Ballymore Eustace, Co. Kildare</city> </customerAddress> </order> <order> <orderId>124</orderId> <items> <item> <itemId>001</itemId> <itemName>Whiskey</itemName> <quantity>1</quantity> <price>18.50</price> </item> <item> <itemId>002</itemId> <itemName>Shotgun</itemName> <quantity>1</quantity> <price>225</price> </item> <item> <itemId>003</itemId> <itemName>Cartridge</itemName> <quantity>1</quantity> <price>1.85</price> </item> </items> <customerAddress> <customerName>Enda Kenny</customerName> <street>A Avenue</street> <city>Castlebar, Co. Mayo</city> </customerAddress> </order> </orders>

  Read the article

• Native ComboBox not displaying choices correctly

  - by anothershrubery

  EDIT: It seems that ListPicker is the way to go but I have had further problems with that detailed Microsoft.Phone.Controls.Toolkit ListPicker throws XamlParseException I have the following ComboBox in code: <ComboBox x:Name="Result" Grid.Column="6" HorizontalAlignment="Center" VerticalAlignment="Center" Foreground="Black" Background="White"> <ComboBoxItem Content="Win" /> <ComboBoxItem Content="Place" /> <ComboBoxItem Content="Lose" /> </ComboBox> But it does not display as I would have expected. When you drop down the ComboBox the options don't appear, it's just like empty items. See below: However, when an item is selected, it displays correctly and the correct index/item is returned. See below: I'm sure there is something simple I have missed but can't put my finger on it. EDIT: Ok I am posting the full code for this. I have a user control, OddsRow, that looks like this: <UserControl xmlns:toolkit="clr-namespace:Microsoft.Phone.Controls;assembly=Microsoft.Phone.Controls.Toolkit" x:Class="MojoPinBetOddsCalculator.OddsRow" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:d="http://schemas.microsoft.com/expression/blend/2008" xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006" mc:Ignorable="d" FontFamily="{StaticResource PhoneFontFamilyNormal}" FontSize="{StaticResource PhoneFontSizeNormal}" Foreground="{StaticResource PhoneForegroundBrush}" d:DesignHeight="480" d:DesignWidth="480"> <Grid x:Name="LayoutRoot" Background="{StaticResource PhoneChromeBrush}"> <Grid.RowDefinitions> <RowDefinition Height="70"/> </Grid.RowDefinitions> <Grid.ColumnDefinitions> <ColumnDefinition Width="50*"></ColumnDefinition> <ColumnDefinition Width="70*"></ColumnDefinition> <ColumnDefinition Width="30*"></ColumnDefinition> <ColumnDefinition Width="70*"></ColumnDefinition> <ColumnDefinition Width="70*"></ColumnDefinition> <ColumnDefinition Width="70*" ></ColumnDefinition> <ColumnDefinition Width="100*"></ColumnDefinition> </Grid.ColumnDefinitions> <TextBlock x:Name="RowNumber" Grid.Column="0" VerticalAlignment="Center" HorizontalAlignment="Center"></TextBlock> <TextBox x:Name="OddsNumerator" Grid.Column="1" Width="90" Height="70" HorizontalAlignment="Center" TextAlignment="Center" VerticalAlignment="Center" MaxLength="3" InputScope="TelephoneNumber"></TextBox> <TextBlock x:Name="Slash" Grid.Column="2" Text="/" HorizontalAlignment="Center" VerticalAlignment="Center"></TextBlock> <TextBox x:Name="OddsDenominator" Grid.Column="3" Width="90" Height="70" VerticalAlignment="Center" TextAlignment="Center" MaxLength="3" HorizontalAlignment="Center" InputScope="TelephoneNumber"></TextBox> <CheckBox x:Name="EachWay" Grid.Column="4" VerticalAlignment="Center" HorizontalAlignment="Center" Margin="10,0,0,0" /> <CheckBox x:Name="Place" Grid.Column="5" HorizontalAlignment="Center" VerticalAlignment="Center" BorderThickness="0" Width="71" Margin="10,0,0,0" Padding="0" /> <ComboBox x:Name="Result" Grid.Column="6" HorizontalAlignment="Center" VerticalAlignment="Center" Foreground="Black" Background="White"> <ComboBoxItem Content="Win" /> <ComboBoxItem Content="Place" /> <ComboBoxItem Content="Lose" /> </ComboBox> </Grid> </UserControl> And it is displayed in the MainPage like so: <phone:PhoneApplicationPage xmlns:my="clr-namespace:MojoPinBetOddsCalculator" x:Class="MojoPinBetOddsCalculator.MainPage" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:phone="clr-namespace:Microsoft.Phone.Controls;assembly=Microsoft.Phone" xmlns:shell="clr-namespace:Microsoft.Phone.Shell;assembly=Microsoft.Phone" xmlns:d="http://schemas.microsoft.com/expression/blend/2008" xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006" mc:Ignorable="d" d:DesignWidth="480" d:DesignHeight="768" FontFamily="{StaticResource PhoneFontFamilyNormal}" FontSize="{StaticResource PhoneFontSizeNormal}" Foreground="{StaticResource PhoneForegroundBrush}" SupportedOrientations="Portrait" Orientation="Portrait" shell:SystemTray.IsVisible="True"> <!--LayoutRoot is the root grid where all page content is placed--> <Grid x:Name="LayoutRoot" Background="Transparent"> <Grid.RowDefinitions> <RowDefinition Height="Auto"/> <RowDefinition Height="*"/> </Grid.RowDefinitions> <!--TitlePanel contains the name of the application and page title--> <StackPanel x:Name="TitlePanel" Grid.Row="0" Margin="12,17,0,28"> <TextBlock x:Name="ApplicationTitle" Text="BET ODDS CALCULATOR" Style="{StaticResource PhoneTextNormalStyle}"/> <TextBlock x:Name="PageTitle" Text="calculate" Margin="9,-7,0,0" Style="{StaticResource PhoneTextTitle1Style}"/> </StackPanel> <!--ContentPanel - place additional content here--> <Grid x:Name="ContentPanel" Grid.Row="1" Margin="12,0,12,0" HorizontalAlignment="Stretch"> <Grid x:Name="Scrollable"> <ScrollViewer> <Grid x:Name="BettingGrid"> <Grid.RowDefinitions> <RowDefinition Height="Auto"></RowDefinition> <RowDefinition Height="*"></RowDefinition> </Grid.RowDefinitions> <Grid x:Name="BetList"> <Grid.RowDefinitions> <RowDefinition Height="30"></RowDefinition> <RowDefinition Height="70"></RowDefinition> <RowDefinition Height="70"></RowDefinition> <RowDefinition Height="70"></RowDefinition> <RowDefinition Height="70"></RowDefinition> <RowDefinition Height="70"></RowDefinition> <RowDefinition Height="70"></RowDefinition> </Grid.RowDefinitions> <Grid.ColumnDefinitions> <ColumnDefinition Width="50*"></ColumnDefinition> <ColumnDefinition Width="70*"></ColumnDefinition> <ColumnDefinition Width="30*"></ColumnDefinition> <ColumnDefinition Width="70*"></ColumnDefinition> <ColumnDefinition Width="70*"></ColumnDefinition> <ColumnDefinition Width="70*"></ColumnDefinition> <ColumnDefinition Width="100*"></ColumnDefinition> </Grid.ColumnDefinitions> <TextBlock Text="EW" Style="{StaticResource PhoneTextNormalStyle}" Grid.Row="0" Grid.Column="4" HorizontalAlignment="Center" /> <TextBlock Text="Place" Style="{StaticResource PhoneTextNormalStyle}" Grid.Row="0" Grid.Column="5" HorizontalAlignment="Center" /> <TextBlock Text="Result" Style="{StaticResource PhoneTextNormalStyle}" Grid.Row="0" Grid.Column="6" HorizontalAlignment="Center" /> <my:OddsRow Grid.Row="1" Grid.ColumnSpan="7" Row="1"/> <my:OddsRow Grid.Row="2" Grid.ColumnSpan="7" Row="2"/> <my:OddsRow Grid.Row="3" Grid.ColumnSpan="7" Row="3"/> <my:OddsRow Grid.Row="4" Grid.ColumnSpan="7" Row="4"/> <my:OddsRow Grid.Row="5" Grid.ColumnSpan="7" Row="5"/> <my:OddsRow Grid.Row="6" Grid.ColumnSpan="7" Row="6"/> </Grid> <Grid x:Name="ControlsGrid" Grid.Row="1"> <Button x:Name="AddRowButton" Background="#BFFFFFFF" BorderBrush="#BFFFFFFF" Foreground="Black" Content="Add Row" FontSize="16" Click="AddRowButton_Click" Height="70" /> </Grid> </Grid> </ScrollViewer> </Grid> </Grid> </Grid> </phone:PhoneApplicationPage> Separately the ComboBox works, and also the code for the OddsRow works as expected... separately. When combined it doesn't display the items. OddsRow.xaml.cs public partial class OddsRow : UserControl { private int m_Row; public OddsRow() { InitializeComponent(); } public int Row { get { return m_Row; } set { m_Row = value; RowNumber.Text = m_Row + " - "; } } }

  Read the article

• Segmentation fault in my C program

  - by user233542

  I don't understand why this would give me a seg fault. Any ideas? This is the function that returns the signal to stop the program (plus the other function that is called within this): double bisect(double A0,double A1,double Sol[N],double tol,double c) { double Amid,shot; while (A1-A0 > tol) { Amid = 0.5*(A0+A1); shot = shoot(Sol, Amid, c); if (shot==2.*Pi) { return Amid; } if (shot > 2.*Pi){ A1 = Amid; } else if (shot < 2.*Pi){ A0 = Amid; } } return 0.5*(A1+A0); } double shoot(double Sol[N],double A,double c) { int i,j; /*Initial Conditions*/ for (i=0;i<buff;i++) { Sol[i] = 0.; } for (i=buff+l;i<N;i++) { Sol[i] = 2.*Pi; } Sol[buff]= 0; Sol[buff+1]= A*exp(sqrt(1+3*c)*dx); for (i=buff+2;i<buff+l;i++) { Sol[i] = (dx*dx)*( sin(Sol[i-1]) + c*sin(3.*(Sol[i-1])) ) - Sol[i-2] + 2.*Sol[i-1]; } return Sol[i-1]; } The values buff, l, N are defined using a #define statement. l = 401, buff = 50, N = 2000 Here is the full code: #include <stdio.h> #include <stdlib.h> #include <math.h> #define w 10 /*characteristic width of a soliton*/ #define dx 0.05 /*distance between lattice sites*/ #define s (2*w)/dx /*size of soliton shape*/ #define l (int)(s+1) /*array length for soliton*/ #define N (int)2000 /*length of field array--lattice sites*/ #define Pi (double)4*atan(1) #define buff (int)50 double shoot(double Sol[N],double A,double c); double bisect(double A0,double A1,double Sol[N],double tol,double c); void super_pos(double antiSol[N],double Sol[N],double phi[][N]); void vel_ver(double phi[][N],double v,double c,int tsteps,double dt); int main(int argc, char **argv) { double c,Sol[N],antiSol[N],A,A0,A1,tol,v,dt; int tsteps,i; FILE *fp1,*fp2,*fp3; fp1 = fopen("soliton.dat","w"); fp2 = fopen("final-phi.dat","w"); fp3 = fopen("energy.dat","w"); printf("Please input the number of time steps:"); scanf("%d",&tsteps); printf("Also, enter the time step size:"); scanf("%lf",&dt); do{ printf("Please input the parameter c in the interval [-1/3,1]:"); scanf("%lf",&c);} while(c < (-1./3.) || c > 1.); printf("Please input the inital speed of eiter soliton:"); scanf("%lf",&v); double phi[tsteps+1][N]; tol = 0.0000001; A0 = 0.; A1 = 2.*Pi; A = bisect(A0,A1,Sol,tol,c); shoot(Sol,A,c); for (i=0;i<N;i++) { fprintf(fp1,"%d\t",i); fprintf(fp1,"%lf\n",Sol[i]); } fclose(fp1); super_pos(antiSol,Sol,phi); /*vel_ver(phi,v,c,tsteps,dt); for (i=0;i<N;i++){ fprintf(fp2,"%d\t",i); fprintf(fp2,"%lf\n",phi[tsteps][i]); }*/ } double shoot(double Sol[N],double A,double c) { int i,j; /*Initial Conditions*/ for (i=0;i<buff;i++) { Sol[i] = 0.; } for (i=buff+l;i<N;i++) { Sol[i] = 2.*Pi; } Sol[buff]= 0; Sol[buff+1]= A*exp(sqrt(1+3*c)*dx); for (i=buff+2;i<buff+l;i++) { Sol[i] = (dx*dx)*( sin(Sol[i-1]) + c*sin(3.*(Sol[i-1])) ) - Sol[i-2] + 2.*Sol[i-1]; } return Sol[i-1]; } double bisect(double A0,double A1,double Sol[N],double tol,double c) { double Amid,shot; while (A1-A0 > tol) { Amid = 0.5*(A0+A1); shot = shoot(Sol, Amid, c); if (shot==2.*Pi) { return Amid; } if (shot > 2.*Pi){ A1 = Amid; } else if (shot < 2.*Pi){ A0 = Amid; } } return 0.5*(A1+A0); } void super_pos(double antiSol[N],double Sol[N],double phi[][N]) { int i; /*for (i=0;i<N;i++) { phi[i]=0; } for (i=buffer+s;i<1950-s;i++) { phi[i]=2*Pi; }*/ for (i=0;i<N;i++) { antiSol[i] = Sol[N-i]; } /*for (i=0;i<s+1;i++) { phi[buffer+j] = Sol[j]; phi[1549+j] = antiSol[j]; }*/ for (i=0;i<N;i++) { phi[0][i] = antiSol[i] + Sol[i] - 2.*Pi; } } /* This funciton will set the 2nd input array to the derivative at the time t, for all points x in the lattice */ void deriv2(double phi[][N],double DphiDx2[][N],int t) { //double SolDer2[s+1]; int x; for (x=0;x<N;x++) { DphiDx2[t][x] = (phi[buff+x+1][t] + phi[buff+x-1][t] - 2.*phi[x][t])/(dx*dx); } /*for (i=0;i<N;i++) { ptr[i] = &SolDer2[i]; }*/ //return DphiDx2[x]; } void vel_ver(double phi[][N],double v,double c,int tsteps,double dt) { int t,x; double d1,d2,dp,DphiDx1[tsteps+1][N],DphiDx2[tsteps+1][N],dpdt[tsteps+1][N],p[tsteps+1][N]; for (t=0;t<tsteps;t++){ if (t==0){ for (x=0;x<N;x++){//inital conditions deriv2(phi,DphiDx2,t); dpdt[t][x] = DphiDx2[t][x] - sin(phi[t][x]) - sin(3.*phi[t][x]); DphiDx1[t][x] = (phi[t][x+1] - phi[t][x])/dx; p[t][x] = -v*DphiDx1[t][x]; } } for (x=0;x<N;x++){//velocity-verlet phi[t+1][x] = phi[t][x] + dt*p[t][x] + (dt*dt/2)*dpdt[t][x]; p[t+1][x] = p[t][x] + (dt/2)*dpdt[t][x]; deriv2(phi,DphiDx2,t+1); dpdt[t][x] = DphiDx2[t][x] - sin(phi[t+1][x]) - sin(3.*phi[t+1][x]); p[t+1][x] += (dt/2)*dpdt[t+1][x]; } } } So, this really isn't due to my overwriting the end of the Sol array. I've commented out both functions that I suspected of causing the problem (bisect or shoot) and inserted a print function. Two things happen. When I have code like below: double A,Pi,B,c; c=0; Pi = 4.*atan(1.); A = Pi; B = 1./4.; printf("%lf",B); B = shoot(Sol,A,c); printf("%lf",B); I get a segfault from the function, shoot. However, if I take away the shoot function so that I have: double A,Pi,B,c; c=0; Pi = 4.*atan(1.); A = Pi; B = 1./4.; printf("%lf",B); it gives me a segfault at the printf... Why!?

  Read the article

• C++/CLI HTTP Proxy problems...

  - by darkantimatter

  Hi, I'm trying(very hard) to make a small HTTP Proxy server which I can use to save all communications to a file. Seeing as I dont really have any experience in the area, I used a class from codeproject.com and some associated code to get started (It was made in the old CLI syntax, so I converted it). I couldn't get it working, so I added lots more code to make it work (threads etc), and now it sort of works. Basically, it recieves something from a client (I just configured Mozilla Firefox to route its connections through this proxy) and then routes it to google.com. After it sends Mozilla's data to google, recieves a responce, and sends that to Mozilla. This works fine, but then the proxy fails to recieve any data from Mozilla. It just loops in the Sleep(50) section. Anyway, heres the code: ProxyTest.cpp: #include "stdafx.h" #include "windows.h" #include "CHTTPProxy.h" public ref class ClientThread { public: System::Net::Sockets::TcpClient ^ pClient; CHttpProxy ^ pProxy; System::Int32 ^ pRecieveBufferSize; System::Threading::Thread ^ Thread; ClientThread(System::Net::Sockets::TcpClient ^ sClient, CHttpProxy ^ sProxy, System::Int32 ^ sRecieveBufferSize) { pClient = sClient; pProxy = sProxy; pRecieveBufferSize = sRecieveBufferSize; }; void StartReading() { Thread = gcnew System::Threading::Thread(gcnew System::Threading::ThreadStart(this,&ClientThread::ThreadEntryPoint)); Thread->Start(); }; void ThreadEntryPoint() { char * bytess; bytess = new char[(int)pRecieveBufferSize]; memset(bytess, 0, (int)pRecieveBufferSize); array<unsigned char> ^ bytes = gcnew array<unsigned char>((int)pRecieveBufferSize); array<unsigned char> ^ sendbytes; do { if (pClient->GetStream()->DataAvailable) { try { do { Sleep(100); //Lets wait for whole packet to get cached (If it even does...) unsigned int k = pClient->GetStream()->Read(bytes, 0, (int)pRecieveBufferSize); //Read it for(unsigned int i=0; i<(int)pRecieveBufferSize; i++) bytess[i] = bytes[i]; Console::WriteLine("Packet Received:\n"+gcnew System::String(bytess)); pProxy->SendToServer(bytes,pClient->GetStream()); //Now send it to google! pClient->GetStream()->Flush(); } while(pClient->GetStream()->DataAvailable); } catch (Exception ^ e) { break; } } else { Sleep(50); //It just loops here because it thinks mozilla isnt sending anything if (!(pClient->Connected)) break; }; } while (pClient->GetStream()->CanRead); delete [] bytess; pClient->Close(); }; }; int main(array<System::String ^> ^args) { System::Collections::Generic::Stack<ClientThread ^> ^ Clients = gcnew System::Collections::Generic::Stack<ClientThread ^>(); System::Net::Sockets::TcpListener ^ pTcpListener = gcnew System::Net::Sockets::TcpListener(8080); pTcpListener->Start(); System::Net::Sockets::TcpClient ^ pTcpClient; while (1) { pTcpClient = pTcpListener->AcceptTcpClient(); //Wait for client ClientThread ^ Client = gcnew ClientThread(pTcpClient, gcnew CHttpProxy("www.google.com.au", 80), pTcpClient->ReceiveBufferSize); //Make a new object for this client Client->StartReading(); //Start the thread Clients->Push(Client); //Add it to the list }; pTcpListener->Stop(); return 0; } CHTTPProxy.h, from http://www.codeproject.com/KB/IP/howtoproxy.aspx with a lot of modifications: //THIS FILE IS FROM http://www.codeproject.com/KB/IP/howtoproxy.aspx. I DID NOT MAKE THIS! BUT I HAVE MADE SEVERAL MODIFICATIONS! #using <mscorlib.dll> #using <SYSTEM.DLL> using namespace System; using System::Net::Sockets::TcpClient; using System::String; using System::Exception; using System::Net::Sockets::NetworkStream; #include <stdio.h> ref class CHttpProxy { public: CHttpProxy(System::String ^ szHost, int port); System::String ^ m_host; int m_port; void SendToServer(array<unsigned char> ^ Packet, System::Net::Sockets::NetworkStream ^ sendstr); }; CHttpProxy::CHttpProxy(System::String ^ szHost, int port) { m_host = gcnew System::String(szHost); m_port = port; } void CHttpProxy::SendToServer(array<unsigned char> ^ Packet, System::Net::Sockets::NetworkStream ^ sendstr) { TcpClient ^ tcpclnt = gcnew TcpClient(); try { tcpclnt->Connect(m_host,m_port); } catch (Exception ^ e ) { Console::WriteLine(e->ToString()); return; } // Send it if ( tcpclnt ) { NetworkStream ^ networkStream; networkStream = tcpclnt->GetStream(); int size = Packet->Length; networkStream->Write(Packet, 0, size); array<unsigned char> ^ bytes = gcnew array<unsigned char>(tcpclnt->ReceiveBufferSize); char * bytess = new char[tcpclnt->ReceiveBufferSize]; Sleep(500); //Wait for responce do { unsigned int k = networkStream->Read(bytes, 0, (int)tcpclnt->ReceiveBufferSize); //Read from google for(unsigned int i=0; i<k; i++) { bytess[i] = bytes[i]; if (bytess[i] == 0) bytess[i] = ' '; //Dont terminate the string if (bytess[i] < 8) bytess[i] = ' '; //Somethings making the computer beep, and its not 7?!?! }; Console::WriteLine("\n\nAbove packet sent to google. Google Packet Received:\n"+gcnew System::String(bytess)); sendstr->Write(bytes,0,k); //Send it to mozilla Console::WriteLine("\n\nAbove packet sent to client..."); //Sleep(1000); } while(networkStream->DataAvailable); delete [] bytess; } return; } Any help would be much appreciated, I've tried for hours.

  Read the article

• PHP Multiple User Login Form - Navigation to Different Pages Based on Login Credentials

  - by Zulu Irminger

  I am trying to create a login page that will send the user to a different index.php page based on their login credentials. For example, should a user with the "IT Technician" role log in, they will be sent to "index.php", and if a user with the "Student" role log in, they will be sent to the "student/index.php" page. I can't see what's wrong with my code, but it's not working... I'm getting the "wrong login credentials" message every time I press the login button. My code for the user login page is here: <?php session_start(); if (isset($_SESSION["manager"])) { header("location: http://www.zuluirminger.com/SchoolAdmin/index.php"); exit(); } ?> <?php if (isset($_POST["username"]) && isset($_POST["password"]) && isset($_POST["role"])) { $manager = preg_replace('#[^A-Za-z0-9]#i', '', $_POST["username"]); $password = preg_replace('#[^A-Za-z0-9]#i', '', $_POST["password"]); $role = preg_replace('#[^A-Za-z0-9]#i', '', $_POST["role"]); include "adminscripts/connect_to_mysql.php"; $sql = mysql_query("SELECT id FROM Users WHERE username='$manager' AND password='$password' AND role='$role' LIMIT 1"); $existCount = mysql_num_rows($sql); if (($existCount == 1) && ($role == 'IT Technician')) { while ($row = mysql_fetch_array($sql)) { $id = $row["id"]; } $_SESSION["id"] = $id; $_SESSION["manager"] = $manager; $_SESSION["password"] = $password; $_SESSION["role"] = $role; header("location: http://www.zuluirminger.com/SchoolAdmin/index.php"); } else { echo 'Your login details were incorrect. Please try again <a href="http://www.zuluirminger.com/SchoolAdmin/index.php">here</a>'; exit(); } } ?> <?php if (isset($_POST["username"]) && isset($_POST["password"]) && isset($_POST["role"])) { $manager = preg_replace('#[^A-Za-z0-9]#i', '', $_POST["username"]); $password = preg_replace('#[^A-Za-z0-9]#i', '', $_POST["password"]); $role = preg_replace('#[^A-Za-z0-9]#i', '', $_POST["role"]); include "adminscripts/connect_to_mysql.php"; $sql = mysql_query("SELECT id FROM Users WHERE username='$manager' AND password='$password' AND role='$role' LIMIT 1"); $existCount = mysql_num_rows($sql); if (($existCount == 1) && ($role == 'Student')) { while ($row = mysql_fetch_array($sql)) { $id = $row["id"]; } $_SESSION["id"] = $id; $_SESSION["manager"] = $manager; $_SESSION["password"] = $password; $_SESSION["role"] = $role; header("location: http://www.zuluirminger.com/SchoolAdmin/student/index.php"); } else { echo 'Your login details were incorrect. Please try again <a href="http://www.zuluirminger.com/SchoolAdmin/index.php">here</a>'; exit(); } } ?> And the form that the data is pulled from is shown here: <form id="LoginForm" name="LoginForm" method="post" action="http://www.zuluirminger.com/SchoolAdmin/user_login.php"> User Name:<br /> <input type="text" name="username" id="username" size="50" /><br /> <br /> Password:<br /> <input type="password" name="password" id="password" size="50" /><br /> <br /> Log in as: <select name="role" id="role"> <option value="">...</option> <option value="Head">Head</option> <option value="Deputy Head">Deputy Head</option> <option value="IT Technician">IT Technician</option> <option value="Pastoral Care">Pastoral Care</option> <option value="Bursar">Bursar</option> <option value="Secretary">Secretary</option> <option value="Housemaster">Housemaster</option> <option value="Teacher">Teacher</option> <option value="Tutor">Tutor</option> <option value="Sanatorium Staff">Sanatorium Staff</option> <option value="Kitchen Staff">Kitchen Staff</option> <option value="Parent">Parent</option> <option value="Student">Student</option> </select><br /> <br /> <input type="submit" name = "button" id="button" value="Log In" onclick="javascript:return validateLoginForm();" /> </h3> </form> Once logged in (and should the correct page be loaded, the validation code I have at the top of the script looks like this: <?php session_start(); if (!isset($_SESSION["manager"])) { header("location: http://www.zuluirminger.com/SchoolAdmin/user_login.php"); exit(); } $managerID = preg_replace('#[^0-9]#i', '', $_SESSION["id"]); $manager = preg_replace('#[^A-Za-z0-9]#i', '', $_SESSION["manager"]); $password = preg_replace('#[^A-Za-z0-9]#i', '', $_SESSION["password"]); $role = preg_replace('#[^A-Za-z0-9]#i', '', $_SESSION["role"]); include "adminscripts/connect_to_mysql.php"; $sql = mysql_query("SELECT id FROM Users WHERE username='$manager' AND password='$password' AND role='$role' LIMIT 1"); $existCount = mysql_num_rows($sql); if ($existCount == 0) { header("location: http://www.zuluirminger.com/SchoolAdmin/index.php"); exit(); } ?> Just so you're aware, the database table has the following fields: id, username, password and role. Any help would be greatly appreciated! Many thanks, Zulu

  Read the article

• Upload File to Windows Azure Blob in Chunks through ASP.NET MVC, JavaScript and HTML5

  - by Shaun

  Originally posted on: http://geekswithblogs.net/shaunxu/archive/2013/07/01/upload-file-to-windows-azure-blob-in-chunks-through-asp.net.aspxMany people are using Windows Azure Blob Storage to store their data in the cloud. Blob storage provides 99.9% availability with easy-to-use API through .NET SDK and HTTP REST. For example, we can store JavaScript files, images, documents in blob storage when we are building an ASP.NET web application on a Web Role in Windows Azure. Or we can store our VHD files in blob and mount it as a hard drive in our cloud service. If you are familiar with Windows Azure, you should know that there are two kinds of blob: page blob and block blob. The page blob is optimized for random read and write, which is very useful when you need to store VHD files. The block blob is optimized for sequential/chunk read and write, which has more common usage. Since we can upload block blob in blocks through BlockBlob.PutBlock, and them commit them as a whole blob with invoking the BlockBlob.PutBlockList, it is very powerful to upload large files, as we can upload blocks in parallel, and provide pause-resume feature. There are many documents, articles and blog posts described on how to upload a block blob. Most of them are focus on the server side, which means when you had received a big file, stream or binaries, how to upload them into blob storage in blocks through .NET SDK.  But the problem is, how can we upload these large files from client side, for example, a browser. This questioned to me when I was working with a Chinese customer to help them build a network disk production on top of azure. The end users upload their files from the web portal, and then the files will be stored in blob storage from the Web Role. My goal is to find the best way to transform the file from client (end user’s machine) to the server (Web Role) through browser. In this post I will demonstrate and describe what I had done, to upload large file in chunks with high speed, and save them as blocks into Windows Azure Blob Storage.   Traditional Upload, Works with Limitation The simplest way to implement this requirement is to create a web page with a form that contains a file input element and a submit button. 1: @using (Html.BeginForm("About", "Index", FormMethod.Post, new { enctype = "multipart/form-data" })) 2: { 3: <input type="file" name="file" /> 4: <input type="submit" value="upload" /> 5: } And then in the backend controller, we retrieve the whole content of this file and upload it in to the blob storage through .NET SDK. We can split the file in blocks and upload them in parallel and commit. The code had been well blogged in the community. 1: [HttpPost] 2: public ActionResult About(HttpPostedFileBase file) 3: { 4: var container = _client.GetContainerReference("test"); 5: container.CreateIfNotExists(); 6: var blob = container.GetBlockBlobReference(file.FileName); 7: var blockDataList = new Dictionary<string, byte[]>(); 8: using (var stream = file.InputStream) 9: { 10: var blockSizeInKB = 1024; 11: var offset = 0; 12: var index = 0; 13: while (offset < stream.Length) 14: { 15: var readLength = Math.Min(1024 * blockSizeInKB, (int)stream.Length - offset); 16: var blockData = new byte[readLength]; 17: offset += stream.Read(blockData, 0, readLength); 18: blockDataList.Add(Convert.ToBase64String(BitConverter.GetBytes(index)), blockData); 19:  20: index++; 21: } 22: } 23:  24: Parallel.ForEach(blockDataList, (bi) => 25: { 26: blob.PutBlock(bi.Key, new MemoryStream(bi.Value), null); 27: }); 28: blob.PutBlockList(blockDataList.Select(b => b.Key).ToArray()); 29:  30: return RedirectToAction("About"); 31: } This works perfect if we selected an image, a music or a small video to upload. But if I selected a large file, let’s say a 6GB HD-movie, after upload for about few minutes the page will be shown as below and the upload will be terminated. In ASP.NET there is a limitation of request length and the maximized request length is defined in the web.config file. It’s a number which less than about 4GB. So if we want to upload a really big file, we cannot simply implement in this way. Also, in Windows Azure, a cloud service network load balancer will terminate the connection if exceed the timeout period. From my test the timeout looks like 2 - 3 minutes. Hence, when we need to upload a large file we cannot just use the basic HTML elements. Besides the limitation mentioned above, the simple HTML file upload cannot provide rich upload experience such as chunk upload, pause and pause-resume. So we need to find a better way to upload large file from the client to the server.   Upload in Chunks through HTML5 and JavaScript In order to break those limitation mentioned above we will try to upload the large file in chunks. This takes some benefit to us such as - No request size limitation: Since we upload in chunks, we can define the request size for each chunks regardless how big the entire file is. - No timeout problem: The size of chunks are controlled by us, which means we should be able to make sure request for each chunk upload will not exceed the timeout period of both ASP.NET and Windows Azure load balancer. It was a big challenge to upload big file in chunks until we have HTML5. There are some new features and improvements introduced in HTML5 and we will use them to implement our solution.   In HTML5, the File interface had been improved with a new method called “slice”. It can be used to read part of the file by specifying the start byte index and the end byte index. For example if the entire file was 1024 bytes, file.slice(512, 768) will read the part of this file from the 512nd byte to 768th byte, and return a new object of interface called "Blob”, which you can treat as an array of bytes. In fact,  a Blob object represents a file-like object of immutable, raw data. The File interface is based on Blob, inheriting blob functionality and expanding it to support files on the user's system. For more information about the Blob please refer here. File and Blob is very useful to implement the chunk upload. We will use File interface to represent the file the user selected from the browser and then use File.slice to read the file in chunks in the size we wanted. For example, if we wanted to upload a 10MB file with 512KB chunks, then we can read it in 512KB blobs by using File.slice in a loop.   Assuming we have a web page as below. User can select a file, an input box to specify the block size in KB and a button to start upload. 1: <div> 2: <input type="file" id="upload_files" name="files[]" /><br /> 3: Block Size: <input type="number" id="block_size" value="512" name="block_size" />KB<br /> 4: <input type="button" id="upload_button_blob" name="upload" value="upload (blob)" /> 5: </div> Then we can have the JavaScript function to upload the file in chunks when user clicked the button. 1: <script type="text/javascript"> 1: 2: $(function () { 3: $("#upload_button_blob").click(function () { 4: }); 5: });</script> Firstly we need to ensure the client browser supports the interfaces we are going to use. Just try to invoke the File, Blob and FormData from the “window” object. If any of them is “undefined” the condition result will be “false” which means your browser doesn’t support these premium feature and it’s time for you to get your browser updated. FormData is another new feature we are going to use in the future. It could generate a temporary form for us. We will use this interface to create a form with chunk and associated metadata when invoked the service through ajax. 1: $("#upload_button_blob").click(function () { 2: // assert the browser support html5 3: if (window.File && window.Blob && window.FormData) { 4: alert("Your brwoser is awesome, let's rock!"); 5: } 6: else { 7: alert("Oh man plz update to a modern browser before try is cool stuff out."); 8: return; 9: } 10: }); Each browser supports these interfaces by their own implementation and currently the Blob, File and File.slice are supported by Chrome 21, FireFox 13, IE 10, Opera 12 and Safari 5.1 or higher. After that we worked on the files the user selected one by one since in HTML5, user can select multiple files in one file input box. 1: var files = $("#upload_files")[0].files; 2: for (var i = 0; i < files.length; i++) { 3: var file = files[i]; 4: var fileSize = file.size; 5: var fileName = file.name; 6: } Next, we calculated the start index and end index for each chunks based on the size the user specified from the browser. We put them into an array with the file name and the index, which will be used when we upload chunks into Windows Azure Blob Storage as blocks since we need to specify the target blob name and the block index. At the same time we will store the list of all indexes into another variant which will be used to commit blocks into blob in Azure Storage once all chunks had been uploaded successfully. 1: $("#upload_button_blob").click(function () { 2: // assert the browser support html5 3: ... ... 4: // start to upload each files in chunks 5: var files = $("#upload_files")[0].files; 6: for (var i = 0; i < files.length; i++) { 7: var file = files[i]; 8: var fileSize = file.size; 9: var fileName = file.name; 10:  11: // calculate the start and end byte index for each blocks(chunks) 12: // with the index, file name and index list for future using 13: var blockSizeInKB = $("#block_size").val(); 14: var blockSize = blockSizeInKB * 1024; 15: var blocks = []; 16: var offset = 0; 17: var index = 0; 18: var list = ""; 19: while (offset < fileSize) { 20: var start = offset; 21: var end = Math.min(offset + blockSize, fileSize); 22:  23: blocks.push({ 24: name: fileName, 25: index: index, 26: start: start, 27: end: end 28: }); 29: list += index + ","; 30:  31: offset = end; 32: index++; 33: } 34: } 35: }); Now we have all chunks’ information ready. The next step should be upload them one by one to the server side, and at the server side when received a chunk it will upload as a block into Blob Storage, and finally commit them with the index list through BlockBlobClient.PutBlockList. But since all these invokes are ajax calling, which means not synchronized call. So we need to introduce a new JavaScript library to help us coordinate the asynchronize operation, which named “async.js”. You can download this JavaScript library here, and you can find the document here. I will not explain this library too much in this post. We will put all procedures we want to execute as a function array, and pass into the proper function defined in async.js to let it help us to control the execution sequence, in series or in parallel. Hence we will define an array and put the function for chunk upload into this array. 1: $("#upload_button_blob").click(function () { 2: // assert the browser support html5 3: ... ... 4:  5: // start to upload each files in chunks 6: var files = $("#upload_files")[0].files; 7: for (var i = 0; i < files.length; i++) { 8: var file = files[i]; 9: var fileSize = file.size; 10: var fileName = file.name; 11: // calculate the start and end byte index for each blocks(chunks) 12: // with the index, file name and index list for future using 13: ... ... 14:  15: // define the function array and push all chunk upload operation into this array 16: blocks.forEach(function (block) { 17: putBlocks.push(function (callback) { 18: }); 19: }); 20: } 21: }); 22: }); As you can see, I used File.slice method to read each chunks based on the start and end byte index we calculated previously, and constructed a temporary HTML form with the file name, chunk index and chunk data through another new feature in HTML5 named FormData. Then post this form to the backend server through jQuery.ajax. This is the key part of our solution. 1: $("#upload_button_blob").click(function () { 2: // assert the browser support html5 3: ... ... 4: // start to upload each files in chunks 5: var files = $("#upload_files")[0].files; 6: for (var i = 0; i < files.length; i++) { 7: var file = files[i]; 8: var fileSize = file.size; 9: var fileName = file.name; 10: // calculate the start and end byte index for each blocks(chunks) 11: // with the index, file name and index list for future using 12: ... ... 13: // define the function array and push all chunk upload operation into this array 14: blocks.forEach(function (block) { 15: putBlocks.push(function (callback) { 16: // load blob based on the start and end index for each chunks 17: var blob = file.slice(block.start, block.end); 18: // put the file name, index and blob into a temporary from 19: var fd = new FormData(); 20: fd.append("name", block.name); 21: fd.append("index", block.index); 22: fd.append("file", blob); 23: // post the form to backend service (asp.net mvc controller action) 24: $.ajax({ 25: url: "/Home/UploadInFormData", 26: data: fd, 27: processData: false, 28: contentType: "multipart/form-data", 29: type: "POST", 30: success: function (result) { 31: if (!result.success) { 32: alert(result.error); 33: } 34: callback(null, block.index); 35: } 36: }); 37: }); 38: }); 39: } 40: }); Then we will invoke these functions one by one by using the async.js. And once all functions had been executed successfully I invoked another ajax call to the backend service to commit all these chunks (blocks) as the blob in Windows Azure Storage. 1: $("#upload_button_blob").click(function () { 2: // assert the browser support html5 3: ... ... 4: // start to upload each files in chunks 5: var files = $("#upload_files")[0].files; 6: for (var i = 0; i < files.length; i++) { 7: var file = files[i]; 8: var fileSize = file.size; 9: var fileName = file.name; 10: // calculate the start and end byte index for each blocks(chunks) 11: // with the index, file name and index list for future using 12: ... ... 13: // define the function array and push all chunk upload operation into this array 14: ... ... 15: // invoke the functions one by one 16: // then invoke the commit ajax call to put blocks into blob in azure storage 17: async.series(putBlocks, function (error, result) { 18: var data = { 19: name: fileName, 20: list: list 21: }; 22: $.post("/Home/Commit", data, function (result) { 23: if (!result.success) { 24: alert(result.error); 25: } 26: else { 27: alert("done!"); 28: } 29: }); 30: }); 31: } 32: }); That’s all in the client side. The outline of our logic would be - Calculate the start and end byte index for each chunks based on the block size. - Defined the functions of reading the chunk form file and upload the content to the backend service through ajax. - Execute the functions defined in previous step with “async.js”. - Commit the chunks by invoking the backend service in Windows Azure Storage finally.   Save Chunks as Blocks into Blob Storage In above we finished the client size JavaScript code. It uploaded the file in chunks to the backend service which we are going to implement in this step. We will use ASP.NET MVC as our backend service, and it will receive the chunks, upload into Windows Azure Bob Storage in blocks, then finally commit as one blob. As in the client side we uploaded chunks by invoking the ajax call to the URL "/Home/UploadInFormData", I created a new action under the Index controller and it only accepts HTTP POST request. 1: [HttpPost] 2: public JsonResult UploadInFormData() 3: { 4: var error = string.Empty; 5: try 6: { 7: } 8: catch (Exception e) 9: { 10: error = e.ToString(); 11: } 12:  13: return new JsonResult() 14: { 15: Data = new 16: { 17: success = string.IsNullOrWhiteSpace(error), 18: error = error 19: } 20: }; 21: } Then I retrieved the file name, index and the chunk content from the Request.Form object, which was passed from our client side. And then, used the Windows Azure SDK to create a blob container (in this case we will use the container named “test”.) and create a blob reference with the blob name (same as the file name). Then uploaded the chunk as a block of this blob with the index, since in Blob Storage each block must have an index (ID) associated with so that finally we can put all blocks as one blob by specifying their block ID list. 1: [HttpPost] 2: public JsonResult UploadInFormData() 3: { 4: var error = string.Empty; 5: try 6: { 7: var name = Request.Form["name"]; 8: var index = int.Parse(Request.Form["index"]); 9: var file = Request.Files[0]; 10: var id = Convert.ToBase64String(BitConverter.GetBytes(index)); 11:  12: var container = _client.GetContainerReference("test"); 13: container.CreateIfNotExists(); 14: var blob = container.GetBlockBlobReference(name); 15: blob.PutBlock(id, file.InputStream, null); 16: } 17: catch (Exception e) 18: { 19: error = e.ToString(); 20: } 21:  22: return new JsonResult() 23: { 24: Data = new 25: { 26: success = string.IsNullOrWhiteSpace(error), 27: error = error 28: } 29: }; 30: } Next, I created another action to commit the blocks into blob once all chunks had been uploaded. Similarly, I retrieved the blob name from the Request.Form. I also retrieved the chunks ID list, which is the block ID list from the Request.Form in a string format, split them as a list, then invoked the BlockBlob.PutBlockList method. After that our blob will be shown in the container and ready to be download. 1: [HttpPost] 2: public JsonResult Commit() 3: { 4: var error = string.Empty; 5: try 6: { 7: var name = Request.Form["name"]; 8: var list = Request.Form["list"]; 9: var ids = list 10: .Split(',') 11: .Where(id => !string.IsNullOrWhiteSpace(id)) 12: .Select(id => Convert.ToBase64String(BitConverter.GetBytes(int.Parse(id)))) 13: .ToArray(); 14:  15: var container = _client.GetContainerReference("test"); 16: container.CreateIfNotExists(); 17: var blob = container.GetBlockBlobReference(name); 18: blob.PutBlockList(ids); 19: } 20: catch (Exception e) 21: { 22: error = e.ToString(); 23: } 24:  25: return new JsonResult() 26: { 27: Data = new 28: { 29: success = string.IsNullOrWhiteSpace(error), 30: error = error 31: } 32: }; 33: } Now we finished all code we need. The whole process of uploading would be like this below. Below is the full client side JavaScript code. 1: <script type="text/javascript" src="~/Scripts/async.js"></script> 2: <script type="text/javascript"> 3: $(function () { 4: $("#upload_button_blob").click(function () { 5: // assert the browser support html5 6: if (window.File && window.Blob && window.FormData) { 7: alert("Your brwoser is awesome, let's rock!"); 8: } 9: else { 10: alert("Oh man plz update to a modern browser before try is cool stuff out."); 11: return; 12: } 13:  14: // start to upload each files in chunks 15: var files = $("#upload_files")[0].files; 16: for (var i = 0; i < files.length; i++) { 17: var file = files[i]; 18: var fileSize = file.size; 19: var fileName = file.name; 20:  21: // calculate the start and end byte index for each blocks(chunks) 22: // with the index, file name and index list for future using 23: var blockSizeInKB = $("#block_size").val(); 24: var blockSize = blockSizeInKB * 1024; 25: var blocks = []; 26: var offset = 0; 27: var index = 0; 28: var list = ""; 29: while (offset < fileSize) { 30: var start = offset; 31: var end = Math.min(offset + blockSize, fileSize); 32:  33: blocks.push({ 34: name: fileName, 35: index: index, 36: start: start, 37: end: end 38: }); 39: list += index + ","; 40:  41: offset = end; 42: index++; 43: } 44:  45: // define the function array and push all chunk upload operation into this array 46: var putBlocks = []; 47: blocks.forEach(function (block) { 48: putBlocks.push(function (callback) { 49: // load blob based on the start and end index for each chunks 50: var blob = file.slice(block.start, block.end); 51: // put the file name, index and blob into a temporary from 52: var fd = new FormData(); 53: fd.append("name", block.name); 54: fd.append("index", block.index); 55: fd.append("file", blob); 56: // post the form to backend service (asp.net mvc controller action) 57: $.ajax({ 58: url: "/Home/UploadInFormData", 59: data: fd, 60: processData: false, 61: contentType: "multipart/form-data", 62: type: "POST", 63: success: function (result) { 64: if (!result.success) { 65: alert(result.error); 66: } 67: callback(null, block.index); 68: } 69: }); 70: }); 71: }); 72:  73: // invoke the functions one by one 74: // then invoke the commit ajax call to put blocks into blob in azure storage 75: async.series(putBlocks, function (error, result) { 76: var data = { 77: name: fileName, 78: list: list 79: }; 80: $.post("/Home/Commit", data, function (result) { 81: if (!result.success) { 82: alert(result.error); 83: } 84: else { 85: alert("done!"); 86: } 87: }); 88: }); 89: } 90: }); 91: }); 92: </script> And below is the full ASP.NET MVC controller code. 1: public class HomeController : Controller 2: { 3: private CloudStorageAccount _account; 4: private CloudBlobClient _client; 5:  6: public HomeController() 7: : base() 8: { 9: _account = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("DataConnectionString")); 10: _client = _account.CreateCloudBlobClient(); 11: } 12:  13: public ActionResult Index() 14: { 15: ViewBag.Message = "Modify this template to jump-start your ASP.NET MVC application."; 16:  17: return View(); 18: } 19:  20: [HttpPost] 21: public JsonResult UploadInFormData() 22: { 23: var error = string.Empty; 24: try 25: { 26: var name = Request.Form["name"]; 27: var index = int.Parse(Request.Form["index"]); 28: var file = Request.Files[0]; 29: var id = Convert.ToBase64String(BitConverter.GetBytes(index)); 30:  31: var container = _client.GetContainerReference("test"); 32: container.CreateIfNotExists(); 33: var blob = container.GetBlockBlobReference(name); 34: blob.PutBlock(id, file.InputStream, null); 35: } 36: catch (Exception e) 37: { 38: error = e.ToString(); 39: } 40:  41: return new JsonResult() 42: { 43: Data = new 44: { 45: success = string.IsNullOrWhiteSpace(error), 46: error = error 47: } 48: }; 49: } 50:  51: [HttpPost] 52: public JsonResult Commit() 53: { 54: var error = string.Empty; 55: try 56: { 57: var name = Request.Form["name"]; 58: var list = Request.Form["list"]; 59: var ids = list 60: .Split(',') 61: .Where(id => !string.IsNullOrWhiteSpace(id)) 62: .Select(id => Convert.ToBase64String(BitConverter.GetBytes(int.Parse(id)))) 63: .ToArray(); 64:  65: var container = _client.GetContainerReference("test"); 66: container.CreateIfNotExists(); 67: var blob = container.GetBlockBlobReference(name); 68: blob.PutBlockList(ids); 69: } 70: catch (Exception e) 71: { 72: error = e.ToString(); 73: } 74:  75: return new JsonResult() 76: { 77: Data = new 78: { 79: success = string.IsNullOrWhiteSpace(error), 80: error = error 81: } 82: }; 83: } 84: } And if we selected a file from the browser we will see our application will upload chunks in the size we specified to the server through ajax call in background, and then commit all chunks in one blob. Then we can find the blob in our Windows Azure Blob Storage.   Optimized by Parallel Upload In previous example we just uploaded our file in chunks. This solved the problem that ASP.NET MVC request content size limitation as well as the Windows Azure load balancer timeout. But it might introduce the performance problem since we uploaded chunks in sequence. In order to improve the upload performance we could modify our client side code a bit to make the upload operation invoked in parallel. The good news is that, “async.js” library provides the parallel execution function. If you remembered the code we invoke the service to upload chunks, it utilized “async.series” which means all functions will be executed in sequence. Now we will change this code to “async.parallel”. This will invoke all functions in parallel. 1: $("#upload_button_blob").click(function () { 2: // assert the browser support html5 3: ... ... 4: // start to upload each files in chunks 5: var files = $("#upload_files")[0].files; 6: for (var i = 0; i < files.length; i++) { 7: var file = files[i]; 8: var fileSize = file.size; 9: var fileName = file.name; 10: // calculate the start and end byte index for each blocks(chunks) 11: // with the index, file name and index list for future using 12: ... ... 13: // define the function array and push all chunk upload operation into this array 14: ... ... 15: // invoke the functions one by one 16: // then invoke the commit ajax call to put blocks into blob in azure storage 17: async.parallel(putBlocks, function (error, result) { 18: var data = { 19: name: fileName, 20: list: list 21: }; 22: $.post("/Home/Commit", data, function (result) { 23: if (!result.success) { 24: alert(result.error); 25: } 26: else { 27: alert("done!"); 28: } 29: }); 30: }); 31: } 32: }); In this way all chunks will be uploaded to the server side at the same time to maximize the bandwidth usage. This should work if the file was not very large and the chunk size was not very small. But for large file this might introduce another problem that too many ajax calls are sent to the server at the same time. So the best solution should be, upload the chunks in parallel with maximum concurrency limitation. The code below specified the concurrency limitation to 4, which means at the most only 4 ajax calls could be invoked at the same time. 1: $("#upload_button_blob").click(function () { 2: // assert the browser support html5 3: ... ... 4: // start to upload each files in chunks 5: var files = $("#upload_files")[0].files; 6: for (var i = 0; i < files.length; i++) { 7: var file = files[i]; 8: var fileSize = file.size; 9: var fileName = file.name; 10: // calculate the start and end byte index for each blocks(chunks) 11: // with the index, file name and index list for future using 12: ... ... 13: // define the function array and push all chunk upload operation into this array 14: ... ... 15: // invoke the functions one by one 16: // then invoke the commit ajax call to put blocks into blob in azure storage 17: async.parallelLimit(putBlocks, 4, function (error, result) { 18: var data = { 19: name: fileName, 20: list: list 21: }; 22: $.post("/Home/Commit", data, function (result) { 23: if (!result.success) { 24: alert(result.error); 25: } 26: else { 27: alert("done!"); 28: } 29: }); 30: }); 31: } 32: });   Summary In this post we discussed how to upload files in chunks to the backend service and then upload them into Windows Azure Blob Storage in blocks. We focused on the frontend side and leverage three new feature introduced in HTML 5 which are - File.slice: Read part of the file by specifying the start and end byte index. - Blob: File-like interface which contains the part of the file content. - FormData: Temporary form element that we can pass the chunk alone with some metadata to the backend service. Then we discussed the performance consideration of chunk uploading. Sequence upload cannot provide maximized upload speed, but the unlimited parallel upload might crash the browser and server if too many chunks. So we finally came up with the solution to upload chunks in parallel with the concurrency limitation. We also demonstrated how to utilize “async.js” JavaScript library to help us control the asynchronize call and the parallel limitation.   Regarding the chunk size and the parallel limitation value there is no “best” value. You need to test vary composition and find out the best one for your particular scenario. It depends on the local bandwidth, client machine cores and the server side (Windows Azure Cloud Service Virtual Machine) cores, memory and bandwidth. Below is one of my performance test result. The client machine was Windows 8 IE 10 with 4 cores. I was using Microsoft Cooperation Network. The web site was hosted on Windows Azure China North data center (in Beijing) with one small web role (1.7GB 1 core CPU, 1.75GB memory with 100Mbps bandwidth). The test cases were - Chunk size: 512KB, 1MB, 2MB, 4MB. - Upload Mode: Sequence, parallel (unlimited), parallel with limit (4 threads, 8 threads). - Chunk Format: base64 string, binaries. - Target file: 100MB. - Each case was tested 3 times. Below is the test result chart. Some thoughts, but not guidance or best practice: - Parallel gets better performance than series. - No significant performance improvement between parallel 4 threads and 8 threads. - Transform with binaries provides better performance than base64. - In all cases, chunk size in 1MB - 2MB gets better performance.   Hope this helps, Shaun All documents and related graphics, codes are provided "AS IS" without warranty of any kind. Copyright © Shaun Ziyan Xu. This work is licensed under the Creative Commons License.

  Read the article

• Node.js Adventure - When Node Flying in Wind

  - by Shaun

  In the first post of this series I mentioned some popular modules in the community, such as underscore, async, etc.. I also listed a module named “Wind (zh-CN)”, which is created by one of my friend, Jeff Zhao (zh-CN). Now I would like to use a separated post to introduce this module since I feel it brings a new async programming style in not only Node.js but JavaScript world. If you know or heard about the new feature in C# 5.0 called “async and await”, or you learnt F#, you will find the “Wind” brings the similar async programming experience in JavaScript. By using “Wind”, we can write async code that looks like the sync code. The callbacks, async stats and exceptions will be handled by “Wind” automatically and transparently.   What’s the Problem: Dense “Callback” Phobia Let’s firstly back to my second post in this series. As I mentioned in that post, when we wanted to read some records from SQL Server we need to open the database connection, and then execute the query. In Node.js all IO operation are designed as async callback pattern which means when the operation was done, it will invoke a function which was taken from the last parameter. For example the database connection opening code would be like this. 1: sql.open(connectionString, function(error, conn) { 2: if(error) { 3: // some error handling code 4: } 5: else { 6: // connection opened successfully 7: } 8: }); And then if we need to query the database the code would be like this. It nested in the previous function. 1: sql.open(connectionString, function(error, conn) { 2: if(error) { 3: // some error handling code 4: } 5: else { 6: // connection opened successfully 7: conn.queryRaw(command, function(error, results) { 8: if(error) { 9: // failed to execute this command 10: } 11: else { 12: // records retrieved successfully 13: } 14: }; 15: } 16: }); Assuming if we need to copy some data from this database to another then we need to open another connection and execute the command within the function under the query function. 1: sql.open(connectionString, function(error, conn) { 2: if(error) { 3: // some error handling code 4: } 5: else { 6: // connection opened successfully 7: conn.queryRaw(command, function(error, results) { 8: if(error) { 9: // failed to execute this command 10: } 11: else { 12: // records retrieved successfully 13: target.open(targetConnectionString, function(error, t_conn) { 14: if(error) { 15: // connect failed 16: } 17: else { 18: t_conn.queryRaw(copy_command, function(error, results) { 19: if(error) { 20: // copy failed 21: } 22: else { 23: // and then, what do you want to do now... 24: } 25: }; 26: } 27: }; 28: } 29: }; 30: } 31: }); This is just an example. In the real project the logic would be more complicated. This means our application might be messed up and the business process will be fragged by many callback functions. I would like call this “Dense Callback Phobia”. This might be a challenge how to make code straightforward and easy to read, something like below. 1: try 2: { 3: // open source connection 4: var s_conn = sqlConnect(s_connectionString); 5: // retrieve data 6: var results = sqlExecuteCommand(s_conn, s_command); 7: 8: // open target connection 9: var t_conn = sqlConnect(t_connectionString); 10: // prepare the copy command 11: var t_command = getCopyCommand(results); 12: // execute the copy command 13: sqlExecuteCommand(s_conn, t_command); 14: } 15: catch (ex) 16: { 17: // error handling 18: }   What’s the Problem: Sync-styled Async Programming Similar as the previous problem, the callback-styled async programming model makes the upcoming operation as a part of the current operation, and mixed with the error handling code. So it’s very hard to understand what on earth this code will do. And since Node.js utilizes non-blocking IO mode, we cannot invoke those operations one by one, as they will be executed concurrently. For example, in this post when I tried to copy the records from Windows Azure SQL Database (a.k.a. WASD) to Windows Azure Table Storage, if I just insert the data into table storage one by one and then print the “Finished” message, I will see the message shown before the data had been copied. This is because all operations were executed at the same time. In order to make the copy operation and print operation executed synchronously I introduced a module named “async” and the code was changed as below. 1: async.forEach(results.rows, 2: function (row, callback) { 3: var resource = { 4: "PartitionKey": row[1], 5: "RowKey": row[0], 6: "Value": row[2] 7: }; 8: client.insertEntity(tableName, resource, function (error) { 9: if (error) { 10: callback(error); 11: } 12: else { 13: console.log("entity inserted."); 14: callback(null); 15: } 16: }); 17: }, 18: function (error) { 19: if (error) { 20: error["target"] = "insertEntity"; 21: res.send(500, error); 22: } 23: else { 24: console.log("all done."); 25: res.send(200, "Done!"); 26: } 27: }); It ensured that the “Finished” message will be printed when all table entities had been inserted. But it cannot promise that the records will be inserted in sequence. It might be another challenge to make the code looks like in sync-style? 1: try 2: { 3: forEach(row in rows) { 4: var entity = { /* ... */ }; 5: tableClient.insert(tableName, entity); 6: } 7:  8: console.log("Finished"); 9: } 10: catch (ex) { 11: console.log(ex); 12: }   How “Wind” Helps “Wind” is a JavaScript library which provides the control flow with plain JavaScript for asynchronous programming (and more) without additional pre-compiling steps. It’s available in NPM so that we can install it through “npm install wind”. Now let’s create a very simple Node.js application as the example. This application will take some website URLs from the command arguments and tried to retrieve the body length and print them in console. Then at the end print “Finish”. I’m going to use “request” module to make the HTTP call simple so I also need to install by the command “npm install request”. The code would be like this. 1: var request = require("request"); 2:  3: // get the urls from arguments, the first two arguments are `node.exe` and `fetch.js` 4: var args = process.argv.splice(2); 5:  6: // main function 7: var main = function() { 8: for(var i = 0; i < args.length; i++) { 9: // get the url 10: var url = args[i]; 11: // send the http request and try to get the response and body 12: request(url, function(error, response, body) { 13: if(!error && response.statusCode == 200) { 14: // log the url and the body length 15: console.log( 16: "%s: %d.", 17: response.request.uri.href, 18: body.length); 19: } 20: else { 21: // log error 22: console.log(error); 23: } 24: }); 25: } 26: 27: // finished 28: console.log("Finished"); 29: }; 30:  31: // execute the main function 32: main(); Let’s execute this application. (I made them in multi-lines for better reading.) 1: node fetch.js 2: "http://www.igt.com/us-en.aspx" 3: "http://www.igt.com/us-en/games.aspx" 4: "http://www.igt.com/us-en/cabinets.aspx" 5: "http://www.igt.com/us-en/systems.aspx" 6: "http://www.igt.com/us-en/interactive.aspx" 7: "http://www.igt.com/us-en/social-gaming.aspx" 8: "http://www.igt.com/support.aspx" Below is the output. As you can see the finish message was printed at the beginning, and the pages’ length retrieved in a different order than we specified. This is because in this code the request command, console logging command are executed asynchronously and concurrently. Now let’s introduce “Wind” to make them executed in order, which means it will request the websites one by one, and print the message at the end.   First of all we need to import the “Wind” package and make sure the there’s only one global variant named “Wind”, and ensure it’s “Wind” instead of “wind”. 1: var Wind = require("wind");   Next, we need to tell “Wind” which code will be executed asynchronously so that “Wind” can control the execution process. In this case the “request” operation executed asynchronously so we will create a “Task” by using a build-in helps function in “Wind” named Wind.Async.Task.create. 1: var requestBodyLengthAsync = function(url) { 2: return Wind.Async.Task.create(function(t) { 3: request(url, function(error, response, body) { 4: if(error || response.statusCode != 200) { 5: t.complete("failure", error); 6: } 7: else { 8: var data = 9: { 10: uri: response.request.uri.href, 11: length: body.length 12: }; 13: t.complete("success", data); 14: } 15: }); 16: }); 17: }; The code above created a “Task” from the original request calling code. In “Wind” a “Task” means an operation will be finished in some time in the future. A “Task” can be started by invoke its start() method, but no one knows when it actually will be finished. The Wind.Async.Task.create helped us to create a task. The only parameter is a function where we can put the actual operation in, and then notify the task object it’s finished successfully or failed by using the complete() method. In the code above I invoked the request method. If it retrieved the response successfully I set the status of this task as “success” with the URL and body length. If it failed I set this task as “failure” and pass the error out.   Next, we will change the main() function. In “Wind” if we want a function can be controlled by Wind we need to mark it as “async”. This should be done by using the code below. 1: var main = eval(Wind.compile("async", function() { 2: })); When the application is running, Wind will detect “eval(Wind.compile(“async”, function” and generate an anonymous code from the body of this original function. Then the application will run the anonymous code instead of the original one. In our example the main function will be like this. 1: var main = eval(Wind.compile("async", function() { 2: for(var i = 0; i < args.length; i++) { 3: try 4: { 5: var result = $await(requestBodyLengthAsync(args[i])); 6: console.log( 7: "%s: %d.", 8: result.uri, 9: result.length); 10: } 11: catch (ex) { 12: console.log(ex); 13: } 14: } 15: 16: console.log("Finished"); 17: })); As you can see, when I tried to request the URL I use a new command named “$await”. It tells Wind, the operation next to $await will be executed asynchronously, and the main thread should be paused until it finished (or failed). So in this case, my application will be pause when the first response was received, and then print its body length, then try the next one. At the end, print the finish message.   Finally, execute the main function. The full code would be like this. 1: var request = require("request"); 2: var Wind = require("wind"); 3:  4: var args = process.argv.splice(2); 5:  6: var requestBodyLengthAsync = function(url) { 7: return Wind.Async.Task.create(function(t) { 8: request(url, function(error, response, body) { 9: if(error || response.statusCode != 200) { 10: t.complete("failure", error); 11: } 12: else { 13: var data = 14: { 15: uri: response.request.uri.href, 16: length: body.length 17: }; 18: t.complete("success", data); 19: } 20: }); 21: }); 22: }; 23:  24: var main = eval(Wind.compile("async", function() { 25: for(var i = 0; i < args.length; i++) { 26: try 27: { 28: var result = $await(requestBodyLengthAsync(args[i])); 29: console.log( 30: "%s: %d.", 31: result.uri, 32: result.length); 33: } 34: catch (ex) { 35: console.log(ex); 36: } 37: } 38: 39: console.log("Finished"); 40: })); 41:  42: main().start();   Run our new application. At the beginning we will see the compiled and generated code by Wind. Then we can see the pages were requested one by one, and at the end the finish message was printed. Below is the code Wind generated for us. As you can see the original code, the output code were shown. 1: // Original: 2: function () { 3: for(var i = 0; i < args.length; i++) { 4: try 5: { 6: var result = $await(requestBodyLengthAsync(args[i])); 7: console.log( 8: "%s: %d.", 9: result.uri, 10: result.length); 11: } 12: catch (ex) { 13: console.log(ex); 14: } 15: } 16: 17: console.log("Finished"); 18: } 19:  20: // Compiled: 21: /* async << function () { */ (function () { 22: var _builder_$0 = Wind.builders["async"]; 23: return _builder_$0.Start(this, 24: _builder_$0.Combine( 25: _builder_$0.Delay(function () { 26: /* var i = 0; */ var i = 0; 27: /* for ( */ return _builder_$0.For(function () { 28: /* ; i < args.length */ return i < args.length; 29: }, function () { 30: /* ; i ++) { */ i ++; 31: }, 32: /* try { */ _builder_$0.Try( 33: _builder_$0.Delay(function () { 34: /* var result = $await(requestBodyLengthAsync(args[i])); */ return _builder_$0.Bind(requestBodyLengthAsync(args[i]), function (result) { 35: /* console.log("%s: %d.", result.uri, result.length); */ console.log("%s: %d.", result.uri, result.length); 36: return _builder_$0.Normal(); 37: }); 38: }), 39: /* } catch (ex) { */ function (ex) { 40: /* console.log(ex); */ console.log(ex); 41: return _builder_$0.Normal(); 42: /* } */ }, 43: null 44: ) 45: /* } */ ); 46: }), 47: _builder_$0.Delay(function () { 48: /* console.log("Finished"); */ console.log("Finished"); 49: return _builder_$0.Normal(); 50: }) 51: ) 52: ); 53: /* } */ })   How Wind Works Someone may raise a big concern when you find I utilized “eval” in my code. Someone may assume that Wind utilizes “eval” to execute some code dynamically while “eval” is very low performance. But I would say, Wind does NOT use “eval” to run the code. It only use “eval” as a flag to know which code should be compiled at runtime. When the code was firstly been executed, Wind will check and find “eval(Wind.compile(“async”, function”. So that it knows this function should be compiled. Then it utilized parse-js to analyze the inner JavaScript and generated the anonymous code in memory. Then it rewrite the original code so that when the application was running it will use the anonymous one instead of the original one. Since the code generation was done at the beginning of the application was started, in the future no matter how long our application runs and how many times the async function was invoked, it will use the generated code, no need to generate again. So there’s no significant performance hurt when using Wind.   Wind in My Previous Demo Let’s adopt Wind into one of my previous demonstration and to see how it helps us to make our code simple, straightforward and easy to read and understand. In this post when I implemented the functionality that copied the records from my WASD to table storage, the logic would be like this. 1, Open database connection. 2, Execute a query to select all records from the table. 3, Recreate the table in Windows Azure table storage. 4, Create entities from each of the records retrieved previously, and then insert them into table storage. 5, Finally, show message as the HTTP response. But as the image below, since there are so many callbacks and async operations, it’s very hard to understand my logic from the code. Now let’s use Wind to rewrite our code. First of all, of course, we need the Wind package. Then we need to include the package files into project and mark them as “Copy always”. Add the Wind package into the source code. Pay attention to the variant name, you must use “Wind” instead of “wind”. 1: var express = require("express"); 2: var async = require("async"); 3: var sql = require("node-sqlserver"); 4: var azure = require("azure"); 5: var Wind = require("wind"); Now we need to create some async functions by using Wind. All async functions should be wrapped so that it can be controlled by Wind which are open database, retrieve records, recreate table (delete and create) and insert entity in table. Below are these new functions. All of them are created by using Wind.Async.Task.create. 1: sql.openAsync = function (connectionString) { 2: return Wind.Async.Task.create(function (t) { 3: sql.open(connectionString, function (error, conn) { 4: if (error) { 5: t.complete("failure", error); 6: } 7: else { 8: t.complete("success", conn); 9: } 10: }); 11: }); 12: }; 13:  14: sql.queryAsync = function (conn, query) { 15: return Wind.Async.Task.create(function (t) { 16: conn.queryRaw(query, function (error, results) { 17: if (error) { 18: t.complete("failure", error); 19: } 20: else { 21: t.complete("success", results); 22: } 23: }); 24: }); 25: }; 26:  27: azure.recreateTableAsync = function (tableName) { 28: return Wind.Async.Task.create(function (t) { 29: client.deleteTable(tableName, function (error, successful, response) { 30: console.log("delete table finished"); 31: client.createTableIfNotExists(tableName, function (error, successful, response) { 32: console.log("create table finished"); 33: if (error) { 34: t.complete("failure", error); 35: } 36: else { 37: t.complete("success", null); 38: } 39: }); 40: }); 41: }); 42: }; 43:  44: azure.insertEntityAsync = function (tableName, entity) { 45: return Wind.Async.Task.create(function (t) { 46: client.insertEntity(tableName, entity, function (error, entity, response) { 47: if (error) { 48: t.complete("failure", error); 49: } 50: else { 51: t.complete("success", null); 52: } 53: }); 54: }); 55: }; Then in order to use these functions we will create a new function which contains all steps for data copying. 1: var copyRecords = eval(Wind.compile("async", function (req, res) { 2: try { 3: } 4: catch (ex) { 5: console.log(ex); 6: res.send(500, "Internal error."); 7: } 8: })); Let’s execute steps one by one with the “$await” keyword introduced by Wind so that it will be invoked in sequence. First is to open the database connection. 1: var copyRecords = eval(Wind.compile("async", function (req, res) { 2: try { 3: // connect to the windows azure sql database 4: var conn = $await(sql.openAsync(connectionString)); 5: console.log("connection opened"); 6: } 7: catch (ex) { 8: console.log(ex); 9: res.send(500, "Internal error."); 10: } 11: })); Then retrieve all records from the database connection. 1: var copyRecords = eval(Wind.compile("async", function (req, res) { 2: try { 3: // connect to the windows azure sql database 4: var conn = $await(sql.openAsync(connectionString)); 5: console.log("connection opened"); 6: // retrieve all records from database 7: var results = $await(sql.queryAsync(conn, "SELECT * FROM [Resource]")); 8: console.log("records selected. count = %d", results.rows.length); 9: } 10: catch (ex) { 11: console.log(ex); 12: res.send(500, "Internal error."); 13: } 14: })); After recreated the table, we need to create the entities and insert them into table storage. 1: var copyRecords = eval(Wind.compile("async", function (req, res) { 2: try { 3: // connect to the windows azure sql database 4: var conn = $await(sql.openAsync(connectionString)); 5: console.log("connection opened"); 6: // retrieve all records from database 7: var results = $await(sql.queryAsync(conn, "SELECT * FROM [Resource]")); 8: console.log("records selected. count = %d", results.rows.length); 9: if (results.rows.length > 0) { 10: // recreate the table 11: $await(azure.recreateTableAsync(tableName)); 12: console.log("table created"); 13: // insert records in table storage one by one 14: for (var i = 0; i < results.rows.length; i++) { 15: var entity = { 16: "PartitionKey": results.rows[i][1], 17: "RowKey": results.rows[i][0], 18: "Value": results.rows[i][2] 19: }; 20: $await(azure.insertEntityAsync(tableName, entity)); 21: console.log("entity inserted"); 22: } 23: } 24: } 25: catch (ex) { 26: console.log(ex); 27: res.send(500, "Internal error."); 28: } 29: })); Finally, send response back to the browser. 1: var copyRecords = eval(Wind.compile("async", function (req, res) { 2: try { 3: // connect to the windows azure sql database 4: var conn = $await(sql.openAsync(connectionString)); 5: console.log("connection opened"); 6: // retrieve all records from database 7: var results = $await(sql.queryAsync(conn, "SELECT * FROM [Resource]")); 8: console.log("records selected. count = %d", results.rows.length); 9: if (results.rows.length > 0) { 10: // recreate the table 11: $await(azure.recreateTableAsync(tableName)); 12: console.log("table created"); 13: // insert records in table storage one by one 14: for (var i = 0; i < results.rows.length; i++) { 15: var entity = { 16: "PartitionKey": results.rows[i][1], 17: "RowKey": results.rows[i][0], 18: "Value": results.rows[i][2] 19: }; 20: $await(azure.insertEntityAsync(tableName, entity)); 21: console.log("entity inserted"); 22: } 23: // send response 24: console.log("all done"); 25: res.send(200, "All done!"); 26: } 27: } 28: catch (ex) { 29: console.log(ex); 30: res.send(500, "Internal error."); 31: } 32: })); If we compared with the previous code we will find now it became more readable and much easy to understand. It’s very easy to know what this function does even though without any comments. When user go to URL “/was/copyRecords” we will execute the function above. The code would be like this. 1: app.get("/was/copyRecords", function (req, res) { 2: copyRecords(req, res).start(); 3: }); And below is the logs printed in local compute emulator console. As we can see the functions executed one by one and then finally the response back to me browser.   Scaffold Functions in Wind Wind provides not only the async flow control and compile functions, but many scaffold methods as well. We can build our async code more easily by using them. I’m going to introduce some basic scaffold functions here. In the code above I created some functions which wrapped from the original async function such as open database, create table, etc.. All of them are very similar, created a task by using Wind.Async.Task.create, return error or result object through Task.complete function. In fact, Wind provides some functions for us to create task object from the original async functions. If the original async function only has a callback parameter, we can use Wind.Async.Binding.fromCallback method to get the task object directly. For example the code below returned the task object which wrapped the file exist check function. 1: var Wind = require("wind"); 2: var fs = require("fs"); 3:  4: fs.existsAsync = Wind.Async.Binding.fromCallback(fs.exists); In Node.js a very popular async function pattern is that, the first parameter in the callback function represent the error object, and the other parameters is the return values. In this case we can use another build-in function in Wind named Wind.Async.Binding.fromStandard. For example, the open database function can be created from the code below. 1: sql.openAsync = Wind.Async.Binding.fromStandard(sql.open); 2:  3: /* 4: sql.openAsync = function (connectionString) { 5: return Wind.Async.Task.create(function (t) { 6: sql.open(connectionString, function (error, conn) { 7: if (error) { 8: t.complete("failure", error); 9: } 10: else { 11: t.complete("success", conn); 12: } 13: }); 14: }); 15: }; 16: */ When I was testing the scaffold functions under Wind.Async.Binding I found for some functions, such as the Azure SDK insert entity function, cannot be processed correctly. So I personally suggest writing the wrapped method manually.   Another scaffold method in Wind is the parallel tasks coordination. In this example, the steps of open database, retrieve records and recreated table should be invoked one by one, but it can be executed in parallel when copying data from database to table storage. In Wind there’s a scaffold function named Task.whenAll which can be used here. Task.whenAll accepts a list of tasks and creates a new task. It will be returned only when all tasks had been completed, or any errors occurred. For example in the code below I used the Task.whenAll to make all copy operation executed at the same time. 1: var copyRecordsInParallel = eval(Wind.compile("async", function (req, res) { 2: try { 3: // connect to the windows azure sql database 4: var conn = $await(sql.openAsync(connectionString)); 5: console.log("connection opened"); 6: // retrieve all records from database 7: var results = $await(sql.queryAsync(conn, "SELECT * FROM [Resource]")); 8: console.log("records selected. count = %d", results.rows.length); 9: if (results.rows.length > 0) { 10: // recreate the table 11: $await(azure.recreateTableAsync(tableName)); 12: console.log("table created"); 13: // insert records in table storage in parallal 14: var tasks = new Array(results.rows.length); 15: for (var i = 0; i < results.rows.length; i++) { 16: var entity = { 17: "PartitionKey": results.rows[i][1], 18: "RowKey": results.rows[i][0], 19: "Value": results.rows[i][2] 20: }; 21: tasks[i] = azure.insertEntityAsync(tableName, entity); 22: } 23: $await(Wind.Async.Task.whenAll(tasks)); 24: // send response 25: console.log("all done"); 26: res.send(200, "All done!"); 27: } 28: } 29: catch (ex) { 30: console.log(ex); 31: res.send(500, "Internal error."); 32: } 33: })); 34:  35: app.get("/was/copyRecordsInParallel", function (req, res) { 36: copyRecordsInParallel(req, res).start(); 37: });   Besides the task creation and coordination, Wind supports the cancellation solution so that we can send the cancellation signal to the tasks. It also includes exception solution which means any exceptions will be reported to the caller function.   Summary In this post I introduced a Node.js module named Wind, which created by my friend Jeff Zhao. As you can see, different from other async library and framework, adopted the idea from F# and C#, Wind utilizes runtime code generation technology to make it more easily to write async, callback-based functions in a sync-style way. By using Wind there will be almost no callback, and the code will be very easy to understand. Currently Wind is still under developed and improved. There might be some problems but the author, Jeff, should be very happy and enthusiastic to learn your problems, feedback, suggestion and comments. You can contact Jeff by - Email: [email protected] - Group: https://groups.google.com/d/forum/windjs - GitHub: https://github.com/JeffreyZhao/wind/issues   Source code can be download here.   Hope this helps, Shaun All documents and related graphics, codes are provided "AS IS" without warranty of any kind. Copyright © Shaun Ziyan Xu. This work is licensed under the Creative Commons License.

  Read the article

• Using R to Analyze G1GC Log Files

  - by user12620111

  Using R to Analyze G1GC Log Files body, td { font-family: sans-serif; background-color: white; font-size: 12px; margin: 8px; } tt, code, pre { font-family: 'DejaVu Sans Mono', 'Droid Sans Mono', 'Lucida Console', Consolas, Monaco, monospace; } h1 { font-size:2.2em; } h2 { font-size:1.8em; } h3 { font-size:1.4em; } h4 { font-size:1.0em; } h5 { font-size:0.9em; } h6 { font-size:0.8em; } a:visited { color: rgb(50%, 0%, 50%); } pre { margin-top: 0; max-width: 95%; border: 1px solid #ccc; white-space: pre-wrap; } pre code { display: block; padding: 0.5em; } code.r, code.cpp { background-color: #F8F8F8; } table, td, th { border: none; } blockquote { color:#666666; margin:0; padding-left: 1em; border-left: 0.5em #EEE solid; } hr { height: 0px; border-bottom: none; border-top-width: thin; border-top-style: dotted; border-top-color: #999999; } @media print { * { background: transparent !important; color: black !important; filter:none !important; -ms-filter: none !important; } body { font-size:12pt; max-width:100%; } a, a:visited { text-decoration: underline; } hr { visibility: hidden; page-break-before: always; } pre, blockquote { padding-right: 1em; page-break-inside: avoid; } tr, img { page-break-inside: avoid; } img { max-width: 100% !important; } @page :left { margin: 15mm 20mm 15mm 10mm; } @page :right { margin: 15mm 10mm 15mm 20mm; } p, h2, h3 { orphans: 3; widows: 3; } h2, h3 { page-break-after: avoid; } } pre .operator, pre .paren { color: rgb(104, 118, 135) } pre .literal { color: rgb(88, 72, 246) } pre .number { color: rgb(0, 0, 205); } pre .comment { color: rgb(76, 136, 107); } pre .keyword { color: rgb(0, 0, 255); } pre .identifier { color: rgb(0, 0, 0); } pre .string { color: rgb(3, 106, 7); } var hljs=new function(){function m(p){return p.replace(/&/gm,"&").replace(/"}while(y.length||w.length){var v=u().splice(0,1)[0];z+=m(x.substr(q,v.offset-q));q=v.offset;if(v.event=="start"){z+=t(v.node);s.push(v.node)}else{if(v.event=="stop"){var p,r=s.length;do{r--;p=s[r];z+=("")}while(p!=v.node);s.splice(r,1);while(r'+M[0]+""}else{r+=M[0]}O=P.lR.lastIndex;M=P.lR.exec(L)}return r+L.substr(O,L.length-O)}function J(L,M){if(M.sL&&e[M.sL]){var r=d(M.sL,L);x+=r.keyword_count;return r.value}else{return F(L,M)}}function I(M,r){var L=M.cN?'':"";if(M.rB){y+=L;M.buffer=""}else{if(M.eB){y+=m(r)+L;M.buffer=""}else{y+=L;M.buffer=r}}D.push(M);A+=M.r}function G(N,M,Q){var R=D[D.length-1];if(Q){y+=J(R.buffer+N,R);return false}var P=q(M,R);if(P){y+=J(R.buffer+N,R);I(P,M);return P.rB}var L=v(D.length-1,M);if(L){var O=R.cN?"":"";if(R.rE){y+=J(R.buffer+N,R)+O}else{if(R.eE){y+=J(R.buffer+N,R)+O+m(M)}else{y+=J(R.buffer+N+M,R)+O}}while(L1){O=D[D.length-2].cN?"":"";y+=O;L--;D.length--}var r=D[D.length-1];D.length--;D[D.length-1].buffer="";if(r.starts){I(r.starts,"")}return R.rE}if(w(M,R)){throw"Illegal"}}var E=e[B];var D=[E.dM];var A=0;var x=0;var y="";try{var s,u=0;E.dM.buffer="";do{s=p(C,u);var t=G(s[0],s[1],s[2]);u+=s[0].length;if(!t){u+=s[1].length}}while(!s[2]);if(D.length1){throw"Illegal"}return{r:A,keyword_count:x,value:y}}catch(H){if(H=="Illegal"){return{r:0,keyword_count:0,value:m(C)}}else{throw H}}}function g(t){var p={keyword_count:0,r:0,value:m(t)};var r=p;for(var q in e){if(!e.hasOwnProperty(q)){continue}var s=d(q,t);s.language=q;if(s.keyword_count+s.rr.keyword_count+r.r){r=s}if(s.keyword_count+s.rp.keyword_count+p.r){r=p;p=s}}if(r.language){p.second_best=r}return p}function i(r,q,p){if(q){r=r.replace(/^((]+|\t)+)/gm,function(t,w,v,u){return w.replace(/\t/g,q)})}if(p){r=r.replace(/\n/g,"")}return r}function n(t,w,r){var x=h(t,r);var v=a(t);var y,s;if(v){y=d(v,x)}else{return}var q=c(t);if(q.length){s=document.createElement("pre");s.innerHTML=y.value;y.value=k(q,c(s),x)}y.value=i(y.value,w,r);var u=t.className;if(!u.match("(\\s|^)(language-)?"+v+"(\\s|$)")){u=u?(u+" "+v):v}if(/MSIE [678]/.test(navigator.userAgent)&&t.tagName=="CODE"&&t.parentNode.tagName=="PRE"){s=t.parentNode;var p=document.createElement("div");p.innerHTML=""+y.value+"";t=p.firstChild.firstChild;p.firstChild.cN=s.cN;s.parentNode.replaceChild(p.firstChild,s)}else{t.innerHTML=y.value}t.className=u;t.result={language:v,kw:y.keyword_count,re:y.r};if(y.second_best){t.second_best={language:y.second_best.language,kw:y.second_best.keyword_count,re:y.second_best.r}}}function o(){if(o.called){return}o.called=true;var r=document.getElementsByTagName("pre");for(var p=0;p|=||=||=|\\?|\\[|\\{|\\(|\\^|\\^=|\\||\\|=|\\|\\||~";this.ER="(?![\\s\\S])";this.BE={b:"\\\\.",r:0};this.ASM={cN:"string",b:"'",e:"'",i:"\\n",c:[this.BE],r:0};this.QSM={cN:"string",b:'"',e:'"',i:"\\n",c:[this.BE],r:0};this.CLCM={cN:"comment",b:"//",e:"$"};this.CBLCLM={cN:"comment",b:"/\\*",e:"\\*/"};this.HCM={cN:"comment",b:"#",e:"$"};this.NM={cN:"number",b:this.NR,r:0};this.CNM={cN:"number",b:this.CNR,r:0};this.BNM={cN:"number",b:this.BNR,r:0};this.inherit=function(r,s){var p={};for(var q in r){p[q]=r[q]}if(s){for(var q in s){p[q]=s[q]}}return p}}();hljs.LANGUAGES.cpp=function(){var a={keyword:{"false":1,"int":1,"float":1,"while":1,"private":1,"char":1,"catch":1,"export":1,virtual:1,operator:2,sizeof:2,dynamic_cast:2,typedef:2,const_cast:2,"const":1,struct:1,"for":1,static_cast:2,union:1,namespace:1,unsigned:1,"long":1,"throw":1,"volatile":2,"static":1,"protected":1,bool:1,template:1,mutable:1,"if":1,"public":1,friend:2,"do":1,"return":1,"goto":1,auto:1,"void":2,"enum":1,"else":1,"break":1,"new":1,extern:1,using:1,"true":1,"class":1,asm:1,"case":1,typeid:1,"short":1,reinterpret_cast:2,"default":1,"double":1,register:1,explicit:1,signed:1,typename:1,"try":1,"this":1,"switch":1,"continue":1,wchar_t:1,inline:1,"delete":1,alignof:1,char16_t:1,char32_t:1,constexpr:1,decltype:1,noexcept:1,nullptr:1,static_assert:1,thread_local:1,restrict:1,_Bool:1,complex:1},built_in:{std:1,string:1,cin:1,cout:1,cerr:1,clog:1,stringstream:1,istringstream:1,ostringstream:1,auto_ptr:1,deque:1,list:1,queue:1,stack:1,vector:1,map:1,set:1,bitset:1,multiset:1,multimap:1,unordered_set:1,unordered_map:1,unordered_multiset:1,unordered_multimap:1,array:1,shared_ptr:1}};return{dM:{k:a,i:"",k:a,r:10,c:["self"]}]}}}();hljs.LANGUAGES.r={dM:{c:[hljs.HCM,{cN:"number",b:"\\b0[xX][0-9a-fA-F]+[Li]?\\b",e:hljs.IMMEDIATE_RE,r:0},{cN:"number",b:"\\b\\d+(?:[eE][+\\-]?\\d*)?L\\b",e:hljs.IMMEDIATE_RE,r:0},{cN:"number",b:"\\b\\d+\\.(?!\\d)(?:i\\b)?",e:hljs.IMMEDIATE_RE,r:1},{cN:"number",b:"\\b\\d+(?:\\.\\d*)?(?:[eE][+\\-]?\\d*)?i?\\b",e:hljs.IMMEDIATE_RE,r:0},{cN:"number",b:"\\.\\d+(?:[eE][+\\-]?\\d*)?i?\\b",e:hljs.IMMEDIATE_RE,r:1},{cN:"keyword",b:"(?:tryCatch|library|setGeneric|setGroupGeneric)\\b",e:hljs.IMMEDIATE_RE,r:10},{cN:"keyword",b:"\\.\\.\\.",e:hljs.IMMEDIATE_RE,r:10},{cN:"keyword",b:"\\.\\.\\d+(?![\\w.])",e:hljs.IMMEDIATE_RE,r:10},{cN:"keyword",b:"\\b(?:function)",e:hljs.IMMEDIATE_RE,r:2},{cN:"keyword",b:"(?:if|in|break|next|repeat|else|for|return|switch|while|try|stop|warning|require|attach|detach|source|setMethod|setClass)\\b",e:hljs.IMMEDIATE_RE,r:1},{cN:"literal",b:"(?:NA|NA_integer_|NA_real_|NA_character_|NA_complex_)\\b",e:hljs.IMMEDIATE_RE,r:10},{cN:"literal",b:"(?:NULL|TRUE|FALSE|T|F|Inf|NaN)\\b",e:hljs.IMMEDIATE_RE,r:1},{cN:"identifier",b:"[a-zA-Z.][a-zA-Z0-9._]*\\b",e:hljs.IMMEDIATE_RE,r:0},{cN:"operator",b:"|=||   Using R to Analyze G1GC Log Files   Using R to Analyze G1GC Log Files Introduction Working in Oracle Platform Integration gives an engineer opportunities to work on a wide array of technologies. My team’s goal is to make Oracle applications run best on the Solaris/SPARC platform. When looking for bottlenecks in a modern applications, one needs to be aware of not only how the CPUs and operating system are executing, but also network, storage, and in some cases, the Java Virtual Machine. I was recently presented with about 1.5 GB of Java Garbage First Garbage Collector log file data. If you’re not familiar with the subject, you might want to review Garbage First Garbage Collector Tuning by Monica Beckwith. The customer had been running Java HotSpot 1.6.0_31 to host a web application server. I was told that the Solaris/SPARC server was running a Java process launched using a commmand line that included the following flags: -d64 -Xms9g -Xmx9g -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=80 -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps -XX:+PrintFlagsFinal -XX:+DisableExplicitGC -XX:+UnlockExperimentalVMOptions -XX:ParallelGCThreads=8 Several sources on the internet indicate that if I were to print out the 1.5 GB of log files, it would require enough paper to fill the bed of a pick up truck. Of course, it would be fruitless to try to scan the log files by hand. Tools will be required to summarize the contents of the log files. Others have encountered large Java garbage collection log files. There are existing tools to analyze the log files: IBM’s GC toolkit The chewiebug GCViewer gchisto HPjmeter Instead of using one of the other tools listed, I decide to parse the log files with standard Unix tools, and analyze the data with R. Data Cleansing The log files arrived in two different formats. I guess that the difference is that one set of log files was generated using a more verbose option, maybe -XX:+PrintHeapAtGC, and the other set of log files was generated without that option. Format 1 In some of the log files, the log files with the less verbose format, a single trace, i.e. the report of a singe garbage collection event, looks like this: {Heap before GC invocations=12280 (full 61): garbage-first heap total 9437184K, used 7499918K [0xfffffffd00000000, 0xffffffff40000000, 0xffffffff40000000) region size 4096K, 1 young (4096K), 0 survivors (0K) compacting perm gen total 262144K, used 144077K [0xffffffff40000000, 0xffffffff50000000, 0xffffffff50000000) the space 262144K, 54% used [0xffffffff40000000, 0xffffffff48cb3758, 0xffffffff48cb3800, 0xffffffff50000000) No shared spaces configured. 2014-05-14T07:24:00.988-0700: 60586.353: [GC pause (young) 7324M->7320M(9216M), 0.1567265 secs] Heap after GC invocations=12281 (full 61): garbage-first heap total 9437184K, used 7496533K [0xfffffffd00000000, 0xffffffff40000000, 0xffffffff40000000) region size 4096K, 0 young (0K), 0 survivors (0K) compacting perm gen total 262144K, used 144077K [0xffffffff40000000, 0xffffffff50000000, 0xffffffff50000000) the space 262144K, 54% used [0xffffffff40000000, 0xffffffff48cb3758, 0xffffffff48cb3800, 0xffffffff50000000) No shared spaces configured. } A simple grep can be used to extract a summary: $ grep "\[ GC pause (young" g1gc.log 2014-05-13T13:24:35.091-0700: 3.109: [GC pause (young) 20M->5029K(9216M), 0.0146328 secs] 2014-05-13T13:24:35.440-0700: 3.459: [GC pause (young) 9125K->6077K(9216M), 0.0086723 secs] 2014-05-13T13:24:37.581-0700: 5.599: [GC pause (young) 25M->8470K(9216M), 0.0203820 secs] 2014-05-13T13:24:42.686-0700: 10.704: [GC pause (young) 44M->15M(9216M), 0.0288848 secs] 2014-05-13T13:24:48.941-0700: 16.958: [GC pause (young) 51M->20M(9216M), 0.0491244 secs] 2014-05-13T13:24:56.049-0700: 24.066: [GC pause (young) 92M->26M(9216M), 0.0525368 secs] 2014-05-13T13:25:34.368-0700: 62.383: [GC pause (young) 602M->68M(9216M), 0.1721173 secs] But that format wasn't easily read into R, so I needed to be a bit more tricky. I used the following Unix command to create a summary file that was easy for R to read. $ echo "SecondsSinceLaunch BeforeSize AfterSize TotalSize RealTime" $ grep "\[GC pause (young" g1gc.log | grep -v mark | sed -e 's/[A-SU-z\(\),]/ /g' -e 's/->/ /' -e 's/: / /g' | more SecondsSinceLaunch BeforeSize AfterSize TotalSize RealTime 2014-05-13T13:24:35.091-0700 3.109 20 5029 9216 0.0146328 2014-05-13T13:24:35.440-0700 3.459 9125 6077 9216 0.0086723 2014-05-13T13:24:37.581-0700 5.599 25 8470 9216 0.0203820 2014-05-13T13:24:42.686-0700 10.704 44 15 9216 0.0288848 2014-05-13T13:24:48.941-0700 16.958 51 20 9216 0.0491244 2014-05-13T13:24:56.049-0700 24.066 92 26 9216 0.0525368 2014-05-13T13:25:34.368-0700 62.383 602 68 9216 0.1721173 Format 2 In some of the log files, the log files with the more verbose format, a single trace, i.e. the report of a singe garbage collection event, was more complicated than Format 1. Here is a text file with an example of a single G1GC trace in the second format. As you can see, it is quite complicated. It is nice that there is so much information available, but the level of detail can be overwhelming. I wrote this awk script (download) to summarize each trace on a single line. #!/usr/bin/env awk -f BEGIN { printf("SecondsSinceLaunch IncrementalCount FullCount UserTime SysTime RealTime BeforeSize AfterSize TotalSize\n") } ###################### # Save count data from lines that are at the start of each G1GC trace. # Each trace starts out like this: # {Heap before GC invocations=14 (full 0): # garbage-first heap total 9437184K, used 325496K [0xfffffffd00000000, 0xffffffff40000000, 0xffffffff40000000) ###################### /{Heap.*full/{ gsub ( "\\)" , "" ); nf=split($0,a,"="); split(a[2],b," "); getline; if ( match($0, "first") ) { G1GC=1; IncrementalCount=b[1]; FullCount=substr( b[3], 1, length(b[3])-1 ); } else { G1GC=0; } } ###################### # Pull out time stamps that are in lines with this format: # 2014-05-12T14:02:06.025-0700: 94.312: [GC pause (young), 0.08870154 secs] ###################### /GC pause/ { DateTime=$1; SecondsSinceLaunch=substr($2, 1, length($2)-1); } ###################### # Heap sizes are in lines that look like this: # [ 4842M->4838M(9216M)] ###################### /\[ .*]$/ { gsub ( "\\[" , "" ); gsub ( "\ \]" , "" ); gsub ( "->" , " " ); gsub ( "\\( " , " " ); gsub ( "\ \)" , " " ); split($0,a," "); if ( split(a[1],b,"M") > 1 ) {BeforeSize=b[1]*1024;} if ( split(a[1],b,"K") > 1 ) {BeforeSize=b[1];} if ( split(a[2],b,"M") > 1 ) {AfterSize=b[1]*1024;} if ( split(a[2],b,"K") > 1 ) {AfterSize=b[1];} if ( split(a[3],b,"M") > 1 ) {TotalSize=b[1]*1024;} if ( split(a[3],b,"K") > 1 ) {TotalSize=b[1];} } ###################### # Emit an output line when you find input that looks like this: # [Times: user=1.41 sys=0.08, real=0.24 secs] ###################### /\[Times/ { if (G1GC==1) { gsub ( "," , "" ); split($2,a,"="); UserTime=a[2]; split($3,a,"="); SysTime=a[2]; split($4,a,"="); RealTime=a[2]; print DateTime,SecondsSinceLaunch,IncrementalCount,FullCount,UserTime,SysTime,RealTime,BeforeSize,AfterSize,TotalSize; G1GC=0; } } The resulting summary is about 25X smaller that the original file, but still difficult for a human to digest. SecondsSinceLaunch IncrementalCount FullCount UserTime SysTime RealTime BeforeSize AfterSize TotalSize ... 2014-05-12T18:36:34.669-0700: 3985.744 561 0 0.57 0.06 0.16 1724416 1720320 9437184 2014-05-12T18:36:34.839-0700: 3985.914 562 0 0.51 0.06 0.19 1724416 1720320 9437184 2014-05-12T18:36:35.069-0700: 3986.144 563 0 0.60 0.04 0.27 1724416 1721344 9437184 2014-05-12T18:36:35.354-0700: 3986.429 564 0 0.33 0.04 0.09 1725440 1722368 9437184 2014-05-12T18:36:35.545-0700: 3986.620 565 0 0.58 0.04 0.17 1726464 1722368 9437184 2014-05-12T18:36:35.726-0700: 3986.801 566 0 0.43 0.05 0.12 1726464 1722368 9437184 2014-05-12T18:36:35.856-0700: 3986.930 567 0 0.30 0.04 0.07 1726464 1723392 9437184 2014-05-12T18:36:35.947-0700: 3987.023 568 0 0.61 0.04 0.26 1727488 1723392 9437184 2014-05-12T18:36:36.228-0700: 3987.302 569 0 0.46 0.04 0.16 1731584 1724416 9437184 Reading the Data into R Once the GC log data had been cleansed, either by processing the first format with the shell script, or by processing the second format with the awk script, it was easy to read the data into R. g1gc.df = read.csv("summary.txt", row.names = NULL, stringsAsFactors=FALSE,sep="") str(g1gc.df) ## 'data.frame': 8307 obs. of 10 variables: ## $ row.names : chr "2014-05-12T14:00:32.868-0700:" "2014-05-12T14:00:33.179-0700:" "2014-05-12T14:00:33.677-0700:" "2014-05-12T14:00:35.538-0700:" ... ## $ SecondsSinceLaunch: num 1.16 1.47 1.97 3.83 6.1 ... ## $ IncrementalCount : int 0 1 2 3 4 5 6 7 8 9 ... ## $ FullCount : int 0 0 0 0 0 0 0 0 0 0 ... ## $ UserTime : num 0.11 0.05 0.04 0.21 0.08 0.26 0.31 0.33 0.34 0.56 ... ## $ SysTime : num 0.04 0.01 0.01 0.05 0.01 0.06 0.07 0.06 0.07 0.09 ... ## $ RealTime : num 0.02 0.02 0.01 0.04 0.02 0.04 0.05 0.04 0.04 0.06 ... ## $ BeforeSize : int 8192 5496 5768 22528 24576 43008 34816 53248 55296 93184 ... ## $ AfterSize : int 1400 1672 2557 4907 7072 14336 16384 18432 19456 21504 ... ## $ TotalSize : int 9437184 9437184 9437184 9437184 9437184 9437184 9437184 9437184 9437184 9437184 ... head(g1gc.df) ## row.names SecondsSinceLaunch IncrementalCount ## 1 2014-05-12T14:00:32.868-0700: 1.161 0 ## 2 2014-05-12T14:00:33.179-0700: 1.472 1 ## 3 2014-05-12T14:00:33.677-0700: 1.969 2 ## 4 2014-05-12T14:00:35.538-0700: 3.830 3 ## 5 2014-05-12T14:00:37.811-0700: 6.103 4 ## 6 2014-05-12T14:00:41.428-0700: 9.720 5 ## FullCount UserTime SysTime RealTime BeforeSize AfterSize TotalSize ## 1 0 0.11 0.04 0.02 8192 1400 9437184 ## 2 0 0.05 0.01 0.02 5496 1672 9437184 ## 3 0 0.04 0.01 0.01 5768 2557 9437184 ## 4 0 0.21 0.05 0.04 22528 4907 9437184 ## 5 0 0.08 0.01 0.02 24576 7072 9437184 ## 6 0 0.26 0.06 0.04 43008 14336 9437184 Basic Statistics Once the data has been read into R, simple statistics are very easy to generate. All of the numbers from high school statistics are available via simple commands. For example, generate a summary of every column: summary(g1gc.df) ## row.names SecondsSinceLaunch IncrementalCount FullCount ## Length:8307 Min. : 1 Min. : 0 Min. : 0.0 ## Class :character 1st Qu.: 9977 1st Qu.:2048 1st Qu.: 0.0 ## Mode :character Median :12855 Median :4136 Median : 12.0 ## Mean :12527 Mean :4156 Mean : 31.6 ## 3rd Qu.:15758 3rd Qu.:6262 3rd Qu.: 61.0 ## Max. :55484 Max. :8391 Max. :113.0 ## UserTime SysTime RealTime BeforeSize ## Min. :0.040 Min. :0.0000 Min. : 0.0 Min. : 5476 ## 1st Qu.:0.470 1st Qu.:0.0300 1st Qu.: 0.1 1st Qu.:5137920 ## Median :0.620 Median :0.0300 Median : 0.1 Median :6574080 ## Mean :0.751 Mean :0.0355 Mean : 0.3 Mean :5841855 ## 3rd Qu.:0.920 3rd Qu.:0.0400 3rd Qu.: 0.2 3rd Qu.:7084032 ## Max. :3.370 Max. :1.5600 Max. :488.1 Max. :8696832 ## AfterSize TotalSize ## Min. : 1380 Min. :9437184 ## 1st Qu.:5002752 1st Qu.:9437184 ## Median :6559744 Median :9437184 ## Mean :5785454 Mean :9437184 ## 3rd Qu.:7054336 3rd Qu.:9437184 ## Max. :8482816 Max. :9437184 Q: What is the total amount of User CPU time spent in garbage collection? sum(g1gc.df$UserTime) ## [1] 6236 As you can see, less than two hours of CPU time was spent in garbage collection. Is that too much? To find the percentage of time spent in garbage collection, divide the number above by total_elapsed_time*CPU_count. In this case, there are a lot of CPU’s and it turns out the the overall amount of CPU time spent in garbage collection isn’t a problem when viewed in isolation. When calculating rates, i.e. events per unit time, you need to ask yourself if the rate is homogenous across the time period in the log file. Does the log file include spikes of high activity that should be separately analyzed? Averaging in data from nights and weekends with data from business hours may alias problems. If you have a reason to suspect that the garbage collection rates include peaks and valleys that need independent analysis, see the “Time Series” section, below. Q: How much garbage is collected on each pass? The amount of heap space that is recovered per GC pass is surprisingly low: At least one collection didn’t recover any data. (“Min.=0”) 25% of the passes recovered 3MB or less. (“1st Qu.=3072”) Half of the GC passes recovered 4MB or less. (“Median=4096”) The average amount recovered was 56MB. (“Mean=56390”) 75% of the passes recovered 36MB or less. (“3rd Qu.=36860”) At least one pass recovered 2GB. (“Max.=2121000”) g1gc.df$Delta = g1gc.df$BeforeSize - g1gc.df$AfterSize summary(g1gc.df$Delta) ## Min. 1st Qu. Median Mean 3rd Qu. Max. ## 0 3070 4100 56400 36900 2120000 Q: What is the maximum User CPU time for a single collection? The worst garbage collection (“Max.”) is many standard deviations away from the mean. The data appears to be right skewed. summary(g1gc.df$UserTime) ## Min. 1st Qu. Median Mean 3rd Qu. Max. ## 0.040 0.470 0.620 0.751 0.920 3.370 sd(g1gc.df$UserTime) ## [1] 0.3966 Basic Graphics Once the data is in R, it is trivial to plot the data with formats including dot plots, line charts, bar charts (simple, stacked, grouped), pie charts, boxplots, scatter plots histograms, and kernel density plots. Histogram of User CPU Time per Collection I don't think that this graph requires any explanation. hist(g1gc.df$UserTime, main="User CPU Time per Collection", xlab="Seconds", ylab="Frequency") Box plot to identify outliers When the initial data is viewed with a box plot, you can see the one crazy outlier in the real time per GC. Save this data point for future analysis and drop the outlier so that it’s not throwing off our statistics. Now the box plot shows many outliers, which will be examined later, using times series analysis. Notice that the scale of the x-axis changes drastically once the crazy outlier is removed. par(mfrow=c(2,1)) boxplot(g1gc.df$UserTime,g1gc.df$SysTime,g1gc.df$RealTime, main="Box Plot of Time per GC\n(dominated by a crazy outlier)", names=c("usr","sys","elapsed"), xlab="Seconds per GC", ylab="Time (Seconds)", horizontal = TRUE, outcol="red") crazy.outlier.df=g1gc.df[g1gc.df$RealTime > 400,] g1gc.df=g1gc.df[g1gc.df$RealTime < 400,] boxplot(g1gc.df$UserTime,g1gc.df$SysTime,g1gc.df$RealTime, main="Box Plot of Time per GC\n(crazy outlier excluded)", names=c("usr","sys","elapsed"), xlab="Seconds per GC", ylab="Time (Seconds)", horizontal = TRUE, outcol="red") box(which = "outer", lty = "solid") Here is the crazy outlier for future analysis: crazy.outlier.df ## row.names SecondsSinceLaunch IncrementalCount ## 8233 2014-05-12T23:15:43.903-0700: 20741 8316 ## FullCount UserTime SysTime RealTime BeforeSize AfterSize TotalSize ## 8233 112 0.55 0.42 488.1 8381440 8235008 9437184 ## Delta ## 8233 146432 R Time Series Data To analyze the garbage collection as a time series, I’ll use Z’s Ordered Observations (zoo). “zoo is the creator for an S3 class of indexed totally ordered observations which includes irregular time series.” require(zoo) ## Loading required package: zoo ## ## Attaching package: 'zoo' ## ## The following objects are masked from 'package:base': ## ## as.Date, as.Date.numeric head(g1gc.df[,1]) ## [1] "2014-05-12T14:00:32.868-0700:" "2014-05-12T14:00:33.179-0700:" ## [3] "2014-05-12T14:00:33.677-0700:" "2014-05-12T14:00:35.538-0700:" ## [5] "2014-05-12T14:00:37.811-0700:" "2014-05-12T14:00:41.428-0700:" options("digits.secs"=3) times=as.POSIXct( g1gc.df[,1], format="%Y-%m-%dT%H:%M:%OS%z:") g1gc.z = zoo(g1gc.df[,-c(1)], order.by=times) head(g1gc.z) ## SecondsSinceLaunch IncrementalCount FullCount ## 2014-05-12 17:00:32.868 1.161 0 0 ## 2014-05-12 17:00:33.178 1.472 1 0 ## 2014-05-12 17:00:33.677 1.969 2 0 ## 2014-05-12 17:00:35.538 3.830 3 0 ## 2014-05-12 17:00:37.811 6.103 4 0 ## 2014-05-12 17:00:41.427 9.720 5 0 ## UserTime SysTime RealTime BeforeSize AfterSize ## 2014-05-12 17:00:32.868 0.11 0.04 0.02 8192 1400 ## 2014-05-12 17:00:33.178 0.05 0.01 0.02 5496 1672 ## 2014-05-12 17:00:33.677 0.04 0.01 0.01 5768 2557 ## 2014-05-12 17:00:35.538 0.21 0.05 0.04 22528 4907 ## 2014-05-12 17:00:37.811 0.08 0.01 0.02 24576 7072 ## 2014-05-12 17:00:41.427 0.26 0.06 0.04 43008 14336 ## TotalSize Delta ## 2014-05-12 17:00:32.868 9437184 6792 ## 2014-05-12 17:00:33.178 9437184 3824 ## 2014-05-12 17:00:33.677 9437184 3211 ## 2014-05-12 17:00:35.538 9437184 17621 ## 2014-05-12 17:00:37.811 9437184 17504 ## 2014-05-12 17:00:41.427 9437184 28672 Example of Two Benchmark Runs in One Log File The data in the following graph is from a different log file, not the one of primary interest to this article. I’m including this image because it is an example of idle periods followed by busy periods. It would be uninteresting to average the rate of garbage collection over the entire log file period. More interesting would be the rate of garbage collect in the two busy periods. Are they the same or different? Your production data may be similar, for example, bursts when employees return from lunch and idle times on weekend evenings, etc. Once the data is in an R Time Series, you can analyze isolated time windows. Clipping the Time Series data Flashing back to our test case… Viewing the data as a time series is interesting. You can see that the work intensive time period is between 9:00 PM and 3:00 AM. Lets clip the data to the interesting period:     par(mfrow=c(2,1)) plot(g1gc.z$UserTime, type="h", main="User Time per GC\nTime: Complete Log File", xlab="Time of Day", ylab="CPU Seconds per GC", col="#1b9e77") clipped.g1gc.z=window(g1gc.z, start=as.POSIXct("2014-05-12 21:00:00"), end=as.POSIXct("2014-05-13 03:00:00")) plot(clipped.g1gc.z$UserTime, type="h", main="User Time per GC\nTime: Limited to Benchmark Execution", xlab="Time of Day", ylab="CPU Seconds per GC", col="#1b9e77") box(which = "outer", lty = "solid") Cumulative Incremental and Full GC count Here is the cumulative incremental and full GC count. When the line is very steep, it indicates that the GCs are repeating very quickly. Notice that the scale on the Y axis is different for full vs. incremental. plot(clipped.g1gc.z[,c(2:3)], main="Cumulative Incremental and Full GC count", xlab="Time of Day", col="#1b9e77") GC Analysis of Benchmark Execution using Time Series data In the following series of 3 graphs: The “After Size” show the amount of heap space in use after each garbage collection. Many Java objects are still referenced, i.e. alive, during each garbage collection. This may indicate that the application has a memory leak, or may indicate that the application has a very large memory footprint. Typically, an application's memory footprint plateau's in the early stage of execution. One would expect this graph to have a flat top. The steep decline in the heap space may indicate that the application crashed after 2:00. The second graph shows that the outliers in real execution time, discussed above, occur near 2:00. when the Java heap seems to be quite full. The third graph shows that Full GCs are infrequent during the first few hours of execution. The rate of Full GC's, (the slope of the cummulative Full GC line), changes near midnight.   plot(clipped.g1gc.z[,c("AfterSize","RealTime","FullCount")], xlab="Time of Day", col=c("#1b9e77","red","#1b9e77")) GC Analysis of heap recovered Each GC trace includes the amount of heap space in use before and after the individual GC event. During garbage coolection, unreferenced objects are identified, the space holding the unreferenced objects is freed, and thus, the difference in before and after usage indicates how much space has been freed. The following box plot and bar chart both demonstrate the same point - the amount of heap space freed per garbage colloection is surprisingly low. par(mfrow=c(2,1)) boxplot(as.vector(clipped.g1gc.z$Delta), main="Amount of Heap Recovered per GC Pass", xlab="Size in KB", horizontal = TRUE, col="red") hist(as.vector(clipped.g1gc.z$Delta), main="Amount of Heap Recovered per GC Pass", xlab="Size in KB", breaks=100, col="red") box(which = "outer", lty = "solid") This graph is the most interesting. The dark blue area shows how much heap is occupied by referenced Java objects. This represents memory that holds live data. The red fringe at the top shows how much data was recovered after each garbage collection. barplot(clipped.g1gc.z[,c("AfterSize","Delta")], col=c("#7570b3","#e7298a"), xlab="Time of Day", border=NA) legend("topleft", c("Live Objects","Heap Recovered on GC"), fill=c("#7570b3","#e7298a")) box(which = "outer", lty = "solid") When I discuss the data in the log files with the customer, I will ask for an explaination for the large amount of referenced data resident in the Java heap. There are two are posibilities: There is a memory leak and the amount of space required to hold referenced objects will continue to grow, limited only by the maximum heap size. After the maximum heap size is reached, the JVM will throw an “Out of Memory” exception every time that the application tries to allocate a new object. If this is the case, the aplication needs to be debugged to identify why old objects are referenced when they are no longer needed. The application has a legitimate requirement to keep a large amount of data in memory. The customer may want to further increase the maximum heap size. Another possible solution would be to partition the application across multiple cluster nodes, where each node has responsibility for managing a unique subset of the data. Conclusion In conclusion, R is a very powerful tool for the analysis of Java garbage collection log files. The primary difficulty is data cleansing so that information can be read into an R data frame. Once the data has been read into R, a rich set of tools may be used for thorough evaluation.

  Read the article

• Wireless not working on XP (with Intel® PRO/Wireless 2200BG Network Connection card and Zyxel router

  - by JohnIdol

  Hi all, I just got a new router, cant' get my old xp laptop (Toshiba Satellite) to connect to it, no matter if it's WPA-PSK or WEP (I was kind of thinking with WEP it would've worked but it doesn't). When I go connect and put in credentials the dialog stays up for a few minutes then disappears with no message and no connection is established. Here's the spec of router and wireless card: Wilreless card: Intel® PRO/Wireless 2200BG Network Connection Router: Zyxel P-660-HW-T1 v3 Everything works fine on my Vista machine (for once). Any help appreciated!

  Read the article

• VirtualBox error VERR_SUPDRV_COMPONENT_NOT_FOUND

  - by ant2009

  I am using Fedora 12, with kernel stv-fedora 2.6.31.12-174.2.22.fc12.i686 I have install VirtualBox OSE v3.1.4-1.fc12. When I start VirtualBox I get the following error: Failed to start the virtual machine WinXP32. Failed to open/create the internal network 'HostInterfaceNetworking-eth0' (VERR_SUPDRV_COMPONENT_NOT_FOUND). One of the kernel modules was not successfully loaded. Make sure that no kernel modules from an older version of VirtualBox exist. Then try to recompile and reload the kernel modules by executing '/etc/init.d/vboxdrv setup' as root (VERR_SUPDRV_COMPONENT_NOT_FOUND). However, if I run /etc/init.d/vboxdrv setup I get this error: -bash: /etc/init.d/vboxdrv: No such file or directory How do I get rid of the error and run VirtualBox?

  Read the article

• RPM Version Issue and won't install

  - by Tiffany Walker

  Get this error when trying to install an RPM: rpm -Uvh rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm warning: rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 6b8d79e6 error: Failed dependencies: rpmlib(FileDigests) <= 4.6.0-1 is needed by rpmforge-release-0.5.3-1.el6.rf.x86_64 rpmlib(PayloadIsXz) <= 5.2-1 is needed by rpmforge-release-0.5.3-1.el6.rf.x86_64 uname -a Linux host 2.6.32-042stab075.2 #1 SMP Tue Mar 5 15:21:53 MSK 2013 x86_64 x86_64 x86_64 GNU/Linux What do I need to do to fix this EDIT: Fixed. I'll answer this in 2 days. I assumed the server was CentOS 6 since I don't use 5 any more. ;/

  Read the article

• How to set up spf records to send mail from google hosted apps to gmail addresses

  - by Chris Adams

  Hi there, I'm trying to work out why email I send from one domain I own is rejected by another that I own, and while I think it may be related to how I've setup spf records, I'm not sure what steps I need to take to fix it. Here's the error message I receive: Technical details of permanent failure: Google tried to deliver your message, but it was rejected by the recipient domain. We recommend contacting the other email provider for further information about the cause of this error. The error that the other server returned was: 550 550-Verification failed for <[email protected]> 550-No Such User Here 550 Sender verify failed (state 14). Here's the response from [email protected] Delivered-To: [email protected] Received: by 10.86.92.9 with SMTP id p9cs85371fgb; Wed, 2 Sep 2009 22:33:32 -0700 (PDT) Received: by 10.90.205.4 with SMTP id c4mr2406190agg.29.1251956007562; Wed, 02 Sep 2009 22:33:27 -0700 (PDT) Return-Path: <[email protected]> Received: from verifier.port25.com (207-36-201-235.ptr.primarydns.com [207.36.201.235]) by mx.google.com with ESMTP id 26si831174aga.24.2009.09.02.22.33.25; Wed, 02 Sep 2009 22:33:26 -0700 (PDT) Received-SPF: pass (google.com: domain of [email protected] designates 207.36.201.235 as permitted sender) client-ip=207.36.201.235; Authentication-Results: mx.google.com; spf=pass (google.com: domain of [email protected] designates 207.36.201.235 as permitted sender) [email protected]; dkim=pass [email protected] DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; s=auth; d=port25.com; h=Date:From:To:Subject:Message-Id:In-Reply-To; [email protected]; bh=GRMrcnoucTl4upzqJYTG5sOZMLU=; b=uk6TjADEyZVRkceQGjH94ZzfVeRTsiZPzbXuhlqDt1m+kh1zmdUEoiTOzd89ryCHMbVcnG1JajBj 5vOMKYtA3g== DomainKey-Signature: a=rsa-sha1; c=nofws; q=dns; s=auth; d=port25.com; b=NqKCPK00Xt49lbeO009xy4ZRgMGpghvcgfhjNy7+qI89XKTzi6IUW0hYqCQyHkd2p5a1Zjez2ZMC l0u9CpZD3Q==; Received: from verifier.port25.com (127.0.0.1) by verifier.port25.com (PowerMTA(TM) v3.6a1) id hjt9pq0hse8u for <[email protected]>; Thu, 3 Sep 2009 01:26:52 -0400 (envelope-from <[email protected]>) Date: Thu, 3 Sep 2009 01:26:52 -0400 From: [email protected] To: [email protected] Subject: Authentication Report Message-Id: <[email protected]> Precedence: junk (auto_reply) In-Reply-To: <[email protected]> This message is an automatic response from Port25's authentication verifier service at verifier.port25.com. The service allows email senders to perform a simple check of various sender authentication mechanisms. It is provided free of charge, in the hope that it is useful to the email community. While it is not officially supported, we welcome any feedback you may have at <[email protected]>. Thank you for using the verifier, The Port25 Solutions, Inc. team ========================================================== Summary of Results ========================================================== SPF check: pass DomainKeys check: neutral DKIM check: neutral Sender-ID check: pass SpamAssassin check: ham ========================================================== Details: ========================================================== HELO hostname: fg-out-1718.google.com Source IP: 72.14.220.158 mail-from: [email protected] ---------------------------------------------------------- SPF check details: ---------------------------------------------------------- Result: pass ID(s) verified: [email protected] DNS record(s): stemcel.co.uk. 14400 IN TXT "v=spf1 include:aspmx.googlemail.com ~all" aspmx.googlemail.com. 7200 IN TXT "v=spf1 redirect=_spf.google.com" _spf.google.com. 300 IN TXT "v=spf1 ip4:216.239.32.0/19 ip4:64.233.160.0/19 ip4:66.249.80.0/20 ip4:72.14.192.0/18 ip4:209.85.128.0/17 ip4:66.102.0.0/20 ip4:74.125.0.0/16 ip4:64.18.0.0/20 ip4:207.126.144.0/20 ?all" ---------------------------------------------------------- DomainKeys check details: ---------------------------------------------------------- Result: neutral (message not signed) ID(s) verified: [email protected] DNS record(s): ---------------------------------------------------------- DKIM check details: ---------------------------------------------------------- Result: neutral (message not signed) ID(s) verified: NOTE: DKIM checking has been performed based on the latest DKIM specs (RFC 4871 or draft-ietf-dkim-base-10) and verification may fail for older versions. If you are using Port25's PowerMTA, you need to use version 3.2r11 or later to get a compatible version of DKIM. ---------------------------------------------------------- Sender-ID check details: ---------------------------------------------------------- Result: pass ID(s) verified: [email protected] DNS record(s): stemcel.co.uk. 14400 IN TXT "v=spf1 include:aspmx.googlemail.com ~all" aspmx.googlemail.com. 7200 IN TXT "v=spf1 redirect=_spf.google.com" _spf.google.com. 300 IN TXT "v=spf1 ip4:216.239.32.0/19 ip4:64.233.160.0/19 ip4:66.249.80.0/20 ip4:72.14.192.0/18 ip4:209.85.128.0/17 ip4:66.102.0.0/20 ip4:74.125.0.0/16 ip4:64.18.0.0/20 ip4:207.126.144.0/20 ?all" ---------------------------------------------------------- SpamAssassin check details: ---------------------------------------------------------- SpamAssassin v3.2.5 (2008-06-10) Result: ham (-2.6 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record -2.6 BAYES_00 BODY: Bayesian spam probability is 0 to 1% [score: 0.0000] 0.0 HTML_MESSAGE BODY: HTML included in message I've registered the spf records for my domain, as advised here Both domains pass validate according to Kitterman's spf record testing tools, so I'm somewhat confused about this. I also have the catchall address set up on the stemcel.co.uk domain here, but I don't have one setup for chrisadams.me.uk. Instead, we have the following forwarders setup [email protected] to [email protected] [email protected] to [email protected] [email protected] to [email protected] [email protected] to [email protected] Any ideas how to get this working? I'm not sure what I should be looking for here.

  Read the article

• Cannot run setups from a vboxsvr mapped network drive on Windows within VirtualBox

  - by Dimitri C.

  I'm trying to run an application setup by double-clicking the setup.exe from within Windows Explorer. The file is located on a mapped network drive, and I'm using Windows 7. This results in the following error message: The specified path does not exist. Check the path, and then try again. The workaround I found is to copy the installer to the main hard drive (c:) and run it from there; however, this is rather inconvenient. I have the impression that the problem only occurs with installers, as everything seemed to work fine with regular exe's. Is there anyone who can explain this odd behavior? Update: After some extended tests I noticed that the problem only occurs with a mapped drive of VirtualBox's "shared folders" (cf. vboxsvr; VirtualBox v3.1.4). Mapping an SMB drive works fine. Update: The same problem occurs on Windows Vista.

  Read the article

• IPSec VPN's being dropped by router and will not re-establish

  - by David Gard

  We have 3 sites, with our two remote sites connection to head office via LAN-to-LAN VPN's. All 3 sites use DrayTek 2900's with firware version v3.3.1.1_RC2 (this is a release candidate that DrayTek suggested I try, but sadly it made no difference). The only way to re-establish the VPN's once they have been dropped is to restart the router. Head office is set to dial out to both sites, with both the 'Always on' and 'Enable PING to keep alive' (pinging a server in the remote offices) options ticked. However, at random intervals the VPN's drop, logging IKE_RELEASE VPN : Dial-out Profile Index = 7, Name = Shepton (for one connection, and '6' & 'Wincanton' for the other connection). I first tried swapping the router with one at another site, and then had all three replaced, but that failed to solve the problem. Is anyone aware of anything that could cause the VPN's to drop randomly like this? Thanks.

  Read the article

• grep is inconsistently defaulting to grep -P?

  - by Sammitch

  I have a script that does some housekeeping that works perfectly well when invoked from an interactive shell, but did nothing when invoked by cron. To troubleshoot this I started a shell with a 'blank' environment with the command: env -i /bin/bash --noprofile --norc Using this blank env I've dug into my script and found that the following grep will not match any files: grep -il "^ws_status\s*=\s*[\"']remove[\"']$" However, when run from an interactive shell the command will return the filenames of the matching files. As a note, the expression is matching lines like: WS_STATUS = "remove" Through trial-and-error I discovered that adding -P to the options [Perl regex] the command started working normally in the 'blank' shell. However, I have no idea why my login shell appears to be defaulted to grep -P. There is only one grep binary, /bin/grep There are no aliases defined for grep=pgrep or grep="grep -P" There is no env variable GREP_OPTIONS defined. What's the deal here? Note: OS is RHEL v5.10, Bash is v3.2.25, grep is v2.5.1

  Read the article

• GlassFish will not start when SNMP is enabled

  - by edarc

  I have a GlassFish v3 app server running on 64-bit Debian Lenny. Everything is running fine, except I would like to monitor GF's JVM instance with SNMP. However, every time I try to enable it by adding the following <jvm-options> in domain.xml: -Dcom.sun.management.snmp.port=10161 -Dcom.sun.management.snmp.acl.file=/path/to/snmp.acl -Dcom.sun.management.snmp.interface=127.0.0.1 GlassFish refuses to start: $ asadmin start-domain Waiting for DAS to start .Error starting domain: default. The server exited prematurely with exit code 1. Command start-domain failed. $ There is also nothing illuminating (well, really nothing at all) in jvm.log or server.log. The snmp.acl file contains: acl = { { communities = public access = read-only managers = localhost } } and is chmod 600 (I know this is not the problem because it will actually fail with an error about the permissions if it is set to anything other than 600) $ java -version java version "1.6.0_0" OpenJDK Runtime Environment (build 1.6.0_0-b11) OpenJDK 64-Bit Server VM (build 1.6.0_0-b11, mixed mode)

  Read the article

• Windows 8 asks DotNet 3.5 to install DotNet 3.5

  - by William.Ebe

  While trying to install some tools in Windows 8, it showed me a nice message saying it requires DotNet framework. Well, it's fine. But the bad part is I have offline installers for v2.0 and v3.5. While I'm trying to install those installers it shows me the same message as it shows above. I can't consider the web installer, because I don't want to download around 100MB. The file I have works fine for other versions like Windows 7 or XP. Any fixes?

  Read the article

• Active Directory using Samba/Open LDAP for user accounts

  - by Jon Rhoades

  I know this is the wrong way round... but Is it possible to use AD in front of Samba for our PC clients, so that the user accounts are in Samba/Open LDAP. Managing our fleet of Windows PC's is becoming more and more difficult with just Samba v3 - until Samba v4 comes along, it would be great if we could leverage Active Directory, but have the accounts stored in Samba/Open LDAP. Windows PC's are a minority in our organisation & Samaba/Open LDAP are used for just about every service (Zimbra/RADIUS/Intranet/SAN/Printing/...) so it will have to remain the definitive account source. Anyway, it probably can't be done, but I thought I would ask for ideas anyway.

  Read the article

• Differencing Disks in VirtualBox

  - by PhilPursglove

  I'm struggling to understand how to do differencing disks in VirtualBox v3.1.0. I've created a Windows 2008 Server, but now I want to use that as a base image for a number of other servers. The help file has a description of what differencing disks are, but I can't find where it actually tells you how to do it. In the Storage dialog for a server I found the Differencing Disks checkbox: but when I check it, I'd expect it to then ask which image should be the parent so I could select my base image. Any pointers you can offer would be greatly appreciated!

  Read the article

• Starting phusion passenger on working apache

  - by fl00r

  Hi! I've got Apache (-v): Server version: Apache/2.0.63 Server built: Nov 29 2009 15:23:34 Cpanel::Easy::Apache v3.2.0 rev4899 I want to start new Sinatra application on passenger. I've just installed passenger gem. So now I need to set up apache configuration. In httpd.conf there are many settings of others applications on server. So I just can't reinstall apache with passenger-install-apache2-module. I need to set up exist Apache with passenger. What have I do now?

  Read the article

• Apache apxs instalation error

  - by MMRUSer

  I'm trying to install Apxs on Redhat EL 5 .. but it gives me an missing dependency error... The dependency is perl-base. But I have installed the perl base perl-base-2.12 still it gives me the same error. warning: apache-apxs-2.2.4-1.i386.rpm: Header V3 DSA signature: NOKEY, key ID 1bbd5459 error: Failed dependencies: perl-base >= 0:5.004 is needed by apache-apxs-2.2.4-1.i386 What is the issue here.. ?

  Read the article

• rpm installation error

  - by JiminyCricket

  im trying to install an RPM compat-db-4.1.25-9 on oracle linux enterprise, since its required to install WebCenter...however the rpm installation is throwing a warning and then not working [root@devsebl downloads]# rpm -i compat-db-4.1.25-9.rpm warning: compat-db-4.1.25-9.rpm: Header V3 DSA signature: NOKEY, key ID 9b3c94f4 [root@devsebl downloads]# rpm -q compat-db-4.1.25-9.rpm package compat-db-4.1.25-9.rpm is not installed any idea what that warning means and why its crashing there? i tried to use Yum, but its not available i guess: [root@devsebl downloads]# yum search compat-db Loaded plugins: security Warning: No matches found for: compat-db No Matches found

  Read the article

• OSX Apache2 Virtual Hosts Proxy Issue

  - by Daven Patel

  I'm using the following httpd-vhosts.conf file to host several sites on my MacBook. Apache Configuration The first two virtual sites (v3.local,ss.local) are giving back the following error messages in the Apache error log: [Thu Aug 30 15:12:04 2012] [error] (61)Connection refused: proxy: HTTP: attempt to connect to [fe80::1]:3002 (localhost) failed [Thu Aug 30 15:12:04 2012] [error] ap_proxy_connect_backend disabling worker for (localhost)] The third site test.local works fine without any issues. I can't seem to find out why the first two sites are responding with the listed issue. What could be causing it and how can it be resolved?

  Read the article

• Scanner Installation: Epson Perfection 3170 Photo, Windows 7 & Adobe Acrobat

  - by Galaxy5727

  I installed the Driver v3.04A from Epson's Web site -- epson12180.exe that is supposedly for Windows 7. Now, the scanner works by itself, i.e. if I hit the scan button I get a dialog menu and can scan with one of the options there but not with Adobe Acrobat. What I need is to have it working from Adobe Acrobat when I go File - Create PDF - From Scanner - Custom Scan... The current state is that I see Scanner: "Please select a device" and no other available devices. I remember I used to see at least 2 other like "WIA Epson Perfection" and smth like "Epson Perfection". "WIA Epson Perfection" is the option that I am hoping to see to move on. Scanner: Epson Perfection 3170 Photo System: Windows 7, 64-bit Software: Adobe Acrobat 9 Pro

  Read the article

< Previous Page | 215 216 217 218 219 220 221 222 223 224 225 226  | Next Page >