This blog is a late post. I was on vacation back to Beijing and Wuhan during the past Christmas time and came back on Jan 16, 2011. During my time in China, unfortunately, I could not write blog since Blogger is blocked. I knew this issue and I scheduled my posts before I left. I have been busy to pick up and organize my stuff in the past week. Now I am back to normal schedule, working, web learning, doing my project and enjoying my life.
This late post is on OWIN: Open Web Interface for .Net. I got this when I listened to Hanselminute podcast #244 one day before in last December. I wrote a note in my Evernote on this(my .Net notebook).
Afterwards, I tried Kayak, an example OWIN library. The result was very exiting. I spent about 1 hour time on a console project. With its running at work, I could access to the service from any box at my work to get HTML results. My box is Windows XP Prof, which suppose not having any IIS. To install a web server is very complicated. With OWIN or Kayak, based on this testing, I could create any web service. OWIN is a very simple framework. I'll further explorer its potentials. Most likely, I will use it as a REST based service to provide data. I have many projects or apps which depends on remote data such as SCADA historian data, SQL server or Oracle database. OWIN may be excellent solution as light weighted layer to provide data.
Sunday, January 30, 2011
OWIN Framework as WebService in .Net
Friday, January 21, 2011
Parse and Transform Text File by Using PowerShell (3)
In my previous blog, I described who I parsed the input text file from my debug result to generate the first report table. Now Let's continue to the second report.
Generate the Second Report
The second report is another summary report, based on the first report. It displays a list of methods and counts of their calls. In other words, it will list distinct method names in the first report, and take the max counter for each method as method call count.
I find out that to get the second report, it is not easy to use just one pipeline with a long chain of segment codes. Still, I'll continue to use pipelines with each one to generate temporary results. I'll use pipelines to get temporary results, and dynamically add properties to objects.
First, initialize some variables:
# second report
# initialize variables
$totalCount = 0
$i = 0
$ht=New-Object Collections.Hashtable
The first pipeline is used to pass the result of the above collection of objects ($results) as input. There will be no result out of the pipeline. Instead, I use the pipeline to update a hash table variable $ht, with the property of method name as its key.
The input objects are piped into the second segment which is a where clause to filter any object with its count property larger than zero.
# Filter out rows with count less or equals 0
$result | where { $_.$propertyNameForCount -gt 0 } `
Then the filtered object is passed to a block of codes, where the hash table variable $ht is updated with the object with the max count value:
| Select-Object -Property 'MethodName', $propertyNameForCount `
| %{
if ( $_ -ne $null ) {
$key1 = $_.'MethodName';
if ( $ht.ContainsKey($key1) -eq $false ) {
$ht.Add($key1, $_)
}
elseif ($_.$propertyNameForCount -gt $ht[$key1].$propertyNameForCount) {
$ht.Set_Item($key1, $_)
}
}
}
The values of the hash table $ht contain objects we need for the report. In addition to those objects, I need a total count of each method call count. This is done by a pipeline to update total count into the variable $totalCount:
$ht.Values `
| %{ `
if ( $_ -ne $null ) {
$totalCount += $_.$propertyNameForCount
}
}
After we get the total count, a new object is created in the same structure of properties as ones in the hash table, then add the new object to the hash table $ht:
# add total count to $ht1 table
$objValue = New-Object PSCustomObject
$objValue | Add-Member -type NoteProperty -Name 'MethodName' -Value '[Total count]'; # use XXX so that sorting to the last
$objValue | Add-Member -type NoteProperty -Name $propertyNameForCount -Value $totalCount;
$ht.Add("XXXX dummy key", $objValue);
Finally, the objects in the hash table are ready for the second report. The report is generated by the last pipeline: sorting by property 'MethodName', adding a sequence number as object property, and appending the table layout report to the output file:
# generate report
$ht.Values `
| Sort-Object -Property 'MethodName' `
| %{ # Add sequence column
$obj = $_;
$obj | Add-Member -type NoteProperty -Name 'No.' -Value $i;
$i++;
$obj;
} `
| ft -AutoSize -Property 'No.', 'MethodName', $propertyNameForCount >> $outputFile; # Format the result and out put to file
In summary, pipeline in PS is a nice-to-have feature. You may achieve the same result without using pipeline. I like PS pipeline's simple, fluent flow and powerful feature. As you can see, PS is also dynamic data type script language. Objects can be created on-fly and properties can be added or removed during the run-time. My parse script codes take the advantage of those two great features.
Friday, January 07, 2011
Parse and Transform Text File by Using PowerShell (2)
With the previous blog on the basic concepts and the project goals on your belt, now it is time to jump into the codes from 6000 feet height.
Define Input Variables
I could set some input parameters for my script module for easy use from command line. However, since my script is only for my own use and I have to update some values frequently when I use it, then I decided to just declare some variables with initial settings as a simple start:
$inputFile = 'C:\Tmp\test\fst_PO20100505.txt';
$outputFile = '{0}.txt' -f $inputFile;
[decimal]$durationLimit = -0.01;
Then some counter variables and a hash table variable $ht for later use are initialized:
$i = 0
$j = 1
$identityExPattern = "*duration: *" # expression pattern as a filter
$exPatternForCount = "*DAO::*" # epxression pattern for count
$identityPropertyName = "Duration"
$propertyNameForCount = "DAO count"
$ht = New-Object Collections.Hashtable;
Next I output a line a header to my output file:
# Output duration limit to the result file
'==== Result of "Duration > {0}" ====' -f $durationLimit >> $outputFile;
Those codes are straightforward. The duration limit is a filter, which is used to list only calls with duration larger than the filter value.
Generate the First Report
The first report is generated by one statement with a long list of chained segments of codes as a pipeline. The result is saved to variable $result.
The first segment in the chain is to get lines from input file:
$result = Get-Content $inputFile `
Then the second segment is a block of codes %{...}. This block takes the input to process and generate empty or a collection of objects as a result. The result can be piped to the next segment. The codes in the block is very simple, it updates the line number in a varable $i, and then takes the input object as it is. $_ is a special variable notation for the input object:
| %{ $i = $i + 1;
$_;
} `
Then each line is piped to the next where constrain statement:
| where {$_.trim().length -gt 0 -and ($_.trim().SubString(0,1) -eq "[") -and ($_.trim() -like $identityExPattern)} `This constrain clause is like a filter to remove un-expected string lines. The result of this filter will be a line which is not empty, the first none-empty char is "[" and the content of the string contains a substring of "duration: ". The interested lines are then piped into the next code block %{...}.
This block contains a lot of codes. They can be divided in to two parts. The first part is to split a line into an array variable $row, and to acuminate count of each method:
$row = (-split $_ ); # split a line into array by space
$bCount = $false;
$method = $row[7]; # example: [ 5/5/2010 9:55:03 AM duration: 0.19 ] AppUserDAO::loginUser
if ( $method -like $exPatternForCount )
{
$c = 1;
# update method in hash table with count value
if ( $ht.ContainsKey($method) )
{
$c = $ht.Get_Item($method) + 1;
$ht.Set_Item($method, $c);
}
else
{
$ht.Add($method, $c);
}
$bCount = $true;
}
The second part is to create an object based on the result of the first part: $row. The object is created by using "Select ... -InputObject ... -Property" statement. The -InputObject take the array of $row as input, and -Property defines a list of properties:
# create an object with properties: sequence, datetime, duration, DAO count, and methodName
$obj = select-object -input $row -prop `
@{Name='No.'; expression={$i;}}, `
@{Name='DateTime'; expression={[DateTime]($row[1] + ' ' + $row[2] + ' ' + $row[3]);};} , `
@{Name=$identityPropertyName; expression={([decimal]$row[5]);} }, `
@{Name=$propertyNameForCount; expression={ `
if ($bCount) { `
$ht.Get_Item($method); `
} `
else { `
0; `
} `
}
}, `
@{Name='MethodName'; expression={($method);} };
$obj; #output object
The last line of $obj will pass the object to the next segment of the pipeline.
The segment is a simple where clause, which filters out any duration smaller than the expected value:
| where { $_.Duration -gt $durationLimit } `The final pipe segment is to create another object based on input object. The purpose of this new object is for the first report, each property for a column in the report:
%{
$obj1 = Select-Object -Input $_ -Property `
@{Name='No.'; expression={$j;}}, `
@{Name='Org No.'; expression={$_.'No.';}}, `
@{Name='DateTime'; expression={$_.'DateTime';};} , `
@{Name=$identityPropertyName; expression={$_.$identityPropertyName;} }, `
@{Name=$propertyNameForCount; expression={$_.=$propertyNameForCount;} }, `
@{Name='MethodName'; expression={$_.'MethodName';} };
$j++;
$obj1; # output obj1
};The result of the above chained segments is a collection of objects for the first report. The result is assigned to variable $result, and then is output to a file:
# get the reults to output file as formatted table
$result | ft -AutoSize >> $outputFile;
Saturday, January 01, 2011
Parse and Transform Text File by Using PowerShell (1)
I have used my DebugLog class to investigate issues in Visual Studio projects. The debug messages are pushed to Visual Studio's output consol. The generated messages may be very extensive huge. For example, I had a case of a Windows application with performance issues of DAO calls. I got about 8233 lines of debug messages just from the start to the main window displayed. I copied the messages to a text file. It is 732K in size. I like to keep the raw messages there; however, it was hard to investigate issues with the extensive raw messages.
What I would like is to generate concise summary reports, for example, a list of method calls with durations in an order and a list of DAO calls with their counts. This is similar to the case to use XPath to parse and to transform an XML content to another format, such as a HTML table list.
PS came to my mind first. PS is a script based language; therefore, it is easy to give it a try. I am not an expert in PS. I just use it and learn it as I need. I spent some time to write codes and finally I completed a script module to get my expected result. Here is my review of the codes.
Basic Concepts
Before I jump deep into my PS codes, I would like to list brief explanations for some basic concepts.
Single value variables are dynamically declared in PS with prefix $. The data type can also be static in the format of [type]var.
Hashtable is a dictionary data type with a key and an associated value. The constant definition is @{[key1=value1,...]} or @{}.
# is used for comments.
Statements can be either separated by line break or terminated by ';' character. ` character is used as a continuing indicator.
Piping or pipeline is a very powerful feature in PS. By using |, twp segments of codes can be chained together, the output or results of the first segment being piped into the next segment of codes as an input. Not only strings can be piped, but objects can also be passed through the pipeline. You can write similar codes without pipelines, but by using it appropriately, your scripts may look much simple and easy to read, and you may like this unique and powerful feature of PS.
There are many great resources on web. For example, the first part of this blog tutorial on PS variables, arrays, and hashtables provides nice hands-on examples on PS basics.
The Goal of my Project
I call it as a project because I want to write script codes to reach my goal. Basically, I copy my debug messages from VS output console and save them to a text file. The goal of the project is to read the text file as input, parse each line and generate a list of reports, actually two reports in this project.
The first report is a table view of methods and their corresponding duration time. The second report is a table view of interested methods and their call counts.
Here is a partial section of the raw data:
[ 5/5/2010 3:12:58 PM ] MainSchedulingTool::posMenuItem_Click
[ 5/5/2010 3:12:58 PM ] POSelectionViewForm::POSelectionViewForm
[ 5/5/2010 3:12:58 PM ] POSelectionViewForm::poStartDateTimeFilterPicker_ValueChanged
[ 5/5/2010 3:12:58 PM duration: 0.00 ] POSelectionViewForm::poStartDateTimeFilterPicker_ValueChanged
[ 5/5/2010 3:12:58 PM ] POSelectionViewForm::poEndDateTimeFilterPicker_ValueChanged
[ 5/5/2010 3:12:58 PM duration: 0.00 ] POSelectionViewForm::poEndDateTimeFilterPicker_ValueChanged
[ 5/5/2010 3:12:58 PM ] PODAO::GetAllPOsByStatusAndOrderByEndDate
[ 5/5/2010 3:12:58 PM ] PODAO::GetPOs
[ 5/5/2010 3:12:59 PM ] PODAO::ReadPOData
[ 5/5/2010 3:12:59 PM duration: 0.00 ] PODAO::ReadPOData
[ 5/5/2010 3:12:59 PM ] PODAO::ReadPOData
[ 5/5/2010 3:12:59 PM duration: 0.00 ] PODAO::ReadPOData
[ 5/5/2010 3:12:59 PM ] PODAO::ReadPOData
[ 5/5/2010 3:12:59 PM duration: 0.00 ] PODAO::ReadPOData
[ 5/5/2010 3:12:59 PM ] PODAO::ReadPOData
[ 5/5/2010 3:12:59 PM duration: 0.00 ] PODAO::ReadPOData
...
Here an example of the first report:
No. Org No. DateTime Duration DAO count MethodName
--- ------- -------- -------- --------- ----------
1 161 5/5/2010 3:12:59 PM 0.38 1 PODAO::GetPOs
2 162 5/5/2010 3:12:59 PM 0.38 1 PODAO::GetAllPOsByStatusAndOrderByEndDate
3 164 5/5/2010 3:12:59 PM 0.41 0 POSelectionViewForm::POSelection...
4 807 5/5/2010 3:13:00 PM 1.56 1 VendorDAO::GetVendorData
5 808 5/5/2010 3:13:00 PM 1.56 1 VendorDAO::GetVendors
6 1450 5/5/2010 3:13:02 PM 1.83 2 VendorDAO::GetVendorData
7 1451 5/5/2010 3:13:02 PM 1.84 2 VendorDAO::GetVendors
8 1452 5/5/2010 3:13:02 PM 1.84 0 POSelectionViewForm::setupSortedPOList
9 2575 5/5/2010 3:13:13 PM 14.13 0 POSelectionViewForm::PopulatePOData...
10 2576 5/5/2010 3:13:13 PM 14.13 0 POSelectionViewForm::POSelection...
11 2729 5/5/2010 3:13:14 PM 0.16 0 MainSchedulingTool::setupC......
The first column is a sequence number. It is a sequence line number in the report. The second is similar to the line number in the raw text file. The remaining columns are the information about each method such date time, duration value, count of DAO method calls, and method names.
The following is an example of the second report:
No. MethodName DAO count
--- ---------- ---------
0 [Total count] 6
1 PODAO::GetAllPOsByStatusAndOrderByEndDate 1
2 PODAO::GetPOs 1
3 VendorDAO::GetVendorData 2
4 VendorDAO::GetVendors 2
...
Posts
