On Software

Microsoft: Silverlight Is (almost) Dead

Following up on my previous post: Ballmer: Microsoft's Strategic RIA Platform is HTML5, it appears that Microsoft is officially switching its strategy away from silverlight and to HTML5.

Highlighted in two articles, from the company and PDC itself:

“Silverlight is our development platform for Windows Phone,” he said. Silverlight also has some “sweet spots” in media and line-of-business applications, he said. But when it comes to touting Silverlight as Microsoft’s vehicle for delivering a cross-platform runtime, “our strategy has shifted,” Muglia told me. Silverlight will continue to be a cross-platform solution, working on a variety of operating system/browser platforms, going forward, he said. “But HTML is the only true cross platform solution for everything, including (Apple’s) iOS platform,” Muglia said.

Read more here:

The big question remaining is what will happen to the Silverlight platform? Will it replace WPF? Or will it head for the graveyard as fast as Windows Phone is destined to? One way or another, it looks like the Microsoft RIA platform is in for an interesting ride, and HTML5 is on it's way into the mainstream.

This is actually really exciting to me. Although HTML5 is in its early days, the future is clearly heading that way. As our customers demand an increasing amount of accessibility in their Single Dealer Platforms, I see HTML5 as the natural long-term solution. People have asked me around Lab49 whether I think that an SDP with realtime data could be built in HTML5 today. My answer has consistently been that if you only support the most modern browsers, than yes. It would be ambitious, you'd be the first, and with that ambition comes a risk higher factor than other alternatives (Flex/Silverlght/WPF), but I believe it is indeed possible, the only question now is who will be the first to do it.

Buckle your seatbelts, RIA is about to be a bumpy ride...

Ballmer: Microsoft's Strategic RIA Platform is HTML5

While browsing around the web the other day, I came across what appeared to be the most popular tech story of the day: here in Engaget:Ballmer: next release of Windows will be Microsoft's 'riskiest product bet' .

 So I went to watch the video, and found what no one seems to be talking about, the real gem of this Ballmer interview:

Interviewer: "HTML or Silverlight, what is Microsoft's lead strategy for developing rich internet applications? Silverlight, HTML5, or acquiring Adobe?"

Ballmer: "If you want to do something that is universal, there is no question, the world is going HTML5... The world is just pushing down that HTML5 path, and so are we... What we've really done [with Silverlight] in very good ways is to repurpose it as an important part of the client platform, so yes, you will embrace HTML5, but it's also clear people want client apps... So we're dedicated to Silverlight, but with a little different concept, and we're embracing HTML5."

Interesting... looks like Ballmer is saying Silverlight is on its way out as an RIA platform, and HTML5 is the companies strategic bet. Clearly Silverlight is not yet a client platform, but if Silverlight is becoming their strategic client platform, where is WPF going in the longer term?

Watch the video here for yourself, the interesting bits start 2 minutes in: http://www.youtube.com/watch?v=iI47b3a9cEI 

On Delivery - Successful Software Delivery and Lifecycle

Given all the recent chatter around Continuous Delivery, I thought it would be good to speak a bit about the practices we prefer in Lab49. Lab49 strives for a model where delivery comes first, and focus on implementing it across our projects. Although this process can vary project to project [due to project, team and environment variability], the main theme of successful delivery remains the same. To some, this model may be very familiar. It has proven to yield high-quality results across a range of projects, and has removed tension from the development process.

There are two components to a successful delivery model. The first component, which I will not talk about in-depth here, is a team which specializes in project and scope management.

The second component is more technical -- it is the mechanism by which we test, configure and deploy all our projects, on a continuous basis, across all environments. This component is what "Continuous Delivery" now refers to, and has been a part of Lab49 projects for a long time.

Putting testing and delivery first has a simple benefit: simply put, software has no value if it cannot be delivered into production. No matter how good of a system you write, without a production deployment, the software has never given anyone any value. Therefore, configuration and deployment are the most valuable components of a software project.

Let’s look at the planned lifecycle of a typical, classic, fixed-time development project:

In theory, this process process looks pretty good, in practice, here’s what happens:

Although planned, development typically eats into the testing, configuration and deployment time, leaving what is the most important step until the end.

This problem, along with the problem of scope changing through the "develop" cycle, is often solved with an agile approach:

Agile is an iterative software lifecycle where every two or three weeks the team checkpoints, shows what they’ve done, and re-prioritizes to make sure they are developing the right software. Teams will occasionally take a "deployment" iteration to get everything ready for deployment and send it out to UAT or production.

This proves to be a much better model. Testing is applied regularly, and the team re-prioritizes work to make sure they are always delivering what the business demands.

Some people take agile one step further, adding configuration and deployment ("delivery") into the development model:

By making delivery part of the cycle, and enforcing a configure + deploy time, we can ensure that whatever we are writing can be delivered to the client at any time in the process, and immediately begin adding value to the business. In an ideal case, configuration + deployment < 10 minutes.

In all Lab49 projects I’ve been part of, I have strived to reach this goal. It is a good benchmark to test yourself again, but is often difficult to achieve. Differences in environments such as change management tickets and other requirements can make it difficult to achieve such a deployment model, but some practices and organizational patterns can help make it easier.

Maintenance of configuration and deployment is the responsibility of every developer on a project. A task should never be considered finished until it can be configured and deployed to all of the environments at hand. Developers should feel involved in the process of delivery, and just like build errors, should receive instant feedback on the quality of their code against the deployment mechanism.

Getting developer involvement can only be achieved by instilling a level of responsibility in each developer towards the deployment process. This can be done through messaging to the team, as well as through the required deployment of a project at the end of each iteration, or, even better, at the end of each day. Minimizing deployment time becomes in the interest of individual contributors because the shorter the deployment time and pain, the faster they are out the door at the end of the day. In this way, the process itself provides positive reinforcement.

But the goal above was not only to provide a daily feedback, but an instant feedback. We are interested in minimizing the time between each line of code written, and the feedback we can provide about that codes integrity.

In a typical test-driven development process, the feedback cycle looks like this:

 Feedback is provided at every step along the way about the integrity of the code and its logic. The missing component, which cannot be tested in this model, is the integrity of the checkin, configuration and deployment.

So far in this document, we’ve reduced the time to deployment testing from the time of the whole project [figure 1, linear project], to a regular process in the development cycle [figure 3, agile with deployment], to a daily process through some constraints on the development team. To provide this feedback even more regularly, we can integrate it into our build process, just like we do with unit or behavior tests:

Upon checkin, a system watching the source repository runs a build and runs the tests [I recommend TeamCity]. Upon successful build/test, the deployment script is the run to the development server. If this succeeds, the last step is to run the integration tests. Integration tests are there to ensure that the deployment was successful. This can be as simple as hitting a few web pages in a web project, and as complex and ensuring standard business workflows are functional. I will admit that while this is nice to have, it is rare in practice to see a project with both good unit/behavior tests as well as good integration tests. In reality, people tend to choose one or the other.

The concepts reviewed here provide a number of distinct advantages over other project configurations:

  • Coders are always adding value, because their code is always deployable
  • Reduces overall project risk.
  • Decreases code’s time-to-market (or time-do-deployment) by ensuring changes can be rapidly released.
  • Configuration and deployment is tested regularly.
  • Developers become aware of the deployment process early, are able to contribute code to deployment, and receive instant feedback when they break the deployment, just as when they break the build.

These advantages increase ROI, reduce risk, get us to market faster, and overall, make software engineering more productive and fun.

Posted by mrdecav | 1 comment(s)

C# @ The MongoDB Conference!

Mongo DB 

I will be speaking at the MongoDB conference in New York City on May 21st. I will cover their .NET/C# Development segment. We have a great demo lined up, and I look forward to seeing you all there.

MongoDB is an excellent open-source database which specializes in document-oriented (object-oriented) storage. You can save complex objects and query them back at lightening fast speeds. I heavily recommend checking it out -- and using it in all your future projects :)

If you're interested in knowing more about MongoDB, follow the link.

If you wish to attend the conference, you can sign up here.

The desktop is dead (really, for real this time)

Time and time again noise builds around the (re)introduction of thin client computing, the death of the desktop OS and the emergence of a single, large networked computer that we all work on.  A super mainframe that is omnipresent across all devices and networks.

But it has not been since the days of the drab green screens of the main frames that we have seen a significant uptake in thin clients over heavy metal. This is now ripe for change.

The stars are aligning in the cloud computing movement, and this time, it won't be another 3com web computing device. The traditional precursors to the explosion of the web device market have been torn down - bandwidth is ubiquitous, price of technology has come down so dramatically that these devices are now comparatively cheap against their full-blown competitors, and most computing services that people have come to expect have an online incarnation.

But the most important change is cultural.  People are demanding cheap alternatives to desktops.  The tell tale signs are there - the netbook craze, the rich web phone interfaces (iPhone, Android), and even the Kindle are all thin clients that are heavily in demand. People have begun to see the value add for a consumer of not having to maintain a desktop OS - they expect it to work like their TV - managed centrally, and pushed down. Zero maintainence time. With a wide range of people now spending most of their computing time in web browsers, and the need for desktop apps marginalized, the consumer demand has shifted towards less expensive, internet-oriented devices.

Even with the netbook fire sale, the market for these devices still hasn't been properly defined. Current netbooks are essentially small laptops with desktop OSes. While these machines hit the price point consumers expect in thin-clients, some of the real value adds a thin client can provide to consumers have not been realized in netbooks. The OS running on these devices is unnecessarially large, boot times are slow, and features like cloud storage are still not out of the box.  Couple this with the poor form factors and tiny keyboards, and there is still a lot to be done in this market.  Consumers know they don't need full blown desktop machines, but so far have only been exposed to "small laptops".

Despite the demand for netbooks, consumers still don't know what these web devices are - there is still an opportunity for proper market definition. There are very few companies who can really make the idea of web devices sit with consumers; excellent marketing coupled with good software and hardware is critical to open the market up.  With so many people having lost their shirts in the web device game over the past decade, chances are slim this company will emerge as a startup. Apple is clearly an industry favorite for defining new markets like this.  With the marketing engine to hit the concept home with consumers and the technical understanding to make it happen, and recent rumors of a "big iPhone" or "Apple Tablet", it's beginning to look like the market definition will start there.  But there are other players who look like they will be emerging as players.  TechCrunch has the "Crunch Tablet", Kindle is looking more and more like a web device daily, and Nokia has expressed serious interest in expanding into net books.

While heavy metal desktops will never become fully eclipsed by web devices, the desktop OS market looks to be in terminal decline.

Update: ... Or maybe Google will be the one to define the market.

Posted by mrdecav | 6 comment(s)
Filed under: ,

Tracking moved files: An interesting tidbit on file references in OS X

Today I had to help someone with a problem with iTunes on Windows.  For those of you that have not had the pleasure of using iTunes on Windows, it sucks, like most software written by Apple for Windows.  That being said, this problem really had nothing to do with iTunes.  The problem was that iTunes was 'losing' music.  As it turns out, the music that the user saw as 'lost' turned out to have been moved from its original location (the desktop) to the 'Music' folder.  iTunes uses absolute paths in its media library to hold reference to the file.  When the file moves, the reference is broken, and thus, the music appears 'lost'.  Without iTunes open all the time (and due to the occasionally unreliable file system watcher), it is simply not possible to track these files.

But then I started wondering... Surely Apple wouldn't let this happen on OSX.  Details like this tend to be a place where Apple shines.  So what in the world does iTunes do if I move something from the 'Music' folder to my 'Desktop' in OSX.  I decided to try.  First I closed iTunes, then I copied a music file, went back to iTunes, and hit play.  Low and behold, it played, no problem.  Not only did this leave me confused, but it also left me interested, so I decided to do a little exploring...

From what I can tell, Apple foresaw this problem with their 'aliases' early on in their original Mac OS, so rather than referencing files by path, you can reference files by a unique ID (if they are on a partition that uses a supported file system).

I'm no expert on Cocoa, but from what I can tell, the details of this can be found in the File Manager APIs and FSRef.

And for some more interesting reading on the topic.

So next time you get the dreaded 'cannot find the target to this shortcut, do you want to try to find the file?' prompt, just think about those 1980's Apple employees and how they solved this problem...

Posted by mrdecav | 2 comment(s)
Filed under: , ,

The Browser Love Affair, RIA and JavaScript as the 'new byte code'

Jump to a section of this article:

Why do consumers love the browser?

One of the things I still have not fully understood is the consumer's affection for the browser.  Unlike desktop applications, consumers seem to feel more comfortable and safe in a browser.

Perhaps it is the familiarity, always knowing where you are with a URL, the 'safety' of not messing up your computer, the security of having data stored in a cloud where it doesn't need backup, or the easy way to get away from where you are with the back button.  Add to this the familiarity of having an interactive 'page' over a bunch of forms and dialog boxes and popup windows, and the element of design and simplicity on the web, web developers were quicker to embrace a "less is more" approach, and web pages are typically thought through and designed by professional designers more often than desktop applications.

When you take all of these things into account, it makes a bit of sense why consumers have shown a preference for web based applications, and it is no surprise that most of the WPF applications we've seen emerge have followed the same basic browser concepts (back button, page like UI, navigation at top, hyperlinks, etc).

For someone who tries to stay away from web development, my recent past has been riddled with all the latest buzz-words on the web - Silverlight, Flash, AIR and JavaScript are all tools that I've had to engage with.  I've built trading systems in Flash and my most recent application is RIA in HTML + JavaScript...  None of these platforms have had the allure of WPF for me, the developer, but its what clients have asked for, and clearly what consumers are interested in.

 

Who is winning the war in RIA (Flex, Silverlight, HTML+JavaScript)?

Why is it that we feel the need to place browser chrome around everything, and when we do, what platform has emerged as dominant for the new breed of complex applications (Google Maps, docs, etc) that we've come to expect in the browser?

With out a doubt, HTML and JavaScript has been a hands-down winner.  Why is it so hard to name a single site that is a pure Flash or Silverlight application.  HTML + JS is certainly not easier to program, and Flash is now ubiquitous, so the concern of requiring a plugin is not there, is there something about interacting with the native browser, not a plugin, that has some appeal. 

I have become a firm believer that as a non-software company, it is your interest to leverage Microsoft-based development technologies as much as possible.  The ability to quickly get things done in .NET and its surrounding technologies is unparalleled.  When you are a software company though, it is much harder to justify.  Tying yourself to Microsoft, one who likely is or will become a competitor in your space is risky.  Hiring good web people (if you're a web startup) is harder because the .NET development culture tends to do things "the Microsoft way". And your acquisition story becomes limited (Google, eBay, Facebook, etc is less likely to by a .NET based system because it doesn't fit in their infrastructure).  For this reason, it may be clear why Silverlight is and will not become the dominant application platform on the web any time soon. Perhaps many web companies have the same fear of Adobe Flash.  By contrast, HTML and JavaScript is "open" and is not dominated by any particular entity.

The decision of most web companies to stick with HTML, despite the rich features of other platforms, is simply a business decision.  HTML will always be there, is relatively reliable, relatively backwards compatible (Adobe is pretty good at breaking builds of older code with newer versions of Flex), and does not tie you to the whims of any one company or platform.

 

JavaScript as Byte Code - C#, Java, etc to JavaScript Compilers and Cross-Compilers

HTML is great for document layout.  Its fast, clean and everyone knows it.  Sprinkle a bit of JavaScript on that, and you have a nice interactive document.  But HTML was never designed to be the basis for an application platform, and really is a pretty poor application platform.  But despite the features of other platforms, and the number of times HTML and JavaScript is pronounced dead, something breaths life back into it.

It started with FireFox, and continued on with the awefully impressive Google Maps.  Google has a large interest in keeping the browser alive and relevant.  The longer it does so, the longer it has a platform on which to compete with Microsoft that is already deployed to 99% of the world's user base.  A "Google Flash" may not be feasible, because of the deployment nightmare involved, but as HTML evolves with offline capabilities, richer CSS and faster JavaScript engines (see Google Chrome), it becomes a platform that can largely compete with the rest.

And for this reason, Google has cleverly turned the browser into their own RIA platform.   The "Google Virtual Machine", as you may call it.  To do this, they've leveraged the existing tools that Java has to offer, and created Google Web Toolkit - a Java compiler that can compile Java into cross-browser JavaScript.  And on the client side, they've added Google Chrome.  You can easily consider Google Web Toolkit and Google Chrome to be a full RIA platform, with JavaScript as the byte code transport.  And not only that, but its an RIA platform fully backwards compatible with the existing deployed base of browsers.

Perhaps the most interesting thing is that Google Web Toolkit is not the only JS or cross-compiler.  A variety for compilers have been built for this purpose.  Microsoft Research has Script#, which compiles C# to JavaScript, and a bunch of projects use JSC, a cross compiler that compiles MSIL into JavaScript (and therefore any .NET language).  With a whole variety of tools out there by different vendors to compile to JavaScript, it appears that JavaScript has become the new universal byte code.  It can be processed by a whole variety of browsers, and produced by a variety of tools - one of the only computing platforms that that is true for.

And compiled JavaScript can also be pretty fast and make applications that are rich and look good.  While it doesn't compare to the offerings of Cocoa, WPF or Silverlight, the quality of application that can be achieved is good enough for most consumers.  In the case of GWT, it is not only fast, but also efficient - and best of all, code can be shared amongst client and server.

 

I don't like HTML as an application platform, and I am not very fond of running every application inside the browser chrome.  But as business strategy and deployability have become increasingly important to me in my recent venture, an RIA application based around the native browser has become my platform of choice.  Cross compilation, in my experience, has proven an effective way to make a very rich client that can even handle push-data.

Introducing PowerShell Pages - Script Style HTML Rendering

Early in the web development world, scripting languages such as ASP or PHP were used to compose pages. Although this proved great for relatively static pages, the dynamic web, filled with rich applications called for a more powerful framework. Thus, frameworks like ASP.NET were born.

ASP.NET solved a good number of problem spaces, but has made creating simple pages (such as a resume or menu, or other primitive list of data) more cumbersome. With the world of COM development becoming less common and less preferable, the gap for a scripting language to replace VBScript/ASP is needed. PowerShell scripting has filled the gap left by the demise of VBScript, but nothing has come along to replace ASP.

PowerShell Pages is an ASP like language, based on the PowerShell runtime. Using a simple HTTP Handler, ASP.NET can render pages scripted using PowerShell script (including cmdlets, and CLR/.NET objects) to the web. Simple, fast and intuitive programming for simple pages that just need to display some content.

The PowerShell Pages project is an open source project that I am starting.  Its implementation will be based on ASP.NET using a simple handler capable of consuming PowerShell HTML (PSH) scripts and writing HTML.  Because the script is hosted in ASP.NET, the ASP.NET HttpContext and the other components of the object model are available.  PSH scripts can work side-by-side with ASPX pages.

Ready to see what PowerShell Pages look like?
Sample Page:
http://decav.com/psp/resume.psh
Sample Page Code: http://decav.com/psp/viewsource.psh?page=resume.psh
View Source Code: http://decav.com/psp/viewsource.psh?page=viewsource.psh

Join the Project - Visit the PowerShell Pages CodePlex project workspace

Posted by mrdecav | 2 comment(s)

Gatsb.com Now Open Source (A Case Study in Geotagging, Workflow Foundation, SMS/MMS Parsing)

After hosting Gatsb.com for a year, I have decided to open source the project.  I haven't had time to promote it the way I wanted to promote it, and it therefore hasn't caught on with too many users.  I've been busy with other projects, other (grander) ideas, and think that this one should now belong to the community.

You can download the source code here: http://code.google.com/p/gatsb/

If you want to contribute to the project, please feel free to contact me at andreblog@decav.com

This project is a great case study of a bunch of new Microsoft technologies.  It includes Workflow Foundation, .NET 3.0, Microsoft Virtual Earth and ASP.NET Ajax.

One of the coolest features is how it keeps open a "session" with an SMS client.  Workflow Foundation is used to persist the state and when a new SMS comes in, it checks if theres an existing session for that SMS.  If there is, then it will revive the workflow and continue the session.  Cool stuff!

If you ever wondered how to create a database of geotagged entries, use ASP.NET ajax to make a snazzier site, or establish a user interface with any mobile device, I suggest checking out the code!

Posted by mrdecav | with no comments
Filed under: , , ,

PowerShell: Repeat Yourself with the Command "Scratch Pad"

Heres a useful snippit of code I wrote as a utility and thought I'd post up for everyone.  Very often when using a command line, you need to repeat a set of commands.  For example, copy several files:

copy "c:\blah.txt" "c:\otherPlace\blah.txt"
copy "c:\dev\blah.txt" "c:\otherPlace\blah2.txt"

History is okay for this, but if you have a set of commands you need to run, you need to run them each individually.  Also, if you go to do something else, they'll eventually lose their spot in the command history.

To solve this, I created a "scratch pad".  Nothing special, but definately useful.

new-scratch "MyFirstScratch"
copy "c:\blah.txt" "c:\otherPlace\blah.txt"
scratch
copy "c:\dev\blah.txt" "c:\otherplace\blah2.txt"
scratch
## Do some other stuff here...
get-scratch # prints out the contents of the scratch.
invoke-scratch

This scratch pad is holds a set of commands that you want to rerun later.  You can then call invoke-scratch to run the scratchpad.  You can have multiple scratch pads, and use them by just adding the name to the end of the function (get-scratch "MyFirstScratch").  By default all scratch pad commands will default to the last used scratch pad.

The example below shows how to shorten the sample above:

new-scratch "MyFirstScratch"
copy "c:\blah.txt" "c:\otherPlace\blah.txt"
copy "c:\dev\blah.txt" "c:\otherplace\blah2.txt"
scratch 2   # Saves the last 2 items to the scratch pad

This can be very useful for writing scripts too.  the Save-Scratch command lets you export your scratch to a file.

Heres the code.  Add it to your profile and go.  Enjoy!

function New-Scratch($name) {
  if ($global:scratchPads -eq $null) {
    $global:scratchPads = @{}
  }
  
  $pad = new-object System.Collections.ArrayList
  $global:scratchPads.Add($name, $pad)
  $global:currentScratchPad = $pad
  $global:currentScratchPadName = $name
}
function Get-Scratch($name=$null) {
  if ($name -ne $null) {
    $global:currentScratchPad = $global:scratchPads[$name]
    $global:currentScratchPadName = $name
  }
  
  return $global:currentScratchPad
}
function Scratch([int]$count=1,$name=$null) {
  if ($name -ne $null) {
    $global:currentScratchPad = $global:scratchPads[$name]
    $global:currentScratchPadName = $name
  }
  
  $hist = get-history
  
  if ($name -eq $null) {
    $name = $global:currentScratchPadName
  }
  
  for ($i=$hist.Length-$count; $i -lt $hist.Length; $i++) {
    $global:currentScratchPad.Add($hist[$i].CommandLine) | out-null
  }
}
function Invoke-Scratch($name=$null) {
  if ($name -ne $null) {
    $global:currentScratchPad = $global:scratchPads[$name]
    $global:currentScratchPadName = $name
  }
  
  foreach ($item in $global:currentScratchPad) {
    write-host ">> $item"
    invoke-expression $item
  }
}
function Save-Scratch($path, $name=$null) {
  if ($name -ne $null) {
    $global:currentScratchPad = $global:scratchPads[$name]
    $global:currentScratchPadName = $name
  }
  
  $global:currentScratchPad | out-file $path
}
Posted by mrdecav | with no comments
Filed under: ,

Invoking PowerShell Scripts from An Application

If you want to run your PowerShell scripts from a piece of code you're writing, and would prefer not to use Process.Start, you can easily use the PowerShell runtime classes to run those scripts for you, completely in-process.

The code snippit below shows how you can create a Runspace and pass it some simple commands.

List<Command> commands = new List<Command>();
commands.Add(new Command("set-location c:\\"));
commands.Add(new Command("./MyScript.ps1 'myParameter'", true));

using (Runspace runspace = RunspaceFactory.CreateRunspace())
{
    runspace.Open();
    using (Pipeline pipeline = runspace.CreatePipeline())
    {
        foreach (Command cmd in commands)
        {
            pipeline.Commands.Add(cmd);
        }
        pipeline.Invoke();
    }
    runspace.Close();
}

One thing to note here is you do not have a PSHost, so if you have a write-host anywhere, it will fail with the following exception:

CmdletInvocationException: Cannot invoke this function becasue the current host does not implement it.

To get around this, you can remove write-host from your scripts and just have them write single lines, or you can implement a simple PSHost (although this is a bit harder than it sounds).

Posted by mrdecav | with no comments
Filed under: , ,

Solution: "Unknown error", LDAP connection strings and System.DirectoryServices

If you've been using System.DirectoryServices.DirectoryEntry class, or the newer System.DirectoryServices.AccountManagement namespace to access your LDAP or Active Directory server, you may have experienced the following error:

COMException: "Unknown error (0x80005000)"

This can happen for numerous reasons, but one of the most frustrating and overlooked reason's I've found for this problem is when your LDAP connection string is malformed.  One of the most common malformations is in the case sensitivity of the LDAP:// component.  For example, LDAP://myServer/cn=users,dc=myserver,dc=com is a valid connection string, however ldap://myServer/cn=users,dc=myserver,dc=com is not.

If you use the Uri or UriBuilder classes, the builder may lowecase your scheme.  Always make sure to recapitalize the scheme when passing it into DirectoryEntry or any other API.

Posted by mrdecav | with no comments
Filed under: ,

Cluster Primitives: MPI, MPI.NET, Large Data, and Passing Classes

The Message Passing Interface (MPI) standard, and its .NET implementation, MPI.NET have been some of the cornerstones of development on compute clusters.  The standard supplies a simple yet primitive way of both sending and receiving data between running compute processes.

The large advantage of MPI has been a mix of its simplicity and speed.  A call to MPI Send on one node and MPI Receive on another block both callers until the operation is complete.  Some more complex calls, such as MPI Scatter and MPI Gather allow a single node to distribute data to a set of nodes or retrieve it from a set of nodes.  An MPI Barrier allows all nodes to stop until they have all reached the agreed upon place in code, then allowing them to continue.  Such primitives allow a distributed set of processes to communicate, do some work, and then share values that each needs to continue with eachother.  Because this is all done with some low level, bare metal socket tricks and/or shared memory, the result is blazingly fast communication.

With this simplicity however, comes a trade off.  MPI has been a standard for nearly 20 years and has changed very little since its inception.  The way we program today has changed drastically, especially with managed languages such as C#.  No longer do we tend to worry about memory allocation, or dealing with raw memory.  Today, most languages have a concept of automatic memory allocations, garbage collection and type safety.  Although the primitives in MPI are unparalleled in simplicity for allowing multiple processors to communicate about a shared set of work, some striking limitations are found once we dig a bit deeper.

When I started with MPI.NET, I found the interface very simple.  An Mpi.Send<T>(obj) would send an object to a waiting client.  Mpi.Receive<T>() would give you back that object.  Nothing could be simpler.  In my example, however T happened to be a class that contained a byte[] of an undetermined size.  Once run on the cluster, the size of the byte[] I was passing increased dramatically, and an unexpected exception occured:

AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory has been corrupted.

After lengthy investigation, I found that MPI.NET was attempting to pin some memory in the .NET GC heap and pass that memory location as a buffer to the underlying MSMPI stack.  In doing so, it did not allocate enough memory for my large byte[], causing the write to try to write into the GC heap, thus throwing the exception.  In my case, to solve this, I created a large enough buffer and passed it into an override of Mpi.Receive<byte>(byte[]).  This overload pins the entire array that was passed in on the GC, and then passes that to the MSMPI stack.  On the send side, I manually serialized my class, checked the length (to ensure I would not overflow the receive buffer) and sent the byte[] instead of the raw class.  This solution does not take into accound messages larger than my expected buffer.  For that, I would have needed to chunk down the data.

The moral of the story here is sending primitives, arrays of primitives or fixed-sized structs over MPI.NET (which is the most common scenario) is a great use of a very fast messaging interface.  Once your demands get more complex, the MPI stack gets less favorable, not because of its inability to send more complex messages, but because of the manual labor involved in serializing and chunking down data.

It is no wonder that the HPC community is moving away from the traditional methods of MPI and communication across a set of processors to a Service Oriented (SOA) model. The benefits of using existing components, such as WCF and its NetTcpBinding, the threading models, serialization and transport models, and other features already provided by these frameworks outweights the possible performance penalty.  Problems such as the one explained above simply do not happen with frameworks like WCF.  Furthermore, although the underlying concepts of MPI and its simple messaging model are very simple and appealing, the overall development, maintainence and debugging of a SOA application is much simpler than that of a MPI application.  The amount of code complexity and custom code drops when compared to an MPI implemenation.

The general industry trend seems to be towards SOA models.  Microsoft Windows HPC Server 2008 is a great example of this.  HPC Server uses WCF to distribute load across the cluster, and can even dynamically scale resources depending on demand of a particular service.  Platform, another industry competior has been building with a SOA model for some time now. 

I'm looking forward to playing with HPC Server 2008 and WCF more as time progresses.  I think that the WCF model will solve a whole bunch of headaches that one incurs when trying to communicate over perhaps over simple primitives such as MPI.  Many models and workloads simply do not require the type of communication MPI provides, and using MPI can be like fitting a square peg into a round hole.  This is not to say MPI does not have its place, many complex processes do require constant communication between a set of workers, however I believe many of the problems we use HPC for today can distributed using SOA in a much simpler fashion.

Ruby, From The Eyes of a C# .NET Developer

I recently got to playing with Ruby, something that some colleagues in Lab49 have been big fans for some time.  I've never been a big fan of scripting languages, but have grown more of an appreciation for functional programming over the past several months and thought I would give it a try.

Ruby is a very smart language, and I can certainly see why it has some appeal.  The "don't repeat yourself" and "defaults over configuration" aspects of the language and its framework are really nice for cranking out simple applications.

ORM and Inference of Properties from database

I am a big fan of the objects that automatically get their properties from the database (a Customer object will automatically be linked with the Customers table when pulled through the ORM).  Things like this make it dirt simple to crank out web projects of moderate size without writing tons of redundant code (as is the case with a classic OO approach, using adapters, abstractions, etc).

Even though this is a nice feature, and great for many projects, I am concerned that this is simply not sufficient for larger enterprise apps.  Very often your app layer and database layer should be different, as one does not always properly map to the other.  I suppose that you could achieve this nicely with views, however for large applications, the lack of abstraction seems a little brittle to me.  At the same time, if you did change something in your database, you'd still have to change your abstraction logic, so maybe this isn't all that bad.

Syntax

The syntax of Ruby is very clean and straight forward.  I like the bare bones structure, nothing more than needed is written.  Reading into it a bit more, I found it interesting that various things can be written in several ways, for example:

while i < 3
   print(i)
end

can also be written

print(i) while i < 3

While conceptually this is nice (the latter "reads" better), it is something I have a really hard time accepting.  Yes it is nice not to have to write things like "end", however, syntactical differences like this (and others, such as the optional parenthesis around method parameters) concern me.  While its great that you can develop your own style, and do what feels comfortable to you, it may be very confusing to other developers.  An argument can be made either way, after call you can do this:

if (i < 3) Console.WriteLine(i);

in C#.  Nothing stops you.  At the end of the day, sloppy code is sloppy code, and its really a matter of having a well trained developer, not a strict language.

Naming Conventions

Here's something that threw me off.  Ruby does a very good job of making names intuitive, however some of their names seem to break their own concepts.  First of all, names like to_s are just sloppy.  If you're making a lanugage thats supposed to be readable, why write cryptic names like this?  Also, whats the deal with properties count and length being synonyms?  Why have both?  It seems relatively dangerous to me to have methods that mean the same thing on objects.  A developer who sees count in one place and length in another may think they are different, something that certainly doesn't help code legibility.

 

Overall, Ruby has been fun to learn (although I'm still a novice), and I can certainly see its value.  I'm not sure if I agree with some of its "friendly" tendencies -- being able to write whatever you feel and have it probably compile is not necessarily good.  Just because code compiles doesn't mean its good code -- some bugs the compiler may catch now become runtime bugs.  On the other hand, things like the ORM make it very easy to build rich apps with little code, something that .NET still comes up short on (just because designers generate LINQ classes doesn't mean that the code isn't there).  Looking forward to playing more!

 

Posted by mrdecav | with no comments
Filed under: , ,

WPF Compute Cluster Monitor Demo - Using 3D, Transparency and Styles

Here's a cool demo that Ronald Lintag and I threw together for Lab49.  The application is supposed to demo a basic GUI that shows the status of nodes and jobs on a compute cluster.  We're showing off the use of WPF 3D and styles to create a cool looking console for a sys-admin.  You can imagine how something developed off this base could turn into a real product.

This demo is a bit rough around the edges, and definately needs polish, but its still worth a look.

  • Press J to bring up the job list.
  • Use the dropdown at the bottom of the job list to add some jobs.
  • Click on job to view details about their tasks.
  • The nodes in the background vary from gray to green to red depending on their utilization

Click here to launch the application (.NET 3.5 required)

Here are a few screenshots for those of you who are not adveturous enough for the ClickOnce...

More Posts Next page »