While doing some reading on Chinese quality of life and pollution last week, I noticed some interesting figures released by the WorldBank. The estimated cost to GDP of pollution emitted by China is a total of 5.8%. That's rediculously high; every year, China is losing 5.8% of their GDP to damage to their environment.
Whats more startling is what this means for total GDP growth of the county. In 2009, China's GDP is expected to grow by about 6.5% -- a numer low due to recession, but still considered high by global standards. Because the cost of pollution is an externality, and to best of my knoledge is not included in the official GDP numbers, this growth does not include environmental pollution. When you subtract the expected loss of GDP due to pollution from the total GDP of China, you're left with dismal 0.7% annual GDP growth. Ouch! 89% of their growth is chopped from this single externality.
China is essentially gutting their natural resources and destroying their habitat with enormous amounts of externalities in order to provide us with inexpensive goods.
In most cases, this scenario leads to market failure. China may be thinking that they can live with temporary loss while they attempt to lift their industry into the modern age - a long term investment using the capital locked in their public land. Certainly the short-term affect of removing the fully-realized costs from their exports and domestic products has been a positive one. But with steadily increasing pollution and dwindling resources, they cannot keep it up forever. At some point, these externalities need to be corrected or the market (and mother nature) will correct it for them. Hopefully she'll keep the fallout off our shores. Scary stuff!
Time and time again noise builds around the (re)introduction of thin client computing, the death of the desktop OS and the emergence of a single, large networked computer that we all work on. A super mainframe that is omnipresent across all devices and networks.
But it has not been since the days of the drab green screens of the main frames that we have seen a significant uptake in thin clients over heavy metal. This is now ripe for change.
The stars are aligning in the cloud computing movement, and this time, it won't be another 3com web computing device. The traditional precursors to the explosion of the web device market have been torn down - bandwidth is ubiquitous, price of technology has come down so dramatically that these devices are now comparatively cheap against their full-blown competitors, and most computing services that people have come to expect have an online incarnation.
But the most important change is cultural. People are demanding cheap alternatives to desktops. The tell tale signs are there - the netbook craze, the rich web phone interfaces (iPhone, Android), and even the Kindle are all thin clients that are heavily in demand. People have begun to see the value add for a consumer of not having to maintain a desktop OS - they expect it to work like their TV - managed centrally, and pushed down. Zero maintainence time. With a wide range of people now spending most of their computing time in web browsers, and the need for desktop apps marginalized, the consumer demand has shifted towards less expensive, internet-oriented devices.
Even with the netbook fire sale, the market for these devices still hasn't been properly defined. Current netbooks are essentially small laptops with desktop OSes. While these machines hit the price point consumers expect in thin-clients, some of the real value adds a thin client can provide to consumers have not been realized in netbooks. The OS running on these devices is unnecessarially large, boot times are slow, and features like cloud storage are still not out of the box. Couple this with the poor form factors and tiny keyboards, and there is still a lot to be done in this market. Consumers know they don't need full blown desktop machines, but so far have only been exposed to "small laptops".
Despite the demand for netbooks, consumers still don't know what these web devices are - there is still an opportunity for proper market definition. There are very few companies who can really make the idea of web devices sit with consumers; excellent marketing coupled with good software and hardware is critical to open the market up. With so many people having lost their shirts in the web device game over the past decade, chances are slim this company will emerge as a startup. Apple is clearly an industry favorite for defining new markets like this. With the marketing engine to hit the concept home with consumers and the technical understanding to make it happen, and recent rumors of a "big iPhone" or "Apple Tablet", it's beginning to look like the market definition will start there. But there are other players who look like they will be emerging as players. TechCrunch has the "Crunch Tablet", Kindle is looking more and more like a web device daily, and Nokia has expressed serious interest in expanding into net books.
While heavy metal desktops will never become fully eclipsed by web devices, the desktop OS market looks to be in terminal decline.
Update: ... Or maybe Google will be the one to define the market.
Today I had to help someone with a problem with iTunes on Windows. For those of you that have not had the pleasure of using iTunes on Windows, it sucks, like most software written by Apple for Windows. That being said, this problem really had nothing to do with iTunes. The problem was that iTunes was 'losing' music. As it turns out, the music that the user saw as 'lost' turned out to have been moved from its original location (the desktop) to the 'Music' folder. iTunes uses absolute paths in its media library to hold reference to the file. When the file moves, the reference is broken, and thus, the music appears 'lost'. Without iTunes open all the time (and due to the occasionally unreliable file system watcher), it is simply not possible to track these files.
But then I started wondering... Surely Apple wouldn't let this happen on OSX. Details like this tend to be a place where Apple shines. So what in the world does iTunes do if I move something from the 'Music' folder to my 'Desktop' in OSX. I decided to try. First I closed iTunes, then I copied a music file, went back to iTunes, and hit play. Low and behold, it played, no problem. Not only did this leave me confused, but it also left me interested, so I decided to do a little exploring...
From what I can tell, Apple foresaw this problem with their 'aliases' early on in their original Mac OS, so rather than referencing files by path, you can reference files by a unique ID (if they are on a partition that uses a supported file system).
I'm no expert on Cocoa, but from what I can tell, the details of this can be found in the File Manager APIs and FSRef.
And for some more interesting reading on the topic.
So next time you get the dreaded 'cannot find the target to this shortcut, do you want to try to find the file?' prompt, just think about those 1980's Apple employees and how they solved this problem...
Jump to a section of this article:
Why do consumers love the browser?
One of the things I still have not fully understood is the consumer's affection for the browser. Unlike desktop applications, consumers seem to feel more comfortable and safe in a browser.
Perhaps it is the familiarity, always knowing where you are with a URL, the 'safety' of not messing up your computer, the security of having data stored in a cloud where it doesn't need backup, or the easy way to get away from where you are with the back button. Add to this the familiarity of having an interactive 'page' over a bunch of forms and dialog boxes and popup windows, and the element of design and simplicity on the web, web developers were quicker to embrace a "less is more" approach, and web pages are typically thought through and designed by professional designers more often than desktop applications.
When you take all of these things into account, it makes a bit of sense why consumers have shown a preference for web based applications, and it is no surprise that most of the WPF applications we've seen emerge have followed the same basic browser concepts (back button, page like UI, navigation at top, hyperlinks, etc).
For someone who tries to stay away from web development, my recent past has been riddled with all the latest buzz-words on the web - Silverlight, Flash, AIR and JavaScript are all tools that I've had to engage with. I've built trading systems in Flash and my most recent application is RIA in HTML + JavaScript... None of these platforms have had the allure of WPF for me, the developer, but its what clients have asked for, and clearly what consumers are interested in.
Who is winning the war in RIA (Flex, Silverlight, HTML+JavaScript)?
Why is it that we feel the need to place browser chrome around everything, and when we do, what platform has emerged as dominant for the new breed of complex applications (Google Maps, docs, etc) that we've come to expect in the browser?
With out a doubt, HTML and JavaScript has been a hands-down winner. Why is it so hard to name a single site that is a pure Flash or Silverlight application. HTML + JS is certainly not easier to program, and Flash is now ubiquitous, so the concern of requiring a plugin is not there, is there something about interacting with the native browser, not a plugin, that has some appeal.
I have become a firm believer that as a non-software company, it is your interest to leverage Microsoft-based development technologies as much as possible. The ability to quickly get things done in .NET and its surrounding technologies is unparalleled. When you are a software company though, it is much harder to justify. Tying yourself to Microsoft, one who likely is or will become a competitor in your space is risky. Hiring good web people (if you're a web startup) is harder because the .NET development culture tends to do things "the Microsoft way". And your acquisition story becomes limited (Google, eBay, Facebook, etc is less likely to by a .NET based system because it doesn't fit in their infrastructure). For this reason, it may be clear why Silverlight is and will not become the dominant application platform on the web any time soon. Perhaps many web companies have the same fear of Adobe Flash. By contrast, HTML and JavaScript is "open" and is not dominated by any particular entity.
The decision of most web companies to stick with HTML, despite the rich features of other platforms, is simply a business decision. HTML will always be there, is relatively reliable, relatively backwards compatible (Adobe is pretty good at breaking builds of older code with newer versions of Flex), and does not tie you to the whims of any one company or platform.
JavaScript as Byte Code - C#, Java, etc to JavaScript Compilers and Cross-Compilers
HTML is great for document layout. Its fast, clean and everyone knows it. Sprinkle a bit of JavaScript on
that, and you have a nice interactive document. But HTML was never
designed to be the basis for an application platform, and really is a pretty poor application platform. But despite the features of other platforms, and the number of times HTML and JavaScript is pronounced dead, something breaths life back into it.
It started with FireFox, and continued on with the awefully impressive Google Maps. Google has a large interest in keeping the browser alive and relevant. The longer it does so, the longer it has a platform on which to compete with Microsoft that is already deployed to 99% of the world's user base. A "Google Flash" may not be feasible, because of the deployment nightmare involved, but as HTML evolves with offline capabilities, richer CSS and faster JavaScript engines (see Google Chrome), it becomes a platform that can largely compete with the rest.
And for this reason, Google has cleverly turned the browser into their own RIA platform. The "Google Virtual Machine", as you may call it. To do this, they've leveraged the existing tools that Java has to offer, and created Google Web Toolkit - a Java compiler that can compile Java into cross-browser JavaScript. And on the client side, they've added Google Chrome. You can easily consider Google Web Toolkit and Google Chrome to be a full RIA platform, with JavaScript as the byte code transport. And not only that, but its an RIA platform fully backwards compatible with the existing deployed base of browsers.
Perhaps the most interesting thing is that Google Web Toolkit is not the only JS or cross-compiler. A variety for compilers have been built for this purpose. Microsoft Research has Script#, which compiles C# to JavaScript, and a bunch of projects use JSC, a cross compiler that compiles MSIL into JavaScript (and therefore any .NET language). With a whole variety of tools out there by different vendors to compile to JavaScript, it appears that JavaScript has become the new universal byte code. It can be processed by a whole variety of browsers, and produced by a variety of tools - one of the only computing platforms that that is true for.
And compiled JavaScript can also be pretty fast and make applications that are rich and look good. While it doesn't compare to the offerings of Cocoa, WPF or Silverlight, the quality of application that can be achieved is good enough for most consumers. In the case of GWT, it is not only fast, but also efficient - and best of all, code can be shared amongst client and server.
I don't like HTML as an application platform, and I am not very fond of running every application inside the browser chrome. But as business strategy and deployability have become increasingly important to me in my recent venture, an RIA application based around the native browser has become my platform of choice. Cross compilation, in my experience, has proven an effective way to make a very rich client that can even handle push-data.
This post is part of a series of essays on things unrelated to software development. You can find all the essays here.
The compensation structures on Wall Street have long been criticized as excessive, lavish and unnecessary. Banks have consistently justified such salaries, noting the executives being compensated brought in big earnings for the bank. When the bank balance sheets were positive, there was little incentive to attempt to correct the salaries, however with most banks now in dire straits, many have begun to take a more firm tone with banks.
This concern was underlined this week by Obama’s new government enforced incentive cap of $500k per year for top executives in banks receiving federal aid. While Obama has found a significant problem in banks, their compensation structure, his cap is not only horribly low, but may also have a significantly negative effect on banks. Without that ability to recruit top talent into executive positions, banks will have more trouble hiring than the offices of Mr. Madoff.
Free markets tend to assume that people will do what they consider in their best interest. Consider a classic example of a big, blue lake, full of fish with multiple fishermen. If the lake is a public entity, it is in the interest of each fisherman to collect as many fish as possible, without regard for environmental impacts or long-term sustainability. If they do not, the other fishermen will fish the entire bounty from the lake, draining it of their resource, and they’ll be left with nothing. If each fisherman owns their own portion of the lake, however, they will want to sustain their portion of the lake for future years, to ensure long-term prosperity.
The banks have begun to look much like the overfished lakes of Wall Street. With huge annual bounties, and little incentive to maintain the overall value of the organization, traders, bankers and executives alike have taken large risky bets at expense of the the organization, in effect hollowing out their balance sheets. Large bonuses rewarded short-term harvests, and threatened long-term prosperity. Even with some compensation supplied in stock options, the incentive to make a few large cash bonuses and move on was so great, that it simply eclipsed the interest in ensuring a banks long-term viability.
Compensation structures should provide incentive for employees to ensure long run profitability over short-term profits. The most obvious way to accomplish this is to lower annual cash bonuses and increase the use of instruments that mature over time, such as stock options. Employees who receive bonus compensation based on the performance of the company in five years are more likely to ensure that the organization maintains long term profitability, and is less likely to take risky bets that undermine the banks sustainability and performance.
Obama is right to point out the broken compensation structures on Wall Street, but the amount of the compensation on Wall Street is not the cause of current economic woes. It is the form of compensation and how that from makes people act. Obama should not be looking at limiting overall executive pay, but rather rewarding success in solidifying a firm over the long run. If a salary cap is necessary to push a bill through congress in the short term, a bonus structure should be provided for any executive who has been instrumental in restoring confidence and a positive cash flow to a bank. Change in the way banks provide incentive to their employees is necessary, but populist politics should not get in the way of proper reform.
Forgive me for borrowing from an old Clintonian campaign slogan, but it seemed appropriate...
Over the past few weeks, more and more information has come out
about the direction of Windows 7. Desperate to be cool once again,
Windows is adding an OS X style dock, multitouch and a range of cool
new UI effects to boot.
With
all this attention to glamorous
interface features, one would be surprised to see the range of negative
articles published on Windows 7 thus far. Much of it, of course, does
have to do with the skepticism cast on Microsoft these days, however
some of it is also pointing out important flaws in what Microsoft is
doing. The features being added to Windows 7 are not the features
users need or are asking for most. They are flashy features Microsoft
thinks will sell Windows, but instead may erode its market share. What
Microsoft needs right now is to prove their critics wrong.
Performance, stability and productivity tools over that extra "wow"
factor.
I
have always been a proponent of Microsoft products. I defended Vista
through its hard times, and have always thought very highly of
Microsoft products. Times have changed. With the start of some Unix
development work, I converted to a more suitable Unix platform, OS X.
With this move, I began to find how far my general beliefs of
development and UI design had diverged from what I had praised as
quality software. There is no doubt: OS X is not only a surprisingly
stable platform, but it delivers the performance and features that
users want - and does so better than its competitor.
Windows
Vista was the perfect example of a monolithic, waterfall software
development project. Many of the features added were to push the
agendas of those running the show, not the customers of the Windows
product. Emphasis was placed on Tablet PC and Windows Media Center
over simplicity, cleanliness, speed and search. This is not to say
Tablet and WMC are bad products -- they both are powerful and
interesting, but they were not the core concerns of consumers. Because
resources were moved to less important aspects of Windows, the
development of core features that users really need suffered.
Vista
search is slow, resume from sleep is buggy at best and not instant, UAC
is annoying and shut off by most advanced users at the beginning, and
the OS overall feels sluggish. While Vista in many ways is a step
above XP, it is arguably also a step backwards. And although they may
have tried to patch up security and speed up search, they did not place
emphisis on it. There is no good reason that Google can search
billions of web pages faster than Windows can search a few million of
my documents. No arguments, no excuses, its pathetic.
Apple's
OS X has taken a different tack than Windows. Every year, a new
version of OS X is released. Every year, incremental new features
(along with a few "pieces of flare") come out of Apple, and every year
the user experience improves. Apple seems to build in the features
users want first, and then focus on the features that impress. Many
Apple advocates focus on the fancy effects, the security, the cool
factor, etc. to promote Apple products. But doing so misses the forest
for the trees. What Microsoft can learn from Apple is not the fancy
demos of Expose, cover-flow and other fun effects. Its real,
value-added features that just work. This is where Microsoft needs to focus its time.
Disclaimer: Don't
get me wrong here. I want Windows 7 to be good. I want it to
succeed. But on that note, I am afraid that it won't. Unless
Microsoft can show business customers real value, Windows 7 may be
destined for the same fate as Vista. And I am scared that a failure of
Windows 7 would hurt a product I do love, .NET.
Here are some of the things I've found since I first started using OS X:
- It turns on instantly from sleep. Always. Open the lid, move the mouse. Its seriously that consistent.
- Search is fast. Lightning fast.
- I
don't miss anything (Except Visual Studio). OS X has a lot of software
that is comparable or better than Windows equivalents. A lot of it is
free too.
- Less visual clutter. Finding features you want is easier.
- No
surprises. No chugging away at the hard drive unexpectedly, no "Please
wait. Configuring updates" junk starting and shutting down.
- Quick
view is priceless. When looking for a file, being able to instantly
see its contents greatly enhances productivity. Why can't Windows do
this?
None of these are flashy features... Just
simple things that work well consistently. Its important. When I open
my machine to show someone something, I expect it to turn on and not
waste my time. Countless times I've had Vista freeze on wake and then
on restart seen "Please wait, configuring updates." Maybe 20 minutes
later I get back to what I wanted to present.
Here are a few
demos of OS X you usually don't see in a typical sales pitch. Things
that just work on OS X. Things that save me time. Things that I want
in Windows.
When
writing software, you need to focus on what users need first. Let's
hope the Windows team pulls more than just the glamorous features from
OS X. Let's hope they find what makes OS X more productive and easier
to use. These are the features we need in Windows. Multitouch is just
a nice-to-have that few people will actually benefit from.
OS X is not going to take over all the Windows market share, but it could make a dent. If Windows 7 fails to appeal to CTO's and consumers alike, it will be 3 years before Microsoft gets yet another shot. In this time, it would not be surprising to see Apple's market share to grow from its current 9% to 20-30%. Even by the time Windows 7 comes out (where Apple will have 10-15% of the market), Microsoft will have to change its game plan. No longer can it ignore the consumers running on Apple platforms -- 15% of the market is just too much to ignore. Proprietary formats and tools will have to bow to interoperable platforms (Silverlight and the new Office XML format). This changes their business model quite a bit.
To those that know me well, this post is of no surprise, however given that I haven't been posting for a while, and probably won't for a while more, I thought it was appropriate to post a quick, personal update.
As of a week ago, I left my colleagues at Lab49 to pursue an exciting new venture which I hope will make the lives of many people easier. Because this venture is based on web technologies, I bought a mac and just to fit in a latte as well, and settled down in a world not as well known to me -- of Open Source and Unix/Linux.
Unfortunately due to the sensitivity of any new venture, my next several months of work will be as secretive as possible. I will not be sharing technical secrets or what the project is about until a few months from now. For that reason, this blog will remain sparse for the next few months.
Early in the web development world, scripting languages such as ASP or PHP were used to compose pages. Although this proved great for relatively static pages, the dynamic web, filled with rich applications called for a more powerful framework. Thus, frameworks like ASP.NET were born.
ASP.NET solved a good number of problem spaces, but has made creating simple pages (such as a resume or menu, or other primitive list of data) more cumbersome. With the world of COM development becoming less common and less preferable, the gap for a scripting language to replace VBScript/ASP is needed. PowerShell scripting has filled the gap left by the demise of VBScript, but nothing has come along to replace ASP.
PowerShell Pages is an ASP like language, based on the PowerShell runtime. Using a simple HTTP Handler, ASP.NET can render pages scripted using PowerShell script (including cmdlets, and CLR/.NET objects) to the web. Simple, fast and intuitive programming for simple pages that just need to display some content.
The PowerShell Pages project is an open source project that I am starting. Its implementation will be based on ASP.NET using a simple handler capable of consuming PowerShell HTML (PSH) scripts and writing HTML. Because the script is hosted in ASP.NET, the ASP.NET HttpContext and the other components of the object model are available. PSH scripts can work side-by-side with ASPX pages.
Ready to see what PowerShell Pages look like?
Sample Page: http://decav.com/psp/resume.psh
Sample Page Code: http://decav.com/psp/viewsource.psh?page=resume.psh
View Source Code: http://decav.com/psp/viewsource.psh?page=viewsource.psh
Join the Project - Visit the PowerShell Pages CodePlex project workspace
After hosting Gatsb.com for a year, I have decided to open source the project. I haven't had time to promote it the way I wanted to promote it, and it therefore hasn't caught on with too many users. I've been busy with other projects, other (grander) ideas, and think that this one should now belong to the community.
You can download the source code here: http://code.google.com/p/gatsb/
If you want to contribute to the project, please feel free to contact me at andreblog@decav.com
This project is a great case study of a bunch of new Microsoft technologies. It includes Workflow Foundation, .NET 3.0, Microsoft Virtual Earth and ASP.NET Ajax.
One of the coolest features is how it keeps open a "session" with an SMS client. Workflow Foundation is used to persist the state and when a new SMS comes in, it checks if theres an existing session for that SMS. If there is, then it will revive the workflow and continue the session. Cool stuff!
If you ever wondered how to create a database of geotagged entries, use ASP.NET ajax to make a snazzier site, or establish a user interface with any mobile device, I suggest checking out the code!
Heres a useful snippit of code I wrote as a utility and thought I'd post up for everyone. Very often when using a command line, you need to repeat a set of commands. For example, copy several files:
copy "c:\blah.txt" "c:\otherPlace\blah.txt"
copy "c:\dev\blah.txt" "c:\otherPlace\blah2.txt"
History is okay for this, but if you have a set of commands you need to run, you need to run them each individually. Also, if you go to do something else, they'll eventually lose their spot in the command history.
To solve this, I created a "scratch pad". Nothing special, but definately useful.
new-scratch "MyFirstScratch"
copy "c:\blah.txt" "c:\otherPlace\blah.txt"
scratch
copy "c:\dev\blah.txt" "c:\otherplace\blah2.txt"
scratch
## Do some other stuff here...
get-scratch # prints out the contents of the scratch.
invoke-scratch
This scratch pad is holds a set of commands that you want to rerun later. You can then call invoke-scratch to run the scratchpad. You can have multiple scratch pads, and use them by just adding the name to the end of the function (get-scratch "MyFirstScratch"). By default all scratch pad commands will default to the last used scratch pad.
The example below shows how to shorten the sample above:
new-scratch "MyFirstScratch"
copy "c:\blah.txt" "c:\otherPlace\blah.txt"
copy "c:\dev\blah.txt" "c:\otherplace\blah2.txt"
scratch 2 # Saves the last 2 items to the scratch pad
This can be very useful for writing scripts too. the Save-Scratch command lets you export your scratch to a file.
Heres the code. Add it to your profile and go. Enjoy!
function New-Scratch($name) {
if ($global:scratchPads -eq $null) {
$global:scratchPads = @{}
}
$pad = new-object System.Collections.ArrayList
$global:scratchPads.Add($name, $pad)
$global:currentScratchPad = $pad
$global:currentScratchPadName = $name
}
function Get-Scratch($name=$null) {
if ($name -ne $null) {
$global:currentScratchPad = $global:scratchPads[$name]
$global:currentScratchPadName = $name
}
return $global:currentScratchPad
}
function Scratch([int]$count=1,$name=$null) {
if ($name -ne $null) {
$global:currentScratchPad = $global:scratchPads[$name]
$global:currentScratchPadName = $name
}
$hist = get-history
if ($name -eq $null) {
$name = $global:currentScratchPadName
}
for ($i=$hist.Length-$count; $i -lt $hist.Length; $i++) {
$global:currentScratchPad.Add($hist[$i].CommandLine) | out-null
}
}
function Invoke-Scratch($name=$null) {
if ($name -ne $null) {
$global:currentScratchPad = $global:scratchPads[$name]
$global:currentScratchPadName = $name
}
foreach ($item in $global:currentScratchPad) {
write-host ">> $item"
invoke-expression $item
}
}
function Save-Scratch($path, $name=$null) {
if ($name -ne $null) {
$global:currentScratchPad = $global:scratchPads[$name]
$global:currentScratchPadName = $name
}
$global:currentScratchPad | out-file $path
}
If you want to run your PowerShell scripts from a piece of code you're writing, and would prefer not to use Process.Start, you can easily use the PowerShell runtime classes to run those scripts for you, completely in-process.
The code snippit below shows how you can create a Runspace and pass it some simple commands.
List<Command> commands = new List<Command>();
commands.Add(new Command("set-location c:\\"));
commands.Add(new Command("./MyScript.ps1 'myParameter'", true));
using (Runspace runspace = RunspaceFactory.CreateRunspace())
{
runspace.Open();
using (Pipeline pipeline = runspace.CreatePipeline())
{
foreach (Command cmd in commands)
{
pipeline.Commands.Add(cmd);
}
pipeline.Invoke();
}
runspace.Close();
}
One thing to note here is you do not have a PSHost, so if you have a write-host anywhere, it will fail with the following exception:
CmdletInvocationException: Cannot invoke this function becasue the current host does not implement it.
To get around this, you can remove write-host from your scripts and just have them write single lines, or you can implement a simple PSHost (although this is a bit harder than it sounds).
If you've been using System.DirectoryServices.DirectoryEntry class, or the newer System.DirectoryServices.AccountManagement namespace to access your LDAP or Active Directory server, you may have experienced the following error:
COMException: "Unknown error (0x80005000)"
This can happen for numerous reasons, but one of the most frustrating and overlooked reason's I've found for this problem is when your LDAP connection string is malformed. One of the most common malformations is in the case sensitivity of the LDAP:// component. For example, LDAP://myServer/cn=users,dc=myserver,dc=com is a valid connection string, however ldap://myServer/cn=users,dc=myserver,dc=com is not.
If you use the Uri or UriBuilder classes, the builder may lowecase your scheme. Always make sure to recapitalize the scheme when passing it into DirectoryEntry or any other API.
The Message Passing Interface (MPI) standard, and its .NET implementation, MPI.NET have been some of the cornerstones of development on compute clusters. The standard supplies a simple yet primitive way of both sending and receiving data between running compute processes.
The large advantage of MPI has been a mix of its simplicity and speed. A call to MPI Send on one node and MPI Receive on another block both callers until the operation is complete. Some more complex calls, such as MPI Scatter and MPI Gather allow a single node to distribute data to a set of nodes or retrieve it from a set of nodes. An MPI Barrier allows all nodes to stop until they have all reached the agreed upon place in code, then allowing them to continue. Such primitives allow a distributed set of processes to communicate, do some work, and then share values that each needs to continue with eachother. Because this is all done with some low level, bare metal socket tricks and/or shared memory, the result is blazingly fast communication.
With this simplicity however, comes a trade off. MPI has been a standard for nearly 20 years and has changed very little since its inception. The way we program today has changed drastically, especially with managed languages such as C#. No longer do we tend to worry about memory allocation, or dealing with raw memory. Today, most languages have a concept of automatic memory allocations, garbage collection and type safety. Although the primitives in MPI are unparalleled in simplicity for allowing multiple processors to communicate about a shared set of work, some striking limitations are found once we dig a bit deeper.
When I started with MPI.NET, I found the interface very simple. An Mpi.Send<T>(obj) would send an object to a waiting client. Mpi.Receive<T>() would give you back that object. Nothing could be simpler. In my example, however T happened to be a class that contained a byte[] of an undetermined size. Once run on the cluster, the size of the byte[] I was passing increased dramatically, and an unexpected exception occured:
AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory has been corrupted.
After lengthy investigation, I found that MPI.NET was attempting to pin some memory in the .NET GC heap and pass that memory location as a buffer to the underlying MSMPI stack. In doing so, it did not allocate enough memory for my large byte[], causing the write to try to write into the GC heap, thus throwing the exception. In my case, to solve this, I created a large enough buffer and passed it into an override of Mpi.Receive<byte>(byte[]). This overload pins the entire array that was passed in on the GC, and then passes that to the MSMPI stack. On the send side, I manually serialized my class, checked the length (to ensure I would not overflow the receive buffer) and sent the byte[] instead of the raw class. This solution does not take into accound messages larger than my expected buffer. For that, I would have needed to chunk down the data.
The moral of the story here is sending primitives, arrays of primitives or fixed-sized structs over MPI.NET (which is the most common scenario) is a great use of a very fast messaging interface. Once your demands get more complex, the MPI stack gets less favorable, not because of its inability to send more complex messages, but because of the manual labor involved in serializing and chunking down data.
It is no wonder that the HPC community is moving away from the traditional methods of MPI and communication across a set of processors to a Service Oriented (SOA) model. The benefits of using existing components, such as WCF and its NetTcpBinding, the threading models, serialization and transport models, and other features already provided by these frameworks outweights the possible performance penalty. Problems such as the one explained above simply do not happen with frameworks like WCF. Furthermore, although the underlying concepts of MPI and its simple messaging model are very simple and appealing, the overall development, maintainence and debugging of a SOA application is much simpler than that of a MPI application. The amount of code complexity and custom code drops when compared to an MPI implemenation.
The general industry trend seems to be towards SOA models. Microsoft Windows HPC Server 2008 is a great example of this. HPC Server uses WCF to distribute load across the cluster, and can even dynamically scale resources depending on demand of a particular service. Platform, another industry competior has been building with a SOA model for some time now.
I'm looking forward to playing with HPC Server 2008 and WCF more as time progresses. I think that the WCF model will solve a whole bunch of headaches that one incurs when trying to communicate over perhaps over simple primitives such as MPI. Many models and workloads simply do not require the type of communication MPI provides, and using MPI can be like fitting a square peg into a round hole. This is not to say MPI does not have its place, many complex processes do require constant communication between a set of workers, however I believe many of the problems we use HPC for today can distributed using SOA in a much simpler fashion.
I recently got to playing with Ruby, something that some colleagues in Lab49 have been big fans for some time. I've never been a big fan of scripting languages, but have grown more of an appreciation for functional programming over the past several months and thought I would give it a try.
Ruby is a very smart language, and I can certainly see why it has some appeal. The "don't repeat yourself" and "defaults over configuration" aspects of the language and its framework are really nice for cranking out simple applications.
ORM and Inference of Properties from database
I am a big fan of the objects that automatically get their properties from the database (a Customer object will automatically be linked with the Customers table when pulled through the ORM). Things like this make it dirt simple to crank out web projects of moderate size without writing tons of redundant code (as is the case with a classic OO approach, using adapters, abstractions, etc).
Even though this is a nice feature, and great for many projects, I am concerned that this is simply not sufficient for larger enterprise apps. Very often your app layer and database layer should be different, as one does not always properly map to the other. I suppose that you could achieve this nicely with views, however for large applications, the lack of abstraction seems a little brittle to me. At the same time, if you did change something in your database, you'd still have to change your abstraction logic, so maybe this isn't all that bad.
Syntax
The syntax of Ruby is very clean and straight forward. I like the bare bones structure, nothing more than needed is written. Reading into it a bit more, I found it interesting that various things can be written in several ways, for example:
while i < 3
print(i)
end
can also be written
print(i) while i < 3
While conceptually this is nice (the latter "reads" better), it is something I have a really hard time accepting. Yes it is nice not to have to write things like "end", however, syntactical differences like this (and others, such as the optional parenthesis around method parameters) concern me. While its great that you can develop your own style, and do what feels comfortable to you, it may be very confusing to other developers. An argument can be made either way, after call you can do this:
if (i < 3) Console.WriteLine(i);
in C#. Nothing stops you. At the end of the day, sloppy code is sloppy code, and its really a matter of having a well trained developer, not a strict language.
Naming Conventions
Here's something that threw me off. Ruby does a very good job of making names intuitive, however some of their names seem to break their own concepts. First of all, names like to_s are just sloppy. If you're making a lanugage thats supposed to be readable, why write cryptic names like this? Also, whats the deal with properties count and length being synonyms? Why have both? It seems relatively dangerous to me to have methods that mean the same thing on objects. A developer who sees count in one place and length in another may think they are different, something that certainly doesn't help code legibility.
Overall, Ruby has been fun to learn (although I'm still a novice), and I can certainly see its value. I'm not sure if I agree with some of its "friendly" tendencies -- being able to write whatever you feel and have it probably compile is not necessarily good. Just because code compiles doesn't mean its good code -- some bugs the compiler may catch now become runtime bugs. On the other hand, things like the ORM make it very easy to build rich apps with little code, something that .NET still comes up short on (just because designers generate LINQ classes doesn't mean that the code isn't there). Looking forward to playing more!
Here's a cool demo that Ronald Lintag and I threw together for Lab49. The application is supposed to demo a basic GUI that shows the status of nodes and jobs on a compute cluster. We're showing off the use of WPF 3D and styles to create a cool looking console for a sys-admin. You can imagine how something developed off this base could turn into a real product.
This demo is a bit rough around the edges, and definately needs polish, but its still worth a look.
-
Press J to bring up the job list.
-
Use the dropdown at the bottom of the job list to add some jobs.
-
Click on job to view details about their tasks.
-
The nodes in the background vary from gray to green to red depending on their utilization
Click here to launch the application (.NET 3.5 required)
Here are a few screenshots for those of you who are not adveturous enough for the ClickOnce...
More Posts
Next page »