I keep looking for the URL’s to get Microsoft Ajax, jQuery, the MVC Ajax libraries and more from our new shiny CDN (check here if you’ve not heard of it)... and I can never remember where they are.
So this post is for my benefit, not yours J
The links to all scripts are at http://www.asp.net/ajax/cdn/.
On the ASP.NET ScriptManager control there is a useful little property that very few people have heard of called “LoadScriptsBeforeUI”. What does it do?
Well, simply put...
· When “true” (the default) it will render <script> tags for each script you reference using the ScriptManager at the top of your HTML.
· When “false” it will render these <script> tags at the bottom of your HTML.
Oh right. Why would I bother?
Well to put it simply it is because all browsers stop parsing and rendering HTML while they are downloading and executing scripts... which means if you put your <script> tags at the top of a web page none of your page will start to render until all scripts are downloaded and dealt with. The bottom line is that while your page is downloading scripts, the page is usually blank, and your user is wondering why the page is taking so long to load.
In a development environment this may not seem particularly important – but deploy your application to somewhere bandwidth and network speed are unknown, and it can often become an issue. Users don’t like sitting waiting for your site to respond, so you could be losing users!
Let’s see it
Let’s see what this looks like in action – download the attached sample solution and view the two pages. One downloads my scripts first; and I’ve made sure one of the scripts takes 5 seconds to load, so it is obvious how slow the page is. The other page loads the script last... and you’ll notice the page renders immediately, without waiting for the script.
Have a look at the HTML source and see where the references to TestScript.js and GetScript.ashx are.
The Gotcha
I’ve just showed you how to improve the perceived performance of your web site by changing one setting from “true” to “false”! Right?
But I’m afraid there is a catch. You’ll also notice there is a button on my test web pages. This is to demonstrate something important; you need to be very aware of when the scripts are being downloaded. This leads to two opposing issues;
1. If you have scripts that execute as soon as they’re downloaded, they probably assume that the DOM has been built. In this case, they usually must be rendered last. I’d recommend using page load events instead, so that you don’t have this requirement.
My sample demonstrates this; only when TestScript.js is loaded after the UI has been rendered does the button work, because TestScript.js assumes the button exists and attaches an event handler. If I put this in pageLoad instead it wouldn’t be an issue.
2. If you have event handlers embedded in your HTML that depend upon some framework scripts you’re downloading later, the user could click the button before your scripts have downloaded and the handler would error! This approach to wiring up event handlers looks something like this;
<button onclick="callMethod()">Click Me</button>
The solution here is to attach event handlers programmatically, as I have done in TestScript, but in the pageLoad event instead. You could even consider defaulting your buttons/controls to a "disabled" state, enabling them once you've attached the event handlers. This can sit well with the principles of Unobtrusive JavaScript and progressive enhancement if your controls have default non-script behaviours, so is a good practice anyway.
The End...
I think this shows a useful technique that can really improve users’ perception of your web site... as long as you design your use of script to work correctly. Enjoy!
I had a fun day debugging some ASP.NET plus jQuery this week, and came across something I’ve known is possible for some time, but that I’ve never actually needed to do... and that was to ensure a bit of JavaScript ran once an UpdatePanel had refreshed as part of a partial render.
It turns out that this is easy; all you need to do is register your JavaScript code block with some methods on ScriptManager, rather than using the standard ClientScript methods.
For example, consider the HTML below (I’ve omitted the rest of the page including the script manager, but it is in the download);
<asp:Button
ID="ClickMeButton"
runat="server"
Text="Click Me to Refresh Below"
onclick="ClickMeButton_Click" />
<div style="background: black; color: White">
<asp:UpdatePanel
ID="MyUpdatePanel"
runat="server"
UpdateMode="Conditional">
<Triggers>
<asp:AsyncPostBackTrigger
ControlID="ClickMeButton"
EventName="Click" />
</Triggers>
<ContentTemplate>
Last Updated:
<asp:Label
ID="UpdateTimeLabel"
runat="server" />
</ContentTemplate>
</asp:UpdatePanel>
</div>
<div id="log">
</div>
Here we have a simple UpdatePanel, with a button that triggers the async postback. There is also an empty DIV with ID “log”.
In the server-side code of the button click event I have some code that looks a bit like this;
string time = DateTime.Now.ToLongTimeString();
ScriptManager.RegisterStartupScript(
UpdateTimeLabel,
typeof(Label),
"UpdateTimeLabelStartupScript",
CreateScript(time),
true);
The CreateScript function simply returns some JavaScript in a string that reads as follows (the {0} is replace with a time);
$('#log').append('Last Server Run: {0}<br />');
This uses jQuery to append a message to our “log” DIV.
There is also an alternative though; using the Sys.Application.add_load method to attach a page load script function, or creating a pageLoad JavaScript function that is a synonym for the same thing. I’ve done the latter;
function pageLoad() {
$('#log').append('pageLoad executed<br />');
}
This function is executed when the page initially renders and when every partial render completes. This means the content of my “log” DIV rapidly starts to look like this;
pageLoad executed
Last Run: 21:56:12
pageLoad executed
Last Run: 21:56:14
pageLoad executed
Last Run: 21:56:19
pageLoad executed
Easy huh?
I hope that adds some clarity to your options for executing script as the result of a partial postback. Check out the attached if you want to try it yourself.
Then tell us what you need! Blaine has blogged a link to a survey that patterns & practices will use to help them understand what challenges that are affecting you the most... so that they can choose the right areas to focus effort in providing guidance.
If you’ve got a couple of minutes spare head over to the survey...
What tasks or areas are difficult? What consumes the most time? What is it that causes you post-deployment headaches? What parts of the process are tricky? What don't you do because it is hard?
Recently I was working on diagnosing a performance issue with a customer’s web site with a colleague (this is one of our favourite engagement types so if you need some help let me know J), and we found that items were being trimmed very regularly from the ASP.NET Cache, causing excessive back-end work, and in turn reduced scalability.
* a “cache trim” is when ASP.NET looks for unused (based on a LRU algorithm) cache entries and deletes them.
So if you’re using the ASP.NET Cache API (or indeed any cache provider) to store custom application data, the moral of the story is to make sure you size your cache appropriately, and monitor it during test and live to see how it behaves.
It is worth knowing that an un-configured ASP.NET Cache takes a memory limit of the minimum of either 60% of physical RAM, or a fixed size RAM. This fixed size is different depending upon your processor architecture - see Thomas Marquardt's post here for some more detail. The idea is that when the limits imposed by configuration are approached, the cache will start trimming.
I’ve attached a very simple ASP.NET application that you can play with to study the cache behaviour while you’re reading the next section. Host it in IIS for the best results.
Setting Cache Limits
It is possible to set limits on how much memory an ASP.NET Application Pool can consume in IIS (see here), and you can also configure behaviour using settings on the cache web.config element. It is important to understand exactly what these mean;
· privateBytesLimit (in web.config, on the cache element): This is the maximum number of bytes that the host process is permitted to consume before the cache is trimmed. In other words, it is not the memory to dedicate to the cache, nor a limit to the memory that the worker process can use... but rather the total memory usage by the w3wp process (in IIS7) at which cache trimming will occur.
· percentagePhysicalMemoryUsedLimit (in web.config, on the cache element): This is the maximum percentage of RAM that the machine can use before the cache is trimmed. So on a 4gb machine with this set to 80%, the cache would be trimmed if Task Manager reported over 3.2gb (i.e. 80% of 4gb) was in use. Note that this is not the percentage memory used by ASP.NET or caching – it is the memory used machine-wide.
· Private Memory Limit (in IIS7 manager, as an advanced property of the App Pool): This is the maximum number of private bytes a worker process can consume before it is recycled (which will of course also empty the cache). If you set this limit lower than privateBytesLimit in web.config, you’ll always get a recycle before a cache trim... which sounds unlikely to be what you would want.
Things to Watch
The remaining content of this post itemises things you should consider monitoring to assess your cache’s performance. Some of it is obvious – but it is surprising how easy it is to overlook the obvious when your boss is breathing down your neck about your slow application J
Overall Application Performance
The best indicator that something might be going wrong is if your application is performing badly. Or maybe it was performing fine, but you’ve changed something or gotten more users, and suddenly it’s not so fast. Make sure you collect timing data in your IIS logs, and use LogParser to parse and analyse them. Also use standard performance counters from ASP.NET, WCF, and whatever other technologies you’re using. Get a feel for where and when things are slow or failing. This isn’t specific to caching, but it is an essential first step!
Hits on the resource you’re caching data from
Do some simple maths on the number of hits your back-end resource is getting. For example, if you have a “GetCities” web service, the results of which are cached for 2 hours, and you have 4 front-end load balanced web servers each with their own cache, you should expect a maximum of 4 hits to that web service every 2 hours. If you’re seeing more than that, alarm bells should be ringing.
Cache Entries
There are some great performance counters for the ASP.NET Cache API, so use those to monitor the state of the cache. Specifically the “ASP.NET Apps\Cache API Entries” counter is the number of items that are in the ASP.NET Cache right now. It should be broadly stable over longer periods with approximately the same load. If you cache an item per user, per region, or anything similar, be aware that this can dramatically affect your cache behaviour and memory consumption... in which case Cache Entries can be quite revealing.
Cache Trims
The “ASP.NET Apps\Cache API Trims” counter is the number of cache items that have been removed due to a memory limit being hit (i.e. they were trimmed). Obviously this should ideally be very low or zero under normal operation. Too many trims could indicate you need to revisit your caching strategy, or manually configure cache memory limits.
Cache Turnover Rate
The “ASP.NET Apps\Cache API Turnover Rate” counter shows the number of newly added and removed/expired/trimmed cache items per second. A target for this number depends on the behaviour of your application, but generally it should be fairly low – ideally items should be cached for quite some time before they expire or are removed. A high turnover rate could imply frequent trims, frequent explicit removals in application code, dependencies firing frequently (e.g. a SqlCacheDependency), or a sudden increase in user load (if, for example, you’re caching something per user).
Private Bytes
Monitoring the “Process\Private Bytes” counter for the w3wp process (assuming you’re using IIS7) tells you how much memory IIS is using for your ASP.NET application. A cache trim is likely to show up as a sudden drop in consumed bytes, and equally you should be able to see how close it is to memory limits for your configuration.
Worker Process Restarts
It is initially a little easy to confuse the cause of a drop in consumed Private Bytes between heavy cache trimming and an Application Pool recycle, so you should also watch the “ASP.NET\Worker Process Restarts” performance counter to ensure you know which happened.
Cache Removed Reasons
When you add items to the ASP.NET cache you can optionally specify a CacheItemRemovedCallback. This means that when the item is removed from the cache the call-back you’ve specified is called, passing in a CacheItemRemovedReason. This is great for adding some debugging instrumentation – if your item was trimmed due to memory pressure then the reason will be “underused”. Other reasons are Expired (you specified a time limit), Removed (you called Cache.Remove) and DependencyChanged (a file, SQL table, or other dependency setup with a CacheDependency was fired).
If you decide to add some logging using this approach I’d recommend adding a switch that enables or disables setting a call-back, as there is a small overhead in dealing with it.
Cache Memory Percentages
There are two slightly mysterious sounding cache counters too. I must admit, at first it took me a few minutes to get my head around exactly what these counters meant.
The first is “ASP.NET Apps\Cache % Machine Memory Limit Used”. This is literally the current physical memory usage by the whole machine as a percentage of the configured (or default) maximum physical memory usage. What?! Well, if you have edited the “percentagePhysicalMemoryUsedLimit” setting to 60%, this means your machine can use up to 60% of its physical memory before cache trimming occurs (not a very realistic example I know!). This counter reports the current usage as a percentage of the maximum... so if your machine is using 40% of available physical RAM, and the limit is 60%, this counter would report 66% (40 divided by 60, multiplied by 100 to get a percentage). It is important to note that this is memory consumed across the whole machine – not just by ASP.NET.
The second is “ASP.NET Apps\Cache % Process Memory Limit Used”. This is the total Private Bytes in use by the current ASP.NET process as a percentage of the limit specified in “privateBytesLimit”. So if you set the limit at 400mb, and your process is currently using 350mb, that would be reported as 87.5% (350 divided by 400, multiplied by 100 to get a percentage).
If either of these counters hits 100% ASP.NET will almost immediately trim 50% of your cache by picking the least recently used items... so obviously you don’t want to be hitting these limits often.
Conclusion
Phew. Well I hope that’s useful stuff. The best advice I can give is to do some real sums. Many customers cache data items per user, plus huge lists of reference data that can grow over time. The end result can be caches that are crippled over a fixed concurrent user level, or take a long time to reload large reference data sets when they have been trimmed. It is more than possible to do some rough maths to work out how much memory your application is using for the cache, and how this changes according to user numbers, regions, languages, locations, roles, user types, base offices, or other parameters – and the results can be very illuminating.
Finally, remember that you should always test using hardware, configuration, and simulated user loads that are as close to live as you can possibly afford, as this gives you the best possible chance of catching problems early.
How do you know when a page is being rendered as the result of a Server.Transfer, rather than a Response.Redirect or the user browsing directly to a page?
Actually it’s quite easy, assuming you’re using the default ASP.NET pipeline. In reality the “thing” that is responsible for handling an HTTP request is aptly called an HttpHandler – that is, they implement the IHttpHandler interface. Of course, you can create your own handlers if you just want to return a document, or your own manually rendered content, or similar.
But... the Page class also implements IHttpHandler, and ASP.NET therefore uses some cleverness to work out which page should be rendered, and then uses an instance of that Page-derived class as the HttpHandler.
So the bottom line is that if you do a Server.Transfer, the HttpHandler for the request will be the page that was originally being rendered... Therefore, the following code when executed from within a page will display “true” if a Server.Transfer has occurred;
TransferredLabel.Text = (Context.Handler != this).ToString();
Easy huh? Try the attached if you want to see this in action.
Unobtrusive JavaScript is a concept that’s been around for quite a while, but it’s now finally starting to make its way into the mainstream. The basic idea is that you should separate your JavaScript from your HTML – so instead of;
<a href="Default.aspx" onclick="alert('Sorry, that is unavailable'); return false;" >Click Here</a>
You should have something like this;
<a id="ClickHereLink" href="Default.aspx">Click Here</a>
... with some matching JavaScript in another file somewhere;
function pageLoad() {
var link = $get('ClickHereLink');
$addHandler(link, 'click', preventClick);
}
function preventClick() {
alert('Sorry, that is unavailable');
return false;
}
I’m not here to tell you why this is so great, but the best discussion and walkthrough of issues I’ve found is this article on onlinetools.org, so check it out if you’re interested.
How do I enforce it?
Recently I found myself cursing some code I’d written a long time ago, when I found it littered with onclick and onload handlers. It was frankly a pain to work out how it works. It was at this point that I thought “I should write something to shout at me when I write don’t write unobtrusive JavaScript”.
Coincidentally I have also been doing some work on HTML parsing, and I have to say I’ve found the XPath-like features in the HTML Agility Pack to be incredibly useful. If you’ve never used it I’d recommend taking a look.
It occurred to me that this was the perfect tool to do what I wanted – so I set about creating a simple MSBuild task that uses the HTML Agility Pack to scan ASPX and ASCX files for Obtrusive JavaScript – basically a form of static code analysis. If you’re interested in customising MSBuild, check this out.
The implementation is very simple;
· It is derived from Microsoft.Build.Utilities.Task.
· There is a property that is an array of ITaskItems, flagged with a [Required] attribute, as a way to pass in a list of files to check.
· There’s a hard-coded (for now) list of validation XPath expressions, with matching explanations.
· In the Execute method override, it loops through all the files.
· For each file, it opens the HTML and applies every XPath to it.
· If there are any matches for the XPath, it creates a build failure.
The key to success here is the simplicity with which the HTML Agility Pack lets me use XPath to find elements. The following is the basics of my code;
HtmlDocument doc = new HtmlDocument();
doc.Load(path);
HtmlNodeCollection results =
doc.DocumentNode.SelectNodes(
validator.SearchXPath);
Easy huh? The XPath expressions themselves mostly look like this;
//*[@onkeypress]
This will match any HTML element that has an onkeypress attribute... which of course means that the developer has specified a JavaScript event handler in the HTML, and broken the rules of unobtrusivitiy (hey, if that’s not a word I’m having first dibs on inventing it J).
Remembering that we’re actually scanning ASP.NET mark-up, not pure HTML, we can also check for ASP.NET attributes that actually create JavaScript event handlers in HTML too;
//*[@onclientclick]
Again, this is not difficult. The tricky one appears when you realise that it is valid to have OnClick in ASP.NET code! Of course, an <asp:button> can have an OnClick attribute referring to the server-side event handler. I wondered about saying “match any with an onclick that don’t have a runat attribute”, but this isn’t quite accurate (HtmlControls can have runat=”server” too remember).
In the end I went for explicitly excluding the ASP.NET controls that I knew were OK to use OnClick;
//*[@onclick and not(
name()='asp:button' or name()='asp:bulletedlist' or
name()='asp:imagebutton' or name()='asp:imagemap' or
name()='asp:linkbutton')]
Finally, I wanted to check that there was no JavaScript inside the page HTML itself... even between <script> tags. In other words, the only valid <script> tags should reference an external file, as follows;
<script language="javascript" type="text/javascript" src="Default.js"></script>
[Of course, you could also use the <asp:ScriptManager> control if you wanted to; in fact I’d prefer it]
To achieve this I check that there is nothing other than whitespace between the start and end script tags;
//script[string-length(normalize-space(text()))>1]
... and that’s it! As usual, the source code for this MSBuild Task is attached to this post, along with a demonstration Web App.
Wiring up the Task
Now we have a simple MSBuild task, all that is left to do is add it to a build. I created a new ASP.NET Web Application, right clicked the project file and chose “unload”, and then edited the XML manually.
The only bit I needed to add was as follows;
<UsingTask
AssemblyFile="..\JsBuildTasks.dll"
TaskName="ValidateUnobtrusive" />
<Target Name="BeforeBuild">
<ValidateUnobtrusive
SourceFiles="@(Content)" />
</Target>
First I import my task (I’ve copied a Release build of the assembly to the parent directory above my Web App project). Next, I override the BeforeBuild target, and add an instance of my ValidateUnobtrusive task. I’ve passed in an expansion of all the Content files in the project. The task filters for *.aspx and *.ascx at the moment – if you wanted to add *.html that would be trivial.
Save the file, and right click it to choose Reload Project. At this point Visual Studio 2008 warns you about a customised MSBuild project file – just choose “Load Project Normally” and you’re away.
From this point on, whenever you build a project containing Obtrusive JavaScript you’ll see errors like these;
Error Use of the 'onclick' JavaScript event on 'a' tag in markup causes Obtrusive Javascript. You should attach this handler in script instead.
Error Failed on 'script' tag . A JavaScript region containing something other than white space was found. You should link to related JavaScript resources files rather than embed script in a page.
Conclusion
I think this works really well so far (although I’m at the early stages of trialling it); if you try it do let me know how you get on. One of the particularly cool things (that I haven’t tried yet as I’m working isolated on my laptop most of the time right now, and I’m short of time) is that this could be wired up into a Continuous Integration build, under Team Build – I guess that would be a good plan if you didn’t like running the validation as part of your local build, or if it starts to perform badly due to the complexity or size of your pages.
I also think it raises other questions and possibilities. For example; how about some static analysis of “.js” files themselves? Could this help check for Directly Accessible Scripts, for example?
I don’t know if you’re the same, but when coding away I often find myself wishing for a String.Unformat function – call it the evil twin of String.Format. With String.Format I can build up strings like this;
var result = String.Format(
"http://{0}:{1}/{2}",
"localhost",
"12345",
"TestPage.aspx");
... which will return "http://localhost:12345/TestPage.aspx".
But what if I want to do the opposite; split out the contents of this URL, in one easy statement, much like String.Format? What if I could do this...?
string input = @"http://localhost:12345/TestPage.aspx";
object[] results = input.Unformat(@"http://{0}:{1}/{2}");
CollectionAssert.AreEquivalent(
new object[] { "localhost", "12345", "TestPage.aspx" },
results);
Of course something similar is already possible with Regular Expressions, but I find them much more unwieldy, they result in longer code, they’re overkill for what I’m trying to achieve, and they’re difficult to remember the syntax (oh, and even harder to get it right when coding late at night!). Take this equivalent example and judge for yourself;
string input = @"http://localhost:12345/TestPage.aspx";
string matchingformat = @"^http://(?<C1>.+):(?<C2>.+)/(?<C3>.+)$";
Regex expression = new Regex(matchingformat);
Match match = expression.Match(input);
Assert.AreEqual<object>("localhost", match.Groups["C1"].Value);
Assert.AreEqual<object>("12345", match.Groups["C2"].Value);
Assert.AreEqual<object>("TestPage.aspx", match.Groups["C3"].Value);
To get around this I have created my own String.Unformat function... all it does is take a simple format string and convert it into a regular expression by applying a number of transforms to it. And how do we do these transforms? With regular expressions, of course J
Now before we dive in here, I must make a point. This is not a replacement for regular expressions. It will be difficult to get the precision you need for anything complex – I’m only after a quick and simple solution to the basic case problem. Knowing this, read on...
How it does it
My sample code (and a bunch of demonstrations in the form of unit tests) is attached. Basically there are five phases to the code;
1. It escapes any special Regular Expression characters (e.g. ^, [, ], $ and so on) with a backslash so that they don’t accidentally affect the matching... they are assumed to be literal in the input string.
2. It replaces the {0} match syntax with regular expression (?<C1>.+) syntax.
3. It adds begin (^) and end ($) markers to the match string.
4. It performs the match.
5. It loops through the results, extracting matches and adding them to an object array to return.
Have a look in the attached and see what you think.
But...
The regular expressions I’m using are compiled where possible... but of course I’m applying multiple transforms (i.e. regular expressions) to a string just to do a simple match.
What does this mean? It means this will always be slower than if you just wrote a regular expression yourself (check out here and here if you want to)... so if performance really is that key to your code, or you’re doing this in a long loop, you might want to avoid it. If on the other hand you can see a great use for this when performance isn’t key – give it a go and let me know what you think!
I have also not had time to write a full suite of real unit tests, so if you find a bug shout up.
Conclusion
What do you think? Have I created a monster, or cooked up a treat?
After writing this I came across an interesting conversation here... it seems I have not been alone in my desires for String.Unformat.
Recently I’ve come across a requirement to easily switch between two Visual Studio 2008 configurations. I might need these two configurations open at the same time (so can’t just manually edit settings in Tools->Options), and don’t want any lengthy scripts, log on/log off activities, or virtualisation involved... mainly because each of those adds effort and is therefore a possible blocker to adoption.
One solution seemed to be using the “RunAs” command, and to configure each user’s settings differently. But I have seen issues with using RunAs – for example, a TFS installation with a customer failed a number of times and we couldn’t work out why (and never did), but logging on instead of using RunAs solved the issue. It can also take a while to do the logon before starting the executable. So I’m a little cautious of recommending it.
After a couple of internal conversations around solutions the Visual Studio SDK was discussed, which suddenly reminded me of the “Experimental Hive”. This is just another place in the Registry that Visual Studio can be instructed to load settings from instead of the usual place, and is generally used when testing customisations to Visual Studio (e.g. Domain Specific Languages) that might cause it to fail – so you don’t want to corrupt your main VS install.
It occurred to me that this is pretty much exactly what I’m after! Two sets of Visual Studio settings that can be used independently without machine virtualisation, user virtualisation, or anything else that might seem like overkill.
Solution
To get this working is quite simple. First, you need the VsRegEx.exe command from the Visual Studio 2008 SDK 1.1. Next, change into the directory it is in (I found it under “C:\Program Files (x86)\Microsoft Visual Studio 2008 SDK\VisualStudioIntegration\Tools\Bin”, but it may differ on your machine), and run the following command;
vsregex GetOrig 9.0 Test ranu
I’ve chosen to call my hive “Test”, but you could call it something else. This creates a new Registry hive under “HKCU\Software\Microsoft\VisualStudio\9.0Test” (alongside the usual “HKCU\Software\Microsoft\VisualStudio\9.0”) and file system storage under “%USERPROFILE%\Application Data\Microsoft\VisualStudio\9.0Test” (alongside the usual “%USERPROFILE%\Application Data\Microsoft\VisualStudio\9.0”).
Next, change to the directory that Visual Studio 2008’s “devenv.exe” command is stored. On my machine this is under “C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE” – again, yours may differ. You can then start Visual Studio with the following command line;
devenv /RootSuffix Test /ranu
This starts VS using your new “Test” hive. You could even setup a shortcut to start this version of Visual Studio running off your different configuration.
The first time it starts Visual Studio will take longer than usual as it is effectively a new install. But don’t worry; next time it will be just as quick as what you’re used to.
You are now free to go into Tools->Options and change your settings in this Visual Studio instance, and they won’t affect your standard Visual Studio install (the one started with “devenv.exe”, or from the standard shortcuts).
Caveats
As always there are some caveats. I have this working, and it seems fine, but I haven’t been using it long term for development. When developing for Visual Studio developers often use this approach for testing the software they have written, but generally not for doing their day-to-day development work in VS. Therefore I’d recommend giving it a try and seeing if you notice any odd behaviour specific to your configuration (Note: personally I think this is pretty unlikely as this approach was designed for testing Visual Studio! But I’m just being cautious J)
The final thing to be aware of is the “run as a normal user” switch (“/ranu”) that is used above. You may need to do some configuration as an elevated administrative user to get Visual Studio working if you’re planning on running VS as an admin (there are some notes on the ranu docs here). The most common reason for running Visual Studio as an admin is if you’re testing web apps in a locally hosted IIS rather than the built in Visual Studio Development Web Server. I haven’t tested this stuff, so try it first if this matters to you.
If you do use this and find anything interesting, feel free to report back here using a comment.
Anyway, I hope that helps – and I hope it saves you some time!
A customer asked me about doing this the other day, and I’d never come across it, although it seems obvious now! If I have a page that makes use of ASP.NET’s server-side ScriptManager AJAX history features but also want to link to an anchor on that page, I get problems.
For example, if I have a page like this;
... imagine (or try the download!) that I can click the “here” link to jump to an anchor defined further down the page as follows;
<h1><a id="bottom">Bottom</a></h1>
<p>
This is at the bottom.
Click <a href="#">here</a> to jump to the top.
</p>
Nothing complex... until I point out that the drop-down list and Person details are in an UpdatePanel, and that when the user selects a different user, server-side code such as the following is run (note that Person.All is a very basic repository of test data);
var selectedId =
Int32.Parse(PeopleList.SelectedItem.Value);
var selected =
Person.All.Where(p => p.Id == selectedId).Single();
if (PageScriptManager.IsInAsyncPostBack)
PageScriptManager.AddHistoryPoint(
PersonIdHistoryKey,
selectedId.ToString(),
String.Format("Viewing {0}", selected.Name));
This adds a “history point” to the URL that I can use to reload the correct Person should the user bookmark the page and revisit another time. The key here is how it encodes it – the fragment of the URL after the hash is used (note I’ve disabled the encryption for this post);
http://myserver/Default.aspx#&&PersonIdHistoryKey=3
Of course, if I now click on my “click here” link shown above, the target anchor (“#bottom”) will replace the AJAX history state data. If you want to see what I mean, download my demo solution and delete the component we’re about to write from the page.
A Solution
I thought about this for a while, and came up with an interesting and reusable solution. I created an ASP.NET Extender component that is used to “extend” the ScriptManager control. This extender emits an ASP.NET AJAX JavaScript behaviour that supplies some additional client-side functionality.
Specifically, when the behaviour is initialised, it performs the following actions;
// add a handler for the navigate event
var navigateDelegate =
Function.createDelegate(this, this.handleNavigate);
Sys.Application.add_navigate(navigateDelegate);
// when the page is loaded, scan for every link that
// targets a "#something" anchor and add a "click"
// javascript handler to it
var clickDelegate =
Function.createDelegate(this, this.handleClick);
Sys.Application.add_load(function() {
$('a[href^=#]').click(clickDelegate);
});
Basically we’re doing two things here – firstly, hooking up to the client-side “navigate” event that is fired when a page loads that has client-side History information embedded in the URL... this is done in the same way as for the server-side history portion we saw above, but after the # and before the && symbols. Have a read of this walkthrough for a bit more info on the client-side aspect to AJAX history.
Secondly we’re adding some code to run once the page has finished loading. This code uses a jQuery selector to find all “a” (anchor) tags for which the HREF begins with a # symbol. Once these have been identified, a click event handler is assigned to them all. This means we’re applying functionality to every link on the page that targets a local anchor mark with a single statement – nice!
There are then two more elements to the solution; handleClick (fired when the user clicks on one of these “a” tags), and handleNavigate (fired when a page loads that contains client-side history markers).
handleClick
When a user clicks on link to a local anchor (i.e. of the form “#something”) we run the following script;
var eventSource = args.target;
var parts = eventSource.href.split('#');
if (parts && parts.length > 1) {
var target = parts[1];
Sys.Application.addHistoryPoint({ jumpTo: target }, document.title);
if (target) {
var targetElement = $get(target)
if (targetElement)
this._scrollToElement(targetElement);
}
else
window.scrollTo(0, 0);
return false;
}
What this does is three things;
1. It finds the link that caused the event, and splits out the portion of the URL after the “#” – which of course is the ID of the anchor we want to navigate to.
2. It adds a client-side history point, including the name of the target anchor we want to display as an expando property called “jumpTo” on an anonymous object. This means we can recreate the behaviour if the user bookmarks the page.
3. It finds the target element and scrolls it into view. If the target is blank, it assumes we’re aiming for the top of the page.
Easy huh?!
handleNavigate
This function is fired when a page loads, if it contains client-side history information.
var target = args.get_state().jumpTo;
if (target) {
var targetElement = $get(target);
if (targetElement)
this._scrollToElement(targetElement);
}
All this does is to extract the data from the history state that we encoded in the handleClick function – we’re after the value of the “jumpTo” expando property. Once it has that, it tries to find the element on the page, and if it is found, scrolls it into view.
Summary
The end result is a URL that can look something like this;
http://myserver/Default.aspx#jumpTo=bottom&&PersonIdHistoryKey=1
... it should be obvious that we have both a client-side (in red) and server-side (in green) portion encoded here.
This component is so easy to use – we can just drop it onto a page, extending the ScriptManager control, and immediately enabling the handling of local anchor references without customising the HTML. Of course, if JavaScript is disabled, it will still work fine.
Let me know what you think – there are plenty of ways to enhance or alter this behaviour if it doesn’t work quite how you want it to (for example, you might not like it returning to the same place after every postback, or you might need to use client-side history for both this and something else at the same time). I’d also love to hear if you use it and have some feedback.
I’ve attached the source (usual disclaimers apply – this is not production strength code) so download it and have play.
This is late news as I’ve been busy recently, but if you’ve been following Blueprints make sure you read Michael’s post here. They’re evolving into something else, that I’m sure will be even more exciting...
I’ll be staying tuned so no doubt I’ll shout up when I hear more.
One of the tasks I used to hate when building a new system was adding admin pages to allow simple create/update functionality for lookup data. For example, maintaining a simple list of Products. It takes ages to write, and is very boring – even if you write templates to generate code it consumes a lot of time for such a commonly repeated and simple task.
ASP.NET Dynamic Data has now arrived, which gives a great way to “scaffold” CRUD pages over an Entity Framework Object Context (or other sources). But what if I have a web site that is mostly a line-of-business system, with complex business logic and custom UI, and I need to administer a small set of tables too?
Well, you can add Dynamic Data to a portion of your existing web site! This means you no longer need to write your own administration pages.
* Before you start off on this merry path, read this to make sure the relevant sections of your Entity Framework model are supported by Dynamic Data... for example, Complex Types are not.
How?
There’s a walkthrough that covers adding Dynamic Data to an existing web site here, so give that a read and you’ll be 80% of the way there. [Note: I also had to add a reference to System.Web.Entity at the end of the walkthrough to get it all working]
Next, make sure you have set ScaffoldAllTables to false – the last thing you want to do is expose your non-lookup data for direct editing by admin staff. All that should be exposed is a few key tables of static data. So, in Global.asax, make sure your data model registration looks like this;
model.RegisterContext(
typeof(OrderDbEntities),
new ContextConfiguration()
{ ScaffoldAllTables = false });
But how does the user get to this administration functionality? I’d recommend putting it under a virtual subfolder in your web site – perhaps named “Admin”. This is really just a route (read up on the new ASP.NET Routing features if this is unfamiliar) and needn’t really exist as a physical folder. So in Global.asax, we set it up as follows;
routes.Add(new DynamicDataRoute("Admin/{table}/{action}.aspx")
{
Constraints = new RouteValueDictionary(
new { action = "List|Details|Edit|Insert" }),
Model = model
});
Note that my route begins with “Admin/...”, which means the user will browse to URLs as follows to get to the admin pages;
http://{add address here}/myfakewebsite/Admin/Product/List.aspx
You can of course link to these pages from elsewhere within your site to save the user typing them in.
At the moment, we’re not scaffolding any of our tables, so none of these URLs will work yet. The next step, then, is to pick a table to expose for administration. I’m going to use a Product table that has ID, Description, Price and CreatedBy/CreatedDate fields. In my EF model the entity also has an OrderItems navigation property which allows me to get all Order Item records that are an instance of an order for the current product...
Due to the way EF supports partial classes, I can tell the Dynamic Data runtime to scaffold the Product table using an attribute (from the System.ComponentModel.DataAnnotations namespace & assembly) on a partial class. I added a C# class to the Web Site with the same namespace as the Entity Framework model, and pasted in the following code;
[ScaffoldTable(true)]
public partial class Product
{
}
Immediately this attribute is added the Product table is available for editing. There are a few problems when I browse to the new page though;
1. It is displaying the link to Order Items even though it isn’t editable (because Order Item isn’t scaffolded).
2. I can edit the columns I’m using for Audit – CreatedBy and CreatedDate. Not good.
It turns out changing this is quite easy. Using some additional attributes and a metadata class as follows customises the model sufficiently (see here for more info on this metadata class approach);
[MetadataType(typeof(ProductMetadata))]
[DisplayColumn("Description")]
[ScaffoldTable(true)]
public partial class Product
{
}
public class ProductMetadata
{
[UIHint("TextReadOnly")]
public string CreatedBy;
[UIHint("TextReadOnly")]
public string CreatedDate;
[ScaffoldColumn(false)]
public EntityCollection<OrderItem> OrderItem;
}
I’ve now instructed the runtime not to display the Order Item navigation property, and to use the Description field should I display Products in a list somewhere (as part of a foreign key drop-down, for example).
I’ve also asked it to use the “TextReadOnly” user control to render my CreatedDate and CreatedBy fields instead of the standard Text control, by placing a [UIHint] attribute on two metadata properties. To create the TextReadOnly control I copied-and-pasted the “Text.ascx” control twice, and renamed the copies to “TextReadOnly.ascx” and “TextReadOnly_Edit.ascx”. This control will now be used whenever the CreatedDate and CreatedBy fields are displayed (note I’ve cheated here – CreatedDate is really a date but as I’m happy with the default ToString display mechanism for it I didn’t bother creating a DateTime ReadOnly control).
As you can see, these user controls are available for you to edit under the DynamicData/FieldTemplates folder that was copied in as part of the original walkthrough. If you want to change this folder to be somewhere else, check out this article.
Finally, we probably want to limit access to the admin pages to a limited set of privileged users. The easiest way to do this is using standard ASP.NET authorisation, by adding a location and authorisation element to web.config;
<location path="Admin">
<system.web>
<authorization>
<allow roles="domain\role"/>
<deny users="*"/>
</authorization>
</system.web>
</location>
Next steps...
There’s plenty more you might like to do, so I would recommend looking further into the features that Dynamic Data provides.
One that is particularly likely to be of interest is customising validation of data that is maintained... a good starting point is the ValidationAttribute.
You probably also want to change Site.master to better reflect your site (or move it, rename it, and update the templates under DynamicData/PageTemplates to use the new one). As long as you have a ScriptManager and the required Content Placeholder on there, it should work great. You can obviously make use of your own CSS too, or even go as far as changing all the field templates.
If you’d like a general overview of how Dynamic Data fits together, have a read of the Infrastructure Overview.
Finally, remember that I’m basing mine on an Entity Framework model, so I could customise the model too – for example, hooking up to the SavingChanges event to perform processing and/or validation on the Product table before it is saved to the database.
I hope all that is useful, and saves you time and boredom for your next system... enjoy!
Is this a question of taste, or something more serious? [Warning: this post won’t change your life J]
I found myself writing the following extension method yet again the other day;
public static void ForEach<T>(
this IEnumerable<T> source,
Action<T> action)
{
foreach (T item in source)
{
action(item);
}
}
This means I can write simple (contrived) code like the following;
entities.ForEach<ObjectDefinition>(o => o.DeleteDbRecord());
This will enumerate all my ObjectDefinition objects and call DeleteDbRecord on each. But why didn’t I just write it like this?
foreach (ObjectDefinition o in entities)
{
o.DeleteDbRecord();
}
I’ve raised this because some code I’ve seen (and written!) has reached a saturation point, where lambdas, extension methods, and other concepts have taken over - meaning the code looks more like it was written in LISP than C#.
But does that matter?
A quick search shows there are all sorts of discussions about this online, which is interesting. It is actually quite an old discussion now!
If you’ve got some XML data in a SQL Server column, how can you flatten it out and query it or present it as though it was relational data? It turns out this is quite easy...
Setup
Let’s create a simple table to hold our data;
CREATE TABLE XmlSourceTable
(
RecordId INT IDENTITY(1,1) NOT NULL PRIMARY KEY,
XmlData XML NOT NULL
)
GO
And we’ll define some straightforward XML to import;
<?xml version="1.0" ?>
<Root>
<Person>
<Name>Simon</Name>
<Age>20</Age>
<Skills>
<Skill>Cooking</Skill>
<Skill>Cleaning</Skill>
</Skills>
</Person>
<Person>
<Name>Peter</Name>
<Age>21</Age>
<Skills>
<Skill>Ironing</Skill>
</Skills>
</Person>
</Root>
Ages may have been changed to protect the innocent J
Next, we’ll import it into my table using one of the mechanisms SQL Server provides – each XML file will be imported into a single row in the target table.
INSERT INTO XmlSourceTable(XmlData)
SELECT *
FROM OPENROWSET(
BULK 'C:\XmlSource.xml', SINGLE_BLOB)
AS ImportSource
GO
After this, if we do a quick query...
SELECT * FROM XmlSourceTable
... we can see that we get a single row back containing an ID and some XML;
RecordId XmlData
----------- -------
1 <Root><Person><Name>Simon</Name><Age>20</Age... (snip)
(1 row(s) affected)
Queries
The simplest way to extract this data is to use the CROSS APPLY keyword, as this executes a function against each row and then adds the returned data to the result set. Combining this with a method that can be called on the XML data type called nodes, we get some great results. A quick query like this;
SELECT
pref.value('(Name/text())[1]', 'varchar(50)') as PersonName,
pref.value('(Age/text())[1]', 'int') as PersonAge,
pref.query('Skills') as PersonSkills
FROM
XmlSourceTable CROSS APPLY
XmlData.nodes('/Root/Person') AS People(pref)
GO
... yields a completely different result set to our last query;
PersonName PersonAge PersonSkills
---------- --------- ----------------------
Simon 20 <Skills><Skill>Cooking</Ski... (snip)
Peter 21 <Skills><Skill>Ironing</Ski... (snip)
(2 row(s) affected)
We can see this query has flattened my single row of relational data with embedded hierarchical XML into two rows and columns of relational data. I’ve also included a subset of the XML as a column, just to show I can! Of course, if I wanted to I could modify this to get a list of people and their skills;
SELECT
pref.value('(Name/text())[1]', 'varchar(50)') as PersonName,
sref.value('(text())[1]', 'varchar(50)') as PersonSkill
FROM
XmlSourceTable CROSS APPLY
XmlData.nodes('//Person') AS People(pref) CROSS APPLY
pref.nodes('Skills/Skill') AS Skill(sref)
GO
What is it Doing?
This SQL can be difficult to understand when you first look at it, but it isn’t really that complex. Breaking it down there are three key concepts that we’ll cover below – using my first query above that fetches PersonName, PersonAge, and PersonSkills as an example.
CROSS APPLY
The first concept is the use of CROSS APPLY, which many people haven’t used before. What this is doing is roughly equivalent to the following steps (note: this is my idiots guide to how I think about it, not a description of how the query optimiser does it);
1. Fetch the rows from the XmlSourceTable table.
2. For each row, call the “nodes” function on the XmlData column. This could be some other function too – it needn’t be on XML data. See the docs on APPLY for more info.
3. Duplicate the XmlSourceTable row once for every row returned by the table valued function “nodes”.
4. Add the columns returned by the “nodes” function to the columns in the result set.
5. Continue doing filtering, joining, and column selection as for any other SQL query.
I hope that makes it a little clearer.
nodes() function
The XML data type in SQL Server defines a number of methods that can be called on it. One of these is “nodes” – and what this does is basically select a list of XML nodes that match an XQuery expression. Knowing this, look at the statement in my SQL;
XmlData.nodes('/Root/Person') AS People(pref)
This is using the path “/Root/Person” to ensure that all Person nodes that exist under the Root are selected. The result is aliased as a table named People, and each resulting XML node will be output as a separate row, in the “pref” column.
Plenty of alternative syntaxes are available for selecting this list of nodes, and this really is the core of how to flatten out the XML. I’ve also used “//Skill” syntax in my second query, for example, which selects every Skill node found in the XML it is used on.
Other XML functions
The last concept is the use of the selector XML functions – in my example I’ve used “value()” and “query()”. Both of these use XQuery expressions to select XML data.
Firstly, I’ve used the value() function to extract a specific value from the XML, and convert it to a SQL Server data type;
pref.value('(Name/text())[1]', 'varchar(50)') as PersonName
The “text()” function here retrieves the inner text from within the XML “Name” node. The “[1]” suffix acts as an indexer, and fetches the first result matched. I know there’s only one name node per person, but I need to make sure SQL Server knows this too. Finally, the second argument is the data type it should be converted to. For an Age, I’ve used ‘int’, but here we’re converting to a varchar type.
The “query()” function allows me to return an XML fragment;
pref.query('Skills') as PersonSkills
This returns the XML that matches the “Skills” node, and is found underneath the current Person element held in “pref” (i.e. the search is relative to the contents of pref). This means it returns fragments such as;
<Skills><Skill>Ironing</Skill></Skills>
Conclusion
None of this is all that difficult once you know how – so I hope this has given you a quick start to using XML data in SQL! Don’t forget to read up more generally on the XML data type and XML indexes.
As the beta 1 drop of Visual Studio Team System 2010 approaches, Brian Harry has a great little summary of the high level new feature areas we’re delivering. Check it out here.
There are some great nuggets in there, so I’m really looking forward to RTM – lab management, gated check-ins, and the architecture tooling are some of my favourites.
