- Test applications on virtual environments and file rich bugs
-
You are testing a complex multi-tiered application. You create different configurations, multiple data sets, and test the different combinations in a user-flow. You come across a hard-to-find bug. You file it and are pleased that your effort was well worth it. But the next morning you realize that your developer has resolved your bug as 'No repro' with the comment 'It works on my box'. You are really frustrated and determined to reproduce the bug again and file it, this time with a screenshot and logs. And the bug ping-pong continues. Does this ring a bell??
Testing and development can be so much more efficient if the bug you filed was rich and actionable in the first place. With Team System 2010, you can automatically create highly actionable bugs with video, screenshots, system info, event logs, and many other types of logs.
Well, there are these nasty configuration issues that will still be hard-to-reproduce for the developer on his environment. What if you could attach the environment snapshot with the bug itself, so the developer can connect to the machines in the same state as they were when you found the bug, and reproduce the problem?? We have good news - with lab management, you can do exactly that. Let's find out how.
After reading this article, you'll be able to:
- Configure your virtual environment to run tests.
- Run manual tests on the application deployed in the environment.
- File a rich bug with your virtual environment snapshot attached to it.
This article is divided into two parts. The first part talks about how to create a virtual environment for testing. And the second part talks about how to run manual tests on the virtual environment and file actionable bugs. Before we end, we'll also show you what the rich bug looks like to the developer and how he can use it.
I. Setup a virtual environment for testing
Here is a block diagram of Lab Management components. To be able to run tests and collect logs from environments, you need to first install and configure a test agent controller with your TFS Team Project Collection. Next, you need to install test agent and lab agent on the virtual machines that'll be a part of your environment. You can import the virtual machine and create environment that is ready for testing. Finally, create a test settings that specifies the different logs to be collected from various roles of the environment while you test your application.
1. In order to run tests and collect logs from environments, you need to first setup your test agent controller. The steps to install a test agent controller and configure it with your TFS Team Project Collection are described in steps 15-17 of the Lab Management Beta1 setup guide.
2. The next step is to install agents on the virtual machines that'll be part of your environment. This can be done when you create virtual machines from SCVMM. You need to install test agent and lab agent on the virtual machines to be able to collect logs and run tests on these machines. Steps 20 and 21 of the setup guide describe how to install lab agent and test agent on the virtual machine.
3. Import virtual machines and create environment: This post describes how to import a virtual machine in lab as a 'virtual machine template' and create an environment with it. These steps can also be founding in the 'Create a virtual environment' section of setup guide.
While you are creating the virtual environment, ensure that on the 'Advanced' page, the testing capability is turned on, and the test agent controller you setup has been selected.
Start the environment and ensure that there are no errors. Once the machines in the environment have started and the testing capability status is 'Ready', you are all set to run tests on the app deployed in the environment.
4. Create test settings: Finally, you need to create a test settings that define what logs to collect (and also where to execute tests in case of automated tests) while you run your tests. You now create a new test settings for the roles in your environment.
- Open Microsoft Test and Lab Manager (henceforth called MTLM) and go to 'Lab Center' -> Test settings. Click on 'New' to create new test settings.
- Enter name and description. Choose testing type as manual (to be used to run manual tests) and select the option to 'Additionally collect data or perform system actions on remote machines using an environment'.
- Go to 'Environments' page. Select the environment you created.
- Skip the 'Test machines' page and go to 'Data and diagnostics'. Here you can define what logs you want to collect while the tests are running. 'Local' refers to the machine from which you'll run the tests using test runner. Click on the various roles and define the logs you need to be collected.
- Verify the settings in the 'Summary' page and click Finish to create your test settings.
So far, we have seen how to configure the virtual environment to run tests and created a test settings. The next section talks about actually running the tests on the virtual environment and filing rich bugs.
II. Run tests and file actionable bugs
You have created your virtual environment and deployed your application on the environment using the workflow. In the meanwhile, the test team has created a test plan for this iteration of the product cycle and added test cases to it. You are all set to test the application. Now, we'll find out how to run manual tests on an application deployed in the virtual environment and create highly actionable bugs, which reproduce for the developer.
Run tests and file rich bugs
1. Go to the Testing Center in MTLM and select the Test tab. Select your test plan and the set of test cases you want to run. Click on "Run with options " and choose your test settings and virtual environment.
2. The Test Runner window is launched (by default it will be docked to the left side of the screen). Click on "Start test". Launch your client window (IE/rich client) and start running the steps in your test case.
3. While running your test cases, when you find a 'hard-to-repro' bug, you can take a snapshot of the environment by clicking on "Take Snapshot" action under "Lab Actions". The snapshot operation will take around 15-30 seconds. Once the snapshot is completed, you can observe the .lvr file (link to the snapshot) in the bottom pane.
4. Now, create a new bug for the problem you just observed by clicking on 'New bug' action in the toolbar. In the new bug dialog, you can see that the repro steps are automatically filled in from your test case. The 'System Info' tab has information about your local machine as well as the test machines in the environment. Also note that the 'Other links' tab has the different logs from your environment as configured in your test settings and the link to the environment snapshot (.lvr file). Enter a title and save the bug.
5. As you are running your test cases, note that you can connect to the virtual machines in the environment using the "Connect to Environment" option under "Lab Actions" menu. Also note that the .lvr file just has a link to the environment snapshot. The link will be resolved dynamically while the developer clicks on it. The size of the link is only around 1 KB. The actual snapshots are stored on the Hyper-V hosts itself, and use the differencing disk technology to store the changes from last snapshot only. Hence you can attach multiple snapshots to a bug, if needed.
6. Once you are done with testing, click "Save and close" to save the results and restore back to MTLM window.
Debugging using environment snapshot
1. Launch Visual Studio 2010. Connect to your Team Project using Team Explorer. Open the query 'Active bugs' under 'Work Items -> Team Queries' and open the bug you just filed.
2. If you go to the "Other Links" section of the bug, you will notice that it contains the link to the snapshot of the environment. Double click on this file to connect to the environment snapshot.
3. You will see a dialog with multiple options to connect to the environment. Let's spend a moment to understand what they mean.
a. "Connect to the saved snapshot in this environment" - Use this option if you want to restore the exact state of the virtual environment at which the snapshot was taken. If you choose this option, you might end up disconnect any user currently using this environment and this might lead to loss of their work.
b. "Connect to the Environment in its current state" - Use this option if you just want to connect to the virtual environment in its current state. There is a possibility that you might still disconnect someone connected to this environment, but you will not restore the environment's state.
c. "Connect to a new instance of this environment" - You can use this option to create your own copy of the environment. This way you will not disturb anyone using the environment currently. However, it will take few minutes for your new environment to be created and it is recommended to use 'Network isolation' capability on environments if you intended to run multiple copies of it.
Note that this option is only available to you if you have stored your virtual environment in the library as a template. You can do so from Lab Center -> Environments and choosing 'Save as Template' action on the environment. If the environment hasn’t been stored in the library, this option is not available.
4. Choose the most appropriate option and click on 'Connect'. This brings up the lab environment viewer and connects you to the machines in the environment. You can now login to the machines, reproduce the error, and debug its root cause.
To sum up, you have seen how you can configure your virtual environment for testing, how to run tests and file rich bugs, with environment snapshots so that the developer can always reproduce the problem and debug it.
We hope you liked this and look forward to hear your comments.
Darshan Desai, Program Manager and Bhuvaneshwari K, SDET
Visual Studio Team Lab Management
- Networking basics for lab management - Part I
-
How are the virtual machines in a Lab Environment connected to each other and to other machines? Can I have physical hosts that are distributed across networks? What is the significance of network location in Team Foundation Administration Console? What if I want more control on how virtual machines are networked? If I turn on network isolation capability, what happens to the network of an environment? We will try to answer questions such as these in this series on 'networking basics for lab management'. Some of this information is useful for lab administrators in planning the physical network topology of the lab. Knowledge of how environments are networked allows development teams understand how their applications might behave when deployed into a virtual Lab Environment.
Physical networking
Before getting into networking of environments, let us spend some time on the networking of lab infrastructure machines - physical hosts, library servers, SCVMM server, TFS server, and Test and build controller machines. Needless to say, there has to be IP connectivity between all these machines. SCVMM communicates with its agents residing on physical hosts and library servers. TFS server communicates with SCVMM and with test/build controller machines.
The clients from which you run "Microsoft Test and Lab Manager" or VSTS need IP connectivity with TFS and with test/build controller machines.
While having IP connectivity between the machines is the minimum requirement, there is more you need to do to get good performance. It is highly recommended that all the physical hosts and library servers have Gigabit Ethernet connectivity to each other. This means that all of them should have Gigabit Ethernet cards, and should be directly connected to a shared Gigabit networking switch. This ensures that the data transfers (think of the large VMs that get copied from one machine to another) between them happen with acceptable performance. As an example, copying an Environment of size 30 GB from a library server to physical host over BITS (Background Intelligent Transfer Service - this is the protocol used by SCVMM to copy VMs) may take about an hour in a 100 Mbps network, and around 10 mins in a Gigabit network. To accommodate larger labs, you can use multiple switches. The idea is to keep the end-to-end bandwidth between hosts and library servers close to a Gigabit.
This guidance implies that all the hosts and library servers are on the same network segment. If you have a larger lab or if you need to distribute physical machines across network segments, Lab Management allows you to do that. You can still get the desired performance by having a Gigabit network within each segment and carefully grouping the hosts that are co-located into a SCVMM host group. By allocating such host groups and co-located library servers to Team Projects, you can ensure that data transfers only happen between machines that are connected on a Gigabit network.
Networking on a host
Each physical host in the lab may be connected to multiple networks. Multiple networks are typical for separating data traffic and management traffic. For each network that a host is connected to, there is a network location that identifies that network. Using SCVMM, you can see the number of networks each host is connected to and their network locations. Select the host in VMM admin console and view its properties. The following figure shows a host that has four network network adapters, but is connected to only one network.
The 'location' of that network in this example is corp.microsoft.com. If the network location is empty, you can click on the 'Override discovered network location' checkbox, and type in the name of a location. Before running lab, you need to ensure that all hosts are connected to a common network location.
Use the TFS admin tool to configure this common location as the 'preferred network location' for lab.
The preferred network location identifies the network to which all virtual machines created by lab should be connected to. And, the network adapter on a host that is connected to the preferred network location is called preferred network adapter for that host.
PS1: In VSTS Lab Management 2010 Beta1, if you change the network location after setting it once, you have to reset IIS and restart the TFS job agent service on TFS machine. This is fixed post Beta1.
PS2: If you change the network location after setting it once, virtual machines that are already deployed are not affected. In other words, they remain connected to the old network location. To make them connect to the new location, you have to store the environments in library, and redeploy them. Or, more simply, you (admin) can use SCVMM and manually change the connectivity of each VM.
One more thing that you need to ensure for every physical host is that it has an external virtual network that is connected to the preferred network adapter. You can check this by using the Hyper-V manager or SCVMM admin console. The figure below shows the Hyper-V manager on a host. By clicking on the Virtual network manager, you will be able to see all the virtual networks that are configured on that host. Hyper-V supports three forms of virtual networks - external, internal, and private. Ensure that there is one virtual network that is marked as external and is connected to the preferred network adapter.
You can easily verify that each host in a host group is properly networked by using the Team Foundation Administration Console. When you add a host group to a Team Project Collection or when you open an existing Team Project Collection that is configured for Lab Management, you can verify that all hosts in the host group satisfy the above networking requirements. Open the Project Collection level Lab Management Settings by clicking on 'Configure Host Groups' as shown in the figure below, and press the 'Verify' button.
Networking for an environment
Now that we have the hosts physically networked and lab configured in TFS, we are all set to understand how environments are networked in VSTS Lab Management. In this post, let us focus on environments that are not network isolated.
Let us say we created an environment with two virtual machines. VSTS Lab Management ensures that each of the virtual machines has one emulated network adapter that is connected to the preferred network location as shown in the figure below. If this means that a new network adapter has to be created and attached to the virtual machine during the creation process, VSTS Lab Management does so.
To walk you through the above figure, there are two physical hosts A and B. The preferred network location configured in Team Foundation Administration Console is 'corp.microsoft.com'. One Lab Environment with two virtual machines has been created. Lab Management placed the first virtual machine (VM1) on Host A, and the second virtual machine (VM2) on Host B. N2 is the preferred network adapter on Host A, since it has a network location that matches the lab's preferred network location. N1 is the preferred network adapter on Host B. Lab Management connects VM1 to an external virtual network of N2 on Host A. VM2 is connected to an external virtual network of N1 on Host B.
More flexible networking for an environment
What if you wanted your environment to be connected to a second network location (say test.microsoft.com in the above example) in addition to the preferred network location? Lab Management does not currently expose this flexibility through its APIs or client. However, you can get around this by using SCVMM. This is what you need to do. When creating the VM in SCVMM, insert a network adapter into the VM and set its network location to 'test.microsoft.com'. The following figure shows how you can do that in SCVMM. While creating a new virtual machine, configure the network location of an adapter to be 'test.microsoft.com' under the 'Configrue Hardware' step.
Now import this VM into lab. When an environment is created from that VM, VSTS Lab Management inserts a new network adapter into the cloned VM with network location set to 'corp.microsoft.com'. As a result, the newly created VM will end up with two network adapters - one connected to 'corp.microsoft.com', and one connected to 'test.microsoft.com.
Questions, Comments, Feedback?
Our goal was to provide a simple experience for the testers and developers using Microsoft Test and Lab Manager, and not to burden them with the intricacies of networking their environments. Hence, we made it easy for the most common case, where all virtual machines in an environment are connected to one network. For certain other complex situations, SCVMM can be used to configure the networking of a VM before it is imported into lab.
We would like to hear from you. Does this networking scheme satisfy your application needs? What other features would you like to see? If you have more questions or comments, please post them to this blog.
In Part 2, I will describe how the networking of a 'network-isolated' Environment looks like.
- Creating and Working with Virtual Environments
-
Today as we build more and more complex applications the complexity of development and testing environments have increased significantly. Development and testing of complex N tier application on single machine results in late discovery of bugs in pre-production or worse in production environments. Ideally we would like to develop and test applications in an environment which is as close to production as possible (due to cost and other infrastructure limitations we may never get identical). Unfortunately today setting up these N tier environments on physical machines is a time consuming and error prone process and keeps dev/tests focus away from their core activities. This results in not only loss of productivity but also impact product quality.
VSTS Lab Management supports creation of virtual environment in minutes as oppose to days. This allows development and test teams to develop and test code against production like environments without spending days in setup which results in improved productivity and quality. As setting up these environments is so easy, it encourages sharing of hardware resources across the team and self service as oppose to dependence on a IT Administrators. With sharing and self service your total cost of ownership of running labs reduces significantly. In this post we’ll cover how to create and work with virtual environments.
A virtual environment is defined as a collection of virtual machines (VM). e.g. a typical 3 tier application environment will have 3 virtual machines namely client , web server and database server. In order to create this environment you first need the individual virtual machines which can play different roles as mentioned above namely client, web server etc.
Since majority of applications will have similar needs as above, Lab Management enables you to create these basic virtual machines (also known as Golden VMs) once and store in the library as virtual machine templates so that any time you want to create an environment you can reuse these virtual machine templates.
Creating Virtual Machine Template
1. First step is to have the base virtual machine which you want to use as virtual machine template (Golden VM) for different environments. If you already have a virtual machine just copy the virtual machine folder with all the necessary files into the System Centre Virtual Machine Manager (SCVMM) library share. In case you do not have a virtual machine you can easily create one using SCVMM Admin Console which comes with Lab Management SKU (we will cover this in more detail in one of the upcoming posts).
2. Now launch “Microsoft Test and Lab Manager” from All Programs -> Microsoft Visual Studio 10.0 -> Microsoft Test and Lab Manager
3. Connect to the TFS Server and your team project. Click on “Don’t Set context” when prompted. Go to Lab Center (from top left) and click on “Virtual Machine Templates” on the ‘Library’ activity.
4. Click “New” to create a virtual machine template. This opens up the wizard where first you need to select the library share where you want Lab Management to store this virtual machine template.
5. Then select the virtual machine from SCVMM library share by clicking the browse button. The below list shows the virtual machines which you have created earlier either using SCVMM Admin console or copied into SCVMM library share (in case you already had virtual machine).
In this list you only see virtual machines available in SCVMM library shares. So if you have existing VHD which you want to create a virtual machine template from, use SCVMM to create a VM using this VHD, and store it in the library.
6. After selecting the virtual machine now you can enter the name and description for this virtual machine template. Also you can assign a role to help select appropriate virtual machine template during environment creation e.g. all the virtual machine templates with different configurations (version, OS etc) of SQL server can be assigned role as “Database Server”. Similarly all the virtual machine templates with different configurations (version, OS etc) of IIS server can be assigned role as “Web Server” etc.
7. That's it!. Just press finish and you have created your first virtual machine template as you can see in your list of virtual machine templates.
8. Repeat above steps to create another virtual machine template for IIS Server.
Creating Virtual Environment
Now as you have created virtual machine templates (Golden VMs) you are all set to create your first virtual environment. Majority of team members in your team can start directly here as once someone creates a set of virtual machine templates everyone can start using them to create environments.
1. Click on “Environment” activity in Lab Center. Click “New” to create a new environment.
2. Provide a name and optionally a description for the environment you are about to create. Let’s give it a name “Dinner Now Test Environment” as we will use this for testing Dinner Now application.
3. Select Virtual as the type of environment
4. Select one of the host groups from the host group drop down list & click “Next”
5. From the list of available roles, select the Database Server role and the Web Server role, by clicking on each and then clicking on the Add button. Note that you can also add your own roles by clicking on the New Role button on top of the roles lists. Click “Next”.
6. Assign a VM template for each of the roles. On the right side you can see the list of VM templates from library as created in earlier step. e.g. Assign VM template Win2k3-IIS to Web Server role and Win2k3-SQL to Database server role.
7. That’s it! Just press finish and your first virtual environment creation is started. Virtual environment creation may take several minutes depending on number and size of VM templates selected above.
Working with Virtual Environment
Like physical environment you can start, stop and connect to different machines of your virtual environment. In addition, virtual environment provides some advantages over physical environments such as snapshot & restore. You can take a snapshot of your complete virtual environment (data and memory state of all machines) at a point in time and later restore that snapshot. This is really useful to debug complex bugs which are difficult to reproduced on a developer machine as now developer has access to the environment with the same configuration, same build and same meta data when the bug was found. With this capability you can significantly reduce the no. of non-reproducible bugs. Stay tuned for more details on this scenario in an upcoming blog post.
To work with any environment you can just double click on any of the virtual machines of the environments and it opens up “Lab Environment Viewer” tool. This tool allows you to connect to different machines of the environment without remembering their names/IP addresses for terminal services (TS) sessions. Also Lab Environment Viewer shows you all the virtual machines on the left and avoid painful switching between TS sessions. While you are working on one of the machines you can also get a quick preview of other machines with the cool thumbnail view.
To start, stop, snapshot or pause an environment you can simply click the necessary action from the environment tool-bar at the top-left corner. when you perform an action it applies to all the virtual machines of the environment. So to snapshot a multi machine environment you just need to take snapshot for the environment as oppose to individually taking snapshot of each machine.
All the snapshots of the environment are available under snapshots tab of environment viewer. At any point of time you can revert your environment to any of these snapshots with a single click of revert and all the machines will be restored to that snapshot. e.g. you may want to take a snapshot after the environment is setup first time calling it “Clean state” and than deploying regular builds on this environment can be simplified significantly as you can revert the environment to clean state before every build deployment. Lab Management also offers great automation to automate regular build, deployment and test cycles.
Also at any point you can access to an older build with a simple revert to the snapshot of that particular build. This is really helpful when you need a prior build for debugging a issue as unlike with physical machines where you may spend hours (to uninstall the current builds and then install new build and even then not sure if everything is fine as uninstall may have issues) now you can get to a prior build with just a button click “Revert to Snapshot”.
In this post we have covered the basic Lab scenario, of creating and working with virtual environment. Stay tuned for the next post about leveraging virtual environment for testing.
- Application build, deploy and test automation in Lab Management
-
Let’s look at an important application lifecycle management (ALM) scenario – end to end automation for building, deploying and testing a multi-tier application. Application deployment for a multi-tier application typically involves
· provisioning of multiple machines
· running cleanup scripts on each machine to get rid of older application binaries and configuration; or reinstalling the OS and application prerequisite stack from a scratch
· running various installation and configuration scripts on each machine
This process is time consuming and error prone, if not fully automated. On top of it, agile development demands that new builds be tested more frequently.
VS2010 Lab management solves this major pain point in a simple and elegant way by leveraging distributed Team Foundation Build workflow. Lab Management workflow activities are bundled with Team Foundation Build Service. You can drag and drop these activities in Windows workflow designer to create custom workflows that allow you to
· quickly provision a virtual environment
· revert to ‘clean’ environment in tens of seconds by using environment snapshot instead of running multiple ‘cleanup’ scripts or reinstalling OS and application prerequisites
· using distributed workflow, run setup and configuration scripts on virtual machines
· Take post deployment environment snapshots, etc.
A typical end to end workflow could be similar to the picture shown below.
To get you jump started with application deployment workflow for a virtual environment, we are providing a lab-specific default workflow template that you can download. The rest of the post covers the following aspects of using lab default workflow template:
· Where to check-in the template file.
· How to use the Lab Default Template as it is.
· How to customize Lab Default Template.
How to Give Permission For Running the Lab Default Template.
Pre-requisites
· Lab is installed and configured on a TFS server. Refer to Lab Management 2010 Beta 1 for further details on setting up lab.
· You have at least one virtual environment either running on a host or stored in a library
Where to check-in Lab default workflow template file?
The lab default workflow template is a XAML file that should be checked-in to TeamBuildProcessTemplates folder under the Team Project you are working in. Once you have successfully checked-in the Lab default template, you will see three files in TeamBuildProcessTemplates folder as shown below.
How to use the Lab Default Template it as is
1. Download the Lab Default Template.xaml. By right click on it and selecting save as.
2. Overview of the Lab Default Template: Let’s look at what Lab Default Template does? Lab Default Template provisions the specified Environment and optionally it restores to a given snapshot name and then runs the application deployment commands on the virtual machines inside environment. When the application deployment completes it optionally takes one more snapshot of the environment and completes the workflow. Lab Default Template consists of Process Parameters and set of Lab Activities. Process Parameters are similar to variables and Activities are similar to functions in a language respectively. There are various process parameters in the Lab Default Template. For example, “Team Project Library Share Name” where you specify the Library share name from where you want to provision your Lab environment. Process Parameters are discussed in detail in the later section. In case you want to customize the deployment workflow beyond process parameters, you can do that as described later in the ‘How to customize it’ section. The next few steps describe how you create a new build definition from the template, schedule it and review the output logs.
3. Create a new build definition
a. In Team explorer pane, do a right mouse click on Builds node under the team project and choose New Build Definition as shown below. This will open the new Build definition window.
b. Enter the name of the build definition (for example here “Lab Deployment Definition”) in the ‘Build definition name” edit field as shown below.
c. Now go to the “Trigger” step listed in the left pane of the wizard and on right hand side select the radio button with text “Manual – check -ins do not trigger a new build” as shown in the screen shots below. This is the most appropriate option for running deployment workflow as you are using a previously generated build as opposed to compiling application code.
d. Now go to the “Build Defaults” step and select the build controller (for example here “Default Controller - TCBCw2K3XB6”) you had specified at the time of the Environment creation in Microsoft Test and Lab Manager. In the field “Copy build output to the following drop folder (UNC path, such as \\server\share)” give a valid shared path location here. Lab default template does not use this field so any shared location path will do as shown below in the screen shot. There is workspace step also but we can ignore it and leave as it is.
e. Now go to the “Process” step. In the right hand side pane go to “Build Process File (Windows Workflow XAML):” field and press on new button, this will open up a dialogue box named” New Build Process Template”. In this dialog box, select the radio button with text “Select an existing XAML file” and then click browse and select the Lab Default Template.xaml file that we checked in the source control and press OK button; and one more OK button as shown in the screenshot below.
f. This will update the “Build Process Parameters:” field as shown below in the screenshot.
Now you can start filling up the Process Parameter Variable as described below.
There are three main headings in the Process parameter pane. Those are:
· Create New Lab Environment From Template.
· Lab Management Parameters.
· Use Snapshot Of Existing Lab Environment
Under each of these heading there are various Process parameters, let’s start by first heading “Create New Lab Environment from Template” under which there are two optional process parameters which allow you to use an existing environment template in the Team Project Library Share and bring it to running state on a host.
· Lab Environment Template Name: Name of the lab environment template stored in a team project library share.
· Team Project Library Share Name: if you specify the “Lab Environment Template Name”, then this process parameter is compulsory. It specifies the Library share in which Lab Environment Template is stored.
The second Heading is “Lab Management Parameters” under which there are five Process parameters.
· Build Location: this is the source path from there deployment scripts will pick up the application binaries and other files; and drop them in the destination locations on various machines that are part of the environment.
· Lab Environment Name: here you specify the Lab Environment Name on which you want to run the Application deployment script. If you specify the “Lab Environment Template Name” and “Lab Environment Name” then the new environment is created from the “Lab Environment Template Name” and given the name specified in the “Lab Environment Name”. If you do not specify the “Lab Environment Template Name” then it means that you will be connecting to an already running environment specified by “Lab Environment Name”.
· Machine Name|Installation Command: here you specify the string which is formed by the virtual machine name (not the computer name) separated by pipe symbol and the installation command that you want to run on that machine. These machine names are the same that you specified when you created the environment in Microsoft Test and Lab Manager. For example, “IBuySpyclient|\\Vg_Hpypervmm\share\install.bat silentinstall” command means that run the Install.Bat silentinstall command on IBuySpyclient machine name. You can specify multiple strings and the commands therein will be run sequentially in the specified order. E.g.
“IBuySpyclient|\\Vg_Hpypervmm\share\install.bat silentinstall “
“IBuySpywebserver|\\Vg_Hpypervmm\share\configureweb.bat “
The screenshot below shows how you enter the values for this parameter.
By default, the working directory for these commands is %windir%\system32. You can override it by specifying working directory after the command string; separated by pipe symbol. For example,
“IBuySpywebserver|\\Vg_Hpypervmm\share\configureweb.bat |c:\temp” will run the configureweb.bat command on virtual machine IBuySpywebserver in C:\temp directory.
· Take Post Deployment Snapshot: if you specify it as true then after running the deployment commands specified by “Machine Name|Installation command” parameters, an environment snapshot will be taken. The name of the snapshot will be Build definition name appended by date and time. The default value of this parameter is false.
· Team Project Host Group: it is the host group on which the Lab environment virtual machines will run. .
· The third Heading is “Use Snapshot of Existing Lab Environment” which has one process parameter “SnapShot Name”. It is an optional parameter. If you specify the snapshot name, the Lab Environment will be restored to it before the app deployment commands specified by the “Machine Name|Installation Command” process parameter are run. The screenshot below shows a sample with all process parameters values filled up.
4. Right click on “Lab Deployment Definition” on the top where it shows “Lab Deployment Definition *” and choose “Save Lab Deployment Definition” to save the definition as shown below in the screenshot. The build definition would show up under the “Builds” node in Team Explorer.
5. In Team explorer, right click on the “Lab Deployment Definition” and choose “Queue New Build” as show below. This will open up the “Queue Build” Dialogue box. Click on the Queue Button.
6. Once you click on the Queue Button in above step it will open up a new sub window named “Build Explorer” as shown below in the screenshot.
7. Inside that window you can double click on the Lab Deployment Definition. This will open up another sub window which will show the Build definition summary. You can go through the summary and if you want to see the detailed logs, click on the "View log” link as shown below.
How to customize Lab Default Template?
1. Go back to the Source Control Explorer. Go to TeamProject and then to TeamBuildProcessTemplates in folders view. Go to right hand side pane and right click on the Lab Default Template.xaml and click on view as shown below in the screenshot.
2. This will open Workflow Designer as shown below in the screen shot.
3. In the workflow Designer window you can see the Lab Default Template workflow. Now go to left hand side and click on Toolbox. This will open up the Toolbox window as shown below in the screenshot. There are many activities listed under different headings. The Lab Activities are listed under “Team Foundation Lab Activities”. Team Build activities are listed under “Team Foundation Build Activities”.
4. To add new activities, drag and drop them from the Toolbox to the workflow designer canvas. You can right mouse click on the activity icon and modify its properties as shown below. You can also delete existing activities in the workflow. If you want to use “If then else” or “switch” statement you can use the activities under “Procedural Heading”. To edit variables associated with an activity, select it and click on the ‘Variables’ field at bottom left corner of workflow canvas. If an activity encompasses more activities, you can browse one level below by double clicking on it.
5. If you made any changes, save them and check in the updated version to source code control.
How to give Permission for running the Lab Default Template.
The Build Agent which is running as the service with the particular login account should have been added under Project Contributor as well as the user who is queuing the Lab default Template for running should be also added under Project Contributor.for How to add the user to Project Contributor Group please follow the link.
- Enable Lab Management features for existing Team Projects
-
If you are upgrading your VSTS 2008 server, you will need to enable Lab Management for existing projects.
So, first upgrade your TFS server. Then you can configure Lab Management using TFS administration console and start using these features for new projects. To enable Lab Management features on existing team projects use the following steps.
Pre-Conditions
· Install SCVMM server and add lab hosts to host groups. For more details see scenario “Installing Lab Management” in Team Foundation Installation guide for Visual Studio Team System 2010.
· Install SCMM admin console on TFS application tier machine. For more details see scenario “Installing Lab Management” in Team Foundation Installation guide for Visual Studio Team System 2010.
· Enable Test Case Management for the project. See Hakan’s blog for a sample script to easily enable Test Case Management features on an existing project.
Step 1 - Configure Lab Management
You can configure Lab Management by using the administration console for Team Foundation.
· Click Start, point to All Programs, and then click Microsoft Team Foundation Server 2010 Beta1.
· Click Team Foundation Administration Console.
· In the tree, click Lab Management under Application Tier.
· On the Lab Management pane, click Configure Lab Management.
· In SCVMM server name, type the fully qualified name of the server that is running Virtual Machine Manager and that you will use to manage the virtual machines.
· Click Test to determine whether Visual Studio Team System Team Foundation Server can communicate with the VMM server. If Team Foundation Server cannot contact the VMM server, a red x and an error message will appear.
· In Network location, type the address of the network to which the virtual machines will connect. If you do not know the name of the network location for each host that you want to include, open the Virtual Machine Manager Administrator Console, view the Hardware properties of each host, and identify the network location for each network adapter.
· Now if you want to use Network Isolation (You can use network isolation to enable multiple copies of a lab environment to run at the same time without causing network conflicts, such as conflicts in computer names and Domain Name System (DNS) registration.) select Network Isolation tab and click “Configure Network Isolation”.
· In DNS suffix, type the suffix of the domain name to be assigned to the environment when an isolated network is created. Each virtual machine in a network-isolated environment is registered with a unique external name and DNS suffix that you specify. You should use a suffix that is configured to support non-secure updates in your DNS server. To confirm that the suffix is correctly configured in the DNS hierarchy, contact your network administrator.
· Click Test to determine whether Visual Studio Team System Team Foundation Server can use the suffix. If the suffix is valid, a green check mark appears. Click OK. If the suffix is not valid, a red x and an error message appear. You must fix the error before you can continue. Click OK.
Step 2 - Configure Lab Management for the Project Collection hosting upgraded projects
· Click Start, point to All Programs, and then click Microsoft Team Foundation Server 2010 Beta1.
· Click Team Foundation Administration Console.
· In the tree pane, click the Team Project Collections node under Application Tier.
· Select the project collection hosting upgraded projects.
· Click on Lab Management tab for the selected project collection.
· In the results pane, click Configure Library Shares.
· Click Add, select one or more VMM library shares that this team project collection will use, and then click Add. You can add a library share to only one team project collection at a time.
· Now to configure Host Groups for this collection, select Host Groups tab.
· Click Add, select one or more host groups that this team project collection will use, and then click Add.
· Click Verify to verify the status of selected host groups.
· Now if you want to use Network Isolation select Network Isolation tab. This option will be available only if you have configured network isolation at application tier in step 1.
· In DNS suffix, type the suffix of the domain name to be assigned to the environment when an isolated network is created. To confirm that the suffix is correctly configured in the DNS hierarchy, contact your network administrator.
· Click Test to determine whether Visual Studio Team System Team Foundation Server can use the suffix. If the suffix is valid, a green check mark appears. Click OK. If the suffix is not valid, a red x and an error message appear. You must fix the error before you can continue. Click OK.
Step 3 - Configure Lab Management for upgraded Team Project(s)
Download the zip file attached to this post, and extract it to a local directory such as C:\Upgrade.
Before you run the script, open “EnableLabManagemet.bat” in a text editor and complete the configuration by specifying values for the following:
- Team Foundation Server URL
- Name of the team project collection
- Name of the team project
- Path to TFSLabConfig.exe utility
- Set ConfigComplete = 1 to indicate that you’ve completed this step
This script will perform following 3 steps to enable Lab Management
· Grant default Lab Management permissions to standard security groups e.g. Readers, Contributors, Project Administrators and Project Collection Administrators.
· Create Project Host Groups
· Create Project Library Shares
Rerun the batch file for each existing project, after updating project name in the batch file.
- VSTS 2010 Beta1 is Released!
-
Today, we have released Beta 1, and it is now available for download. We encourage customers and partners to download and evaluate the Beta, which ultimately results in changes and improvements to the final product.
Beta 1 represents a substantial amount of the functionality that will be in the final shipping version of the products. I may say we have about 95% of the functionality that we plan for our Version 1 of Lab Management.
Lab Management is part of this Beta, so you can start using it right away.
To find out how to download the beta and where to share your feedback, please visit the Visual Studio 2010 Product Page.
To ramp up real quick, follow the video on downloading & installing. Once you setup TFS, enabling Lab Management is pretty easy.
So, get it on your machines, find the productivity boost that you can get from it, and if you see any issues, please report them to us.
Note: we will not provide upgrade for Lab Management from Beta1 to Beta2 or the 2010 official release, so once Beta 2 will be available, you will need to reinstall lab. So don’t enable lab on your production Team System server at this time.
- VSTS 2010 Lab Management – Basic Concepts
-
This is first of a series of blogs we will be publishing over the next few weeks to cover breadth and depth of VSTS 2010 Lab Management.
Visual Studio Team System 2010 Lab Management is an integrated solution to give you all the benefits of virtualization for application lifecycle management. In this post, we will cover high level architecture of Lab Management and some basic concepts that are key to virtualization and Lab Management.
Lab Management High Level Architecture
The picture below shows a high level architecture diagram for Lab Management.
On the server side, Lab Management service is one of the many services running inside Team Foundation Server (TFS). This is what makes the Lab Management solution unique for software testers and developers. Now you can map your lab resources, such as, hosts, virtual machines and storage to Team Project Collections and Team Projects; thus aligning lab hardware needs with the business needs for the projects you are working on.
The lab management service in TFS uses System Center Virtual Machine Manager (SCVMM) for management of lab infrastructure and provisioning of virtual machines across multiple virtualization platforms. You get a copy of SCVMM with Lab Management.
Microsoft Test and Lab Manager is a Windows Presentation Foundation based rich client. The Lab Center in Test and Lab Manager allows you to
- Create and manage virtual or physical environments
- Take environment snapshots or revert to existing snapshots for virtual environments
- Interact with the virtual machines in the environments through environment viewer
- Define test settings for the environments
You can define test plans, test suites and test cases in the Testing Center and execute them on the lab environments.
Basic Concepts
Similar to Internet, hardware virtualization is a disruptive technology that is changing the face of computing. Therefore, it is important to understand some of the basic concepts around virtualization and how these are used in Lab Management to understand this paradigm shift.
Virtual Machine
A virtual machine (VM) is a computer within a computer, implemented in software. A VM emulates a complete hardware system, from processor to network card, in a self-contained, isolated software environment, enabling the simultaneous operation of otherwise incompatible operating systems on a single physical computer. Each operating system runs in its own isolated software partition.
Virtual Machine Snapshot
A virtual machine snapshot is a file-based snapshot of the state, disk data, and configuration of a VM at a specific point in time. A VM snapshot is similar in functionality to laptop hibernation state with the additional flexibility that a VM supports multiple snapshots. You can roll back the VM to any of the previously taken snapshots and continue operating from there. The picture below shows a snapshot tree for a Hyper-V VM. Using snapshots, you can accomplish some of the testing scenarios, such as, rolling back the VM to a clean baseline state, before installing a new build, in tens of seconds which otherwise might take tens of minutes or hours.
Host
A host is a physical computer that hosts one or more virtual machines.
Host Group
A host group is a custom group of virtual machine hosts, which an administrator can create in SCVMM for ease of monitoring and management. Host groups can be used to allocate and determine the resources reserved for various team projects. For example, an administrator could create a host group named “Global Bank Hosts” for a team that works on “Global Bank” project and bind it to the corresponding team project in Team Foundation Admin Console.
Library Share
One of the beauties of virtual machines is that you don’t need to tie up a host if you are not actively using a VM. You can store it on a disk and bring it back to life on a host in a few minutes. SCVMM supports the concept of a library share where you can store virtual machines and other resources, such as, ISO images. The library share is nothing but a file share that is accessible to all the hosts. Similar to host groups, you can create multiple library shares for ease of management. For example, you could have a library share for storing pristine or golden OS images. Another library share could be used for storing VMs that have various application software components installed.
Environment
A typical multi-tier application consists of multiple roles, such as, Database Server, Web Server, Client, etc. Each role could be running on one or more computer. You could also have multiple roles running on a single computer. An environment is a set of roles that are required to run a specific application and the lab machines to be used for each role.
Managing environments for multi-tier applications is an error prone task today. Replicating the same environment at same or another site is even a bigger problem.
Lab Management surfaces environments as a first class entity. See the picture below for an example. It shows a list of environments running in the lab. The “DinnerNow Integration Testing 2” environment consists for two virtual machines, named, DinnerNow-Web and Dinner-SQL.
Environment brings with it a ‘strong’ group notion. That is when you do an operation on an environment, such as, start, stop, take snapshot, etc., that operation is applied on all the virtual machines that are part of the environment. In the next post, we will show you how to create, start, stop, delete, save, take snapshot and interact with rich self-documenting virtual environments using Lab Management.
- TechEd India 2009
-
We've had a phenomenal last couple days at Tech Ed India 2009 (including some bloopers -- find it in the pics below :-) )
Starting from Steve Balmer’s keynote to sessions by Amit Chatterjee (on VSTS 2010), Tejasvi (on TFS), Infosys (Best Practices with TFS) & Neelesh (VSTT 2008 Load Testing) which went off really well. We had a full day of showcasing test & lab management at the "technology tents" where we had packed house and in some sessions barely enough standing room.
We also met with a lot of customers, partners and had some great discussions around ALM/Team System & TFS.
Team Lab Technology Tent
Team Test Technology Tent
And the blooper:
ALM gives you lifestyle in the lifecycle :-)
- Tech-ed and some announcements
-
Tech-ed North America is going on right now in Los Angeles and we from VSTS and lab management team have a big presence there. In case you are attending Tech-ed, we have 4 VSTS booths in the DPR/ALM area of the Technical Learning Center. One of these booths is dedicated for Test Edition, and we have a machine setup with lab management. Please stop by to see the latest bits in action and to discuss any questions you may have. Also, we have a session on lab management on May 14th, Thursday at 4:30 pm (DTL317: Using Virtualization to Improve Application Quality with Team System - Lab Management).
There have been a few important announcements this week. We announced the name for the generalist testing tool used for testing and lab management as "Microsoft Test and lab manager" (formerly codenamed Camano). You can read more about this in Jason's blog post. Also, there have been some announcements about Windows Server virtualization and Windows 7 with respect to release timelines and new features. You can read all about it in the virtualization team blog.
- Step In Hyderabad Chapter meet
-
stepinforum
stepinforum
StepIn Hyderabad Chapter meet
Venue: Microsoft Campus
Date: 27th April 2009-04-29
Time: 5pm
Approximately 100 odd software professionals from the city met at Microsoft campus to attend the StepIn Hyderabad Chapter meet. Two sessions were conducted with the theme of virtualization. A short brief of the sessions below:
1. “Green Test Labs” by AB Kishore, Wipro
Virtualization and green IT initiatives are the new ways in which companies are doing business. The talk started with the need to go “Green” for test labs. The optimization hot spots of the test process were identified and various challenges in a typical SI (Service Integrators, the kind of company Wipro is) shop pointed out. Some of them are the need to be platform agnostic, test interface agnostic etc. Wipro had developed a test automation framework that was overviewed during this session. The highlights were that it has a service based model; supports open test automation and virtual scripting. The talk ended with two demos. In the first one, test execution was optimized and automated and in the second one, resources i.e. lab hardware was commissioned and decommissioned, thus increasing reusability.
2. “Improving software quality and hardware utilization by using Virtual Labs” by Vinod Malhotra, Group Program Manager, Visual Studio Team System – Lab Management, Microsoft
Among the biggest challenges in software development today are poor software quality and lack of business alignment. On digging deeper, it can be seen that developers and testers are unable to collaborate and there is overall inefficiency in the hardware utilization. Key areas for improvement identified were build automation extending to environment provisioning, build deployment and testing, increasing hardware utilization and building the tools required for developers and testers to work together.
A demo of the Visual Studio Team System Lab Management was shown where each bug could have detailed information including video of the bug, extensive logs and screen shots; managers could get accurate reporting on the status of the project and test cases could be easily automated. To the traditional build work flow of compile, deploy, run tests, Team Lab adds “Restore environment” to restore the test environment to a pristine state. This is enabled by the capability to “Take Checkpoint” of the environments while testing or during development, so you can attach those to a bug and later restore those checkpoints and debug the system.
Stay tuned to the StepIn Hyderabad chapter here http://www.stepinforum.org/index.php?option=com_magazine&Itemid=434
- Trends in Software Testing
-
Tanuj Vohra (Partner Director of Program Management, Visual Studio Test Business) has recently published a nice article on Trends in Software Testing on ITMagz.com. It is summarizing the current trends and techniques of software testing. Test Labs Virtualization is of course one of the items there, as well as other solutions that are part of VSTS 2010, and some other trends that VSTS is not addressing.
Hope you will find it an interesting read.
-- Shay Mandel
Lab Management Team
- Getting ready for Beta1 and VSTT 2008 Quick Reference Guide Released
-
As we continue to prepare for Beta 1, we continue to talk with customers, demo the product and get feedback. The feedback is usually:”Hey, this is great stuff, when can I get it? Can I start work with Beta1 for my new project that is just starting?” – well, the answer is that we will have Beta1 released in about 2 months from now, but we don’t suggest that you will take a risk and start your new project on it. The reason is mostly the fact that the upgrade from Beta1 to Beta2 will not be bullet-proof as we would like it to be when we tell you to start working on your project.
But you should try to work on Lab for a part of your team, enjoy the productivity gains that you can get, learn about the product and get us feedback so we can improve it for Beta2.
So what can you do while you are waiting for the Beta? get the hardware ready. You should have at least one host that is Hyper-V compatible (required Hyper threading processor), preferably with good amount of memory – 4-8GB, so you will be able to run a few VMs on it. Get it allocated, so when the Beta is released you can start right away.
In the mean time, our Rangers team has released the VSTT 2008 Quick Reference Guide, that will help you leverage your current testing solutions in VSTT 2008. With this reference, you can take VSTT to the next level, and create professional test solutions for your existing projects. The guide contains several topics:
- SETUP CONSIDERATIONS
- WEB TEST CONSIDERATIONS
- WEB SERVICE TEST CONSIDERATIONS
- UNIT TEST CONSIDERATIONS
- LOAD TEST CONSIDERATIONS
- LOAD TEST RIG CONSIDERATION
- PERFORMANCE DATA COLLECTION AND USAGE
- LOAD TEST RESULTS STORE INFORMATION
- TEST CUSTOMIZATION
- ITEMS CHANGED OR FIXED IN VSTS 2008 SP1
- GENERAL COMMANDS AND TRICKS (NOT VSTS SPECIFIC)
The guide is published on CodePlex, so you can contribute to it from your experience as well.
You can download the guide here.
-- Shay Mandel
Lab Management Team
- Cool Visual Studio 2010 Video
-
Long time has passed since the last post…
The team is working hard to get our bug count to zero, getting Beta1 ready for customers. A lot of hard work polishing the product and making sure it will be stable and have good performance.
There is still some work left for Beta2, and we also reserve some time for the valuable feedback we will get from you after we will release Beta1 (no official date, but it will be 2-3 months from now).
On a lighter note, here is a funny video about VS 2010 features. You’ll like it, especially if you have young children.
- Lab Management in the overall ALM picture
-
I (Shay Mandel) have recently made a series of customer visits in Israel. I have met R&D managers, QA managers and CTOs from 3 ISVs/Start-Ups, 2 IT organizations (one large insurance company and one governmental department), and 2 MVPs - Guy Kolbis and Leon Langleyben from SRL which is a Microsoft VSTS implementation partner. Guy has also posted about Lab in his blog after our meeting.
It was very good to see that most of the development organizations today either use virtualization as part of their development and testing process, or that they intend to use it in the coming year. This shows that we have a great timing for the product.
I have also presented to the Israel ALM (Application Lifecycle Management) User Group. I have shown how lab can be woven into the overall quality process of ALM. While preparing to it, I saw that we have a lot of places that we can help:
- By getting developers an environment that emulates complexity of a multi-layer, production-like deployment in a matter of minutes, so that their testing will be much more realistic.
- Providing an easy to use Library of VM and Environment templates that they can use
- Removing the hassle of finding the hardware resources to run a VM on, as we introduce the notion of Machine Pools that contain several physical hosts, on which we automatically allocate the best machine to run on
- By making the Gated Check-in build to run on a clean environment, using our built-in workflow activity for reverting an environment (in this case the build environment) to a checkpoint. If you are not familiar with Gated Check-in yet, you can see a demo of it (and many more cool features of VSTS 2010) on Cameron's Skinner's PDC session: Lap Around VSTS 2010
- Similarly, we can help to make the Continuous Integration environment more controlled and more similar to the complexity you will find your real life scenario. We can also revert this environment to a clean state in a matter of minutes before deploying the new build and running the integration tests on it
- Testers do get a lot of attention in our stories. We allow to deploy an environment for every tester in a matter of minutes, be it a single machine or a complex, clustered set of machines that simulate the real production set.
- Testers are encouraged to take checkpoints when they find a bug. By simply clicking on a button in the Test Runner, a test engineer can capture the state of all the machines which comprise a system. This includes the memory state of the processes, the active windows, the logs and events in the system and more. All this information is captured as snapshots on the hosts, and we provide a simple link to all this setup. When a developer clicks on the link, he can revert to the exact same state, attach a remote debugger and pinpoint the bug, as if it was revealed just a second ago! This is what call Actionable Bugs!
- Another good use of lab in the testing arena is for testing complex installations and upgrades. The problem with such installation is that after you test once, your system is not 'clean' anymore and you need to reinstall the OS or face the risk that the bugs you will file after the next time you run the install test will not be because of a real malfunction of the installer/upgrader, but rather because your environment is not real. Using checkpoints, you can overcome this. Simply take a checkpoint before the installer runs for the first time, and make it as 'pre-install state'. Then you can always revert back to this state, and it will be a real pre-install state, as it is a copy of the machine as it was at the time the checkpoint was taken.
I got a lot of great feedback which I will try to share in the next posts, after I share it with the team and we decide on what will get into the product. Feel free to add your own feedback and questions by commenting on this post.
If you are interested in hearing more about ALM and Lab Management, especially for Agile development process, I am going to present on the subject: The Next Generation of Agile Software Development - Virtualization for faster development cycles in Virtualization Congress which will be held on May in Las Vegas. Please vote for it and come hear it.
And if you are interested in that, you should also vote for Vijay Machiraju's topic - Technology choices for virtual lab automation. Vijay will share with the audience the choices we had and the decisions we took for how to use the right technologies for implementing ALM processes in a development and testing organization.
Aseem Bansal from our development team will talk about Testing & debugging made easy using virtual lab automation , and will share from his dogfooding experience how bugs can be more easily reproduced and fixed faster.
Please do vote for those sessions if you think they are interesting (just click on the up arrow near the subject name when following the links above).
Shalom!
- Lab Management in Virtual Tech Days
-
I (Vishal) have just completed the Team Lab session in Virtual Tech days using the slide deck you can find as attachment to this post.
Session went well with lots of questions and participation, we had to end the session due to time constraints but questions just keep on pouring. I was happy to see that our product is really generating a lot of excitement. As we could not cover all questions due to lack of time please feel free to post your questions as comments to this post, we will try to answer those here.
We had a live demo of our latest bits (yesterday) and all the demos ran successfully except in one case I forgot to share the VM so participants were not able to see.
From the Polls collected during the session from participants, it was evident that almost everyone is facing the challenge of non reproducible bugs.
Another interesting data from the poll was related to common causes for non-reproduction of bugs. Environment configuration (Software, Hardware) mismatch came as the number one issue, followed by data mismatch and finally build mismatch and few selected Others category.
Fortunately with Lab Mgmt integration with testing (Tester can take checkpoint from the testing tool and the link to environment checkpoint is automatically associated with the bug without any additional step) and development experience (Developers can access the environment with same state in which bug was found by simply clicking the link to checkpoint in the environment) using the checkpoint and restore technologies we can handle the first 3 categories of common causes.
It was a good experience and a nice reassurance to our product. I hope that the people who attended the session enjoyed it as well.
