Our application (web based) is built in .Net and Infragistics. It has 6 modules. One module is similar to product management, in this we need to enter product title, product details, etc. Newly created product names appear on the homepage. So while testing, in the "Product Title" field I entered this text: <script>alert ("test")</script> and then I saved the Products page. After saving an alert popup with text "test" appeared. And when I went to the homepage, the product name was not appearing but a popup with text "test" appeared. Whenever any user goes to the homepage, an alert popup appears. Then I insert a for loop in the product title and now the popup appeared 5 times. It was quite irritating =)
This is my favorite bug. Actually it is a security loophole (cross side scripting). Here one can call malicious scripts also.
-- Sumit Kalra
Do you have a bug whose story you love to tell? Let me know!
The one that says that it is erasing my old email when it really isn't!
I like to erase old email. For some reason the web mail I am using will not allow me to delete my old email. One day I spent over an hour attempting to delete old email, when I realized the old email had not been deleted at all. "What in the world is this?" I spoke to myself. Then like lightning from on high, I realized my bug was human. What else could it be? Some one out there, wants my old email. So, I don't want it any more! Let 'em have it!
-- Thelma Whitehorton
Do you have a bug whose story you love to tell? Let me know!
Our team uses a web-based tool to test our product. In one page, we can input tree-like structured data. In some conditions, we have to create a child under one node of that tree by clicking a drop-down list to select the type of the child object. The amazing thing is, we have to click many times to create the child successfully. We have to click, click, click, and don't know which time we can succeed, sometimes, it never works. After being confused for several days, one tester finally found the defect: just at the left side of the drop-down list, there is an invisible button! If we click that button by accident, we can create the child object. If we click carefully on the list, we will have no chance of success. Sometimes, carelessness helps!
-- Donny Liu
Do you have a bug whose story you love to tell? Let me know!
In one application there is a small line of whitespace between a custom toolbar and the menu; clicking this whitespace causes an unhandled exception due to a null pointer. I found this by accident just trying to access certain options. This is my favorite because I like finding unexpected bugs.
-- Sherry Chupka
Do you have a bug whose story you love to tell? Let me know!
Of all the bugs I've came across I'll never forget this one because it was so severe and easy to miss. Later, when I was a Test Lead it was the first thing I added to my sign-off checklist… A few years ago, our product was practically out the door when we learned that setup would fail if the computer clock was fast forwarded. Basically, a year or so after shipping the product would no longer install, because some time-bomb code remained in the setup custom actions. The team had performed timebomb tests. But did them by moving the clock forward *after*, and not before, installing the product. Easy mistake to make, if you aren't aware of the setup timebomb or that setup can have custom actions. For me, it's a good illustration of a common reason why teams miss bugs: knowledge/communication gaps.
-- Craig
Do you have a bug whose story you love to tell? Let me know!
When my first test manager left Microsoft he told the team this was his favorite bug and it is easy to see why (testers are often asked about their favorite bugs when they leave). On his first week on the job one of my friends hit a crash in the speech recognition engine. He had no idea what to do next and went over to ask his developer. His developer was busy and so asked him to create a bug and write down *exactly* what he did. Being new and nervous he entered the bug exactly as it happened, including the *exact* words he spoke to crash the recognizer. Being a colorful character the exact words he wrote in the bug included many curse words! The bug bounced between several external teams and later I heard they all had a good laugh. The bug is still in the database today and it was his first ever bug at Microsoft (the bug did get fixed). In retrospect, I'm sure he later realized he could've easily found some PG words that would've reproduced the crash. One takeaway: try to find the minimum steps to reproduce the bug!
-- Craig
Do you have a bug whose story you love to tell? Let me know!
The most interesting recent bug was a Vista upgrade error that was initially assigned to my component. We had a couple of Watson reports where the symptom was there would be an error logged during upgrade which caused a fatal error and a roll-back. Unfortunately there was no more information present and none of the reports we were getting were on a kernel debugger. We had no way to investigate further. This was occurring just as Vista SP1 was getting ready to ship so it was very consternating to our team - even though we were convinced it was not an issue in our code! I borrowed a computer over the holidays from a person in another area of campus who had a repro, but it would not repro in my office under a debugger. Finally it boiled down to me doing upgrade after upgrade of Vista to SP1 in my office - desperately seeking a repro. That is the unglamorous side of testing to say the least. Eventually I hit the issue and our team was able to really start the investigation after which the bug unraveled quite quickly. It turned out to be a bug in setup. Coincidentally right after we "root caused" the bug, a timing change made it more likely to occur and it turns out it would have caused something like 3-4% of Vista upgrades to fail. When you're talking about Windows, that would have been a huge number of customers affected! (Of course, we would never have shipped without a fix.) On the lighter side, we got a "good job" email from a partner who hit the bug. The mail thanked the teams involved for providing a private fix to "turn-around" the issue less than 24 hours after he hit it. It was nice to hear, but he was not aware of the weeks of work we'd put in tracking down the repro – it just looked really fast. :)
-- Tom Whalen
Do you have a bug whose story you love to tell? Let me know!
There was this bug that I remember even 4 years later now. Since we expect our users to install our product on operating systems of different languages, we do globalization testing on this. So for the Beta release, I installed our product on a Japanese (JPN) Windows 2003 (W2K3) box and tested that the basic functionality was all doing good. Of course, it already worked well on an English W2K3 system - so we were quite happy with the way it went.
Late into the cycle, just for kicks, I installed my product on a German machine and tried to kick off a build. Boom! Crash on just kicking off a build! It was my first product shipping experience - I totally panicked on finding this so late. :) Turned out that the code was checking for user having admin rights by comparing against the hard coded string "Administrator". The bug slipped through the JPN testing because in Japanese OS-es, the user groups are not translated and remain "Administrator" while in German, the user groups are translated into "Administratën" :) Well - lessons learnt: grep through the code to detect hardcoded strings to ensure they are appropriate and don’t do glob testing on a single platform to sign off. We now have clear buckets of issues that can be caught in each language - but you never know what else is missing!
-- Anutthara
Do you have a bug whose story you love to tell? Let me know!
During Windows 95, I was a beta tester. It was the first time I'd seen a 'Recycle' bin on a Microsoft platform. (We had some undelete functionality in DOS that required you to know the first letter of the file you wanted to undelete on the FAT file system. Sometimes it worked pretty well, but I digress...) Anyway, just for fun, I drug the recycle bin icon on the desktop and then dropped it into itself. This immediately crashed the machine.
-- Bruce Shankle
Do you have a bug whose story you love to tell? Let me know!
Problem:
Release To Web (RTW) for Office Online site and we were seeing features breaking randomly with no reason at all. A lot of debugging led to no indication of what the problem was.
Cause:
Testing features before RTW, testers did not catch the issue and also our test environments had not really mimicked the production environment. All features were tested as a standalone unit and they all worked perfectly but not as a whole.
Issue:
The problem was the hardware (H/W) load balancer in the production environment that directed all requests to available servers for processing, including cookies/headers that we set per feature. The load balancer had a upper limit for cookie size as 4 KB, and we were exceeding that limit when all the features were used on the site. When this happened, the load balancer was truncating the cookie set first to keep it below 4 KB. This caused some features that were used first to not function anymore due to lack of cookies required.
Result:
We immediately fixed the issue by re-thinking our cookie story and keeping it under the limit.
Lessons learnt:
Always mimic your test environments with production in terms of H/W and design.
- Test your features as a complete set with the entire product and not as a unit.
- This led to discovery of more potential issues over the next couple of releases that opened our eyes to how we write and use cookies.
- Reduce cookie size to reduce the performance impact and improve end–user experience.
- Discard cookies when not needed to keep it down to a reasonable size.
-- Vamshidar Rawal, SDET
Do you have a bug whose story you love to tell? Let me know!
Problem:
We had instrumented our code to spit out timing markers at various points of pages that were tracked via automation and measured actual end-user download times 24/7 on the production site. The results showed that 25% of the time the download times were in the excess of 20 Seconds and it did not matter if it was peak usage time or over the weekend. That did not make any sense and all debugging and investigation showed no cause for the behavior and it was really hurting our end user experience.
How was the cause determined:
We had no cause nor a solution even after a few months. I monitored the live site end user experience for some time and noticed that during a couple of times a month the issue with 20 second download times now dropped to 5 seconds. This triggered another investigation on what happened or if any changes occurred on the live site during these 2 hour windows where the performance issue was better. We came to know that during these 2 hour windows twice a month we were updating the production site with newer content/code. During these times our operations team takes out half the servers out of rotation and updates code/content and when done they swap the servers with rest of them being out of rotation. During these times the long download times were resolved. It stunned us to see that the site worked better on just half the number of servers and that didn’t make sense.
Issue:
- Ultimately more investigation showed us that actual problem was the number of unique OLEDB connection strings that were created and persisted on each of our front end servers.
- We had 42 Front End Servers (FEs) hitting 28 Back End (BE) servers that contained 47 language databases (DBs).
- So the end result was that each FE server had to create and persist (28*47 = 1316) unique DB connection strings.
- The problem is this number of unique connection strings exceeded way over the recommended limit (at around 100).
- During the time for updating the site, the number of FEs hitting BEs was reduced by half (21 FE hitting 14 BE servers that contained 47 DBs.)
- This reduced the unique connections strings by half (14*47 = 658).
- That in turn dropped the long delays of 20 seconds to just around 5 seconds.
Result:
This led us to discuss a better way of arranging and linking our FE and BE servers in production so the number of connection strings was reduced. We ended up with a site that now had 6 servers in the middle tier between the FE and the BE that now solved the issue with the excessive connection strings.
- Now we had 42 FE – 6 Search Web Servers – 28 BE servers (47 DBs).
- This now reduced the number of connection strings on each FE to 42 * 6 = 252.
- The issue with long download times was completely fixed and our download times were sub-second 100% of the time.
Lessons learnt:
- Production site can never be mimicked and issues occurring on the site will almost never be clear.
- Every aspect of the production environment must be thought about comprehensively.
- All parts of the production environment can and will impact performance and end user experience if not well thought about.
-- Vamshidar Rawal, SDET
Do you have a bug whose story you love to tell? Let me know!
Elfriede Dustin sent me a review copy of Implementing Automated Software Testing (IAST), which she and her colleagues Thom Garrett and Bernie Gauf recently published. I am familiar with some of Elfriede's work on test automation, so I was looking forward to an in-depth explanation of how she does test automation.
That is not what this book is.
Part way through IAST I realized that it describes the process the authors use to design and build test automation frameworks, not the details of their implementation. Once I wrapped my head around that, IAST started making sense to me. The authors lay out a sensible approach to designing a test automation framework and managing its implementation, one which bears a distinct resemblance to the heavier software design processes. While you can get that from any process book, the value in this book lies in the test-specific additions, such as detailed instructions for calculating the expected ROI from test automation, and reviews of test case data and logic.
One point the authors make repeatedly is that writing test automation is a software development project and should be treated like one. I am happy they do, for one mistake I repeatedly see product teams make is treating test automation as a second-class citizen that any yahoo can throw together. Software is software, regardless of who writes it.
I found other statements to be happy about as well: the importance of testing test infrastructure was discussed several times, as was the importance of explicitly deciding which tests to automate and which to not. I also found much to disagree with. Throughout the first half of the book, in fact, I found myself saying "No no no no no!" at least as much as I said "Yes yes yes yes yes!" Most egregious to me were the continual statements that manual testing "can hardly do" specific types of testing. Eventually I decided this meant that these types of testing were difficult to do manually, not impossible.
Mostly, though, I was confused what point the authors were meaning to make. Were they attempting to convince me that automated testing was important? Or sell me on a specific way to do automated testing? Or describe to me their process for managing the development of test automation? In the end I decided on the latter, however I am still not quite sure.
If your team wants to start up (or revive) a test automation project and does not know where to start, you might find this book helpful. Browse it in your local bookstore before you purchase it to see whether it fits for you. If you find that it does - or doesn't - let me know why: michael dot j dot hunter at microsoft dot com.
My favorite bugs are the errors that generate themselves recursively. I have two examples:
The first involved an error logging mechanism that logged each exception in a database. At some point our database started throwing exceptions because the disk on the database server was nearly full. These exceptions caused more logging to occur, which lead to ever more exceptions, etc...
Another time I had a problem with Microsoft's own error reporting tool, the one that asks you to submit the bug to Microsoft. Well, it crashed. So this error in turn brought up a new dialog asking me again to submit that error to Microsoft. And then that dialog crashed too. Etc...
-- Jurgen Appelo
Do you have a bug whose story you love to tell? Let me know!
I was happy after I upgraded my laptop's memory from 1GB to 2GB until I noticed that sometimes XP just won't shut down when I try to hibernate. It sits there at "Save personal setting" forever. One day I put it to hibernate and before I saw it shutdown, I put it in my notebook bag and went home. After half an hour, I took it out, and it was so hot that it hurt my finger. When I turned it on, the battery was not empty. So I guess eventually the machine turned off power when the temperature was too high. The good thing is, this feature still functions. Otherwise, it would be on fire. BTW I did search online and applied some packs to try to fix that bug. But it didn't work.
-- Xu Wang
Do you have a bug whose story you love to tell? Let me know!
I opened a bug 1.5 years ago about an issue where the product would stop displaying updates to the user in a rare scenario dealing with boundaries; we have gotten customer reports about a similar issue that could be the same root cause and are linked to the bug. Unfortunately we were very close to shipping at the time and the fixes that were brought to the table were pretty destabilizing and required a lot of rework so the bug got punted. Moving forward to the next release and the bug is still sitting, waiting... and then it gets mark as a Won't Fix because there is not enough customer feedback. I kindly disagree with the PM saying that there is customer feedback attached to said bug and it gets reactivated. About a month later it gets resolved as Won't Fix pending customer feedback again; again I reactivate it and point out the customer feedback. Currently the bug is still in deliberation and has yet to be fixed, although it looks like it might be punted yet again. Not my favorite bug, but possibly my most frustrating.
-- Scott Banning
Do you have a bug whose story you love to tell? Let me know!
