Download Research Tools
Research published in academic journals is trustworthy. Or is it? This question is being asked more and more these days. While few doubt the integrity of the researchers, many in the scientific community are concerned about the inability to reproduce experiments. If the work is not reproducible, how can its reliability be judged?
Making research reproducible is far from trivial. It involves sharing not just the results, methods, and data, but also the implied knowledge of the original researcher, without which it’s difficult to independently reproduce the complete results. The sharing of results and methods has become established practice over the last 350 years through academic publication. Data sharing is coming of age, and huge efforts by the research and wider community on open data are starting to bear fruit. But that still leaves a big problem: how to share the original researcher’s implied knowledge. Fortunately, the possibility of attaining completely reproducible computational experiments is drawing closer, as we’ve seen this week (December 8–11, 2014) during the Recomputability 2014 workshop at the 7th IEEE/ACM International Conference on Utility and Cloud Computing in London.
During the workshop, Tom Crick of Cardiff Metropolitan University, Benjamin Hall of the University of Cambridge, Samin Ishtiaq of Microsoft Research, and I presented new ways of thinking about reproducibility. We’ve been considering what it takes to make reproducible research not only possible but attractive to researchers. Much of the current work around data sharing and reproducibility focuses on the person trying to reproduce the work. As such, it often fails to take into account the intense pressures under which the original researcher labors: the stress of having to produce and publish results. We therefore propose building systems that make researchers more productive during their day-to-day work: automated systems that help make their work reproducible. In computational domains, researchers have many cloud-based tools that make our lives easier, such as Github, Visual Studio Online, figshare, Office 365, and OneDrive. We use these tools every day, but usually in a disconnected way. We envisage a new world of “reproducibility as a service,” wherein these disparate services are brought together to make it much easier for researchers to think, develop, test, and publish their computational work wherever they are. Read more about our thoughts in our workshop paper, “Share and Enjoy: Publishing Useful and Usable Scientific Models.”
Tom Crick from Cardiff Metropolitan University and Lars Kotthoff from University College Cork kick off deep discussions at Recomputability 2014.
"This is an exciting area of research and one that could have a profound impact on the way that computational science is performed. By rethinking how we develop, use, benchmark, and share algorithms, software, and models, alongside the development of integrated and automated e-infrastructure to support recomputability and reproducibility, we will be able to improve the efficiency of scientific exploration as well as promoting open and verifiable scientific research," says Crick.
We are also excited to be working with Ian Gent and his team at the University of St Andrews on Recomputation.org, whose work on using virtual machines in the cloud to freeze, and later unfreeze, computational experiments is very promising. It’s like creating an exact copy of Michael Faraday’s Lab and then being able to reproduce and extend his experiments. During the summer, we had a fantastic time exploring what’s possible; read more about our experiences trying to reproduce several experiments. These efforts provide a great starting point, although even with access to detailed lab notebooks, it is very difficult to know every detail of the original experimenter’s experience and implied knowledge. So this approach is not the whole solution, but certainly a move in the right direction.
All of these steps toward reproducibility are becoming more realizable due to rapid developments in the cloud, and we were excited to share our own experiences with the research community during the conference. Ant Rowstron of Microsoft Research described the need to think at rack scale—that is to say, to think in terms of racks, rather than individual servers, as the building blocks in our data centers. Ross Smith from Skype and Lori Ada Kilty of Microsoft Research showed how crowdsourcing and gamification can improve online conferencing systems, such as Lync and Skype. I was delighted to discuss where we are heading with hyper-scale cloud computing, from efficiency, consistency, and productivity points of view. And it was great to see that Claudio Rosso from the Microsoft Innovation Centre in the Istituto Superiore Mario Boella (ISMB) received the best poster award for the FLOODIS flood information project, using Microsoft Azure: an example of how the cloud can augment mobile devices for helping citizens in times of crisis. To round it off, we had fun with dozens of researchers as they learned about Microsoft Azure and Azure Machine Learning and explored how these platforms can help with their work.
All and all, it was an amazing week in London. We were thrilled to share how cloud computing is making a difference for researchers today—and the promise it holds for the future. If you’re interested in how cloud computing can help with your research, we invite you to learn more through our Azure for Research program, which offers a host of tools and tips, not to mention substantial grants of cloud resources for qualified projects.
—Kenji Takeda, Solutions Architect and Technical Manager, Microsoft Research
The power of computing can go a long way toward solving some of the world’s most vexing problems. This is why we are committed to fostering the next generation of technology-savvy researchers, through programs such as the Microsoft Research Asia Fellowship Program. Launched in China in 1999 and expanded to the entire Asia-Pacific region in 2002, this program recognizes exceptional Ph.D. students in computer science, electrical engineering, information science, and applied mathematics, empowering them to realize their full potential as researchers.
The fellowship rewards doctoral students who have demonstrated exceptional research aptitude through concrete achievements at this early stage of their career—students like Yongtae Park of Korea University, whose research focuses on enabling mobile devices to communicate with the emerging Internet of Things, and Seunghoon Hong of Pohang University of Science and Technology, whose work aims to achieve semantic understanding of video content.
The Microsoft Research Asia 2014 fellows and advisors were honored during the Computing in the 21st Century conference in Beijing. They are shown here with Jeannette Wing, corporate vice president of Microsoft Research (tenth from the left); Hsiao-Wuen Hon, managing director of Microsoft Research Asia and Chairman of the Microsoft Asia-Pacific R&D Group (tenth from the right); and Tim Pan, director of Microsoft Research Asia University Relations (far left).
This year, 83 outstanding doctoral candidates from 45 leading research universities were nominated for the fellowship. The nominees included students from mainland China, Hong Kong, Taiwan, Japan, South Korea, and Singapore. Their credentials were thoroughly evaluated by a review committee, and 27 finalists were invited to Beijing for an on-site interview. The top 12 finalists received the full Microsoft Research Asia Fellowship. Each of these new fellows will have the opportunity to complete an internship at Microsoft Research Asia in Beijing, during which they will actively participate in cutting-edge research.
The 2014 fellows and their mentors were formally introduced during the sixteenth Computing in the 21st Century conference, which took place on October 29, 2014, at Peking University. Jeannette Wing, corporate vice president of Microsoft Research, and Hsiao-Wen Hon, managing director of Microsoft Research Asia and chair of the Microsoft Asia-Pacific R&D Group, presented the fellowship awards.
The new fellows enjoyed a roundtable discussion with 1992 Turing Award winner Butler Lampson (sixth from the left) and Tim Pan, director of Microsoft Research Asia Connections (fifth from the left).
In addition, the new fellows engaged in a roundtable discussion with Turing Award winner Butler Lampson and Tim Pan, director of Microsoft Research Asia Connections, which gave the students an opportunity to discuss their own research and, more broadly, to explore the means for attaining the highest quality research results. We’re eager to see what these talented young researchers achieve.
—Rui Li, University Relations Manager, Microsoft Research Asia
Sign language has long been a tool used by many of the world’s estimated 360 million people with severe hearing loss. But since the majority of hearing individuals do not understand sign language, the hearing world does not always have the capability to engage in real-time, unwritten communication with people with hearing loss. Now, technology is poised to make such interactions more feasible.
Hoping to facilitate communication between the signing and non-signing communities, Microsoft Research in February 2012 initiated the Kinect Sign Language project in collaboration with the Chinese Academy of Sciences (CAS) and Beijing Union University. The Kinect Sign Language Translator enables real-time conversations between signing and non-signing participants by turning sign language into words spoken by a computer and simultaneously changing spoken words into sign language rendered by an avatar.
Some of the attendees at the Kinect Sign Language Working Group's inaugural event
Early last month, the Kinect Sign Language Working Group, a research community that includes a website for sharing data and algorithms, was established at the Institute of Computing Technology, CAS in Beijing. P. Anandan, managing director of Microsoft Research Outreach, attended this inaugural event, as did other dignitaries representing the community’s founding members: the CAS, Beijing Union University, and Microsoft Research. We are encouraging experts from other research institutions, schools for the deaf and hard of hearing, and non-government organizations to join the Kinect Sign Language Working Group.
The community’s vision is to advance research in sign-language recognition.
The community’s vision is to advance research in sign-language recognition. As a first step, we are opening to academia the DEVISIGN, Chinese Sign Language Database. Compiled by the Visual Information Processing and Learning (VIPL) group of the Institute of Computing Technology, under the sponsorship of Microsoft Research Asia, the DEVISIGN covers about 4,400 standard Chinese Sign Language words based on 331,050 pieces of vocabulary data from 30 signers (13 males and 17 females). The vocabulary data comprises RGB video (in AVI format), and depth and skeleton information (in BIN format). The DEVISIGN thus provides sign-language researchers with a rich store of data for training and evaluating their algorithms and for creating state-of-the-art practical applications, such as solutions for training the system to adapt to an unknown signer.
In the near future, we hope to expand the sign-language database with contributions from new community members, which will help advance the research and development progress for this and potentially other sign language translations. In addition, we intend to organize workshops and to post sign-language-recognition algorithms from researchers worldwide.
Microsoft Research Asia Director Tim Pan expects the Kinect Sign Language project to provide cost-effective and reliable communication between deaf and hearing users.
No single field of expertise can fulfill such an expansive mission. Doing so requires “the collaboration of experts in such diverse fields as machine learning, sign language, social science, and more,” noted Microsoft Research Asia Director Tim Pan during the community’s inaugural event. “In the long run,” he added, “the community will work together to turn ideas into reality, and we fully expect the Kinect Sign Language project to provide cost-effective, easy, and reliable communication between deaf and hearing users.”
—Guobin Wu, Research Program Manager, Microsoft Research Asia