An earlier article derived the Knowledge Investment Curve.

Information sharing is an integral part of the R&D process. Thus, decision makers affect the pace of scientific progress when they determine the fraction of R&D dollars dedicated to sharing knowledge. Think of it this way: A program for sharing knowledge derived from scientific research has much in common with a scientific research program itself in that they share the common goal of advancing science. 

 By Walter L Warnick and Peter M. Lincoln

The Office of Scientific and Technical Information (OSTI) maintains several collections of scientific and technical information (STI) that can be employed to help achieve the President’s national objectives for the U.S. Department of Energy.

 OSTI’s databases are important resources for scientists and engineers working to strengthen America’s role as the world leader in science and technology, reduce greenhouse gas emissions, promote energy security and enhance nuclear security.

by David Wojick and Peter Lincoln

As OSTI Director Walt Warnick likes to say, today's Web is like the Model T Ford -- revolutionary but ready for vast improvement. This is especially true when it comes to making the Web work for science and technology. In that spirit I want to describe a new kind of Web Portal, one which has yet to be built. It is called the X-Portal.

Science.gov is a one-stop portal for federal government science information.  Over 200 million pages of science information from 14 federal agencies may be searched through a single query.  How far we have come in the past decade!

You may not be aware that Science.gov was developed and is governed by the Science.gov Alliance, a group of science information managers who began working together to overcome the stovepipes of agency information in 2001.

by David Kaiser (MIT) and Luis Bettencourt (LANL)

For some time, we and OSTI have been interested in the question of how new scientific ideas spread. What does it take for the “next big thing” to leap from one person’s head to an active community of researchers? Do those shared ideas or techniques bind the community together more tightly than before, perhaps even helping to define a new research field that didn’t exist before? And if so, how might we detect and measure such shifts in the space of researchers and ideas

Unique and interesting insights into U.S. Department of Energy (DOE) Research and Development (R&D) accomplishments are available in a special collection that features research of DOE and its predecessor agencies, the Energy Research and Development Administration (ERDA) and the Atomic Energy Commission (AEC).

This special collection contains historically significant government documents that have been specially selected and digitized to make them accessible via the Web. Landmark documents such as The Eightfold Way: A Theory of Strong Interaction Symmetry and The First Weighing of Plutonium are among approximately 300 specially-selected documents included in the database. Additionally, documents are aggregated with related aspects of the collection into more than sixty (60) Feature Topic pages with diverse topics such as Video Games -- Did They Begin at Brookhaven? and Human Genome Research: Decoding DNA.

By Walt Warnick and David Wojick

Every scientist knows that science advances only if knowledge is shared. Mathematically, this statement implies that the advance of science is a function of both the sharing of research results, as well as doing the original research. In principle, therefore, decision makers face the problem of deciding how much to spend on original research and how much to spend on sharing the knowledge that comes out of research.

WorldWideScience provides a one-stop search engine to mine global scientific databases in the deep web

The internet has revolutionized society by changing the way people communicate, find information, and enjoy entertainment. But a standard internet search misses at least 90 percent of the information available. 

The internet is separated into two unequal pools of information. The surface web contains pages of information that are utilized by popular search engines. The second pool of information is locked away in the deep web, which consists of countless databases world wide.

Did you know that science information is available via web "mashups"? Web "mashups" combine multiple products/services into a single application for the purpose of consolidating information with an easy-to-use interface.

The Department of Energy (DOE) Office of Scientific and Technical Information (OSTI) uses “mashups” to return search results from Science Accelerator, Science.gov, and WorldWideScience.org. These "mashups" include external sources of information, in these cases from Wikipedia and EurekAlert!, that are provided as a service to the user for help with additional background information or with the ability to further study their topic.

Did you know:

• That now you can find research from China when you search within WorldWideScience.org?  OSTI was in Ottawa June 10 helping formalize the addition of China to the WorldWideScience Alliance? In addition, now you can quickly narrow your WorldWideScience.org results list to the research you need, share them on social networking sites, bookmark your search, and set up alerts.
• That now you can learn about OSTI tools and services on our OSTI YouTube site launched in June?

Dr. Warnick, you travel quite a bit and make numerous presentations about OSTI's innovative work. What drew you to speak at the Computers In Libraries (CIL) Conference?

Energy Secretary Steven Chu aims to better integrate basic and applied sciences.[1] OSTI's mission [2], products, and services support this goal. Secretary Chu sees the need to build networks to connect research within national labs, universities, and industry. A number of OSTI's research offerings directly facilitate open communication. Secretary Chu seeks to expand international science collaboration. WorldWideScience.org, conceived at OSTI, promotes such collaboration. Secretary Chu wants to link research with solutions to our nation's pressing problems. OSTI, through the DOE SBIR program, encourages small businesses to develop and commercialize technologies that advance the acceleration of science. Secretary Chu values development of engineering talent. OSTI is developing technology that will improve science education.

[Note: This article first appeared in the Federated Search Blog. ]

This is the story of how one organization of the Federal government came to recognize the potential of federated search and then set out to deploy it and encourage its maturation.

Along the way, considerable progress has been made. More science is freely findable on the web today than has ever before been available to the public. Yet, much more progress remains to be made.

On March 2nd I wrote an article for the Federated Search Blog: On credibility  of search results.

The article asserts that a federated search engine is only as good as the quality of the content to which it provides access. While the major consumer-oriented search engines may provide more search results overall, it is left to the user of the search application to sift through the search results to identify which content represents credible scientific and technical information.

As Isaac Newton said, "If I have seen further than others, it is by standing on the shoulders of giants." Newton was not alone on those shoulders. Everyone in science, from his day to ours, draws on the work of others.

Science is all about the flow of knowledge:  New methods, instruments, techniques, concepts, results, questions, data, etc.  The flows are endless, complex and in all directions.  It is rightly called a diffusion process.  This concept is reflected in a host of statutes that form the legislative basis for OSTI.

OSTI is dedicated to the principle that to advance science, research must be shared. OSTI  is increasingly recognized for its contributions to the sharing and exchange of scientific and technical information.  OSTI works to accelerate discovery by speeding access to knowledge.

OSTI’s users conducted 80 million information transactions last year.  Spreading responsibility for these transactions across the OSTI staff implies that each OSTI employee was responsible for 12 information transactions per minute throughout the work year, which I view as an incredible metric of success.   And we have only just begun.  OSTI’s traffic increases significantly each year. 

It is exciting to announce that an array of new search and retrieval features and capabilities have been added to ScienceAccelerator.gov, providing new options for customizing your search experience.

Photo Credit:
Lawrence Berkeley National Laboratory
Roy Kaltschmidt, Photographer             

In conjunction with the recent selection of Dr. Steven Chu as the next United States Secretary of Energy by President-elect Barack Obama, a quality high-level compilation of information about Dr. Chu and his research has been made available on the web at http://www.osti.gov/accomplishments/chu.html.  This web page includes scientific documents that he authored, including his Ph.D. thesis supported by the Energy Research and Development Administration (ERDA); interviews, speeches, and presentations; and links to related web sites.

The Office of Scientific and Technical Information (OSTI) for the U.S. Department of Energy (DOE) has achieved another milestone in our mission of accelerating the diffusion of knowledge to advance science.  Albert Einstein once said that “[t]he only source of knowledge is experience.”  What better way for OSTI to advance science and accelerate the diffusion of knowledge than by joining the Web 2.0 world of social networking.  So, come increase your own knowledge and experience the OSTI Page on Facebook.

Innovation needs information -- the medical innovation model

The goal of energy transformation can take a lesson from America's rapid deployment of innovative medical technologies. It may take a long time to get a new drug or device approved, but once this happens the deployment is very rapid. America's spectacular success in fielding new medical technologies is anchored in the innovative Web resources of the National Library of Medicine. For a modest $350 million a year NLM supplies vast amounts of innovation information to America's scientists, doctors and consumers.

OSTI is driven! We are fully committed to providing scientists and researchers with the social networking tools and services that can make it easier for them to more rapidly advance their scientific research. We have a number of exciting ongoing initiatives in support of accelerating the evolution of science. Here are ten that come to mind:

by Walt Warnick and Sol Lederman

OSTI has embraced a new paradigm for sharing scientific and technical information (STI). Historically, OSTI has fulfilled its mission of providing STI to scientists, researchers, and the public by hosting, or collecting, documents and/or metadata. OSTI's new paradigm is to make content searchable that is often hosted by others; today, OSTI connects those seeking the content with the organizations that host it.

Two hundred years ago the gulf between the rich and the common people was huge, illustrated by these photos.  On the left is Doughoregan Manor at Ellicott City, Maryland, largely built in the 1700’s, home of Charles Carroll of Carrolton, a signer of the Declaration of Independence.  On the right is the cabin where Abraham Lincoln was born in 1809 located at Hodgenville, Kentucky.  Today, the public view of these times is influenced by the fact that many of the fabulous mansions yet survive, while the cabins that housed common people have long since been replaced.

Sources: Wikipedia articles "Abraham Lincoln" and "Doughoregan Manor".

by Walt Warnick and David Wojick

The immediate goal of science is understanding, rather than social utility.  In the rush of day-to-day activity it is easy to overlook how science allows us to live better. The path from science to better lives is complex and often takes decades.  In broad terms, basic research first hands off its results to applied research.  Applied research then hands off to technology research and development, which then flows into entrepreneurship and finally manufacturing and distribution.  Then, and only then, are great benefits realized from science knowledge.

On October 14, OSTI announced that the People's Republic of China had joined the WorldWideScience Alliance. The press release making the announcement described, and hinted at, the importance of China's contribution:

China, a major producer of journals and conference proceedings, is offering searches of key Chinese English-language scientific literature through WorldWideScience.org. The Chinese resource enables searching of over 6,000 journals.

by Walt Warnick and Sol Lederman

While federated search is a core technology that OSTI employs to tackle challenges of sharing knowledge, the technology isn't perfect.  OSTI aggressively uses federated search because it does what no other search technology can do—inexpensively making dozens of non-Googleable databases searchable via a single query.  Two nagging limitations of federated search are that it can take 30 seconds to execute a search—which seems slow in the digital age—and hit lists are not exhaustive. 

Our team is excited to announce that you can now join in or start discussions on important DOE research and development in the Information Bridge collection.

Our new social networking feature, Document Discussion, has been added to the Information Bridge to provide a forum for moderated, substantive commentary on DOE research and development.

Because we live in a digital world, many people mistakenly believe all research is easily available online. Not only is this a false assumption, it’s not even an easy task to digitize the volume of research currently available in paper format and get it posted online.

That’s why OSTI is pleased to announce that we’ve recently posted 15,000 DOE research reports heretofore only available in paper or microfiche.

by Walt Warnick and David Wojick

Compared to the pre-Web world of the early 1990s, OSTI now enables about a thousand-fold more information transactions. An information transaction occurs when the customer receives information he requested, such as delivering the results of a search or following a link clicked to display a document. But the mind-boggling growth in the number of transactions is only part of the story.

Science.Gov 5.0 is now available!

The first thing you’ll notice is the new main page design. The same elements are there, but reconfigured to update the website look and feel. We have also added seven deep web sources (see DOE press release) into the search.

by Walt Warnick and David Wojick

The life of every person in the world today has been shaped by successive technological transformations. The printing press transformed communication and education, beginning in the mid 15th century. Sailing and navigation technology of the late 15th century allowed Europeans to learn about other continents, beginning the global network of trade. Metal tools and firearms technology of the early 17th century enabled Europeans to colonize other continents and spread the fruits of European technology around the world. Railroads transformed transportation beginning in the early 19th century, and the telephone transformed communication in the latter part of that century. The automobile transformed transportation beginning early in the 20th century. These are but a few of the notable transformations that profoundly reshaped the way people live.

Just as science progresses only if knowledge is shared, accelerating the sharing of knowledge accelerates science. All of us engaged in disseminating science knowledge have the opportunity and obligation to do our jobs better, for to do so accelerates science itself. 

To this end, I propose a grand challenge—to make more science available to, and searchable by, more people than ever before. A momentous milestone will be achieved once we enable everyone with web access the ability to search with unparalleled precision a billion pages of authoritative science. Already, considerable progress has been made. 

Scientific history was made today in Seoul, Korea, as 13 founding members of the WorldWideScience Alliance committed their talents and resources to promoting the global sharing of science.

Our OSTI team recently completed digitizing and uploading to the Information Bridge database about 10,000 documents issued by the U.S. Department of Energy (from 1991 to 1994). This means that citizens can now search this database and rapidly download these documents in full text – all for free.

Today, all of OSTI's information products are on the web. This is in sharp contrast to the situation as recently as the mid-1990s, when OSTI had no products on the web.

First becoming popular in 1994, the web quickly emerged as a transformational technology, and its potential for reshaping OSTI was apparent. Recognizing the opportunity to advance the OSTI mission, OSTI set out to capitalize on it as quickly as resources would allow by producing web applications to disseminate all manner of scientific and technical information (STI). A steady progression of new OSTI products addressed the various forms of STI: technical reports, e-prints, conference proceedings, accomplishments, patents, and project descriptions . To make it easy for users who want to search through all these products at once, we introduced the DOE Science Accelerator, which is powered by our special web architecture called federated search. Reaching out beyond DOE, we initiated a collaboration with other agencies to allow users to search their R&D results along with DOE's; thus emerged Science.gov. Most recently, we took collaboration world wide by federating the best information sources from governments around the world, WorldWideScience which makes searchable about the same quantity of science as does Google.

by Kristin Bingham and Sol Lederman

On June 22, 2007, OSTI opened WorldWideScience.org, a global science gateway, to the public. WorldWideScience.org was an ambitious undertaking and OSTI was the perfect organization to take on the technical, administrative, and organizational challenges to take a powerful idea and bring it to fruition.

When it comes to science and technology development, OSTI people are writing one of the biggest Internet success stories. Everyone talks about how the Internet is changing science but OSTI is making it happen, and doing it on a shoestring budget.

This is the third, and final, article in a series. The first article provided an overview of the E-print Network. The second article discussed the special harvested component of the E-print Network in depth. This article provides a tour of the E-print collections which are federated. Hopefully, once you finish reading this article and this series, you will appreciate the innovation and hard work that has gone into producing the premier federated search application for searching E-prints.

by Walt Warnick and Sol Lederman

This is the final article in a series about the limitations of the crawl and index approach to searching scientific content and the advantages of federated search. Part 1 identified a number of issues with Google and the other crawlers, and showed why researchers and the science attentive citizenry don't rely on "Googleable" content to meet their needs for quality scientific and technical information. Part 2 explained how federated search, by providing access to "non-Googleable" content, overcomes the Google limitations. This article highlights three important applications, developed and maintained by OSTI, that demonstrate how federated search is going beyond crawling to advance science.

by Walt Warnick and Sol Lederman

This is the second in a three part series of articles about the deficiencies of web crawling and indexing, the superiority of federated search to the serious researcher, and the value of OSTI federated search applications in advancing science. Part 1 identified a number of serious limitations of Google and the other crawlers. This article shows how federated search overcomes these limitations. The final article in the series highlights a number of federated search applications and databases that OSTI makes available to the public.

by Walt Warnick and Sol Lederman

The web is growing.

For providing searchable access to the content that matters the most to scientists and researchers, Google and the other web crawlers can't keep up. Instead, growing numbers of scientists, researchers, and science attentive citizens turn to OSTI's federated search applications for high quality research material that Google can't find. And, given fundamental limitations on how web crawlers find content, those conducting research will derive even more benefit from OSTI's innovation and investment in federated search in the coming years.

In Part 1 of this series I provided an overview of the technology that drives the E-print Network. In this article I will provide some detail about how the harvested collection, the "E-prints on Web Sites" component of the E-print Network, is constructed. In Part 3, I will discuss the technology of the portion of the E-print Network that relies on federated search of databases.

 Part of OSTI's R&D aims at understanding how scientists use information. This goal was originally articulated by OSTI's Thurman Whitson, who has since retired. To that end we have begun to look at the different kinds of information provided by the different Web-based science resources. Different kinds of information imply different uses. It is not that one resource is better than another overall, it is that they are very different and support different uses.

The E-print Network is one of OSTI's most popular and powerful research offerings yet few of its users know about the advanced technology that drives it and makes it simple to use. Professional researchers in basic and applied science are able to access over 5 million e-prints gathered from nearly 28,000 world-wide databases and web-sites. Numerous OSTI innovations ensure that the E-print Network's documents are of extremely high quality, are highly relevant to researchers, and are easy and quick to find. This is the first in a series of articles about the technology behind this very important component of the Science Accelerator. This article serves as an overview; subsequent articles will provide more technical information.

Federated search is very much at the heart of OSTI's ability to realize its mission. OSTI provides a simple description of what federated search is and how it works in the OSTI environment. The best way to experience the tremendous value of federated search at OSTI is to try several of OSTI's flagship applications:

• Science Accelerator


• Science.gov


• WorldWideScience.org
  
  

OSTI provides searchable access to vast collections of DOE research results, project descriptions, and accomplishments. Making R&D findings available and usable is central to OSTI's mission:

The mission of the Office of Scientific and Technical Information (OSTI) is to advance science and sustain technological creativity by making R&D findings available and useful to Department of Energy (DOE) researchers and the American people.

Allan Cormack

A young father of two lies in a hospital bed seriously ill. The physician admits he has no treatment. The pancreas is secreting substances that are digesting itself and destroying surrounding tissue. Some patients recover on their own; others simply expire. Only time will tell which fate awaits the young father.

As a staff member involved in OSTI’s Web presence, it was personally satisfying today to hear Google’s J.L. Needham mention OSTI in testimony to the Senate Homeland Security and Governmental Affairs Committee.

Yesterday my son had an emergency appendectomy – these days a pretty routine procedure. But far from routine was the array of drugs offered to get him through the long night ahead.

In 2005, the idea of creating a global science gateway for the web was conceived at OSTI.  It would make the best collections of scientific information from nations around the world act as if they were a single enormous collection.  It would be searchable via a single query, and it would be available at no cost to anyone anywhere with web access.