Current and Future Information Landscape – Challenges and Solutions
The "Google" Perception – Everything is "there"
We've come a long way in 20 years, and we hope to be around for another 20 years and longer, which requires us to look at the challenges that lie ahead for programs such as ETDE. Certainly one of those challenges is what I call the perception that all meaningful information can be found through the major commercial search engines, such as Google, Yahoo!, and MSN.
Reality of surface web versus deep web
The fact, however, is that the deep web, where significant databases such as ETDEWEB are found, is perhaps hundreds of times larger than the surface web, and it is widely accepted that the deep web has much more reliable and authoritative scientific information. Because of the dynamic content within these deep web databases, their content isn't accessible to most search engines' crawling.
"Members Only" Access
Besides the deep web issue, almost anyone in the world can use Google or a similar search engine without any access restrictions or registration requirements. Many deep web databases have access restrictions of some kind, and users opt for the path of least resistance, which further inhibits wide usage. ETDE relies on its member countries for information inputs, and their reward for providing that input is that they gain access to other member countries' information. For this model to work, we have to retain this benefit, but a registration requirement inherently limits a system's usage. This is yet another challenge that databases such as ETDEWEB must overcome.
Economics of producing bibliographic records
Another challenge is the simple economics of producing bibliographic records for a system such as ETDEWEB. It is a costly proposition for each member country to create bibliographic records for all the energy R&D literature within its borders, and this could certainly be short-circuited if all of ETDEWEB's contents had full text, where the full text could simply be automatically indexed, but that is not the case since much of ETDEWEB's contents are about copyrighted full text which doesn't reside within our database.
Proliferation of distributed portals – maintaining an identity
Another challenge is the proliferation of distributed portals where many sources are searched simultaneously. In such a federated approach, the user may not know or have an appreciation for which sources are being searched, and, thus, sources face the risk of losing their identities and branding, which are important, again, to ensure that both users and funders of these systems retain the sense that a certain system is important and should be sustained.
Multiplicity of languages – The rise of non-English literature
And, finally, nations are building on their research output continually, and much of this is published in each country's native language. For a system, such as ETDEWEB, which has at least the metadata in English, this is a challenge to adequately cover this literature, and with several non-English speaking countries, such as China and Russia, not belonging to ETDE, we are missing some of the world's meaningful research output.