Visualization of information: a proposal to improve the search and access to digital resources in repositories
Visualización de información: una propuesta para mejorar la búsqueda y los accesos a los recursos digitales en repositorios
P. A. Gaona García1, S. Sánchez-Alonso2, C. E. Montenegro Marín3
1Paulo Alonso Gaona García. Ph.D. (c) Information and Knowledge Engineering at University of Alcalá, Spain. M.Sc. in Information Science and communications at Distrital University, Colombia. Affiliation: Professor of Engineering Faculty of Distrital University, Colombia.
2Salvador Sánchez Alonso. Ph.D. in Computer Science at the Polytechnic University of Madrid in 2005, Spain. Affiliation: Assistant Professor, Computer Science Department, University of Alcalá, Spain.
3Carlos Montenegro Marin. Ph.D., Computer Science, Oviedo University, Oviedo, Spain. Affiliation: Professor, Engineering Faculty of Distrital University, Colombia.
How to cite: Gaona, P., Sánchez-Alonso, S., Montenegro, C., Visualization of information: a proposal to improve the search and access to digital resources in repositories., Ingeniería e Investigación, Vol. 34, No. 1, April, 2014, pp. 83 – 89.
Presently, the most notable challenges associated with repositories in resolving problems of searches of digital resources, lie in providing an understanding of resource classification according to a knowledge representation scheme and the relationship between them. However, one of the areas with very little research in the field is the study of visual search interfaces that provide access to relevant materials in digital repositories; more specifically, in the access of digital resources according to areas and sub-areas of a particular knowledge domain through a taxonomy classification. In this article, we focus in proposing a best practice for the search and access to relevant digital resources in repositories through visualization techniques. The article presents a prototype tool as one possible approach to facilitate searches and access to digital resources. Finally, we present the conclusions and future work in the field in order to improve access to relevant materials in digital repositories.
Keywords: Visualization techniques, data visualization, user interfaces, repositories, digital resources.
En la actualidad, los desafíos más notables asociados a los repositorios para resolver los problemas en la búsqueda de los recursos digitales, se enfocan en ofrecer un mayor entendimiento de la clasificación de los recursos digitales, de acuerdo a un esquema de representación de conocimiento y relación entre ellos. Sin embargo, una de las áreas con pocos estudios en este campo, es el estudio de interfaces visuales que proporcionen acceso a los materiales relevantes en repositorios digitales. Es necesario prestar mayor atención al acceso de los recursos digitales, teniendo en cuenta un conjunto de áreas y sub áreas de dominio del conocimiento específico, a través de una clasificación taxonómica. En este artículo, nos enfocamos en plantear una propuesta de mejores prácticas para la búsqueda y acceso a recursos relevantes en repositorios digitales, a partir de técnicas de visualización. El artículo presenta un prototipo de herramienta como un posible enfoque para facilitar el acceso a los recursos digitales. Finalmente, se presentan las conclusiones y el trabajo futuro en el campo, con el fin de mejorar el acceso a materiales relevantes en repositorios digitales.
Palabras clave: Técnicas de visualización, visualización de datos, interfaces de usuarios, repositorios y recursos digitales.
Received: August 7th 2013 Accepted: February 3th 2014
One of the greatest challenges that face the specialized data repositories is how best to facilitate access to the digital resources. There are studies (Marchionini, 2006; White & Roth, 2009) that focused specifically on strategies for searches and browsing capabilities to locate digital resources on this type of interfaces, demonstrated that conventional search and exploration strategies are not sufficiently robust and flexible to facilitate access and location of a collection of digital resources. In terms of efficiency, these results exhibited a loss of all semantic capacity available through some specialized repositories by defining metadata, although these requirements are the key to enrich the processes of finding digital resources in a specific knowledge area (Cechinel, Sánchez-Alonso, & Sicilia, 2009; Fernández, 2001; Pastor, 2009).
This visualization tool is part of a proposal. As a result of our research (Gaona, Martin-Moncunill, Fermoso, & Sanchez-Alonso, 2014; Martin-Moncunill, Sanchez-Alonso, Gaona, & Marianos, 2013) into how through visualization techniques, we can help creators of digital repositories provide better services for users to: i) locate materials of a precise fashion and effective form through an exten-sive collection of digital resources, ii) locate materials ac-cording to a thematic structure or knowledge area, iii) identify thematic cover-age of a term according to the number of digital resources, and finally, iv) identify effective interfaces to performing browsing and searching processes over digital resources.
The motivation of this article is to analyze if through visualization information, we can help users improve access to a collection of digital resources hosted by repositories. To carry out this study, we developed a tool based on four visualization techniques (tree, radial, category, hyperbolic) (Herman, Melancon, & Marshall, 2000). In order to obtain real results, we took a sample collection of (42.800) digital resources extracted from Europeana digital li-brary, such as concepts and topics of art, culture, and heritage defined by Art and Architecture Thesaurus (AAT), as a knowledge representation scheme. Finally, we present the results of a usabil-ity study in order to evaluate usefulness and efficiency of visual search interfaces based on a hierarchical navigation structure. The total of participants were (16) with basic knowledge of web searching.
The main purpose of the study focused on evaluating the use of information visualization related to i) the relevance of hierarchical classification of all visualization techniques applied in order to search and access a collection of digital resource (visual percep-tion), ii) the effect of visual interfaces to explore digital resources (effectiveness), and finally iii) the effect of aesthetic design of visual interfaces to carry out the access of digital resources (usefulness).
The second section of this paper introduces a theoretical back-ground, previous efforts in the field, and related investigations. Section 3 explains the methodology used to carry out the devel-opment of visual tool interfaces, followed by usability test. Section 4 presents the results of usability study through user's interaction. Finally, we present an analysis of evaluation with the final conclusions of the study.
In recent years, digital repositories have had a representative impact on their technological development due to exponential in-crease in the number of digital resources published (Margaryan & Littlejohn, 2008). This growth of digital resources has led to the development of several strategies in different areas: i) (in terms of technology) the development of distributed repositories, heterogeneous repositories and federations of repositories as central access points to each of them (McGreal, 2008), ii) (in terms of semantics) the use of linking knowledge classification schemes by using ontologies and thesauri to provide a better understanding and organization of digital resources, and finally iii) (in terms of access) the design of strategies to offer metadata description. This latter strategy turns out to be an essential condition in order to search relevant results within a digital repository (Cechinel, et al., 2009; Park, 2009).
Empirical studies have been conducted to show that deficiencies in the design of interfaces can interrupt user activity throughout information search processes (Tsakonas & Papatheodorou, 2008). One of the first proposals to improve access to learning objects using visual strategies was conducted by Card (Card, Mackinlay, & Shneiderman, 1999). In this study, users were assisted by various zooming techniques, and so they were able to approach the most relevant learning objects restricted by additional search criteria applied to the number of remaining objects. Kim (Kim, 2006) conducted another study to measure the level of user satisfaction through the use of two digital libraries and an eprint repository. The findings revealed serious design-related interfaces problems, particularly when defining input formats in queries and search results for display in the browser.
There are a few studies concerned with the interaction of users through information visualization in open access digital libraries. Tsakonas (2008) conducted a study to determine if the content and system features signiﬁcantly aﬀect the usefulness and usability levels of an Open Access (OA) repository. As a result, the study found that attributes related to the level of relevance of information and learning facilities directly affect the ability of interaction and user satisfaction.
MACE project (Stefaner et al., 2007) is a European project related with the search of digital resources associated with architecture. The results of the field search suggest that principles of navigation with multiple facets facilitates immersion processes of search through collaborative tagging (Stefaner et al., 2009).
Information visualization is a visual representation of complex information using appropriate graphical spaces and structures in order to facilitate rapid assimilation and comprehension. Its development has marked a strong line of research disciplines making headway on major trajectories in the field of computer graphics. This is the case of Human Computer Interaction HCI, a discipline in constant growth based on user requirements and execution times in order to avoid mistakes, and thereby provide product satisfaction to users (Marchionini, 1997; Shneiderman & Ben, 1998).
Information visualization has several areas of study, which have been addressed from the graphical user perception through research to determine the influences linked to visual variables such as position, length, color and shape, impacting on the effectiveness of data visualization (Cleveland, 1984; Simkin & Hastie, 1987; Vesey & Galletta, 1991). There have been studies of visual perception models, where some principles are raised from the point of view of computer graphics (Branch & Olague, 2001). Other studies to work with advanced techniques such as eye tracking visual analysis, strategies for evaluating product designs in the Web (Heer & Bostock, 2010; Simkin & Hastie, 1987), and a diverse list of ap-plications and research oriented production lines (von Landes-berger et al., 2011).
There are several types of techniques used in the visualization of data that are more appropriate for certain graphical structures (Herman, et al., 2000). There are different types of techniques used in data visualization, but for our purposes of study, we performed a selection of techniques according to: i) graphical representations structures (Schulz, Hadlak, & Schumann, 2011), ii) thematic clustering strategies (Shneiderman, Feldman, Rose, & Grau, 2000), and finally iii) through hierarchical representations of categories (Khoo, Kusunoki, & MacDonald, 2012; Tenopir, 2003).
Based on these selection criteria, we adapted and integrated four interfaces based on four information visualization techniques. Next, we describe each of them.
Tree visualization type
It is a classic type of visualization implemented to locate resources hierarchically using different levels (Fig.1). For this strategy we implement a navigational structure (Plaisant, Grosjean, & Bederson, 2002) in order to explore concepts trough hierarchical structure.
Radial visualization type
Is a circular visualization technique that permits the use of graphical components in order to represent nodes (Fig. 2). This technique uses links that identify a navigation structure according to a previously defined classification (Eades, 1990).
Hyperbolic visualization type
These are radial type structures, whose differences lie in the use of focus and context techniques based on hyperbolic geometry for visualizing and manipulating large hierarchies (Lamping & Rao, 1996) (Fig. 3).
Category visualization type
Also known as folder navigation, this type of visualization is appropriate for handling hierarchies and classifications (Fig. 4). The main purpose of this visualization technique is to solve problems of semantic interoperability (Noik, 1993).
For our study, we defined three phases. In the first phase we selected a set of AAT thesaurus terms in order to define concepts of navigation structure. Then, we use Europeana API (Europeana, 2013) to carry out the connection of metadata and linking digital resources by external content providers. The second phase, the interfaces were integrated according to four visualization techniques (tree, radial, hyperbolic and category). To carry out the development of interfaces, we defined navigation structure according to knowledge representation scheme of AAT thesaurus. Finally, the third phase we proposed a usability study to make quantitative and qualitative data, to assess effectiveness and usefulness at level of accessibility interfaces and use of navigation structures.
The relationship between utility and usability are mutually dependent, as exposed by Dillon (Dillon & Morris, 1999): "Usability represents the degree to which the user can exploit the utility." The purpose of usability analysis is to reduce user frustration when it comes to performing tasks (Norman, 2005).
For our case study, as a reference, we took the search of relevant digital resources according to taxonomic classification and specific knowledge area of art, culture and European heritage by AAT thesaurus.
Fig. 5 presents the work model for the data transformation process to design taxonomical structures of navigation and connection with the API of Europeana in order to obtain digital resources related with the topics of knowledge representation schemes defined by AAT thesaurus.
We evaluated four visualization techniques for the design and integration of visual interfaces. Each interface was loaded with the same taxonomic structure of terms related to the topic of art, culture and European heritage by the AAT thesaurus. These terms were connected with the number of digital resources explored in Europeana. By clicking on a node, users can display a representation of each term given by the thesaurus. Users can view the classification of thematic areas, the number of resources associated with each term, and digital resources listed through a in a paging system classified by language, content provider, format and copyright.
Fig. 6 presents the structure of the extensible architecture for visualizing collections of digital resources.
Fig. 7 presents an example of a visualization tool with the representation of radial interface
Fig. 7 presents: (1) the mechanism to autocomplete textual search through AJAX component.; the terms represented a topic of art, culture and European heritage according to terms of AAT thesaurus, (2) the central area shows the area of navigational search through structure of taxonomy representation by AAT and (3) at the top left corner of the image, the figure presents different interfaces (visualization techniques) that the user can select in order to search digital resources. Finally, users obtain the resources related to the topic selected, and the software tool shows information associated with the title, description, type, content provider and language of resources found in the results information page.
Results of usability study
In this section we present the results of the usability study. According to the objectives of our analysis and to the recommendations defined for usability studies (Nielsen, 1994a, 1994b), 16 participants were selected for the tests. All participants were middleaged, with a good fluency in handling Web applications by Internet searches.
Given the nature of the study, it was not necessary that users exhibit extremely specific knowledge at the level of taxonomic structures. Instead, careful consideration was given to their knowledge of search methods and interfaces, which along with the other aforementioned data were collected in a questionnaire. The participant distribution was as follows: 4 researchers (25%), 4 graduate students (25%), 4 undergraduate students (25%) and 4 high school students (25%), for a total of 16 participants.
There are specific proposals for the principles of usability focused on level of consistency (Shneiderman & Ben, 1998), heuristics factor (Nielsen, 1994b), error handling and recovery (Polson & Lewis, 1990), among others. As a proof-of-concept, we carried out an analysis based on how easily users can search digital resources through the use of visual search interfaces; for that reason, we used digital resources based on European heritage, through the use of Europeana API as a case study. The main goal is to see if through these visualizations, users can locate digital resources in a more interactive way. To carry out this study, we analyzed all in-terfaces, through descriptive statistics and the ANOVA test to show the relevance of attributes of usability related with the fol-lowing hy-pothesis: "The hierarchical classification of a graphical interface will have a positive effect on access to a collection of digital resources".
To verify our hypothesis, we performed a Pearson correlation analysis to identify if the taxonomy of these interfaces, affects access to a collection of digital resources (Table 1).
According to the Pearson correlation, R is an index that measures the linear relationship between two random quantitative variables. The maximum is 1 and a minimum of 0.6 is considered acceptable. As R is increased (R = 0.695) is considered an acceptable result, but not ideal. In the same way, (R2 = 0.46) and less than p-value (p-value <0.05) are ideal. These results indicate that the null hypothesis is rejected and thereby, it is accepted that there is a re-lationship between taxonomy and accuracy.
Therefore, we check that the hierarchical classification attribute, affects the access to a collection of digital resources, but in a con-siderable way. Being the category interface, the highest degree of effectiveness (mean = 4.38 SD = 0,744).
The effect of hierarchical classification
This attribute refers to the evaluation of taxonomic structure for each interface, that is the classification structure of a navigational search at the graphical level that user perceived, according to his experience with the interface selected.
In general, according to the results of visual perception the hier-archical classification of all interfaces presented a good level of sat-isfaction for all users with (53.1%) good ratings, followed by (18,8%) regular, (12,5%) very low, (9,4%) excellent, and (6,3%) low ratings. However, in order to identify the ease of use of the visual search interface according to hierarchical classification, users have preference for tree and category interfaces. Table 2 presents an analysis of the mean and standard deviation, to analyze the perception of interfaces evaluated at the level of hierarchical classification.
At the hierarchical classification level, we can identify that the tree interface has a high average with respect to other interfaces (mean = 4.11 SD = 0.609).
The effect of effectiveness
Related to this aspect, we evaluated the effectiveness with which users found the concept or topic of his election in order to search digital resources. Table 3 presents the results associated with this evaluation criterion.
In this case, the interface that demonstrates better effectiveness in order to locate concepts and topics in the navigational structure was the category interface with (mean = 4.375 and SD = 0.517) and the tree interface with (mean = 4.00 and SD = 0.7011). Finally, Fig. 8 presents the average time required by each user in order to search a term or topic in the navigational structure.
The time average of each user shows tree and category interfaces as the better interfaces for effectively locating terms within a navigation structure.
The effect of usefulness
In Table 4 we present results of usefulness associated with ease of use of interface to carry out the search process according to navigation structures defined by AAT thesaurus classification.
As a result, still the visual interfaces category (mean = 4.11 SD = 0.609) and tree (mean = 4.11 SD = 0.609), were the best ratings in comparison to other interfaces.
Based on results from other research that applied strategies of taxonomic structure in a graphic visualization (Graham, Kennedy, & Benyon, 2000). The users' preference for the visual representa-tion of information on-screen, strongly reflected their own mental model of the information rather than the actual underlying struc-ture of the information. However, for our case study this condi-tion didn't apply for all techniques of visualization development. We found that without good definitions of usability in the inter-faces, for example (definition of events to mark the navigation paths), the user easily leaves the navigation process that he or she is performing (hyperbolic). Yet, interfaces that have good defini-tions of these terms of usability (radial, tree, category), have proven to be of great advantage for the location of resources within the navigation scheme, and therefore have allowed us to know the hierarchical classification structure to continue with the explora-tion process.
The size of the graph is a typical problem in data visualization. Few techniques can be effectively treated with thousands of nodes, alt-hough the application in this order of magnitude are in a wide va-riety of applications and display technique combinations (Blanch & Lecolinet, 2007; Muelder & Ma, 2008) that can address the data accurately. In this case, at the level of taxonomy, the deeper the level of hierarchy, the lower the access to digital resources.
In conducting a correlation analysis, we found that the hypothesis proposed is not discarded, because all p-values are less than 0,05 and the correlation coefficient was in all cases, greater than 0.5. The Pearson correlation in the hypothesis obtained was acceptable (R = 0.695), which indicates that the attributes of taxonomy could be a key factor to improve access to digital resources. However, it is necessary that participants understand the navigation structure in order to facilitate the search for digital resources on a hierarchical structure within a specific knowledge area. For this case, it is important to define a good strategy of navigation, in order to explore the hierarchical classification categories and locate resources according to this classification.
Future work should focus on integrating components to perform additional searches defined by metadata that allow the connection of the most relevant digital resources. In this direction, is important to identify the community of users and knowledge area of digital resources, in order to define a knowledge representation scheme like an ontology or thesaurus, to implement a navigational search structure.
This work was partially supported by Organic.Lingua CIP-ICT-PSP.2010.6.2 project, reference number 27099. Special thanks to Distrital University by their support through the teacher training program.
Blanch, R., Lecolinet, E., Browsing Zoomable Treemaps: Structure-Aware Multi-Scale Navigation Techniques., Visualization and Computer Graphics, IEEE Transactions on, Vol. 13, No. 6, 2007, pp. 1248-1253.
Branch, J., Olague, G., La visión por computador: Una aproximación al estado del arte., Revista Dyna, No. 133, 2001.
Card, S. K., Mackinlay, J. D., Shneiderman, B., Readings in information visualization: using vision to think., San Francisco, CA, Morgan Kaufmann Publishers Inc., 1999.
Cechinel, C., Sánchez-Alonso, S., Sicilia, M. Á., Empirical analysis of errors on human-generated learning objects metadata., 2009.
Cleveland, W. S., Graphs in scientific publications., The American Statistician, Vol. 38, No. 4, 1984.
Dillon, A., Morris, M., P3: modeling and measuring the human determinants of information systems usage., Paper presented at the Proceedings of the 43rd Annual Meeting of the Human Factors and Ergonomics Society, Paper presented at the Annual Meeting of HFES in Texas, Santa Monica, CA: HFES, September, 1999.
Eades, P., Sugiyama, K., How to Draw a Directed Graph., J. Information Processing, Vol. 13, No. 4, 1990, pp. 424-434.
Europeana., Europeana professional, available at: http://pro.europeana.eu, (last access october 17 2013).
Fernández, J. M., Modelos de Recuperación de Información basados en Redes de Creencia. A PhD Thesis, 2001.
Gaona, P., Martin-Moncunill, D., Fermoso, A., Sanchez-Alonso, S., A usability study of taxonomy-visualization user interfaces in digital repositories., Journal Online Information review, Vol. 38, No. 2, 2014.
Graham, M., Kennedy, J., Benyon, D., Towards a methodology for developing visualizations., International Journal of Human-Computer Studies, Vol. 53, No. 5, 2014.
Heer, J., Bostock, M., Crowdsourcing graphical perception: using mechanical turk to asess visualization design., 2010.
Herman, I., Melancon, G., Marshall, M. S., Graph visualization and navigation in information visualization: A survey., Visualization and Computer Graphics, IEEE Transactions on, Vol. 6, No. 1, 2000, pp. 24-43.
Khoo, M., Kusunoki, D., MacDonald, C., Finding Problems: When Digital Library Users Act as Usability Evaluators., Paper presented at the System Science (HICSS), 45th Hawaii International Conference, 4-7 jan., 2012.
Kim, J., Finding documents in a digital institutional repository., DSpace and Eprints, Vol. 42, No. 1, 2006.
Lamping, J., Rao, R., The Hyperbolic Browser: A Focus+Context Technique for Visualizing Large Hierarchies., Journal of Visual Languages and Computing, Vol. 7, No. 1, 1996, pp. 33-55.
Marchionini, G., Information seeking in electronic environments., Vol. 9, Cambridge University Press, 1997.
Marchionini, G., Exploratory search: from finding to understanding., Communications of the ACM, Vol. 49, No. 4, 2006, pp. 41-46.
Margaryan, A., Littlejohn, A., Repositories and communities at crosspurposes: issues in sharing and reuse of digital learning resources., Journal of Computer Assisted Learning, Vol. 24, No. 4, Blackwell Publishing Ltd., 2008. Retrieved from http://dx.doi.org/10.1111/j.1365-2729.2007.00267.x
Martin-Moncunill, D., Sanchez-Alonso, S., Gaona, P., Marianos, N., Applying visualization techniques to develop interfaces for educational repositories: the case of Organic. Lingua and VOA3R., Paper presented at the Proceedings of the Learning Innovations and Quality: The Future of Digital Resources, Rome, 2013.
McGreal, R., A typology of learning object repositories., Handbook on Information Technologies for Education and Training, 2008, pp. 5-28.
Muelder, C., Ma, K. L., A treemap based method for rapid layout of large graphs., Paper presented at the Visualization Symposium, 2008. PacificVIS '08. IEEE Pacific, Tokio.
Nielsen, J., Guerrilla HCI: Using discount usability engineering to penetrate the intimidation barrier, 1994, Academic Press, Inc.
Nielsen, J., Usability engineering., San Francisco, California, USA, Morgan Kaufmann Publishers Inc., 1994.
Noik, E., Exploring large hyperdocuments: fisheye views of nested networks., Proceedings of the 5th ACM conference on Hypertext, 1993, pp. 192-205.
Norman, D. A., Human-centered design considered harmful., Interactions, Vol. 12, No. 4, 2005, pp. 14-19.
Park, J. R., Metadata quality in digital repositories: A survey of the current state of the art., Cataloging & Classification Quarterly, Vol. 47, No. 3-4, 2009.
Pastor, J. A., Diseño de un sistema colaborativo para la creación y gestión de tesauros en Internet basado en SKOS., a PhD thesis, 2009.
Plaisant, C., Grosjean, J., Bederson, B. B., Spacetree: Supporting exploration in large node link tree, design evolution and empirical evaluation. Paper presented at the INFOVIS 2002., IEEE Symposium on Information Visualization, 2002, USA.
Polson, P., Lewis, C., Theory-based design for easily learned interfaces., Human-Computer Interaction, Vol. 5, No. 2-3, 1990, pp. 191-220.
Schulz, H. J., Hadlak, S., Schumann, H., Point-based visualization for large hierarchies., Visualization and Computer Graphics, IEEE Transactions on, Vol. 17, No. 5, 2011, pp. 598-611.
Shneiderman, B., Ben, S., Designing the user interface., Pearson Education India, 1998.
Shneiderman, B., Feldman, D., Rose, A., Grau, X. F., Visualizing digital library search results with categorical and hierarchical axes., Paper presented at the Proceedings of the 5th ACM conference on Digital libraries, Rugensburg, 2000.
Simkin, D., Hastie, R., An information-processing analysis of graph perception., Journal of the American Statistical Association, Vol. 82, No. 398, 1987.
Stefaner, M., Dalla Vecchia, E., Condotta, M., Wolpers, M., Specht, M., Apelt, S., Duval, E., MACE-enriching architectural learning objects for experience multiplication., Creating New Learning Experiences on a Global Scale, Vol. 4753, 2007, pp. 322-336.
Stefaner, M., Wolpers, M., Memmel, M., Duval, E., Specht, M., Börner, D., Klemke, R., MACE: Joint Deliverable "Evaluation of the MACE system"., 2009.
Tenopir, C., Use and users of electronic library resources: An over-view and analysis of recent research studies., Paper presented at the Council on Library and Information Resources, Washington, 2003.
Tsakonas, G., Papatheodorou, C., Exploring usefulness and usability in the evaluation of open access digital libraries., Information processing & management, Vol. 44, No. 3, 2008, pp. 1234-1250.
Vessey, I., Galletta, D., Cognitive fit: An empirical study of information acquisition., Information systems research, Vol. 2, No. 1, 1991.
von Landesberger, T., Kuijper, A., Schreck, T., Kohlhammer, J., van Wijk, J. J., Fekete, J. D., Fellner, D. W., Visual Analysis of Large Graphs: State-of-the-Art and Future Research Challenges., Computer Graphics Forum, Vol. 30, No. 6, 2011. Retrieved from http://dx.doi.org/10.1111/j.1467-8659.2011.01898.x.
White, R. W., Roth, R. A., Exploratory Search: Beyond the Query-Response Paradigm., Synthesis Lectures on Information Concepts, Retrieval, and Services, Vol. 1, No. 1, 2009, pp. 1-98.