 |
DIH - Distributed Index Handler |
In order to create a completely scalable solution that delivers high performance and high availability, Autonomy provides the ability to efficiently split and index extremely large quantities of data into multiple Intelligent Data Operating Layer servers using the Distributed Index Handler. The Distributed Index Handler (DIH) provides a flexible way of transparently batching, routing and categorizing the indexing of internal and external content into the IDOL server™. The DIH is completely transparent to both IDOL servers and Connectors and can therefore be easily added into any existing system.
The Connector sends the index command to the Distributed Index Handler. The DIH then passes the commands to each individual IDOL server™
The DIH has several modes to control the way it handles index commands:
Batching
In some instances, an IDOL server™ could be slowed down considerably if data is being indexed into it at very frequent intervals, for example once a second. The DIH can be set up to batch the data based on certain criteria which is used to increase the efficiency of indexing many small files to an IDOL server™:
- Batch up index commands to one or more IDOL servers
- Configurable batching based on file size
- Configurable batching based on number of documents
- Scheduling of IDOL server™ index requests, allowing for scheduling of indexing for "quiet" periods to improve performance
- Batching of smaller index requests with time-restrained flushes to improve performance
Replication
Integrating the DIH into existing Autonomy Infrastructure improves system redundancy through replication. The DIH can be configured to replicate all index commands data to multiple IDOL servers. In addition, to enable transparent architecture administration, the DIH includes a virtual database-mapping feature. This allows the abstraction of the IDOL server™ architecture from Connector components, and the routing of documents into different IDOL servers™ to be centrally administrated.
Virtual Database Mapping
The Distributed Index Handler (DIH) allows several IDOL servers to be virtually combined to form a single virtual IDOL server™ that is transparent to the application software. A virtual database enables the administrator to specify one database name to represent a set of databases spread over separate IDOL servers. Virtual database mapping allows the abstraction of the IDOL server™ architecture from connector components, and the routing of documents into different IDOL servers™ to be centrally administrated without the need to update multiple connectors.
Load Balancing
To complement the batching and replication functionality, the Distributed Index Handler delivers scalability enhancement through Load Balancing. The DIH can be used to manipulate the indexing process into one or more IDOL servers. The DIH listens on a port that can be treated as an IDOL server™ index port, this in turn distributes data to be indexed evenly across multiple IDOL servers™. Furthermore, DIH allows queuing of index requests to the IDOL server™ for efficiency, plus queue polling for maximum throughput.
Categorized based Indexing
To enhance automatic categorization, DIH seamlessly integrates with Autonomy's categorization module enabling categorized information to be efficiently indexed across multiple IDOL servers. The DIH can also route automatically categorized documents to a specific database or collection of databases in the form of a Virtual Database.
Cross-lingual Indexing
In order to create a cross-lingual Autonomy solution, organizations need to aggregate data in various languages. Already providing language independent technology, Autonomy now enables content to be dynamically translated into multiple languages at the DIH level using plug-ins to 3rd party translation software.
Each plug-in allows the language settings to be configured, i.e. translate source language (original content) to target language (translated content). Further configuration parameters enable you to specify which IDOL server™ or database the translated content should be indexed to. Translated content is typically indexed into a cross-lingual training database to optimize the performance of the cross-lingual system. Autonomy currently provides plug-ins for translation software providers SDL and Softissimo. Please note that other translation plug-ins can be written by third party developers to support other translation software solutions as required. Contact Autonomy for more information.
DiSH Compliant
DIH is Distributed Service Handler (DiSH) compliant, enabling the crucial maintenance, administration, control and monitoring of DIH from a centralized location. See the DiSH section for more details.
