Big Picture

To better understand the issues and the structure of the problem we outline the overall picture of solutions.

Component Model

Data Harvester

The main task of this component is to feed raw system data (documents) on a regular basis - adding new and updating old.

• collects data from defined sources
  • supports protocols such as OAI-PMH, HTTP / HTTPS, FTP, file-system
  • supports all common video and text formats (JPG, TIFF, TXT, PNG, PDF, DOC, HTML, XML, ...)
• does basic validation, transformation and conversion to mediation (exchange, labor) formats
•  define (add) technical and organizational metadata - source, time of harvest, the producer, format, size, ...
•  scheduler to schedule (regular) data collection for each source
•  allows a one-time, immediate collection

Open Data Enhancer

Component obtain of the input document the maximum amount of relevant information (metadata) and evaluate their quality (completeness, accuracy, relevance, ...)

• in the case of non-text document attempts to recognize text part (subcomponent OCR)
• classified according to the type of document (content), for example. contract, invoice, ...
• in the text document (also obtained via OCR) recognizes the fundamental (content) attributes of a document (by document type) - a "text mining" and "text recognition" respectively. occupancy documents
• using the "spellchecker" and defined dictionaries (registers authorities) to complete or correct metadata

Open Data Repository

Central repository for data, metadata (content, technical and operational) and indices (for full-text and faceted search). It consists of

• data storage (repository) - saved a document or link directly to it
• metadata repository (repository), eg. RDF format
• indexing server (support for fulltext, veneers, sorting, filtering)
• image server - as needed for any storage and delivery of large image data

It also allows

• allocation and management of unique and persistent ID documents for maintaining identity, addressing duplication and facilitate internal and external links between documents.
• management & monitoring, ie. management and maintenance of data-level system administrator

Open Data Services & API

Services and open interfaces to support third-party applications, create mashups etc. over all the processed data in the system.

Portal & Application

Web user interface, its main tasks are:

• basic information about Open Data activity
• make available all documents processed at a basic level:
  • find documents - browse on, fulltext, filtering, searching by metadata
  • view documents (and metadata) in a suitable view - list, detail, table (later map, timeline, graph, ...)
• community and collaboration tool for all stakeholders (data providers, application providers, application users, donors, the general public, ...)
• provide the basis of the lists and statistics - sources and their quality, contributors, trends, most requested / most discussed documents, the most active group ...
• allows for feedback on the documents and resources, including the system for repairs and complete metadata (eg crowd sourcing)

Links

Automatic extraction of metadata from published documents