By Juan Chamero

By Juan Chamero




    Darwin algorithms may deal with a vast scope of Applications mainly those derived from IR, Information Retrieval and IIR, the same IR faced and empowered with AI, Artificial Intelligence tools. Content Generation is a leading application of today as it is supposed we are living in the Content Era. The actual state of the art of it is restricted to aids and facilitations of Content Generation performed by humans, like Specialized Editors and Compilers with access to Tutorials, Guides, Quotations, Cites, References, Similar, Main Thematic Trends, Thesauruses, Glossaries and Dictionaries.
    If you imagine a Control Panel where from you may access all these resources and you are an expert writer as well your intellectual production will be many times empowered, However in order to transfer this talent to agents it would be necessary to execute and save all possible sequences of actions and of their corresponding outcomes and all steps clearly understood and explained at its deepest detail. Believe me this is still a cyclopean task!. As an example of this talent steps and micro-steps please go back to see our analysis about K-side dissection where we present examples of querying Search Engines, one of the simplest tasks onto the above mentioned Control Panel. As we will see agents could perform any computable task as long it is perfectly and clearly understood and explained.
    Darwin may behave in some instances like an efficient Differential Data Mining implying that once you have unveiled the inherent and most times hidden intelligence of a huge data reservoir there is no more need of “mining”. Darwin agents only need to process the last shell of aggregated data!. Darwin processes by “de facto” evolve by themselves.

Established Knowledge Realm
    We may find examples of this side in all type of data reservoirs either concentrated or distributed. Within the first type we have Catalogs and within the second the Web. However Search Engines provide databases to facilitate the search process. Darwin Ontology is well suited to work on any of these scenarios. Its acronym that stands for Distributed Agents to Retrieve the Web Intelligence tells us about its versatility.

Cataloging and Catalogs Harmonization
    Its first aim is to facilitate the semantic ordering of the established side building natural hierarchic indexes. Giving for instance a Catalog of Materials it may check its consistency and harmony, suggesting changes to improve both. Giving a list of materials needed Darwin agents may be committed to go to permitted databases and data reservoirs to extract those Catalog trees and sub trees that match needs in order to build feasible Catalog models.

Information Offer versus Demand Optimization
    Given a well defined information “offer” database it could be cloned in as many market front ends exist and study the different Offer versus Demand matchmaking interfaces in order to: a) have a better knowledge of the true demand and b) adjust its global and distributed offer accordingly!.

Potential Demand
    The potential demand could be also tracked in the absence of a particular offer. Darwin algorithm could be settled and adjusted to study demand without offer. For example in the Cyber space every day almost one thousand million users from all over the world are continuously trying to learn, to teach and to broadcast their ideas and opinions. We may tune up this info brewing to focus on a small window of the demand spectrum in a “hearing at” attitude. We may also then browse the whole Web space to account for the available offer to satisfy this potential demand. Non Satisfied Demand could be then next Darwin step.

SSSE
    The WWW, World Wide Web is a network of routers, servers, and databases created to satisfy the World Information Demand. Search Engines facilitate this complex task by concentrating in their databases meaningful summaries of all documents dispersed along the Web. Is supposed that there are about 20 billion documents located in this net and its number grows at a terrific pace. From this data asset only about 10 billion are actually “classified” by the most powerful search engines. This classification is rather primitive, mostly by “words”, with all documents sharing only one semantic level of thematic specificity. In order to facilitate and make more efficient the search process the trivial solution would be to classify documents thematically. Darwin algorithm may perform this task uprising the thematic diversity from “zero ground” to as much as thirteen levels. Darwin may use the same information search engines have but providing users with a sort of “semantic glasses” ® to “see” the Web better, as ordered in up to thirteen levels instead of a fuzzy and noisy zero ground. .

People's Knowledge Realm

People’s Behavior Patterns Inferences
    The simplest scenario is users trying to learn as much as possible from a given Information Offer as it is the case of people using search engines. In the near future search engines will make sound actual and potential Information Demand inferences as long as people interact with their services freely. With SSSE’s users will be encouraged/induced to question via concepts (semantic chains in our approach), and as concepts match specific subjects it is relatively easy to make probabilistic inferences about what users are trying to find out.
   Users’ will question via ”established concepts” or via their own “people concepts”. People concepts play a similar role to established concepts, and should match their corresponding people’ subjects. These subjects are strongly related to “people’ information needs”. So “people information’ needs” are in some extent equivalent to “established subjects”. People query established information sources trying to satisfy their information needs. These needs are also matter of analysis within Darwin Ontology that states as a strong Conjecture that people tend to specialize in Major Activities and presupposes that these major activities, similar to disciplines in the established side, structure themselves as trees. Querying chains as semantic chains tend to structure probabilistically onto these trees. Most frequent querying chains suggest significant activities perhaps a Major Activity. It is supposed that people that belong to a given Major Activity when querying search engines will use, in the long range, a similar kit of querying chains.
    Darwin algorithms are anthropic, that is some steps are necessarily performed by humans until the retrieving talent is precisely defined and then transferred to agents. For example Darwin algorithm that work onto K-side has about 80 steps being most of them (75) performed by agents. Darwin algorithms working in the next future work onto K’-side will initially have steps that necessarily must be performed by humans because it is a realm less known than K. .

Some Darwin byproducts

    Eventually a Darwin algorithm may be located as a smart interface of a SDI, Selective Distribution of Information system as in the figure above. The main outcomes, The Hot information is secure and selectively distributed in time and form over the Organization hierarchy creating by de facto an Organization Strategic Infoduct.

    Darwin byproducts are many and fruitful. Along the distilling process mentioned above as in the oil industry appear byproducts out coming from intermediate steps. For example to build the Art Map it was necessary to process about 500,000 documents, many of them authorities.  The pure textual sequences needed to unveil keywords is a corpus a huge and linguistically structured set of texts that could be used for linguistic statistical analysis, text of validity of linguistic rules, concordance analysis, etc.  (in English in this case). To give an idea of its importance BYE, the Corpus of American English: It has more than 360 million words in nearly 150,000 texts, including 20 million words each year from 1990-2007. The “Art Corpus” that could be delivered by Darwin may have about 1,000 million words corresponding to 500,000 texts.
    Another important byproduct would be K’-side versus K-side rotation. If in K-side we suppose hosted a meaningful sample of the actual Human Knowledge, in K’-side we may imagine the actual People Knowledge flowing and we may ask ourselves: how many times K’-side must “rotate” to change K-side content?. At large how to know more about the life cycles of both realms?. As a gym about it some speculations follows. 

     People brewing in K’: 500,000 daily in the average, with an average bandwidth of 100 KB each which gives us a flow of  5x10**10 Bytes per day. As we do not have any metric yet we may dare a figure that estimate the number of days to change K significantly, for example 100,000 days. We ignore how is the relative “volatility of the disciplines of the human knowledge, some extremely volatile like news, fashion, computing and some other extremely stable and inert to changes like religion and to take into account this inertia we have to estimate a redundancy factor, which would stand for cycling insistence to effectively generate a change. If we bid for a redundancy equal 10 we finally would need an accumulatde traffic of 5x10**16 Bytes in order to change K completely.
    Now let’s get a reasonable estimation for K-side size concerning this problem. Out of the whole Website documents we should only take into consideration authorities that we estimate in 1000,000,000 with 2,000 words each in the average that taking 5 characters per word and a byte per character gives us a Web volume size of 10**13 Bytes. Finally we are going to need 5,000 rotations of K’ over K in order to change K completely. Of course this calculation is only a speculation to test order of magnitudes because real rotation has to be measured.

ByJuan Chamero



Darwin versus Data Mining
    Data mining is a sorting process over huge amounts of data (the mine) trying to extract significant “data patterns”. It pre supposes no previous knowledge about data.  It belongs to the field of “Knowledge Discovery” and their procedures root deep into statistics.  It also pre supposes that is being used as an analytical tool for data analysis in man-machine environments.  Patterns then may lead to discover “behaviors”, in this case users’ behaviors. Its main field of application rests on real world data with unknown interrelations. This characteristic carries an unavoidable weakness: the critical data that allegedly built detected patterns could never be observed. Concerning this weakness data mining always require ulterior and costly “data dredging” studies.
    However documents are human generated data and as such they are somehow intelligently structured and suited to a better approach pre supposing that we know something about their inner structure that is to say we may choice a model about its inner structure. So Choice Modeling is an adequate tool for making probabilistic predictions about how humans decide and document. In this sense Darwin technology may be considered within the Choice Modeling realm.

The Darwin distilling analogy
 
    From Art raw data we may imagine a distillation process that proceeds along distillation paths (instead distillation steps and towers) In this figure the product “Rigoletto” experiment 11 distillation steps. In this case Rigoletto is a “leave” of the tree and at the same time a subject and a keyword. Some other like “Fantasy novel” and “Paella” are subjects that hare not leaves but roots of sub-trees like types of Fantasy novels within the literature and types of Paella by cooking and regions of Spain within gastronomy (these sub-trees are not represented here). Node Keywords are not shown here, only nodes that are subjects and remember that all subjects are keywords but not all keywords are or become subjects. For the Art map In the average we may find about 50 keywords per node.
    

    Darwin Ontology Conjectures intent to model: a) how new concepts are generated; b) how knowledge hierarchy evolves, and; c) the documentation process. With this model “in mind” Darwin anthropic algorithm proceeds to unveil along a sort of industrial distilling process the semantic hidden structure and authoritativeness behind data. As in the oil industry raw data is, like crude oil, thousand trillions of words accumulated here and there in billions of semantic containers (documents) and generated by humans following established and well known literary routines (chemical processes).
    First we have to eliminate wastes and heavy by-products, leaving only distillable oil (Web pages   are transformed in text strings). Then Darwin algorithm has to process those text strings in order to separate two groups of semantic particles: Common Words and Expressions from Specific Concepts (and of course as in any distillation process wastes appear). Then all concepts are “purified” and reclassified within their specific clusters, namely the core of the industrial semantic distillation process.
    To complete the analogy it is important to emphasize the role of Chemical Analysis.  We may distill crude oil because we know “in advance” its chemical analysis at its deepest detail. What we know in advance as TRUE apart from our Darwin Conjectures are Conventional Search Engines Indexes. Some Search Engines like Google index all Web documents by word, no matter if they are well or wrongly written!. So we may know “in advance” crucial information about their semantic composition, something equivalent to the crude oil chemical analysis.        
    Summarizing Darwin may upgrade Search Engines indexes to become directly retrievable in only one click. We may also apply a version of this algorithm working within the same ontology to properly index all documents at registering time, something equivalent to books classification. For each Web document we may automatically and autonomously compute its “semantic fingerprint”, something equivalent to its semantic spectrum described in concepts dealt with. Next generation search engines will have incorporated this feature as standard.  From this moment onwards we may sustain that “established knowledge is known” and may be properly used by all people. 
We may also say from now onwards that the Web content has been tamed and we may talk of it as the Semantic Web. 
    Paraphrasing Tim Berners Lee the Web creator this is equivalent to know the Web Thesaurus, a whole meaningful Web mapping; and as in a huge Virtual Library documents will be perfectly classified and retrieved directly at will. We may also argue that the Web version of the Established Human Knowledge is perfectly identified. However this identification strongly depends on “authorities”, those entities that rules what the established knowledge is at a given moment. But why?: Because via their authoritativeness authorities rule both meanings and names.

Towards the second Semantic Big Step
 
    We have worked with this image before: It depicts the K-K’ equilibrium. Initially the Web could be considered semantically “flat”, a huge reservoir of almost 12 billion documents not thematically indexed. First step would be to build the Semantic Web that is the whole Web mapped as depicted in the lower part of the figure. Initially the intelligent interface between these two realms whether existent (in yellow) would work at half of its possibilities broadcasting information and intelligence (in yellow arrow) on way from K to K’ but only receiving not analyzed users’ queries (in white arrow)). Once K is mapped both sides of an intelligent interface would work and enabling, at large, the K’ mapping.

    In Darwin Ontology the Web Thesaurus is only the first big semantic step to pursue the truth. To obtain the best approximation to it we need to perform the second big step: to know as much as possible the People’s Truth and People’ Thesaurus. 
    We may talk of a sort of collective brewing truth in one side trying to change in some extent the Established Truth existing at the other side in a process that resembles thermodynamic equilibrium.
    We may also imagine this equilibrium like a man-machine game where: