Imagine having to read 7,000 documents, each with an average length of 200 pages. That is a total of 1,400,000 pages: a huge number, if we consider that all the Harry Potter books together have fewer than 4,000 pages. But imagine subsequently having to classify them according to the topics they cover and then relate them to other documents covering similar topics.Such an activity would require a dedicated research team and months of work. Each researcher would obviously apply a certain amount of subjectivity, making further work necessary to standardise the results. Let's also imagine that, once the work is finished, the documents are updated with a new, perhaps even completely different, version... we would have to start all over again from scratch.

Text mining and AI

This is a situation where text mining techniques become essential, because automatic systems can be used to extract structured information from texts. However, such an artificial intelligence tool has to learn to perform the task and understand the context in which it is to operate; the system has to be “trained”. This is done by showing it the results of the operation on a small subset of documents that is representative enough to reduce potential future errors.

A schools project

CSI applied these automated text analysis techniques to the PTOF (three-year school development plans) that Italian schools publish annually.

We are talking about a large number of documents, approximately 7,000, that have been analysed and structured to support INDIRE (National Institute for Documentation, Innovation and Educational Research) in increasingly effectively targeting interventions for strategic initiatives, such as inclusion, gender equality, teaching respect and the fight against bullying and cyberbullying.  

This is also an important technological and cultural milestone; it is no longer just humans who define a set of rules, which other humans or computers then apply. The machine itself is able to learn from the data and identify the best paths to take based on this. Combined with the processing of texts or unstructured content in general, this leads to the generation of a large amount of new information and knowledge.