Text Analytics – Page 2 – Shibani's blog

Notes: Automatic recognition of conceptualization zones in scientific articles

Reference:
Liakata, M., Saha, S., Dobnik, S., Batchelor, C., & Rebholz-Schuhmann, D. (2012). Automatic recognition of conceptualization zones in scientific articles and two life science applications. Bioinformatics, 28(7), 991-1000.

Background:

Scientific discourse analysis helps in distinguishing the nature of knowledge in research articles (facts, hypothesis, existing and new work).
Annotation schemes vary across disciplines in scope and granularity.

Purpose:

To build a finer grained annotation scheme to capture the structure of scientific articles (CoreSC scheme).
To automate the annotation of full articles at sentence level with CoreSC scheme using machine learning classifiers (SAPIENT “Semantic Annotation of Papers: Interface & ENrichment Tool” available for download here).

Method:

Data:

265 articles from biochemistry and chemistry, containing 39915 sentences (>1 million words) annotated in three phrases by multiple experts.
XML aware sentence splitter SSSplit used for splitting sentences.

Scheme:

First layer of the CoreSC scheme with 11 categories for annotation:
- Background (BAC), Hypothesis (HYP), Motivation (MOT), Goal (GOA), Object (OBJ), Method (MET), Model (MOD), Experiment (EXP), Observation (OBS), Result (RES) and Conclusion (CON).

Implementation:

Text classification:
- Sentences classified independent of each other.
- Uses Support Vector Machine (SVM).
- Features extracted based on different aspects of a sentence: location within the paper, document structure (global features) to local features. For the complete list of features used, refer the paper.
Sequence labelling:
- Labels assigned to satisfy dependencies among sentences.
- Uses Conditional Random Fields (CRF).

Results and discussion:

F-score: Ranges from 76% for EXP (Experiment) to 18% for the low frequency category MOT(Motivation) [Refer complete results from runs configured with different settings and features in Table 2 of the paper].
Most important features: n-grams (primarily bigrams), Grammatical triples (GRs), verbs, global features such as history (sequence of labels) and section headings (Detailed explanation for the features
Classifiers: LibS has the highest accuracy at 51.6%, CRF at 50.4% and LibL at 47.7%.

Application/Future Work:

Can be applied to create executive summaries of full papers (based on the entire content and not just abstracts) to identify key information in a paper.
CoreSC annotated biology papers to be used for guiding information extraction and retrieval.
Generalization to new domains in progress.

Notes: Computational analysis of move structures in academic abstracts

Reference:

Wu, J. C., Chang, Y. C., Liou, H. C., & Chang, J. S. (2006, July). Computational analysis of move structures in academic abstracts. In Proceedings of the COLING/ACL on Interactive presentation sessions (pp. 41-44). Association for Computational Linguistics.

Background:

Swales pattern for research articles: Introduction, Methods, Results, Discussion (IMRD) and Creating a Research Space (CARS) model.
Studying the rhetorical structure of tests is found to be useful to aid reading and writing (Mover tool notes here).

Purpose:

To automatically analyze move structures (Background, Purpose, Method, Result, and Conclusion) from research article abstracts.
To develop an online learning system CARE (Concordancer for Academic wRiting in English) using move structures to help novice writers.

Method:

Processes involved:

TANGO Concordancer used for extracting collocations with chunking and clause information – Sample Verb-Noun collocation structures in corpus: VP+NP, VP+PP+NP, and VP+NP+PP (Ref: Jian, J. Y., Chang, Y. C., & Chang, J. S. (2004, July). TANGO: Bilingual collocational concordancer. In Proceedings of the ACL 2004 on Interactive poster and demonstration sessions (p. 19). Association for Computational Linguistics.)
- TANGO Tool accessible here.
Data: Corpus of 20,306 abstracts (95,960 sentences) from Citeseer. Manual tagging of moves in 106 abstracts containing 709 sentences. 72,708 collocation types extracted and manually tagged 317 collocations with moves.
Hidden Markov Model (HMM) trained using 115 abstracts containing 684 sentences.
Different parameters evaluated for the HMM model: “the frequency of collocation types, the number of sentences with collocation in each abstract, move sequence score and collocation score”

Results:

Precision of 80.54% achieved when 627 sentences were qualified with following parameters: weight of transitional probability function 0.7 , frequency threshold for a collocation to be applicable – 18 (crucial to exclude unreliable collocation).

Conclusion:

CARE system interface created for querying and looking up sentences for a specific move.
System is expected to help non native speakers write abstracts for research articles.

Notes: Visualizing sequential patterns for text mining

Reference:

Wong, P. C., Cowley, W., Foote, H., Jurrus, E., & Thomas, J. (2000). Visualizing sequential patterns for text mining. In Information Visualization, 2000. InfoVis 2000. IEEE Symposium on (pp. 105-111). IEEE.

Background:

Mining Sequential patterns aims to identify recurring patterns from data over a period of time.
A pattern is a finite series of elements from the same domain A -> B -> C -> D
Each pattern has a minimum ‘support’ value which indicates the percentage of pattern occurrence. (E.g. 90% of people who did this process, did the second process, followed by the third process)
Sequential pattern vs association rule:
- Sequential pattern – studies ordering/arrangement of elements E.g. A -> B -> C -> D
- Association rule – studies togetherness E.g. A+B+C -> D

Purpose:

Presenting a visual data mining system that combines pattern discovery and visualizations.

Method:

Datasets:

Open source corpus containing 1170 news articles from 1991 to 1997 and harvested news of 1990 from TREC5 distribution.

Pre-processing:

Topic Extraction: Identifies the topic in documents based on the co-occurrence of words. Words separated by white space evaluated – stemming done, prepositions, pronouns, adjectives, and gerunds ignored.
Multiresolution binning: Bins articles with the same timestamp (E.g. Binning by day, week, month, year)

Discovery of sequential patterns by Visualization:

Plotting topics/ topic combinations over time.
Strength: Can quickly view overall patterns and individual occurrence of events.
Weakness: No knowledge on exact connections that make up the pattern and statistical support on the individual patterns.

Discovery of sequential patterns by Data mining:

Building patterns on n-ary tree with elements as nodes.
Patterns are valid if the support value is greater than threshold.
A sample pattern mining from given input data is given in Figure 2 of the paper.
Strength: Provides accurate statistical (support) values for all weak and strong patterns.
Weakness: Loses temporal and locality information, large number of patterns produced in text format making human interpretation harder.

Visual Data Mining system:

Combining visualization and data mining to compensate each others’ weaknesses (Refer Figure 4 & 5 in the paper to see the pattern visualizations).
Binning resolution can be changed to see different patterns based on day, week, month, year etc.
Patterns associated to a particular topic can be picked.

Result/Discussion:

Strength of pattern is not easily identifiable from the visualization without statistical measures. Pattern mining gets enhanced by graphical encoding with spatial and temporal information.
Knowledge discovery by humans is aided by combining statistical data mining and visualization.

Future Work:

Handling larger data sets using secondary memory support and improve display.
Integrating more techniques like association rules into visual data mining environment.

Notes: Mover – a Machine Learning tool to analyze technical research papers

Reference: Anthony, L., & Lashkia, G. V. (2003). Mover: A machine learning tool to assist in the reading and writing of technical papers. IEEE transactions on professional communication, 46(3), 185-193.

Background:

Identifying the structure of text helps in reading and writing research articles.
The structure of research article introductions in terms of moves is explained in the CARS model (Ref: J. M. Swales, “Aspects of Article Introductions,” Univ. Aston, Language Studies Unit, Birmingham, UK, Res. Rep. No. 1, 1981.).

Problem:

Identifying the moves in a particular type of article.
Time-consuming identification of moves by raters (manual annotation) with no immediate feedback.

Purpose:

To provide immediate feedback on move structures in the given text.

Method:

Using supervised learning to identify moves from 100 IT research article (RA) abstracts.
Machine readable abstracts were further pre-processed with subroutines to remove irrelevant characters from raw text.
Data labelled based on the modified CARS model which had three main moves with further steps under each move as below: (Ref: L. Anthony, “Writing research article introductions in software engineering: How accurate is a standard model?,” IEEE Trans. Prof. Commun., vol. 42, pp. 38–46, Mar. 1999.)
1. Establish a territory
  1. Claim centrality
  2. Generalize topics
  3. Review previous research
2. Establish a niche
  1. Counter claim
  2. Indicate a gap
  3. Raise questions
  4. Continue a tradition
3. Occupy the niche
  1. Outline purpose
  2. Announce research
  3. Announce findings
  4. Evaluate research
  5. Indicate RA structure

Supervised Learning System – Implementation details:
- Bag of clusters representation was implemented:
  - Dividing input text into clusters of 1-5 word tokens to capture key phrases and discourse markers as features.
  - Bag of words model does not consider word order and semantics by splitting input text into word tokens – not useful in the discourse level.
  - E.g. “Once upon a time there were three bears” clusters –> “once”, “upon”, “once upon”, “once upon a time”
  - Not useful clusters (noise) removal using statistical measures – Information Gain (IG) scores used to remove clusters below threshold.
  - ‘Location’ feature added to take note of preceding and later sentences – position index of sentence in the abstract.
    - Additional training feature for the classifier – probability of common structural step groupings.
- Naive Bayes learning classifier outperformed other models.
- Tool available for download as AntMover.

Results:

Evaluation of Mover:
- Training: 554 examples, Test: 138 examples
- Five fold cross validation, Average accuracy: 68%
- Classes (steps within the structural moves) with few examples had lower accuracy. Incorrectly classified steps were mostly from the same move (note similarity among 3.1, 3.2. 3.3)
- Features to improve accuracy:
  - When two most likely decisions are used (instead of predicting only one class) using the Naive Bayes probabilities, accuracy increased to 86%.
  - Flow optimization effectiveness improved accuracy by 2%.
  - Manual correction of steps adding new training data (second opinion of students on the moves classified by the system used for retraining the model).

Discussion:

Based on two practical applications in classroom, the usage of ‘Mover’ assisted students to
- identify unnoticed moves in manual analysis.
- analyze moves much faster than manual analysis.
- better understand own writing and prevent distorted views.
Implications:
- Important vocabulary can be identified for teaching from the ordered cluster of words.
- Trained examples can be used as exemplars.
- Aid for immediate analysis of text structure.
Future Work:
- Increasing the accuracy of Mover.
- Expanding to more fields – currently implemented for engineering and science text types.

Notes: NLP Techniques for peer feedback

Reference: Xiong, W., Litman, D. J., & Schunn, C. D. (2012). Natural language processing techniques for researching and improving peer feedback. Journal of Writing Research, 4(2), 155-176.

Background:

Feedback on writing is seen to improve students’ writing, but the process is resource intensive.
Possible options to reduce the workload in giving feedback:
- Direct feedback using technology assisted approaches (from grammar checks to complex computational linguistics).
- Peer Review [Considered in this paper].
Peer review:
- Good feedback from a group of peers is found to be as useful as the instructor’s feedback and even weaker writers are seen to provide useful feedback to stronger writers (See references in original paper).
- When providing feedback on other students’ work, students become mindful of the mistakes and improve their own writing.
- Some web-based peer review systems: PeerMark in turnitin.com, SWoRD (used in this study) and Calibrated Peer Review.

Problem:

Challenge lies in the form of feedback provided by peers – peer feedback might not be in a form useful to make revisions. Key features identified to aid revisions:
1. Localized information (Providing exact location details like paragraph, page numbers or quotations).
2. Concrete solution (Suggesting possible solution rather than just pointing the problem).
Research problem: Studying peer review is hard with a large amount of feedback data.
Practical problem: Identifying useful feedback for students and possible interventions to help them provide good feedback.

Purpose:

To automatically process peer feedback and identify the presence or absence of the two key features (Providing feedback on feedback for students and automatically coding feedback for researchers).
Refer prototype shown in figure 1 of the original paper that suggests students to provide localized and explicit solutions.

Technical – How? (Details explained in study 1 and study 2)

Building a domain lexicon from common unigrams and bigrams in student papers.
Counting basic features like domain words, modals, negations, overlap between comment and paper etc. from each feedback.
Creating a logic model to identify the type of feedback (Contains localization information or not/ contains explicit solution or not) – classification task in machine learning.

Method and Results:

Study 1 – Localization Detection:

Each feedback comment represented as a vector of the four attributes below:
1. regularExpressionTag: Regular expressions to match phrases that use location in a comment (E.g. “on page 5”).
2. #domainWord: Counting the number of domain-related words in a comment (based on the domain lexicon gathered from frequent terms in student papers).
3. sub-domain-obj, deDeterminer: Extracting syntactic attributes (sub-domain-obj) and count of words like “this, that, these, those” which are demonstrative determiners.
4. windowSize, #overlaps: Extracting the length of matching words from the document to identify quotes (windowSize) and words overlapped.

Weka models to automatically code localization information. The decision tree model had better accuracy (77%, recall 82%, precision 73%) in predicting if a feedback was localized or not. To refer the rules that made up the decision tree, take a look at Figure 2 of the original paper.

Study 2 – Solution Detection:

Feedback comment represented as vectors using the three types of attributes(Refer table 2 in the original paper for details).
- Simple features like word count and the order of comment in overall feedback.
- Essay attributes to capture the relationship between the comment and the essay and domain topics.
- Keyword attributes semi-automatically learned based on semantic and syntactic functions.

Logistic Regression model to detect the presence/ absence of explicit solutions (accuracy 83%, recall 91%, precision 83%). Domain-topic words followed by suggestions were highly associated with prediction. Detailed coefficients of attributes predicting presence of solution can be referred in Table 3 of the original paper.

Study 3: Can Research Rely on Automatic Coding?

Comparing automatically coded data to hand coded data to see if the accuracy is sufficiently high for practical implementation.
Helpfulness ratings by peers and 2 experts (content, writing experts) on peer comments at a review level.
To account for expert ratings:
- Regression analysis using feedback type proportions (praise only comments, summary only comments, problem/solution containing comments), proportion localized critical comments, and proportion solution providing comments as predictors.
- 10 fold cross validation – SVM best fit.
- To check if same models are built using machine coded and hand coded data – 10 stepwise regressions. Refer Table 4 in the original paper to see the feedback features commonly included in the model by the different raters – Different features were helpful for different raters.
- Overall regression model is similar to hand coded localization data (Most of the positivity, solution and localization were similar between hand coding and automatic coding).

Discussion:

Predictive models for detecting localization and solution information are statistical tools and do not provide deep content insights.
To be integrated into SWoRD to provide real time feedback on comments.
Technical note: Comments were already pre-processed – segmented into idea units by hand; data split by hand into comment type (summary, praise, criticism).
Future work:
- Examine impact of feedback on feedback comments
- Obtaining generalization across courses
- Improving accuracy of prediction