Notes: Mover – a Machine Learning tool to analyze technical research papers

Reference: Anthony, L., & Lashkia, G. V. (2003). Mover: A machine learning tool to assist in the reading and writing of technical papers. IEEE transactions on professional communication, 46(3), 185-193.

Background:

  • Identifying the structure of text helps in reading and writing research articles.
  • The structure of research article introductions in terms of moves is explained in the CARS model (Ref: J. M. Swales, “Aspects of Article Introductions,” Univ. Aston, Language Studies Unit, Birmingham, UK, Res. Rep. No. 1, 1981.).

Problem:

  • Identifying the moves in a particular type of article.
  • Time-consuming identification of moves by raters (manual annotation) with no immediate feedback.

Purpose:

  • To provide immediate feedback on move structures in the given text.

Method:

  • Using supervised learning to identify moves from 100 IT research article (RA) abstracts.
  • Machine readable abstracts were further pre-processed with subroutines to remove irrelevant characters from raw text.
  • Data labelled based on the modified CARS model which had three main moves with further steps under each move as below: (Ref: L. Anthony, “Writing research article introductions in software engineering: How accurate is a standard model?,” IEEE Trans. Prof. Commun., vol. 42, pp. 38–46, Mar. 1999.)
    1. Establish a territory
      1. Claim centrality
      2. Generalize topics
      3. Review previous research
    2. Establish a niche
      1. Counter claim
      2. Indicate a gap
      3. Raise questions
      4. Continue a tradition
    3. Occupy the niche
      1. Outline purpose
      2. Announce research
      3. Announce findings
      4. Evaluate research
      5. Indicate RA structure

 

  • Supervised Learning System – Implementation details:
    • Bag of clusters representation was implemented:
      • Dividing input text into clusters of 1-5 word tokens to capture key phrases and discourse markers as features.
      • Bag of words model does not consider word order and semantics by splitting input text into word tokens – not useful in the discourse level.
      • E.g. “Once upon a time there were three bears” clusters –> “once”, “upon”, “once upon”, “once upon a time”
      • Not useful clusters (noise) removal using statistical measures – Information Gain (IG) scores used to remove clusters below threshold.
      • ‘Location’ feature added to take note of preceding and later sentences – position index of sentence  in the abstract.
        • Additional training feature for the classifier – probability of common structural step groupings.
    • Naive Bayes learning classifier outperformed other models.
    • Tool available for download as AntMover.

Results:

  • Evaluation of Mover:
    • Training: 554 examples, Test: 138 examples
    • Five fold cross validation, Average accuracy: 68%
    • Classes (steps within the structural moves) with few examples had lower accuracy. Incorrectly classified steps were mostly from the same move (note similarity among 3.1, 3.2. 3.3)
    • Features to improve accuracy:
      • When two most likely decisions are used (instead of predicting only one class) using the Naive Bayes probabilities, accuracy increased to 86%.
      • Flow optimization effectiveness improved accuracy by 2%.
      • Manual correction of steps adding new training data (second opinion of students on the moves classified by the system used for retraining the model).

Discussion:

  • Based on two practical applications in classroom, the usage of ‘Mover’ assisted students to
    • identify unnoticed moves in manual analysis.
    • analyze moves much faster than manual analysis.
    • better understand own writing and prevent distorted views.
  • Implications:
    • Important vocabulary can be identified for teaching from the ordered cluster of words.
    • Trained examples can be used as exemplars.
    • Aid for immediate analysis of text structure.
  • Future Work:
    • Increasing the accuracy of Mover.
    • Expanding to more fields – currently implemented for engineering and science text types.