Contextualizable learning analytics for writing support

Recently I gave a talk on Augmenting pedagogical writing support with contextualizable learning analytics at the CRLI seminar series in the University of Sydney.  It was a great opportunity to share and discuss ideas from my PhD research, and indeed a privilege to be invited to present at this seminar. Long time slot means less time constraints, so I enjoyed doing the 1 hour+ session. The talk is recorded and available for viewing on Youtube, and the slides are here. This post is a summary of the key ideas from this talk and an upcoming paper on ‘Contextualizable Learning Analytics Design (CLAD)’.

Big data, learning analytics and education:

Big data and artificial intelligence are changing many ways we do things to improve our lives (for better or for worse). Companies around the world including Facebook, Google, Apple and Amazon use data everyday to get big insights to support us. What can the more traditional organizations like educational institutions use data for? Can we harness this technology and data to improve learning? To answer these questions, Learning Analytics (LA) emerged as a field to attempt tackling huge amounts of data in education. Although data was previously available in education research for decades, different granularities of data from multiple sources in authentic scenarios and technical affordances of new tools can now support many causes which were not previously plausible. This root cause for the inception of the field has probably been a reason for its emphasis on ‘big impact’ and generalizable solutions that can cater to and scale up to huge numbers. Massive Open Online Courses (MOOCS) are a classic example of how we can scale teaching to a large number of learners using technology. However, the problem with scalable, generalizable solutions in learning analytics is that education is inherently contextual, and a one-size-fits all approach would not work in all contexts the same way. This has led to the argument on moving from big data to meaningful data for learning analytics.

Bringing in the context:

To bring the educational context to Learning Analytics (LA), it must be coupled with pedagogical approaches. This involves the integration of LA in pedagogical contexts to augment the learning design and provide analytics that are aligned with the intended learning outcomes. Learning Design (LD) describes an educational process, and involves the design of units of learning, learning activities or learning environment which are pedagogically informed. LA can provide the necessary data, methodologies and tools to test the assumptions of the learning design, and LD can add value to the analytics by making it meaningful for the learner. By bringing LA and LD together, they can contribute to each other and close the gap between the potential and actual use of technology.

Contextualizable Learning Analytics Design:

We introduce the Contextualizable Learning Analytics Design (CLAD) model in a forthcoming article by bringing together the elements of LA and LD for context. The educators are involved with LA developers to co-design this contextualization. This involves LD elements of assessment and task design, and LA elements of features and feedback working dynamically and in sync for different contexts, rather than being rigidly fixed. The CLAD model is demonstrated by implementing the Writing Analytics tool ‘AcaWriter’ in different learning contexts (Law essay writing, Accounting business report writing). AcaWriter, developed by the Connected Intelligence Centre, UTS provides automated feedback on student writing based on rhetorical moves. To contextualize the use of this LA tool for students, the elements of the CLAD model were employed as follows:

  • Assessment formed the basis of contextualization to align AcaWriter with the intended learning outcomes.
  • The features of data that are important for the context were picked so that AcaWriter can bring them to the attention of the learners.
  • The feedback from AcaWriter was tuned to make it relevant for the context of writing by mapping it back to assessment criteria.
  • Task design ensured that AcaWriter activities are relevant to the learner and grounded by pedagogic theory.

With such contextualized LA, the educator has agency to design learning analytics that is relevant to the learning context, and the learner finds it meaningful due to its embedding in the curriculum. This ensures that LA contributes to learning in authentic practice by augmenting existing good pedagogic practice. The approach scales over multiple learning contexts by transferring good design patterns from one learning context to another (for example from law essay writing to accounting business report writing).

More details on the above can be found in the following article, and related resources are available on the HETA project website.

References:

Tools for automated rhetorical analysis of academic writing

Alert – Long post!

In this post, I’m presenting a summary of my review on tools for automatically analyzing rhetorical structures from academic writing.

The tools considered are designed to cater to different users and purposes. AWA and RWT aim to provide feedback for improving students’ academic writing. Mover and SAPIENTA on the other hand, are to help researchers identify the structure of research articles. ‘Mover’ even allows users to give a second opinion on the classification of moves and add new training data (This can lead to a less accurate model if students with less expertise add potentially wrong training data). However, these tools have a common thread and fulfill the following criteria:

  • They look at scientific text – Full research articles, abstracts or introductions. Tools to automate argumentative zoning of other open text (Example) are not considered.
  • They automate the identification of rhetorical structures (zones, moves) in research articles (RA) with sentence being the unit of analysis.
  • They are broadly based on the Argumentative Zoning scheme by Simone Teufel or the CARS model by John Swales (Either the original schema or modified version of it).

Tools (in alphabetical order):

  1. Academic Writing Analytics (AWA) – Summary notes here

AWA also has a reflective parser to give feedback on students’ reflective writing, but the focus of this post is on the analytical parser. AWA demo, video courtesy of Dr. Simon Knight:

  1. Mover – Summary notes here

Available for download as a stand alone application. Sample screenshot below:

antmover

  1. Research Writing Tutor (RWT) – Summary notes here

RWT demo, video courtesy of Dr. Elena Cotos:

  1. SAPIENTA – Summary notes here.

Available for download as a stand alone java application or can be accessed as a web service. Sample screenshot of tagged output from SAPIENTA web service below:

sapienta-outputAnnotation Scheme:

The general aim of the schemes used is to be applicable to all academic writing and this has been successfully tested across data from different disciplines. A comparison of the schemes used by the tools is shown in the below table:

ToolSource & DescriptionAnnotation Scheme
AWAAWA Analytical scheme (Modified from AZ for sentence level parsing)-Summarizing
-Background knowledge
-Contrasting ideas
-Novelty
-Significance
-Surprise
-Open question
-Generalizing
Mover Modified CARS model
-three main moves and further steps
1. Establish a territory
-Claim centrality
-Generalize topics
-Review previous research
2. Establish a niche
-Counter claim
-Indicate a gap
-Raise questions
-Continue a tradition
3. Occupy the niche
-Outline purpose
-Announce research
-Announce findings
-Evaluate research
-Indicate RA structure
RWTModified CARS model
-3 moves, 17 steps
Move 1. Establishing a territory
-1. Claiming centrality
-2. Making topic generalizations
-3. Reviewing previous research
Move 2. Identifying a niche
-4. Indicating a gap
-5. Highlighting a problem
-6. Raising general questions
-7. Proposing general hypotheses
-8. Presenting a justification
Move 3. Addressing the niche
-9. Introducing present research descriptively
-10. Introducing present research purposefully
-11. Presenting research questions
-12. Presenting research hypotheses
-13. Clarifying definitions
-14. Summarizing methods
-15. Announcing principal outcomes
-16. Stating the value of the present research
-17. Outlining the structure of the paper
SAPIENTAfiner grained AZ scheme
-CoreSC scheme with 11 categories in the first layer
-Background (BAC)
-Hypothesis (HYP)
-Motivation (MOT)
-Goal (GOA)
-Object (OBJ)
-Method (MET)
-Model (MOD)
-Experiment (EXP)
-Observation (OBS)
-Result (RES)
-Conclusion (CON)

Method:

The tools are built on different data sets and methods for automating the analysis. Most of them use manually annotated data as a standard for training the model to automatically classify the categories. Details below:

ToolData typeAutomation method
AWAAny research writingNLP rule based - Xerox Incremental Parser (XIP) to annotate rhetorical functions in discourse.
MoverAbstractsSupervised learning - Naïve Bayes classifier with data represented as bag of clusters with location information.
RWTIntroductionsSupervised learning using Support Vector Machine (SVM) with n-dimensional vector representation and n-gram features.
SAPIENTA Full articleSupervised learning using SVM with sentence aspect features and Sequence Labelling using Conditional Random Fields (CRF) for sentence dependencies.

Others:

  • SciPo tool helps students write summaries and introductions for scientific texts in Portuguese.
  • Another tool CARE is a word concordancer used to search for words and moves from research abstracts- Summary notes here.
  • A ML approach considering three different schemes for annotating scientific abstracts (No tool).

If you think I’ve missed a tool which does similar automated tagging in research articles, do let me know so I can include it in my list 🙂