Tag: learning analytics

LAK25 at Dublin – Publications and Reflections

It was great to reconnect with the Learning Analytics community at LAK25 in Dublin (before coming straight back to a busy teaching semester). LAK15 was the first conference I ever attended, and I have fond memories of this first scholarly experience. I went down memory lane after 10 years, reflecting on how it is probably the reason why I chose to do a PhD and stay in academia. My publications at LAK25 are discussed below, but also check out the full UTS line-up from the Connected Intelligence Centre (CIC) website! 

In the writing analytics workshop I co-organised, our discussions spanned new methods that can inform and use generative AI for writing (tools and analytics), but we also asked some fundamental questions on the nature and evolving role of writing itself.

Rianne Conijn, Antonette Shibani, Laura Allen, Simon Buckingham Shum & Cerstin Mahlow (2025) Writing Analytics in the age of Large Language Models: Expanding horizons for assessing and designing writing with AI. In companion proceedings of the 15th International Conference on Learning Analytics & Knowledge (LAK25) pg 436-439.

Abstracts of the presented papers are on the workshop website, and you can access the organiser slides and the shared notes as well. As someone who values writing quite a bit (I’ve always found academic writing really helpful for my thinking, which enhances my own ideas as they evolve in my head), the diminishing role of writing especially for student assessments worries me (and many other folks I know, we had a great question on it at the workshop). But, having heard Prof. Rupert Wegerif talk about dialogic theory and education servicing technologies (Yes, that’s right!), I’ve come to peace with other ways of thinking. While writing is still important, it may become less essential; and maybe that’s okay. Because people think differently, writing is not the only medium for it (for example, Indigenous Knowledges have been around for centuries in various other forms). For those who prefer writing, it can continue to be of significant value, and I agree, the college essay is not dead (manifesto), we just need better, more relevant ones. I believe that we should still teach students how to write well, especially in finding their voice and being authentic (with text starting to sound very similar everywhere, I’ve personally started craving for more originality in tone of writing; Sounding professional is great, but looks like I could only take so many AI-generated emails)!

I was invited to take part in the full day Grand Challenges workshop, where workshop participants had a lot of fun identifying and voting for the grand challenges that the field of Learning Analytics should tackle in the next decade. Thanks, Sasha for the shoutout on LinkedIn.

The workshop had interesting follow-up events: an interactive poster, and an interactive panel session (keynote 3), and Prof. Simon Buckingham Shum was able to convince most (voted #1 challenge in the workshop), or me atleast, that LA should prioritize the soceity’s most pressing challenges. I’ve asked Simon how exactly we can make this happen (I can draw basic connections between learning analytics empowering learners to tackle skill challenges but I cannot extend it to how it can help save the world in this polycrisis!). He promised that he would write about it in detail, so I’m looking forward to reading it. Sidenote: I was reading Tomitsch (Head of School, TD School, UTS) and Baty’s Designing Tomorrow book and all about the climate challenges during my flight to Dublin, so it made sense – highly recommended read; it completely changed how I view Human-centred design!

Another challenge around defining core principles for LA in the rapidly changing GenAI context was something I contributed to quite a bit (stemming from the many conversations at LAK around futures and past failures of LA). There’s a lot happening around GenAI pretty quickly; its affordances are rapidly improving – but this is all the more the reason to be mindful and cautious in what we research and implement in education. We’ve contributed to some starter discussions in an upcoming editorial for the JLA Special Section on Generative AI and Learning Analytics – look out for this in the coming weeks!

At another workshop From Data to Discovery: LLMs for Qualitative Analysis in Education, Lisa did a fabulous job of presenting our team’s work (unfortunately, it conflicted with my writing analytics workshop, so I couldn’t make it). The title is self-explanatory, and the full paper has all the details:

Aneesha Bakharia, Antonette Shibani, Lisa-Angelique Lim, Trish McCluskey and Simon Buckingham Shum (2025) From Transcripts to Themes: A Trustworthy Workflow for Qualitative Analysis Using Large Language Models. In CEUR proceedings of the workshop ‘From Data to Discovery: LLMs for Qualitative Analysis in Education’. Companion proceedings of the 15th International Learning Analytics & Knowledge Conference (LAK ’25).

There’s another detailed blog post on this work, so I’ll add its link here:

Theme Explorer: LLM-augmented Inductive Coding

We also presented a poster that expands analytics on writing for potential feedback on AI-human collaboration (student work, extended from CoauthorViz):

Antonette Shibani, Vishal Raj and Simon Buckingham Shum (2025). Towards Analytics for Self-regulated Human-AI Collaboration in Writing. In Companion proceedings of the 15th International Learning Analytics & Knowledge Conference (LAK25) pg 120-122.

There are ongoing discussions at LAK and beyond on the use of the term “collaboration” to denote human-AI partnerships (and “partnerships” as a term, too but we mean partners in cognition like this 1991 paper). Various disciplines have always interpreted terminologies differently (especially computer science, which makes loose analogies with human psychology – e.g., hallucinations), and it is an open question if we can even agree on how we must perceive terms. Maybe we need better definitions for what we mean by regulation, hybrid, augmenting, or just human-AI interaction? More on this is likely to come later!

I also chaired two sessions at LAK (one where the best papers from sister conferences were invited to present, and another on learning design). The Q&A parts were super interesting there and throughout LAK. We could really see the diversity of the audience’s backgrounds here – the constant switch between statistics, practitioners, LA, and LLMs is mind-boggling!

And more photo dumps because… why not 🙂

Team UTS

The Long Room

Book of Kells Experience

CHI’24 research publications

I attended the prestigious Human-Computer Interaction (HCI) conference CHI’24 at Hawaii, Honolulu in May 2024. While I’m quite familiar with the field of HCI, it was my first time attending the conference because it is much wider than my main area of research (Learning Analytics, AI in education, and Writing Analytics). The sheer scale of the conference (~2K to 3K attendees) and the broad range of topics it covers (check out this full program) is almost impossible to fully grasp!

TLDR; Go to the end for the list of paper from CHI’24.

My personal highlight was the Intelligent Writing Assistants Workshop, which was running for the third time at CHI, organized by a bunch of fun people who are all super keen about researching the use of AI to assist writing. Picture from our workshop below (thanks, Theimo, for the LinkedIn post)

Pictured: Participants of the Intelligent Writing Assistants CHI’24 workshop at the end of the session

The workshop had many mini presentations on the overall theme of Dark Sides: Envisioning, Understanding, and Preventing Harmful Effects of Writing Assistants. I presented my work with Prof. Simon Buckingham Shum on AI-Assisted Writing in Education: Ecosystem Risks and Mitigations, where we examined key factors (in the broader socio-technical ecosystem which are often hidden) that need consideration for implementing AI writing assistants at scale in educational contexts.

This was actually a deep dive into the Ecosystem aspect of a larger piece of work we presented at CHI on A Design Space for Intelligent and Interactive Writing Assistants. The full design space from our full paper mapped the space of intelligent writing assistants reviewing 115 papers from HCI and NLP, with a team of 36 authors, led by Mina Lee.

Figure: Design space for intelligent and interactive writing assistants consisting of five key aspects—task, user, technology, interaction, and ecosystem from our full paper.

An interactive tool is also presented to explore the literature in detail.

I also had a late-breaking work poster presentation on Critical Interaction with AI on Written Assessment (I have a seperate post about it!) where we explored how students engaged with generative AI tools like ChatGPT for their writing tasks, and if they were able to navigate this interaction critically.

A cherished memory to hold on to was also the time I spent with my friend Vanessa, who is currently a Research Fellow at Monash university during this trip in Hawaii. Vanessa and I started our PhD together at th Connected Intelligence Centre at UTS ~8 years ago, and it was really nice to catch up after a long time (along with few others). I had also just visited Monash university’s CoLAM a week before for a talk and meeting fellow Learning Analytics researchers, hosted by her and Roberto. The group do interesting work in Learning Analytics that is worth checking out.

6 years apart… On the left: Vanessa and I in 2018 while attending AIED/ ICLS 2018 in London; On the right: Us while attending CHI in 2024 in Hawaii.

TLDR -> Research publications:

Here are all the papers from the work we presented at CHI’24:

Mina Lee, Katy Ilonka Gero, John Joon Young Chung, Simon Buckingham Shum, Vipul Raheja, Hua Shen, Subhashini Venugopalan, Thiemo Wambsganss, David Zhou, Emad A. Alghamdi, Tal August, Avinash Bhat, Madiha Zahrah Choksi, Senjuti Dutta, Jin L.C. Guo, Md Naimul Hoque, Yewon Kim, Simon Knight, Seyed Parsa Neshaei, Agnia Sergeyuk, Antonette Shibani, Disha Shrivastava, Lila Shroff, Jessi Stark, Sarah Sterman, Sitong Wang, Antoine Bosselut, Daniel Buschek, Joseph Chee Chang, Sherol Chen, Max Kreminski, Joonsuk Park, and Roy Pea, Eugenia H. Rho, Shannon Zejiang Shen, Pao Siangliulue. 2024. A Design Space for Intelligent and Interactive Writing Assistants. In Proceedings
of the CHI Conference on Human Factors in Computing Systems (CHI ’24),
May 11–16, 2024, Honolulu, HI, USA. ACM, New York, NY, USA, 33 pages.
https://doi.org/10.1145/3613904.3642697

Antonette Shibani, Simon Knight, Kirsty Kitto, Ajanie Karunanayake, Simon Buckingham Shum (2024). Untangling Critical Interaction with AI in Students’ Written Assessment. Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI ’24), May 11-16, 2024, Honolulu, HI, USA. doi.org/10.1145/3613905.3651083

Antonette Shibani & Simon Buckingham Shum (2024). AI-Assisted Writing in Education: Ecosystem Risks and Mitigations. In The Third Workshop on Intelligent and Interactive Writing Assistants @ CHI ’24, Honolulu, HI, USA. https://arxiv.org/abs/2404.10281

New Research Publications in Learning Analytics

Three of my journal articles got published recently, two on learning analytics/ writing analytics implementations [Learning Analytics Special Issue in The Internet and Higher Education journal], and one on a text analysis method [Educational Technology Research and Development journal]. that I worked on earlier (many years ago in fact, which just got published!).

Article 1: Educator Perspectives on Learning Analytics in Classroom Practice

The first one is predominantly qualitative in nature, based on instructor interviews of their experiences in using Learning Analytics tools such as the automated Writing feedback tool AcaWriter. It provides a practical account of implementing learning analytics in authentic classroom practice from the voices of educators. Details below:

Abstract: Failing to understand the perspectives of educators, and the constraints under which they work, is a hallmark of many educational technology innovations’ failure to achieve usage in authentic contexts, and sustained adoption. Learning Analytics (LA) is no exception, and there are increasingly recognised policy and implementation challenges in higher education for educators to integrate LA into their teaching. This paper contributes a detailed analysis of interviews with educators who introduced an automated writing feedback tool in their classrooms (triangulated with student and tutor survey data), over the course of a three-year collaboration with researchers, spanning six semesters’ teaching. It explains educators’ motivations, implementation strategies, outcomes, and challenges when using LA in authentic practice. The paper foregrounds the views of educators to support cross-fertilization between LA research and practice, and discusses the importance of cultivating educators’ and students’ agency when introducing novel, student-facing LA tools.

Keywords: learning analytics; writing analytics; participatory research; design research; implementation; educator

Citation and article link: Antonette Shibani, Simon Knight and Simon Buckingham Shum (2020). Educator Perspectives on Learning Analytics in Classroom Practice [Author manuscript]. The Internet and Higher Education. https://doi.org/10.1016/j.iheduc.2020.100730. [Publisher’s free download link valid until 8 May 2020].

Article 2: Implementing Learning Analytics for Learning Impact: Taking Tools to Task

The second one led by Simon Knight provides a broader framing for how we define impact in learning analytics. It defines a model addressing the key challenges in LA implementations based on our writing analytics example. Details below:

Abstract: Learning analytics has the potential to impact student learning, at scale. Embedded in that claim are a set of assumptions and tensions around the nature of scale, impact on student learning, and the scope of infrastructure encompassed by ‘learning analytics’ as a socio-technical field. Drawing on our design experience of developing learning analytics and inducting others into its use, we present a model that we have used to address five key challenges we have encountered. In developing this model, we recommend: A focus on impact on learning through augmentation of existing practice; the centrality of tasks in implementing learning analytics for impact on learning; the commensurate centrality of learning in evaluating learning analytics; inclusion of co-design approaches in implementing learning analytics across sites; and an attention to both social and technical infrastructure.

Keywords: learning analytics, implementation, educational technology, learning design

Citation and article link:  Simon Knight, Andrew Gibson and Antonette Shibani (2020). Implementing Learning Analytics for Learning Impact: Taking Tools to Task. The Internet and Higher Education. https://doi.org/10.1016/j.iheduc.2020.100729.

Article 3: Identifying patterns in students’ scientific argumentation: content analysis through text mining using LDA

The third one led by Wanli Xing discusses the use of Latent Dirichlet Allocation, a text mining method to study argumentation patterns in student writing (in an unsupervised way). Details below:

Abstract: Constructing scientific arguments is an important practice for students because it helps them to make sense of data using scientific knowledge and within the conceptual and experimental boundaries of an investigation. In this study, we used a text mining method called Latent Dirichlet Allocation (LDA) to identify underlying patterns in students written scientific arguments about a complex scientific phenomenon called Albedo Effect. We further examined how identified patterns compare to existing frameworks related to explaining evidence to support claims and attributing sources of uncertainty. LDA was applied to electronically stored arguments written by 2472 students and concerning how decreases in sea ice affect global temperatures. The results indicated that each content topic identified in the explanations by the LDA— “data only,” “reasoning only,” “data and reasoning combined,” “wrong reasoning types,” and “restatement of the claim”—could be interpreted using the claim–evidence–reasoning framework. Similarly, each topic identified in the students’ uncertainty attributions— “self-evaluations,” “personal sources related to knowledge and experience,” and “scientific sources related to reasoning and data”—could be interpreted using the taxonomy of uncertainty attribution. These results indicate that LDA can serve as a tool for content analysis that can discover semantic patterns in students’ scientific argumentation in particular science domains and facilitate teachers’ providing help to students.

Keywords: text mining, latent dirichlet allocation, educational data mining, scientific argumentation

Citation and article link:  Wanli Xing, Hee-Sun Lee and Antonette Shibani (2020). Identifying patterns in students’ scientific argumentation: content analysis through text mining using Latent Dirichlet Allocation. Educational Technology Research and Development. https://doi.org/10.1007/s11423-020-09761-w.

London Festival of Learning 2018

I attended the London Festival of Learning this year from June 22nd-30th, which brought together three conferences: the 13th International Conference of the Learning Sciences (ICLS), the Fifth Annual ACM Conference on Learning at Scale (L@S) and the 19th International Conference on Artificial Intelligence in Education (AIED).  It was great to see the convergence of ideas and academics from these three fields that generally work towards enhancing educational practices with technology. I could see overlaps and similarities in the topics of research being studied by these communities, but I also noticed they were divergent in terms of the main foci of their research. The festival was huge with over a 1000 attendees, and also involved edtech companies that wanted to develop evidence-informed products.

Throughout the conferences, I found an emphasis and move towards making more use of human ability and intelligence to augment what artificial intelligence can do for education in many keynotes and talks. This included concepts like giving importance to our internally persuasive voice and the power of negotiation in addition to “datafied” learning, and embracing imperfections from machines by adding in human context. A critical stance on what Artificial Intelligence can and cannot do was seen, with more conversations happening around the ethical use of learner’s data.

(Excuse me for the blurry pictures, I was not in a good spot to take pictures)

In the sessions, I could see a lot of research on developing intelligent tutoring systems, agents, intervention designs and adaptive learning systems for teaching specific skills, and advances made in their techniques. The majority of data comes from online settings i.e, students’ trace data from their usage with such systems. Recently, multi-modal data is getting more attention where sensors and wearables collect data from learner’s physical spaces as well. One best paper award winning work on Teacher-AI hybrid systems showcased the power of mixed-reality systems for real-time classroom orchestration. The cross-over session and the ALLIANCE best paper session showcased interesting research cutting across the three communities; it’s a shame we couldn’t attend both sessions since they ran in parallel.

Simon Knight presented our work on Augmenting Formative Writing Assessment with Learning Analytics: A Design Abstraction Approach at the cross-over session where he explained how we can augment existing good practices with learning analytics, and use design representations for standardizing these learning designs. I presented our poster on studying the revision process in writing in AIED, where I used snapshots of students’ writing data to study their drafting process at certain time intervals. I also participated in the collaborative writing workshop earlier in ICLS where many interesting tools to support writing were discussed. I shared about AcaWriter – a writing analytics tool providing automated feedback on rhetorical moves, developed by the Connected Intelligence Centre, UTS  which is now released open-source.

Overall, it was a great place to learn, network and follow work from related disciplines (with some catching up to do on the presented work, coz we can only be at one place at one time during the parallel sessions). I did feel a bit exhausted a the end of it (maybe I’m better off attending one conference at a time 🙂 ), but I guess that’s natural, and you can’t complain when your brain gets so much to learn in a week!

LAK 2018 in Sydney

This post is on the exciting week of the Learning Analytics and Knowledge Conference LAK 2018, held in Sydney. LAK is a prestigious conference dedicated for sharing work in Learning Analytics across the globe. LAK coming down under was something we were looking forward to for quite some time. LAK is in fact the very first international conference I’ve ever attended (back in 2015), so it is always extra special 🙂

I started off with a Writing Analytics workshop, which we organized in Day 1 of LAK. We used a Jupyter notebook which runs Python code to demonstrate the application of text analysis for writing feedback and the pedagogic constructs behind designing such applications for learning analytics. Our aim was to bridge the gap between pedagogic contexts and the technical infrastructure (analytics) by crafting meaningful feedback for students on their writing, and to do so by developing writing analytics literacy. The participants were quite engaged in this hands on approach and we had good discussion on the implications of such Writing Analytics techniques.

The next day, I participated in the Doctoral Consortium, which is a whole day workshop where doctoral students present their work, discuss and receive feedback on their work from experts and other students. To know more about a Doctoral Consortium, read this. My doctoral consortium paper published in the companion proceedings is available here:

The new workshop for school practitioners was of interest to many educators working in K-12 learning analytics applications, and the Hackathon continues to be of wide interest. After the pre-conference events, the main conference officially started with the first keynote by Prof. David Williamson Shaffer on ‘The Importance of Meaning: Going Beyond Mixed Methods to Turn Big Data into Real Understanding’. David talked about how data is not scarce anymore, and to analyze such a sheer volume of data for learning, how we have to go beyond traditional quantitative and qualitative approaches. He gave examples of logical fallacies where statistics is likely to be misused while interpreting the concepts in learning, and introduced the notion of quantitative ethnography which can close the interpretive gap between the model and the data.

If you want to hear the full talk, all the keynotes are available along with the slides here: https://latte-analytics.sydney.edu.au/keynotes/ 

In general, there was great interest in the development of theories around designing dashboards, discussing how to and how not to develop dashboards for students.

Aligning learning analytics with learning design was increasingly emphasized. The demo paper which I presented that day exemplifying this in a Writing Analytics context is here (bonus pic with the supervisors):

The second day of the main conference (aptly on International women’s day) started with Prof. Christina Conati’s keynote on user adaptive visualizations, where she talked about adaptive interactions.

She showed how visualizations can be personalized for users by building user models based on eye tracking features.

Visualization in general was another key topic which gathered growing interest in the LAK community, along with other topics like Discourse analysis and Writing Analytics, many of them moving towards more near real-time applications.

I attended the SOLAR executive meeting for the first time to see what’s happening around SOLAR. It felt great to be part of a very welcoming community of researchers and practitioners. That’s where they announced this:

We also celebrated Women’s day:

It was quite an eventful day ending with the conference banquet in a Sydney harbour cruise.

The final keynote on the last day touched upon a number of criticisms around learning analytics and how we can progress the field further taking into account the key aims of learning analytics.

Multi modal learning analytics, MOOCS, Ethics and Policies, Theories, Self-regulated learning and Co-designing with stakeholders are other areas which continued to be discussed throughout the conference.

And then to wrap it up, happy hour!

To read all the interesting papers from LAK, follow this link.

For more tweets from the awesome LAK community, check #LAK18, #LAK2018, @lak2018syd

Note: Initially created as a private post for my own reference notes, this post was later made publicly available from 23 May 2018.

 

Notes: XIP – Automated rhetorical parsing of scientific metadiscourse

Reference: Simsek, D., Buckingham Shum, S., Sandor, A., De Liddo, A., & Ferguson, R. (2013). XIP Dashboard: visual analytics from automated rhetorical parsing of scientific metadiscourse. In: 1st International Workshop on Discourse-Centric Learning Analytics, 8 Apr 2013, Leuven, Belgium.

Background:

Learners should have the ability to critically evaluate research articles and be able to identify the claims and ideas in scientific literature.

Purpose:

  • Automating analysis of research articles to identify evolution of ideas and findings.
  • Describing the Xerox Incremental Parser (XIP) which identifies rhetorically significant structures from research text.
  • Designing a visual analytics dashboard to provide overviews of the student corpus.

Method:

  • Argumentative Zoning (AZ) to annotate moves in research articles by Simone Teufel.
  • Rhetorical moves tagged by XIP – partly overlap and partly different from AZ scheme: SUMMARIZING, BACKGROUND KNOWLEDGE, CONTRASTING IDEAS, NOVELTY, SIGNIFICANCE, SURPRISE, OPEN QUESTION, GENERALIZING
  • Sample discourse moves:
    • Summarizing: “The purpose of this article….”
    • Contrasting ideas: “With an absence of detailed work…”
      • Sub-classes: novelty, surprise, importance, emerging issue, open question
  • XIP outputs a raw output file containing semantic tags and concepts extracted from text.
  • Data: Papers from LAK & EDM conferences and journal – 66 LAK and 239 EDM papers extracting 7847 sentences and 40163 concepts.
  • Dashboard design – Refer original paper to see the process involved in prototyping the visualizations.

Tool:

  • XIP is now embedded in the Academic Writing Analytics (AWA) tool by UTS. AWA provides analytical and reflective reports on students’ writing.