FactShala Media Literacy Initiative in India: An Impact Evaluation

"The spread of both mis- and disinformation, especially via social media, is a common problem in India. Contributory factors include a lack of credible digital media literacy programs and loose regulation of social media platforms."

This report shares the findings of an evaluation of the FactShala media literacy programme in India, which is designed to help individuals from small cities and villages across India navigate online information in multiple languages. The study is intended as a contribution to the growing body of research to establish effective approaches to building media and information literacy globally, as well as a tool to aid the FactShala team in their efforts to improve and evolve the programme (described at Related Summaries, below).

As explained in the report, "Misinformation is an ongoing problem in the global media landscape dominated by digital and social media. More than 600 million Indians use the internet. Over 34 million have come online in the past year alone. Digital literacy and regulation, however, have yet to catch up with this massive growth. People with affordable phone and broadband connections have access to news and information flowing from different digital sources but are not equipped to assess the veracity of claims usually made in these messages."

The FactShala media literacy training programme, which was launched by Internews in collaboration with DataLEADS and with the support of Google.org and the Google News Initiative, seeks to tackle this challenge. From September 2020 to January 2021, 253 trainers delivered the FactShala training programme to 23,347 adults in communities across India - primarily in Tier 2 and 3 cities with low literacy levels and in villages. The training was delivered in more than 10 languages and dialects, mainly through online sessions. Most participants were students, teachers, representatives from non-governmental organisations (NGOs) and civil society organisations (CSOs), and self-employed people. The curriculum was designed based on the findings of an internet user study conducted at the beginning of the initiative. It is comprised of three modules: understanding the information ecosystem; developing critical thinking around information; and fact-checking and verifying information.

The programme evaluation, which was conducted by the Centre for Media Studies (CMS) in July-August 2021, had the following objectives: (i) assess how far FactShala improved trainees' awareness of, and ability to identify, online misinformation and disinformation; (ii) determine how far FactShala motivated trainees to share this knowledge with others; (iii) capture best practices in curbing misinformation and disinformation at the individual, organisational, or community level; and (iv) provide insights to help design or improve future training modules, curricula, and capacity-building sessions.

The research methodology included a mix of quantitative and qualitative methods: (i) quantitative: a quasi-experimental study to assess the impact of FactShala training on participants' knowledge retention and handling of online information several months after the training; (ii) quantitative: pre- and post-training assessments to assess participants' knowledge and ability to critically analyse online information; and (iii) qualitative: methods to collect data on, and insights into, training participants' and other stakeholders' perceptions and experience of online information.

The research results outlined in the report first describe study participants' media habits, smartphone use, occupation, education level and gender. For example, on average, respondents across all quantitative samples reported spending 3.9 hours a day online, with men generally reporting spending more time online than women. WhatsApp is the most used and most trusted online information source across all groups of respondents, followed by Facebook and YouTube.

Key findings related to the impact of the programme are analysed against the programme's five expected learning outcomes:

1. Training participants are aware of the concept and extent of misinformation and disinformation, especially on social media - The results showed that the FactShala programme generated significant awareness about mis- and disinformation among trainees. Going beyond awareness, both the quasi-experimental study and the pre- and post-training assessments showed that respondents who had attended FactShala training had a clearer understanding of both terms, and how they differ.
2. Training participants are introduced to critical thinking in order to create discerning information users who can check supporting evidence - FactShala-trained respondents demonstrated a greater ability to critically assess information than other respondents. After the training, around 70% of respondents stated they would not automatically trust all online information and would question or check its authenticity.
3. Training participants are able to spot misinformation and disinformation and verify information using one or more methods - FactShala-trained respondents were better able than other respondents to identify various types of mis- and disinformation and knew more about how to check it. They also reported applying this knowledge more than other respondents.
4. Training participants are able to protect themselves from harm caused by misinformation and disinformation - Results showed that FactShala-trained respondents demonstrated being significantly more able to protect themselves from negative impacts of inaccurate or fraudulent online information than other respondents. They were also much more likely to have put these skills into practice.
5. Training participants help to curb the spread of misinformation and disinformation by reporting or deleting content, alerting others, or sharing fact-checked information - Both the quantitative and qualitative findings indicate that FactShala-trained respondents were more proactive than other respondents in taking steps to reduce the spread of misleading online content.

To further highlight the impact of the programme, the report includes the results of in-depth interviews with trained respondents. Six stories are shared that describe training participants' ability to use and share their learning to protect themselves and their communities from misinformation and disinformation.

The report concludes with a list of recommendations for the FactShala team to incorporate into the next iteration of the programme:

• Incentivise testimony from trained participants: Trained participants should be encouraged to share the action they have taken to curb or restrict the spread of misinformation. This testimony could guide and motivate social media users to alert fellow citizens about misinformation and fake news.
• Prioritise in-person training: To have more impact, particularly in rural areas, face-to-face training is preferred. It overcomes connectivity challenges and is more effective in sustaining attention and creating opportunities for social learning.
• Facilitate follow-up: To sustain change among participants, further training or follow-up training sessions should be held regularly to generate sustainable practice.
• Expand the focus to news media literacy: It is recommended that future iterations of FactShala have an increased focus on news media literacy - creating awareness of the role of news media organisations in driving the spread of mis- and disinformation.