Why don't you tell me what I need to know? Self-flipped classroom and students' personal epistemology.

-This is a full research paper addressing the crucial element of understanding students when creating learning envi­ ronments. It is for instance important to be aware of how students appreciate our way of teaching and to consider consequences of students not understanding or plainly disliking a setup. This paper addresses the negative experiences of students in a peer­ learning environment named the self-flipped classroom. Through the lens of a theory of personal epistemology we investigate course evaluation reports and observations from anonymous students. Results indicate that the personal epistemology framework indeed gives some answers to students disliking the self-flipped aspect of the course, and that some students would rather be told what to learn in detail as in the quote: "why don't you tell me what I need to know?". Finally the paper presents some ideas on ways forward.


INTRODUCTION
Three years ago a new course was launched within the Department of IT at the University of Uppsala by a team of computing education researchers together with the teachers' team. The course itself was inspired by recent ideas in the field of computing education research and underpinned by ideas of peer-learning, student activated learning and the self flipped classroom. Building on these pedagogical approaches, many of which had been proved to be successful in other circumstances, we were sure that we had created an effective learning environment, addressing the interesting area of how to design, implement, maintain and procure complex IT systems in large organisations. Unfortunately, the responses from the student cohorts showed a significant split in their opinions of the course. When answering the final course evaluation some described the overall impression of the course as very good, whereas some students disliked it and had a very negative overall opinion. Not one student responded with an "average" assessment; in short, the course seemed to be a "love it or hate it" course, with substantially polarised student evaluation. The students' reactions to the course has made us curious, and in this paper we investigate the negative reactions from the 2020 cohort through the theoretical lens of personal epistemology.
This split in opinions of the course might not be so surpris ing, since it is a well known phenomena that students do not always appreciate innovative teaching approaches involving their own participation and contribution [1]. Recent examples include programming courses requiring active participation [2] and cooperation with industry in courses [3]. The importance of understanding the student perspective is perhaps especially high in cases where the educational settings are innovative and differ from what the students are traditionally used to.
The paper explores the self-flipped classroom learning en vironment, which has recently emerged as a concept in the computing education research area [4]- (6]. We will base our findings on five different formative and summative student evaluations of the recent course, as well as student comments to teachers while it was being delivered.
One aspect of student psychology that may provide insight into their reactions to the course and go some way to explain the range of learner reactions to the self-flipped classroom is the personal epistemology [7] of the individual students. Personal epistemology is the study of those elements of the learner's perception of knowledge, its justification and its boundaries, that have an impact on the acquisition of knowledge, skills and competencies in general. It can be considered to be "the subjective counterpart of philosophical epistemology" which investigates how the ideas found in the philosophical discipline of epistemology affect the individual at a psychological level (8]. It may be considered to have originated in the developmental psychology of Piaget [9], and its application to tertiary education began in the mid-l 960s with the work of Perry (10]. It is worth noting that, while many researchers acknowledge the name of the subject to be slightly incongruous when describing the field, it has nevertheless been retained by the academic community as a general term for the focus of epistemic and ontological considerations which have an impact on learning.
The research question addressed in this paper concerns the value of using the theory of personal epistemology as a lens to understand attitudes to attitudes to learning environments. Specifically, to what extent do the personal epistemological beliefs of students in a self-flipped classroom moderate the learning of students and contribute to their perception of value for the learning experience. In terms of contribution, the paper seeks to understand the role and importance of these affective or dispositional aspects of student perception when developing new type of learning environments which are heavily dependent on peer learning.

II. BACKGROUND
We will present the idea of the self-flipped classroom in this section followed by a presentation of relevant theories on personal epistemology.

A. The Self-Flipped Classroom Concept
The self-flipped classroom approach was developed as a blend of several pedagogical approaches, most notably con tributing student pedagogy (11] with an emphasis on student creation of multimedia artefacts, and the flipped classroom (12]. Note that the word 'self' in the name 'self-flipped' refers to the use, and reuse, of student-created materials for peer teaching, while the second part of the name, 'flip', refers to the mode of that teaching -the flipped classroom.
The first principle of self-flipped classroom -contributing student pedagogy -advocates learning through engaging stu dents as co-creators of learning resources [ 13]. The defining characteristic of contributing student pedagogy is the explicit creation by students of identifiable artefacts which can be used by other students to promote learning. The advantages of the creation of such teaching materials can be understood from a constructionist perspective (14] and by using a learning by-teaching approach (15]. The theory of constructionism posits that the effectiveness of subject learning is enhanced by activities in which students create tangible and shareable artefacts that they believe to be meaningful. Learning by teaching reinforces this by requiring students to rethink the newly acquired knowledge in order to communicate it to others. Empirical studies have demonstrated numerous benefits of taking part in activities centred around artefact creation, including higher level of thinking and deeper learning (16], (17], as well as development of various 21st century skills such as creativity and critical thinking (18], communication and teamwork (19].
The second principle of self-flipped classroom -the flipped classroom -is a form of learning where course material is delivered to students in the form of audio-video recordings and reading materials via digital and online media (12]. This has also been extensively reported as beneficial for teaching various subjects. Such benefits include the fact that students can learn at their own pace and use classroom time for more creative activities rather than the reception of content. This provides students with opportunities for creativity and allows instructors to spend more time with students on solving authentic problems (20].
Building on these tested pedagogical approaches, the self flipped classroom promises to be an effective learning envi ronment. Recent examples of the application of self-flipped classroom in computer science education have demonstrated great potential with this approach allowing students to increase motivation and engagement (21], enhance learning experiences [5], promote better content understanding and develop multiple lifelong learning skills [ 4], [ 6], (22].

B. Personal Epistemology
The role played in the learning process by a student's perception of the nature of knowledge, its justification, and the ways in which it can be acquired, has been an active research area in applied philosophy and educational psychology since the late 1960s. During that time, a significant amount of research has concluded that epistemic beliefs, that is, be liefs about knowledge, play an important part in moderating learning dispositions, and that this significantly impacts the educational process. There is evidence that such epistemic con siderations play an important role in the processes by which students become self-regulated learners. Moreover, there is evidence that a more "sophisticated" set of epistemic beliefs, together with a wider set of ontological commitments (i.e. what students perceive as valid objects of study within the subject area) are often correlated with higher-order learning outcomes, e.g. depth of scientific understanding (23]. Research links such understanding with a range of academic outcomes including academic achievement (24], and it is claimed that epistemic factors play an important role in the solution of ill-structured problems [3], (25]. Such work would therefore suggest that there are good arguments for treating epistemic concerns as important when trying to understand educational development across a range of different domains.
This interest in personal epistemology, or what should perhaps more precisely be termed epistemic cognition (i.e. the cognitive process related to knowledge, its nature, acquisition and justification) can be seen in work done by educators in a variety of subjects, ranging from science and engineering (26] to music theory [27]. Such work has increased in recent years and there is now a significant and growing body of research that suggests that the topic has important implications for teaching within a discipline. For example, work done by Dohn (28] on the epistemological presuppositions of reflective activities has led to a greater understanding of reflective activities as essentially situated and context-dependent. a) History of the Concept: From a historical perspective, questions about the nature of knowledge, the justification and evaluation of claims to such knowledge and the boundaries of what can be known, have always been a fundamental subject of investigation within the western philosophical tradition, as well as its counterparts elsewhere. In the 20th century, the extension of this study within the context of the learning sciences was first motivated by the developmental psychology of Piaget and was subsequently applied to university education with the work of Perry (10] in the 1960s, and developed further under the guise of "personal epistemology" in the 1990s, e.g. see [7], (29].
One issue that presents itself almost immediately on enter ing this field is a lack of both clarity and precision in the terminology used to describe basic elements of the subject. Briell et al, in their review (30], describe a dozen frequently employed terms and almost thirty less-frequent synonyms for the general concept. The more popular terminology includes such descriptors as "personal epistemology", "epistemic be liefs", "reflective judgement" and "ways of knowing". While we use the most common term "personal epistemology" as a general description for the field of study, it has been pointed out by Greene (31] that both this term and its common alter native, "epistemological beliefs", should more meaningfully refer to a belief that an individual has about the study of knowledge and the process of knowing (i.e. epistemology). This is not actually what work in this areas actually does; it is more common to focus on beliefs about knowledge itself rather than beliefs about the theory of knowledge. This has led to the suggestion that a more appropriate title for the area of study should be something like "epistemic beliefs". Unfortunately, even this term fails to adequately capture the central area of study since much of the work done tries to address not just questions concerning beliefs about knowledge but questions about the processes by which those beliefs about knowledge are formed. Such processes then influence learning through activities as the articulation of learning tasks, the selection of strategies for solution, and metacognitive self regulation (32]. Indeed, it is precisely these elements which are most important from a pedagogical perspective. Because of this, Kitchener (33] introduced the term "epistemic cognition" to cover both knowledge and the processes involved in its definition, acquisition and use. Unfortunately, as an informal, catch-all term, the phrase "personal epistemology" appears still to be fairly widespread within the literature, and, despite the objections noted above, is used in this paper as a convenient short-hand to describe this general field of study. b) Models of Personal Epistemology: The literature on personal epistemology provides two basic operational models for the concept. Some researchers e.g.
[34]-(36] propose that epistemic development occurs in a one-dimensional framework characterised by a series of general stages, from what are considered to be relatively unsophisticated "objectivist" per ceptions of knowledge (i.e. having a dualistic understanding of a subject based on true/false propositions) to more nu anced, complex, "evaluatist" conceptions in which knowledge depends both on individual cognitive processes (such as the subjective interpretation of events or facts) and contextual judgement. In contrast, other researchers, such as Hofer et al. [8], (29] and Schommer-Aikins (37], propose multidimen sional models in which development occurs at different times and at different rates along a number of quasi-independent "dimensions". Nevertheless, development is also characterised by a progression from positions generally described in the same kinds of language as the unidimensional model, i.e. from nai"ve, objectivist, views to more sophisticated beliefs based on judgement and interpretation. Within such models, there is a range of proposals as to the degree to which development along different dimensions is correlated or can happen independently.
One example of a multidimensional belief model is that proposed by Schommer(-Aikins) [37]-(39]. Here, personal epistemology is seen as a multidimensional belief system, the dimensions of which may be only weakly bound to each other. She retained the idea of a developmental continuum in which learners progressed from "nai"ve" views to more "sophisticated" ones, but applied it to each of the epistemo logical dimensions, in each of which development could occur at different rates. Epistemic development was therefore better described by a trajectory in a multidimensional space rather than by a linear progression.
Schommer-Aikins' model itself developed over time, both in the identification of the dimensions, and indeed, their number, but her multidimensional system (39] which gave rise to her influential Epistemological Beliefs Questionnaire, recognised five dimensions.
• The first of these can be termed the structure of knowl edge and refers to how students think about the structure, relationship and organisation of knowledge in a particular domain. The developmental continuum for this dimension ranges from lower levels in which knowledge is seen as discrete, unambiguous parcels of information, through to an idea of knowledge as a highly integrated and interdependent set of concepts. • A second dimension was stability of knowledge. This tracked how students think about the contingency of knowledge and the way theories may change over time.
At the lower end, knowledge was seen as made up of el ements which were clearly established and not subject to change. The contrasts with more developed understanding in which knowledge is seen as contingent and subject to possible revision and change. • A third dimension, the source of knowledge, recorded where students think domain knowledge originates. At the lower end, it may be seen as being handed down by some authority, whereas its more developed notion sees knowledge as being derived from empirical evidence and more critical reasoning. • The fourth dimension was the speed of learning, and recorded how students think about the rate at which they acquire knowledge. This ranged from a dualistic, naYve view in which an individual either had, or did not have, an understanding of some subject element, and so learning occurred quickly or not at all, to a more "sophisticated" view of learning as a gradual, cumulative process. • Finally, Schommer-Aikins proposed a fifth dimension, that of control of learning ( often termed the ability to learn) which tracked the degree to which students think they themselves control the acquisition of knowledge. This ranged from a view that the ability to learn is fixed at birth to a view that it can be significantly improved over time. The first three dimensions of Schommer-Aikins' model were influenced by Perry's original work. The speed of learning dimension was based on the work of Schoenfield [40], and the control of knowledge dimension was influenced by Dweck's work on implicit intelligence [41].
Other multidimensional models have been proposed, with varying numbers of dimensions, e.g.
[29], [30] and char acterised by slightly different categories, but each retains a family resemblance to the Schommer-Aikins model in that they preserve the notion that epistemic development consists in the movement from a position characterised by true/false dualism, to one of contextual relativism.
The problems with the definition of the central construct, as well as with disagreements on the nature of the process of epistemic growth and development, is also reflected in the lack of firm quantitative evidence for the reliability of individual models.
Because of this, Greene [42], drawing on the tradition of philosophical epistemology with its focus on the key process of justification of belief, called attention to the fact that learning often involves making ontological commitments to structural components of the subject, commitments which develop in sophistication over time. A student may start with an acknowledgement of certain facts about a subject domain but then admit the introduction of interpretational elements such as judgements, which expand the ontological categories that define their understanding of what knowledge is. Consequently, the factors in the multidimensional models that purport to track the nature of knowledge should, instead be treated as demonstrating varying beliefs about the categories of knowledge elements and knowledge claims in academic domains. Thus, within a disciplinary domain, there is not so much a movement from nai"ve to sophisticated beliefs, but rather an expansion of the ontological categories that counts as knowledge. Greene at al argued that, for example, belief in simple dualistic conceptions of knowledge actually indicated a simplistic ontology with limited or unhelpful categories. The development of different perspectives on the stability of knowledge (static/unchanging to contingent) indicates an expansion of the learners ontology from one that deals with knowledge claims within some categories which are rela tively static, to one in which claims in other categories may potentially change over time. Learning, therefore involves both epistemic and ontological development (epistemic and ontological cognition, in the terminology of Greene et al,[43]) Kuhn [44], indicated some practical implications of this kjnd of analysis for teaching. Consider the development of high-level metacognitive skjlls such as those associated with critical thinking. At early stages of epistemic cognition, claims about knowledge are seen as facts which are either correct or incorrect. Within an ontology in which facts are the primary, if not only, kind of item, a process such as critical thinking is perceived to be a straightforward evaluation of a comparison between such statements and reality, in order to determine their truth or falsity. At this stage, interpretations of facts or asser tions about the theory in which the facts sit, are considered to be mere opinions, none of which is more compelling than any other, and so any one of which may be selected based on personal preference. Critical thinking is largely irrelevant at this stage as justification, over and above comparison of propositions to established facts about reality, is limited to the statement of subjective views. However, when the ontology expands to include categories of interpretation and judgement, evaluation of assertions about facts and interpretations of theory are considered to be knowledge elements that can be appraised by argument and with reference to evidence. As a consequence, it is primarily at this stage that critical thinking, seen as a method for promoting coherent, logical argument, will be considered useful.
One major difficulty with the study of personal epistemol ogy is that we do not observe these beliefs directly but only infer them from behaviour. Consequently, these dimensions are conceptual constructs which are hidden from direct ob servation. Moreover, several behaviour patterns could result from the same belief. For example, according to Schommer Aikins [45], if one considers the dimension for "structure of knowledge", the naive view is that knowledge is essentially simple, e.g. reducible to truth values of propositions about facts which are the only things with ontological status. At this level, complexity, if it arises, is due to a lack of an appropriately thorough analysis of the situation rather than any inherent conceptual ambiguity in the information or the interrelationships involved. If a person held this view, there may well be a tendency to oversimplify complex information. This in turn might manifest itself in two ways: they could tend to focus on one aspect of the problem and neglect others, or else they could artificially reduce the complexity of the relationships between the constituents of the problem by a process of inappropriate abstraction or compartmentalisation. The views about structure of knowledge therefore give rise to two different subsets of observable behaviour. Reconstructing the underlying epistemic viewpoint would be difficult but something that could, in principle, be done through appropriate documentation and analysis. Such a reconstruction, if carried out comprehensively, would then enable one to infer the student's epistemic and ontological framework which gave rise to the behaviour. This is obviously a difficult task but it is the thinking behind the various personal epistemology inventories that were developed in the 1990s and early 2000s. These tried to use a variety of statistical techniques (exploratory and confirmatory factor analysis, principal component analysis, etc) to gauge levels of personal epistemological development in students. They consisted of surveys consisting of various epistemological statements on which students were invited to rate agreement using Likert-scale responses. While these had some success, there were recurring reliability issues with the levels of internal consistency of such questionnaires (as measured by, say, the Cronbach alpha statistic). In addi tion, there were concerns that the statements made in the questionnaires failed to capture nuances in discipline-specific investigations where epistemic categories and methods might differ considerably, e.g. history and physics.
In what follows, we try to plot a middle way by using a qual itative analysis inspired by the dimensional models without resorting to the quantitative instruments that were developed. Specifically, we use the dimensions from Schommer-Aikins questionnaire as a basis for the epistemic categories used in the present analysis but seek evidence for epistemic development in these categories using the qualitative analysis of student comments.

Ill. THE COURSE
The course discussed here had run for three years in the "Master in Information Technology" program as a manda tory course for an engineering specialization called "Human, Machine and Society". It is delivered at the end of the first year of a two-year master degree in Computing and IT. The course corresponds to 116th of the expected study load for one semester. A typical enrollment of the course is 45-65 students, with the 2020 cohort comprising 45 students.

A. The Course Setup
The course was specifically designed to emphasize active, student-centered learning (46] with elements of peer-learning [47], (48], co-creation of videos (49] and the flipped classroom [ 12]. The learning objectives of the course were related to development, acquisition, implementation and maintenance in the life cycle of complex IT systems in large organizations. The main learning outcomes of the course were: • the ability to describe potential challenges that arise in connection with the development and introduction of IT systems in large organizations, as well as appropriate methods for addressing them; • the ability to describe challenges and problems that occur during acquisition or development of systems aimed at different groups of users and methods to deal with them; • the ability to propose an appropriate solution for a given problem situation, as well as the ability to discuss ad vantages, disadvantages and applicability of the proposed solution. It should be noted that these learning objectives are given as a legal frame in the National setting and that the course uses teaching strategies to deal with the wicked problem [50] of developing, acquiring, implementing and maintaining complex IT systems in large organisations.
The course's teaching materials included reading a popular novel about software engineering problems, "The Phoenix Project: A novel about IT, DevOps and helping your business win" [51] and scientific articles in the field. The course was delivered in a self-flipped classroom style, based on artefact creation exercises and active learning seminars. The course assessment comprised a combination of group project and individual exam.

B. Self-Flipped Classroom Aspects
The self-flipped classroom element of the course involved students working on a group project assignment and creating multimedia materials that were used for peer-teaching. The assignment was to conduct an interview with a selected IT expert and, based on the received answers, create a rich but concise description of the expert's work. This description had to be in the form of a short instructional video and a text report which would be suitable for other students to learn from. Each group, comprising three to four students, were given a particular aspect of the IT system life cycle (e.g. development, procurement, implementation or maintenance), as well as a particular type of organisation where the system was in use (e.g. healthcare services, a commercial organisation, or public authority). Then, based on the selected combina tion (e.g. implementation plus commercial organisation, or maintenance plus healthcare), each group received contact details of a corresponding field expert, with whom the course faculty had an agreement for participation in the study activity. Consequently, each selected combination was only addressed by one group of students.
At the start of the semester, the students received training on how to conduct workplace interviews, and the faculty helped the students prepare the interview questions for the field experts. The students also learned how to make instructional videos and were given access to equipment such as video cameras and microphones, if needed. One member of faculty did group coaching, and the students had 3-4 weeks to complete the task of preparing and conducting an interview and creating a film. One of the faculty's recommendations for the final videos was to avoid using video coverage of the actual interviewee. Instead, the students were encouraged to either re enact the interview or present it in another format, synthesising the experts' answers into a short and clear informational artefact. According to the faculty's evaluation and the student feedback, the best videos in the course were made in the forms of an interview re-enacting, a news report, or a TV talk show. Some of the less popular videos were made using presentation slides with a voiceover recording, or a one-person narration.
In the second part of the semester, after all of the student groups completed their projects, the videos and text reports were digitally shared among all the students in the class in order for them to learn about all the processes of IT systems life cycles in all of the studied organisational contexts. In this way, each group project contributed to building a bigger collective picture of various IT system lifecycles. After watching the collectively-created videos and reading the text reports, students took part in discussions during one two hour seminar using the techniques of constructive controversy [52] and one two-hour seminar using the affinity diagram technique [53] to stimulate peer learning. One of the seminars was focused on different processes and the students met other groups that had worked on the same process, e.g. development.
In the other seminar the students met with other groups who had done interviews and videos with people from the same organisational context, e.g. healthcare. In both these workshops the groups also discussed and related the films to the scientific papers and the book that they had previously read in the course.
The learning theory of constructive controversy posits that learners can find a starting point to understanding complex issues and solving difficult problems through discussions and controversies in which they seek to reach a general agreement between conflicting information, ideas or opinions (54]. At the same time learners also discover new facts and develop critical thinking skills. The affinity diagram technique was used to support students in classifying and organising the information they found as the result of their group projects. Both of these techniques stimulated learning through productive sharing of the student group project results. In contrast to simple in-class presentations, the two techniques aimed at providing a better opportunity for students to actively learn from each other. The individual exam that the students had to do at the end of the course included the information presented in videos and the text reports produced by all of the groups.

IV. METHOD
Throughout this instance of the course, four formative course evaluations and one summative evaluation were per formed. The questions in the course evaluations were open questions, and the data used in this paper comes from the question: "What can be improved about the course" which was a part of all five course evaluations. The formative course evaluations were done online anonymously in the classroom. Almost all 45 students were present and answered the survey on all occasions. The summative evaluation was distributed by mail after the course had ended and 12 students answered. In addition to the course evaluations, we have collected data through comments and conversations with students throughout the semester.
One way of investigating whether the personal epistemology of the students has a bearing on their learning disposition within the course is to analyse their comments within a framework which seeks to identify and correlate aspects of their response with the students underlying epistemic beliefs. There are several ways that this could be done but, as we have said, a straightforward approach is to analyse the comments paying attention to, say, the dimensions of belief found in the models of . This is not to endorse her particular multidimensional approach, per se, but merely to acknowledge that the five dimensions she described do provide reasonable coverage of the fundamental areas ad dressed by the field of personal epistemology. As such, they allow for at least a preliminary discussion of these aspects within the comments, by situating them in an established conceptual framework. Moreover, while it may be the case that Schommer-Aikins' model of multidimensional development may not have sufficient levels of reliability or robustness to provide a good quantitative analysis, it is the case that the categories she used to label the dimensions -source of knowledge, structure of knowledge, etc -have become central to the subject in one form or another, and, as educational concepts, do have underlying support from other areas of educational psychology, e.g. control of knowledge acquisition is very closely related to the self-theories and the concept of a growth mindset developed in the work of Dweck (41]. It therefore seems reasonable to examine the comments within this framework while remaining agnostic as to whether her particular model best describes quantitative features of the discipline.

V. RESULTS RELATED TO STUDENTS' PERSONAL EPISTEMOLOGY AN D THE SELF-FLIPPED CLASSROOM
Applying our data to the dimensions suggested by Schommer-Aikins, we made the following observations.

A. Structure of Knowledge
This dimension includes how students think about the struc ture, relationship and organisation of knowledge in a particular domain.
Some students thought that the course was too easy, and did not require much effort at all. They seemed to not connect the different aspects of the complexities involved when designing, maintaining, procuring and implementing complex IT systems in large organisations. In this way of thinking they often focused on one aspect of the problem such as "you need to be good at communication" or "the requirements specification needs to be clear" and then seemed to be unable to explore what this means and see why it is difficult to be good at communication or to write a clear requirements specification. They failed to see the interrelated and integrated parts of working with complex IT systems in large organisations, but focused on small aspects hence reducing the complexity to simple pieces.
Other aspects that were often stressed by students with this mindset was that the interviews were very limited, and that interviews by others only gave second-hand information which was not worth as much to them. For instance, this was expressed by stating that there should be less focus on the interviews as they limit people to work done by others.

B. Stability of Knowledge
This dimension included how students think about the contingency of knowledge and the way theories may change over time.
Many students had problems with the fact that people in the interviews had different opinions both about what the problems are when working with software development, and what solutions should be used. There were also contradictions in what the book, the research papers and the interviewees said about a topic. Interesting discussions arose in the seminars with the students related to areas such as trust in people and their knowledge versus micro-management and control in IT projects. The students perceived the micro-management approach and distrust in programmers to be true as it was presented by a very charming and knowledgeable interviewee. Faculty expressed some doubt about this approach in the discussions with the students, but they showed unwillingness to question and critically reflect on the interviewee's views.
Some students thought that the course set up was trou blesome and believed this led to ambiguous findings. They appeared to have an expectation that their findings from different sources should present them with clear answers. They assumed that knowledge could be certain and that there was one universal truth. As a result, they were unsettled and dissatisfied with having to rely on diverse, self-and peer produced information. Comments of the following kind were common: "I think it's still a bit unclear what your [the teacher] stance is on all the processes and best practises etc. Right now it feels more like we are getting information from each other -and there is no "real truth". I would have appreciated some more course literature or a small pdf just summarizing what each process (usually) contains etc.

C. Source of Knowledge
This dimension is related to where students think domain knowledge can come from.
The course material is an important source of knowledge related to a course unit to students. In this case, the course material consisted of a) a novel about software engineering, b) scientific papers and c) interviews with IT professionals. In addition to this, students were encouraged to make use of the lectures, teachers and resources in their learning process.
A general comment, made by many students, was that they would prefer the course setup to be more traditional with a course book divided into a number of traditional seminars. In this alternative course setup, faculty would present one or several chapters of the course book during the seminars. Often, the idea is that students read the course book before the lecture, and then there are a number of study questions provided in the book. One student described it in this way in the course evaluation: "Create a logical course structure based on a good course book. " This comment is interesting since it indicates that knowl edge packed in a book is more valid than knowledge from research papers and interviews with IT professionals. Some other comments related to this dimension are: • Why don't you tell me what I need to know?
• What will the exam look like? How is it possible to have an exam on self-flipped material based on interviews? • The quality of the created materials varies and this of course affects the learning of the creators and their peers.

D. Speed of Learning
This dimension includes how students think about the speed at which they acquire knowledge and skills.
One observation from members of faculty was that many students thought that they already knew everything there was to know about the course topic from previous courses and that mastering the course contents only required minimal effort and time. It appeared that, compared to other more difficult, technical courses, some students believed that knowledge in this course was easily gained. The students' stance in this respect is in direct contrast to that of the faculty who believed the course subject to be an area requiring (and offering an opportunity for) constant, lifelong learning. In addition, from faculty's perspective, it would be argued that the course actually required more effort from students than more technical courses as knowledge and understanding about the course topic required them to assimilate and critically reflect on information from a variety of heterogeneous sources.
That being said, students' perception of the course as easy may be connected to their lack of understanding of the structure of knowledge dimension and stability of knowledge dimension within the course.

E. Control of Learning
This dimension is related to how students think about their capacity to control the acquisition of knowledge and skills.
Criticism of the unequal quality of self-and peer-produced course contents suggests that students may believe they have low levels of control over their knowledge acquisition through out the course. Indeed, since they have to rely on -from their perspective -unverified third-party material, they may feel that their ability to achieve a satisfactory learning outcome is out of their hands. This may be connected to their being used to more traditional forms of teaching, in which the course substance is directly conveyed to students; they do not have to make any effort to acquire it (only to assimilate it). The view that the course should be set up in the same way as other courses is probably related to this factor.
From faculty's perspective, it may be argued that, as the course requires students to put together their own course material, they have an even greater control over their knowl edge acquisition; however, this increased control is dependent on increased effort, since students need to actively work on gathering and reflecting over course materials. These two aspects of the course -greater control at the price of greater effort -seems to be overlooked by most students (since, as discussed in section 5.3, they assessed the course as easy). As a result, the learning outcomes were often below faculty's expectations.

VI. DISCUSSION AND LESSONS LEARNT
This paper focused on the observation that students were sceptical about the course setup. It has sought to shed some light on factors influencing these students' mindset by analysing negative comments using a personal epistemology perspective, especially . We believe that our analysis, despite its limitations, contributes to a deeper understanding of the reasons behind students' negative evaluations of the course. Moreover, reflecting on students' evaluations and comments through the dimensions of personal epistemology can be a helpful tool for understanding how to address students with such beliefs.
For instance, it appeared that we have students embracing the position that there is one universal solution to a complex real-world problem, suggesting that they have difficulty un derstanding the concept of wicked problems. This relates to the structure of the knowledge dimension. Another problem we have found is that some students have difficulties with the stability of knowledge dimension, in the sense that they struggle with ambiguous information. In addition, a view that activities should yield single, unambiguous solutions, preferably acquired quickly, seemed to be present among our students. The self-flipped classroom approach is, of course, problematic for students with this belief.
Another issue with the self-flipped classroom approach was that of the quality of the learning material produced, since appears to have led to worries related to the source of knowledge and the control of learning dimensions. This could, to a degree, be dealt with by presenting the activity of judging the quality of the produced material as an explicit learning objective. The issue of quality of the learning material might have been mitigated by introducing a quality control phase into the workflow performed by teachers.
The view of knowledge as facts is problematic on a deeper level, since we are interested in a wider aspect of "knowledge", often captured by the "competency" concept. A recent report describes a framework for capturing aspects of a professional competency in relation to educational settings that could be valuable in this regard (55].
Viewing competence as a construction which subsumes knowledge allows for reasoning about learning objects that is different from knowledge of facts, and especially how as sessment can be done regarding these learning objectives. This is important as many comments are related to the assessment issue. That assessing professional competencies is a challenge is clear (56] and is in part due to the complexity of what a professional competency really is. It is clear that we, as educators, need to know more about how to deal with the tension between the student perspective and the teachers' perspective. This includes an understanding related to the dimensions of personal epistemology, but it is certainly a more complex issue. The reasons for disliking learning from peers does not need to be (solely) about quality of the material, but also includes not wanting to produce learning material or being of the opinion that it is the task of the teacher to present what is to be learnt. Indeed, having to put effort into acquiring course material in addition to having to learn and reflect over it may be off-putting to students, who are not used to having control over their learning materials.
Our observations regarding student behaviour are also de pendent on students having learning strategies that are less concerned with actual learning and more to do with figuring out how to pass our examinations.
There might also be other, substantially different, reasons for students to be quite negative, for instance: • The course was taught by two female faculty; • The subject of the course was in the area of human computer interaction and not very technical in nature .
The first aspect is discussed in (57], (58] and the latter aspect has been illuminated in Peter's thesis (59]. It is highly likely that these reasons do play a part in negative evaluations. We do, however, claim that the presented view of students personal epistemology with regard to knowledge is a factor behind the "hate it" attitude, and we believe that a similar study looking at comments related from a more open view of what knowledge is important would be a reason for the "love it" attitude.