The Multimodal Workshop was organized by Jim Martin
& Rick Iedema at the University of Sydney, December 1997.
The following program has been provided by
Rick Iedema.
1 Sydney Multimodal
Workshop 1997
Program
Multimodal Workshop
University of Sydney
15-17 December 1997
Workshop Program
New Education Building
Rooms: 351 (keynotes); 521 & 524 (parallel papers)
Keynote Papers: Monday
1. Monday 9:00-10:30 (Main Lecture Theatre 351)
Gunther Kress
Culture Communication and Societies (CCS) Group
Institute of Education
20 Bedford Way
LONDON WC1H 0AL
Tel: + 44 171 612 6502/Fax: + 44 171 612 6177
"Modes of representation and local epistemologies: the presentation of Science in Education".
The development of the notion of multimodality is opening quite new perspectives on issues such as ' literacy across the curriculum', or pedagogy. In the case of the former, the concept opens up the possibility of a profound and fruitful engagement on their own ground by teachers in Art, Drama, Science, English on the issue of representation. In the case of the latter, it is becoming possible to examine anew the pedagogic/rhetorical strategies of teachers in the course of the presentation of sections of a curriculum. Assuming that communication does not proceed in one mode only (ie "language", or even in two modes, speech and writing) but rather in a multiplicity of modes simultaneously, allows us to ask about the pedagogic / rhetorical purposes and effects of the uses and foregroundings of particular modes at specific points in a lesson (or lesson - span ).
This talk consequently examines briefly some aspects of a multimodal approach to communication, focussing in particular on the issue of the representational/ communicational specialization of modes. It then proceeds to exemplify these ideas on hand of some examples -- science textbooks, and if possible, a brief extract of a video recording of a science classroom.
2. Monday 11:00-12:30 (Main Lecture Theatre 351)
David McInnes
Communication and Media
University of Western Sydney, Nepean
PO BOX 10
Kingswood
NSW 2747
Ph: 02 96787216
Fx: 02 96787399
Em: d.mcinnes@kilo.uws.edu.au
"Performing performativity: the limits and excesses of a systemic account of movement text"
If we conceive of identity as an on-going, iterative practice
- a process of performativity - then an analytics that
investigates
"identities/subjectivities" and the practices through
which they are formed, must attempt to describe the complex ways
in which identity is enacted. A focus on performance data, script
text and performance text, provides a way to consider two aspects
of a systemic account of such iterative practices. A systemic
based approach to social semiosis that couples a contextualised
understanding of the deployment of linguistic, bodily and spatial
resources with an account of semogenesis produces the beginnings
of a framework for documenting and critiquing composite
movement/linguistic text.
This paper will use the "composite" text of a homosexual encounter to foreground the iterative functioning of performance text and the necessity for understanding the relationship between semiotic modes deployed in such texts. It will then go on to propose a two level - connotative and denotative, account of the body movement resources of kinesics, gaze and proxemics, using a systemic based approach to modelling context and text. The value of such an approach will be understood in two respects. Firstly, via the descriptive potential of such a framework, in comparison to other approaches, and secondly via the capacity of a social semiotics of body movement to work with notions of production, iteration and performativity.
3. Monday 1:30-3:00 (Main Lecture Theatre 351)
Gillian Fuller
Media and Communications
University of New South Wales
"Meta-Multi Meaning Hype(r)... Multimedia interactives in exhibitionary space"
New technologies enable new configurations of meaning and construct new forms of literacy. This paper will explore some of the rhetoric around these new literacies and the semiotic work being done to describe such literacies. It will consider questions such as: How effective is linguistics as a base model to describe the meaning functions of multimedia environments? What kind of metalanguage can semiotics develop? From where should it source itself?
This paper will examine the uses and potentials of one multi-modal site-the computer interactive in another multi-modal site- the museum exhibition.
4. Monday 3:30-5:00 (Main Lecture Theatre 351)
Christian Matthiessen
Media, Communication and Linguistics
Macquarie University
"Multimodality in Computational Modelling and Social Semiotics: From Diagrams and Maps to Paintings"
[please contact presenter for abstract]
Keynote Papers: Tuesday
5. Tuesday 3:30-5:00 (Main Lecture Theatre 351)
Michael O'Toole
Division of Social Sciences,
Humanities and Education
Murdoch University,
Perth, Western Australia 6150
<otoole@central.murdoch.edu.au>
"From Systems to Hypertext: Navigating Semiotic Space in the Visual Arts and Poetry'"
Rembrandt's "The Night Watch" is generally acclaimed
as a breakthrough in the well-established Dutch genre of group
portraits. Yet, apart from a recapitulation of Wölfflin's broad
distinctions of linear/painterly or planar/recessional, most of
the discourses about this painting in books, journals and its
presentation in the Rijks museum itself focuses itself entirely
on the accuracy of the
representation. If we analyse separately the Representational,
Modal and Compositional functions in this painting and then note
the focuses of interplay between the functions, we may come up
with a far richer, better proven - and more revolutionary -
interpretation of Rembrandt's achievement.
This demonstration of the power and flexibility of the
systemic-functional semiotic model for the analysis and
interpretation
of painting developed in my book The Language of Displayed Art
(1994) will open up a discussion of the scientific principles of
'open dynamic systems', a topological approach to
multidimensionality in art, and the more recent concept of
'hypertext'.
My recent work applying the principles of interactive multimedia to the analysis and interpretation of paintings suggests that the classic systemic-functional chart of three distinct "metafunctions" working through systems at four ranks of textual unit (Halliday, 1973; O'Toole, 1990) is a kind of map for hypertext. The detailed study of individual systems in the visual array promotes exploration and comparison of features across the boundaries of the metafunctions and a 'shuttling' strategy between elements at different ranks and the whole work.
In the context of reading an art work this enables one to enact and record not merely the patterns of meaning which one takes to have been encoded by the painter according to generic or stylistic norms, but the patterns of meaning which result from the interplay between those norms and the expectations, assumptions and mood of the viewer. In other words, the supposed barriers between 'formal' analysis and 'reader-centred' or 'contextual' criticism are no longer valid.
Like the svstemic model of visual 'grammar', topological approaches to the study of semiotic space (Atkin, 1981; O'Toole, 1980) start from the specification of features at particular ranks (or 'sets' and 'cover-sets'). They quickly move, however, to the mapping of a whole 'backcloth' of patterns of features against which the knowledge and preoccupations of the receiver/viewer/reader become a kind of 'traffic'. Tracing and recording this through the operations of hypertext appears to bring alive the processes of semiosis and highlights the functioning of a work of art as an open dynamic system interacting with the social semiotic of the culture and the individual viewer/reader.
This semiosic process becomes particularly interesting when we consider our readings of Mexican mural paintings, where the 'backcloth' provided by formal analysis is subjected to the 'traffic' of cultural (Mexican/non-Mexican), political (Socialist/non-Socialist), situational (the interaction of painting with architecture; the time of the painting's production/ viewing) and institutional presuppositions.
I will finish with a brief consideration of how these notions of hypertext, semiotic space and traffic illuminate the process of interpretation of a famous poem.
References
Atkin, Ron (1981) MultidirnensionaI Man ,Harmondsworth: Penguin Books
Halliday, M.A.K. (1973) Explorations in the functions of
Language, London: Edward Arnold
(for Chart
of systems and functions in language, see Table 1, p. 141)
O'Toole, Michael (1980)'Dimensions of semiotic space in
narrative', Poetics
Today ,
Vol I, No. 4, pp.135 -49
(1990)'A functional model of analysis for the visual arts', Semiotica ,82, 3/4, pp. 185- 209
(1994) The Language of Displayed Art ,London: University
of Leicester Press/Pinter Publishers
(Chart of systems and functions in painting on p. 24)
Keynote Papers: Wednesday
6. Wednesday 11:00-12:30 (Main Lecture Theatre 351)
David Rose
University of Adelaide
"Encoding Ideology in Western Desert Art & Text: the Semogenesis of Abstraction"
This paper begins by looking at how the traditional art of Australia's Western Desert employs a simple visual grammar of polysemous symbols to encrypt abstract principles of social order. It then looks at how the same is achieved in oral texts, in a language that is semantically rich but has no written mode or lexicon for abstraction. The question we go on to ask is how, why and when the potential for abstract meaning evolves over three semantic timescales: logogenesis, ontogenesis and phylogenesis. The latter is then linked back to the evolution of art and ritual in hunting-gathering cultures in general, with implications for our conceptions of the phylogenesis of modern human cultures.
7. Wednesday 1:30-3:00 (Main Lecture Theatre 351)
Joan Rothery
University of Sydney
and
Maree Stenglin
Manager Educational Services
The Australian Museum
"The Role of Images in Indigenous Australian Cultures"
This paper examines visual images on display in the Indigenous Australians exhibition at the Australian Museum. The perspective taken is that of exploring them as a resource for learning how knowledge is socially constructed in some areas of Indigenous Australians' cultures. This perspective has been developed in three units of work for primary school students. These units are to be used in conjunction with a visit to the Australian Museum. The first unit deals with hand stencils in a rock shelter, the second with a group of rainbow serpent paintings and the third with a mural called 'Goomadeer Dreaming' painted by Thompson Yulidjiri on a rock wall. Through visual images, the authors have aimed to introduce students to a complexity of Indigenous knowledge and values.
8. Wednesday 3:30-5:00 (Main Lecture Theatre 351)
Theo van Leeuwen
The London Institute
"The Semiotics of Kinetic Design"
The semiotics of kinetic design explores how human artifacts
and built environments facilitate action. It looks at such
objects and
environments, not as texts to read and interpret, but as objects
and environments to do something with (in, on, against, etc.),
and this includes textual objects such as paintings, sculptures,
books, etc.
A system network will be presented that details the key options in the kinetic design of toys, and these options will be exemplified by particular toys and the ways they are designed to move and/or be moved.
The principle of kinetic design is then extended into design
of everyday objects (e.g. chairs), architecture, and traditional
as well
as contemporary (computer-based) 'textual objects'. It will be
argued that the semiotics of kinetic design is of particular
importance for the latter, in the form of interface design.
Parallel Papers: Tuesday
1a. Tuesday 9-9:45am (Room 521)
Anne Cranny-Francis
English and Cultural Studies
Macquarie University
Sydney, NSW 2109
Australia
"Rhizome or Tree: Theorising textual production and
consumption on the internet (and elsewhere)"
This paper examines practices of textual production and consumption with reference to Deleuze and Guattari's notion of the rhizome. In particular, the paper looks at the way this concept is useful in understanding hypertext.
Critics working on information technology are torn between its liberatory potential and its potential as a super-panopticon. Will it be a means for global democratisation - or global domination? This paper approaches this issue at the level of text, where those questions are transformed into questions about the relative freedom of hypertext in comparison with paper texts. Do hypertexts links represent a new kind of freedom? Do they signal our freedom from dependence on the tree - as provider of both medium and message? How does the multi-modality of hypertext figure in this issue?
1b. Tuesday 9-9:45am (Main Lecture Theatre 351)
Bill Greaves
Dept of English
Glendon College
Toronto
"The Music of English: Intonational Patterns in Spoken Spontaneous Discourse"
English is a sung language, and this is reflected in the lay terms often used to describe its sound: "rhythm", "tempo", "pitch", "phrasing" and so forth. Often this aspect of English is seen as an "add on", as "icing on the cake", as something that is attached to the "real" matter of speech--its lexis and grammar. Halliday stands out as a linguist who fully understands that this is not the case--that English exploits its spoken modality at the heart, not the fringes, of its networks of semantic and lexicogrammatical choice.
This session will introduce Halliday's framework, with some attention to the use of CECIL, a free software program, as a resource for exploring the intonational contours of English. The paper will explore the way the "music" of English determines interpersonal and textual meanings in the Pretonic parts of tone groups--a relatively unexplored part of our spontaneous phrasing.
2a. Tuesday 9:45-10:30 (Room 521)
Noboru Yamaguchi, Fukushima University, Japan
"Verbal and Visual Conspiracy in Mass Media Texts"
The present paper is an attempt to show that the conspiratorial relationship between verbal and visual elements in mass media texts can construe/construct and stabilise (sometimes ideologically problematic) social realities more effectively (and somtimes more cryptically) than when they are isolated. Many texts in mass media, eg. news articles in newspapers or advertisments in magazines, very often consist not only of verbal elements but also visual elements, eg. pictures or drawings, which are not simply ornamental appendage to the verbal. More often than not, the verbal and visual elements in these texts are realisations of the same contextual elements, values or ideologies, which tend to probabilistically skew the choices from a system of language and a system of visual images in their own semiotic domain. Value-laden texts of same kind, repeatedly produced by the mutally helping conspiracy between language and visual images, can perpetuate certain types of social realities, which will pass unquestionable among ideologically naive people, and which will make any potentially problematic ideologies behind them pathologically invisible.
2b. Tuesday 9:45-10:30 (Main Lecture Theatre 351)
Ichiro Kobayashi
"Multimodality and 'Information Fusion': A Systemic-Functional Approach"
[please contact presenter for abstract]
3a. Tuesday 11:00-11:45 (Room 521)
Louise Ravelli
Linguistics Unit
University Of New South Wales
Kensington 2052
EMAIL: l.ravelli @unsw.edu.au
"BEYOND SHOPPING: multi-modality and the 2000 experience"
If multi-modality can be taken as the co-deployment of multiple semiotic resources, then 3-dimensional experience must be the ultimate example. This paper examines the various semiotic resources deployed in the Olympic Store (Pitt St, Sydney), a souvenir shop for the Sydney 2000 Olympic Games, including the language, spatial layout, visual design, and organisation of goods. It is argued that the store generates something much more than a shopping experience; more even than that of a 'lifestyle' store. Through overt intertextual references, it mimics the display practices of museums, and thereby generates its own cultural significance. Additionally, foregrounded features in each of the different semiotic systems create a kind of inter-semiotic convergence: all signs can be seen to point in the same direction, so to speak, at least for the compliant reader. The destination is that of a particular experience of/attitude to the Games, one which is a crucial element in pre-Games advertising, but which is not necessarily replicated in all Olympic sites.
3b. Tuesday 11:00-11:45 (Room 524)
Zeng Licheng, Ichiro Kobayashi & Christian Matthiessen:
"Generating Multimodal Presentations by Computer: The Systemic Func-tional Multiomodal Generator 'Multex'"
This talks presents a computational model of the semiotic system of map, as an example of the general approach to modelling multimodality in MULTEX, a multilingual and multimodal text generation system developed at Macquarie University.
Our baseline assumption about multimodality is fairly straightforward. Since a modality systematically construes certain types of meaning, it should be treated as a semiotic system on a par with a linguistic system within the systemic theoretical framework. Multimodality means the capability of a computational system to communicate meanings between different semiotic systems.
With this assumption, we are then able to set up the structure as well as the operational model of maps by systematically reinterpreting the systemic theoretical categories such as system, instance, semantic stratum, lexicogrammatic stratum, with regard to the map semiotic system. Software modules can be developed accordingly to interpret these theoretical cateogires in computational terms.
In this talk, we will:
1) explicitly interpret the map semiotic system in the light of the systemic theory.
2) show how this interpretation can be computationally represented and brought into play in Multex.
3) discuss possible pitfalls in our approach.
4a. Tuesday 11:45-12:30 (Room 521)
Rick Iedema
Meaning Research/University of NSW
"Camera Style and the Filmic Representation of Fashion"
Exclusive fashion aims to achieve distinction or cutting edge as 'haute couture'. This is not merely brought out by the exclusiveness of the fashion designs, but also by its presentation (newsmaking fashion shows) and its re-presentation (fashion magazines and fashion television).
This talk will look at the re-presentation of fashion in fashion television, with the aim of characterising the semiotic/representational features which contribute to the construction of fashion as 'cutting-edge'.
These features will be described in terms of television camera
devices, the editing of such products, and to a lesser extent,
the
semiosis that attaches to the practice of modelling. Here, highly
culture-sensitive issues to do with significance of colour in
film, clarity of the image (degree of content/context
distinction), degree of framing of the fashion item, and
interpersonal positioning of the model are taken into
consideration.
4b. Tuesday 11:45-12:30 (Room 524)
Wu Canzhong & Christian Matthiessen:
"Lexicogrammatical Text Analysis, Modelling Multimodal Domains and Visualising Results"
[please contact presenters for abstract]
Parallel Papers: Wednesday
5a. Wednesday 9-9:45am (Room 521)
Joan Phillip
School of Teacher Education
Charles Sturt University
Bathurst NSW 2795
"So now you have learned to read! Semiotic readings of the picture storybook, Beware, Beware by Susan Hill, illustrated by Angela Barrett (Walker Becks, London, 1993)"
In this paper the relationships between the visual and
linguistic elements of the picture storybook, Beware, Beware by
Susan Hill, illustrated by Angela Barrett (Walker Books, London,
1993) are read from a series of semiotic perspectives which
include systemic linguistics, the application of systemic theory
to visual representations, especially in the location of viewing
points (Kress and van Leeuwen 1990) and psychoanalytical theory
as it illuminates
the 'surveillance of the gaze' and the construction of desire.
The foundation of the paper is a child's reading which is
traced and mirrored by the semiotic readings. Through the
mirrored readings I suggest the possible function of narrative
texts in the dynamics of cultural reproduction and the social
regulation of female lives, not just through ideological
'recruitment' (Althusser 1969) but through corporeal and psychic
inscriptions (Walkerdine 1984; Grosz 1994). There follows a coda
in which, using psychoanalytical theory, I
present the ambiguities of my initial pleasure in the text and my
later disquiet, exploring the possibility of recovering the text
as a feminist project.
This paper is concerned with the disruption of commonsense views of reading and presents possible relationships between the semiotics of narrative, linguistic and visual, and a child' s becoming a cultural subject.
5b. Wednesday 9-9:45am (Room 524)
Radan Martinec
The London Institute
London, U.K.
"Towards a Semiotics of Action"
In this paper, I will present an outline of three kinds of
action, or movement, which can be seen as parts of an overall
action system. I will discuss the similarities and differences
among the different kinds of action and show that despite
differences in organization in terms of strata, the different
kinds of action share certain parts of their meaning potential.
The first kind of action I will discuss will be action oriented
mainly towards some practical task. The second kind of action is
that which is explicitly intended to communicate and which can do
so without being supported by a simultaneously occurring verbal
utterance. The third kind of action depends on a simultaneously
unfolding verbal text for its full meaning. I will show that the
meaning potential of the three kinds of action can be mapped out
by the systems and structures of systemic-functional semiotics
and will present realization rules relating such meanings to
observable forms. I will argue that many, perhaps most,
interactions are contributed to by all three kinds of actions,
and that being able to analyze and
interpret them is thus essential to getting at the interactions'
meaning. I will illustrate my talk by examples from film and
video.
6a. Wednesday 9:45-10:30am (Room 521)
Jon Callow
University of Western Sydney, Macarthur
Faculty of Education
PO Box 555
Campbelltown. NSW. Australia. 2560
Ph: 61 2 9772 9630
Fax: 61 27721565
e-mail: j.callow@uws.edu.au
Len Unsworth
Senior lecturer
School of Teaching and Curriculum Studies
Faculty of Education
The University of Sydney
"Equity in the Videosphere"
To the videosphere, the era of the visual..... We are there.
- Regis Debray, The Three Ages of Looking, p.532
As Debray declares, we are in the era of the visual. While
people have produced and viewed images for thousands of years,
the present era confronts us with a multitude and choice of image
that has been unachievable and unimaginable in the past. If
images are so pervasive and 'valued' by our culture(s) it follows
that we, as a society, should be quite visually Iiterate. But is
this the case? Is visual literacy understood by teachers and
taught in an equitable fashion,
allowing students to meet the demands that both the schooling
system and the culture in general puts upon them?
After a brief overview of systemic theory in the visual context, this paper will examine some of the viewing demands that are placed on Australian students, in terms of national profiles and state syllabus outcomes in English K-10 and in the formal context of the Year 12 New South Wales Higher School Certificate English examination.
By analysing both the questions that ask students to critically analyse images and the sample answers and comments provided by the New South Wales Board of Studies (the independent body responsible for setting and administering the examination), some of the implicit expectations that are placed upon students in interpreting images will be drawn out. The final section of the paper will explore how some of the systemic theorising could be applied to further assist students in analysing and critiquing these and other images.
6b. Wednesday 9:45-10:30 (Room 524)
David Butt
Media Communication and Linguistics
Macquarie University
"The Visual Interpretation of Change and the Semiotics of Process"
[please contact presenter for abstract]
The following note on the workshop was
written by C. Matthiessen after the event
Many of us (perhaps around 60) have just attended the Multimodal
Workshop at Sydney University this week (15 Dec. through 17 Dec.)
organized by Jim Martin and Rick Iedema.
Those of you who were fortunate enough to be able to attend will know what I mean when I say that we're are very very grateful to Jim and Rick for organizing this extraordinary workshop. It was a high-powered and high-energy event with a wide range of contributions and a wide range of contexts of applications. The founding fathers of the systemic functional semiotic theory of visual semiotics were all there -- Michael O'Toole, Guenther Kress and Theo van Leeuwen, all giving talks presenting new materials not included in their very recent books and all appropriately representationally, modally and compositionally centred in a pannel session where Jim and Rick presented questions they had distilled from contributions elicited from sys-func readers and then members of the audience also contributed. It was a unique opportunity to get a sense of past products, present problems and future fisions (should be "visions", but I wanted to maintain the pattern -- just a matter of agnation in voicing ...).
When they were asked what they saw as the current research agenda and what issues they would identify as problems that needed to be tackled urgently, they responded:
• Semiotic systems — The need to engage with and describe non-European forms of visual art;
• Semiotic systems — The challenge of interpreting 3-dimensional forms of art and movement;
• Kinds of system, the level of materiality — our sensory understanding of that aspect of the world;
• Semiotic systems, Representational resources — in particular, vectors and diagonals, framing and continuity, strategies of salience;
• Semiotic systems — patterns of co-occurrence in visual systems;
• Semiotic systems, levels of organization — connexion between forms of knowledge and forms of representation;
• Multimodal typology — is the visual as articulated as the verbal?
• Metasemiotic resources —
multimedia as a means of carrying on the discourse in a more
sophisticated form.
In the course of the workshop, these issues and many others
came up. See the abstracts listed above.