SCM Projects

Overall

The course is centered around three projects:
  1. Mini project I: literature search. Search for literature on javascript and related subjects (details in class Sep 10th), summarize your findings and pose one or more research questions.
  2. Mini project II: the empirical method. Perform exploratory studies of some specific area, form a model and hypothesis, perform confirmatory studies.
  3. Survey project. From a given starting point (typically a scientific paper), perform a comprehensive survey and analyze the use of scientific methods in the surveyed papers.
Each project is documented in a report, see exam and report for formal requirements and evaluation.

Both mini projects are done in groups. Groups are around 3 people (plus minus one) and formed for the first mini project at the beginning of the project, with in-class assistance by the teacher for those who do not form groups by themselves. The second mini project can use same or different groups that should be formed by the students. Group can advantageously be the same as for the SE and ROB courses. The first responsiblity of a group is to add itself to the list of "Mini project groups" on the wiki found under course documents on Blackboard.

You are encouraged to use latex for writing your report, see also the small starter example.

Mini project I: literature search
The purpose of this project is to get experience in searching for literature on a specific topic and posing critical questions on what you are reading. Concretely, we have an invited talk by Mads Ager from Google (see course schedule) which we will use as inspiration for the project. You must search for literature on "jacascript" and related subjects (details in class Friday Sep 10th, see the announcement of his talk for more keywords to use as a starting point). This search can take you in many different directions, from the first object-oriented languages to state of the art in modern virtual machines. The report must summarize your findings, that is:
  1. a list of literature that seems relevant
  2. a short summary of each piece of literature
  3. arguments for the relevance of each piece of literature
Moreover, the report must pose one or more research questions that you answer either based on your findings or based on Mads' talk. In the context of this project, a research question is a non-trivial question that requires a scientific approach to answer. The question(s) are whatever open or unclear issues you encounter as you are reading your literature. The question(s) can be answered by the literature you are reading or alternatively by Mads. If the questions could not be answered through literature and did not seem relevant to what Mads is talking about, leave them open.

Details on mini project I:

Assessment criteria (in no particular order)
Written presentation of report. Cohesiveness and breadth of the research literature found from search. Quality of summaries and rigor of arguments for relevance. Relevance of research question(s) to the literature and rigor of the answer(s).
Work plan (suggestion)
(1) Group inception meeting. (2) Find a common starting point and related literature. (3) Each person reads selected literature. (4) Write summary and relevance for each piece of literature. (5) Combine into a single common report, (6) Use project log to estimate contributions for work cateogies.
Report structure (suggestion)
2-3 pages: Introduction, Summary of Papers, Overall Assessment, Research Questions
Work categories (for estimating contributions, suggestion)
(1) Initial literature exploration, (2) Detailed reading and writing of summaries, (3) Common parts of report.
Questions or comments regarding Mini Project I should be directed to the mailing list.

For an example of what the report can look like, take a look at the mini project 1 example included in the materials directory.

Mini project II: the empirical method

The purpose of this project is to get experience in working with the empirical method, specifically using statistical methods. Concretely, you pick one of the two projects below (or something else, at the discretion of the teacher), perform exploratory studies in this area while working to accomplish the given task, form a causal model and hypotheses regarding your work, and perform confirmatory studies.

Suggested project topics:

AI1: What is the best way of mounting and using the light sensors?
A fundamental problem with behavior-based robotics compared to more conventional robotics is that it is diffcult to ensure that the robot will work not only once, but every single time. This problem arises from the fact that in behavior-based robotics the robot relies heavily on sensors for controlling its behavior. This makes the robot sensitive to sensor noise and changes in the environment which may influence sensor readings. For example, a robot following a line using light sensors is influenced by dirt on the line, surrounding light, the position of the sensor among other things. This means that if you have seen the robot follow the line one day it may not another day, because environment conditions have changed. So how do you systematically improve your robot so once you fix one problem it doesn't break another aspect of the behavior you actually wanted.....(two path here: test under different environment conditions even some you are sure will make the robot fail, do a statistical comparison between the average number of errors across the different environments).
SE1: How to improve the scalability of the ZAM zoo example?
The ZAM zoo example has a fairly small number of threads and agents, but what happens when this number increases? Imagine modelling all the zoos of the world: there are over 1000 major zoos all over the world, each of which could easily have hundreds of animals, plus there are probably numerous smaller zoos with e.g. 50 animals. What is the limit of the ZAM system, for example in terms of memory requirements? Can this be improved? How about performance: how to measure performance (throughput, responsiveness) in a system driven by a real-time clock? How to improve it for large scenarios?
Alternatively you can also use some other scenario, but you must discuss this with Ulrik and get his approval first. Ordinarily, you must use on of the two topics above, but if for some reason that is not possible, we could e.g. select some other software system already known to you where you can work with scalability, or perhaps pick some other topic from another course.

The report must document your work, and although short (recommended size is again "a few pages") should have the structure of the prototypical academic paper within empirical science:

  1. Introduction: describes context, problem and solution (optionally including state of the art, for extra points)
  2. Methodology: describes what you are doing how and why
  3. Experiments: describes what you actually did
  4. Conclusion: summarizes, evaluates, assesses, perspectivates
The report should of course e.g. cite literature from the corresponding course, when appropriate.

Details on mini project II:

Assessment criteria (in no particular order)
Written presentation of report. Cohesiveness and breadth of any research literature included. Proper use of overall empirical methodology. Correct use of statistical methods.
Work plan (suggestion)
(1) Group inception meeting. (2) Decide what to do and how (both what you are doing and how to use empirical methods to assess). (3) Follow the "general preparation" guidline from Cohen. (4) Do pilot experiments while completing task, or perhaps complete experiments while completing task. (5) Combine into a single report, (6) Use project log to estimate contributions for work cateogies.
Report structure (suggestion)
2-3 pages: Introduction, Methodology, Experiments, Conclusion
Work categories (for estimating contributions, suggestion)
(1) SCM-specific implementation tasks, (2) General preparation, (3) Various pilot experiments, (4) Real experiments, (5) Data analysis, (6) Report writing.

Survey project
From a given starting point (a scientific paper or a general topic), perform a comprehensive survey and analyze the use of scientific methods in the surveyed papers. A list of suggested papers/starting poitns will be made available, taking inspiration from AI1 and SSE0/SSE1 as well as Ulriks area of research. The paper SCM-01 (An insightful comparison between experiments in mobile robotics and in science by F Amigoni, M Reggiani, V Schiaffonati - Autonomous Robots 2009 - Springer) is actually a fairly good example of the kind of work we want you to do: survey an area, obtain a general understanding, evaluate the use of scientific methods. Only I want you to put more focus on the survey part (and thus the general understanding) and less on the evaluation part (but it should still be there as a significant part of your report).

Suggested starting topics:

The report must document your survey and your analysis of the use of scientific methods. The recommended size is about 7 pages, and the paper should have a structure resembling that of SCM-01 An insightful comparison between experiments in mobile robotics and in science:

  1. Introduction: describes context, problem and solution (both domain and report)
  2. Survey: the main body of the paper, summarizing and critically (yet respectfully) assessing the articles that you have read
  3. Evaluation: your evaluation of their use of scientific methods
  4. Conclusion: summarize and perspectivate
Naturally, the report should cite relevant scientific literature. The literature page contains references to a number of survey papers that you can serve as examples.

Details on the main project:

Assessment criteria (in no particular order)
Written presentation of report. Cohesiveness and breadth of the literature survey. Quality of the assessment of the individual articles. Quality of the evaluation of the use of scientific methods.
Work plan
Oct 2 (latest): email Ulrik with your project description and your list of starting points or selected literature. Oct 9: email Ulrik a progress report describing what you have done and what you are doing, as well as your current draft (if any). Oct 30: project report due
Project description
The project description is typically a one-line description of what you have in mind, perhaps including a one or more papers that will serve as your starting point. The purpose is to make sure that you don't start in the wrong direction
Progress report
The progress report is a few lines describing what you have done nso far and what you plan on doing, perhaps including a more comprehensive list of papers (or maybe even your final list of papers) and a copy of the draft of your report (if you have made a draft). The purpose is to enable Ulrik to let you know if you are somehow goin in the wrong direction or if you are dangerously far behind schedule

General advice on projects

The mini projects are intended to be small and you should be able to do them in a few hours in addition to the time we spend on them in class. Keep in mind the web resources on report writing.